Well, after wasting much time and developing a nice headache, there seems to be no solution for getting a pdf to open in a modal dialog.
Now you would think something like this would work:
myWindow.caller = window.showModalDialog("/myReport.pdf")
but it doesn't work. All you get it a blank page. So lets try something else:
myWindow.caller = window.showModalDialog("/showReport.aspx")
and lets say we have the following in our System.Web.UI.Page.Load event handler:
Response.ClearContent()
Response.ClearHeaders()
Response.ContentType = "application/pdf"
Response.WriteFile(MapPath("myReport.pdf"))
Response.Flush()
Response.Close()
Ok, that doesn't do anything either. You still get a blank page. So after hours and hours of trying to make this work and looking at other peoples solutions, there is no solution. There is only a workaround. The work around is to create an IFrame on showReport.aspx, and the source of the IFrame would be the location of your pdf file. Viola! That works! But yeah, it sucks. Now you have to scroll the IFrame instead of the window itself, and printing becomes a worry. Not only that, but I need this to work with another third party software program that most likey won't work with an IFrame, and I don't even want to go through the troubles of seeing if thats going to work out alright. For those of you who don't know and use the product (sorry, can't tell you which one it is), you might think "We'll of course it will work. Its just an IFrame, anyways". We'll, the product, as awesome as it is, doesn't quite work like that.
So, in conclusion, what was my solution? window.open. Didn't want to do it this way, but maybe if I set the window size properties from screen.width and screen.height and take up the full screen, they'll be more inclined to close down the window rather than just minimize it out of the way.