As humbling as it may seem, I created this problem and it was actually more straightforward than I originally had thought. As an enhancement to the last version of the import, I added a xml status file which simply acted like a log file. Where did I put this file? I put it in the bin directory. After some digging I was seeing that the app was unloading quite often. It was unloading even when an import wasn't occuring. What was happening is that during debugging, I found that when I would delete the status file (in essence resetting the system) that the .NET worker process was keeping tabs on the bin and any changes to that directory would unload the application. So any updates to this file during the import was causing the app to restart which was causing the import to take longer due to the excessive restarts. This would also explain the somewhat sporadic behavior of my errors. If the timing of it all worked just right things would import ok, albeit slower than it should (which I only know now because I see the speed increase). However, there would be certain points where the the application would quit. Each time it was in a different location and I couldn't pin it down. My thinking is at this point that it couldn't necessarily be pinned down since it was an internal timing thing.
I've written a fairly slick class library that is used from a web app that imports content from a third party source into our application. The source provides two types of feeds, full and incremental. The only difference in the feeds, is the file size. Structurally they are the same.
I'm attempting to test the process, from downloading the feed, extracting and then importing, but along the way I get a ThreadAbortException thrown when I try to do an import of the full feed (rougly ~100MB).
Any ideas where this is coming from? Resource utilization? If I knew where it was coming from I could try to clean it up and work around it.