I started using .NET 2.0 a couple of months ago and I knew I had this problem coming up soon and I was hoping the threadpool was enhanced. It doesn't look like it was.
Or, am I wrong and is there a more advanced threadpool or one already well written out there?
Basically, I don't want to throw a million items into the threadpool uncontrolled.
Here is going to be the flow:
1) Pick up all of the unprocessed records in the queue from the database. I'd probably limit it to 10000 or so at a time.
2) Limit it to processing 20 or so at a time.
3) After they are all completed, go back to the database and get the next 10000.
4) If a request comes in to stop, I want it to gracefully shut down and finish the 20 remaining items, rather than calling the join method or whatever it is to stop all threads immediately.
The main problem is when you use the .net threadpool and call adduserworkitem it will internally queue up all of the items and there is no way to see how many are completed, or to stop them gracefully.
The main thing is I don't want items processing while I perform the call to get the next 1000 records, that is why I want to make sure all are fully completed before going back to step 1.
I actually wrote a threadpool manager that did this but it had some problems where it would stop processing. I wanted to see if there is an elegant solution built-in, or anyone had any suggestions.