Here's some psuedo code...
int[] ids = new int[1000];
string[] names = new string[1000]
for (int i = 0 ; i < 1000 ; i++)
{
// open customer001txt through customer999.txt
// update the id and name array avariables
}
This is a VERY simplified version.. the real project consists of opening up hundereds of encrypted database files that must be decrypted, parsed, and then arrays in the parent class/thread need to be updated with info...
The problem is the single thread version now works fine, but takes forever.. most of the burden is decrypting and unpacking the database files.. if I could multi-thread this I could cut down processing time big time I believe..
Right now I am trying to use QueueUserWorkItem() and just load up all the file names using the object state var.. but because my worker function is static, I cant figure out how to update the arrays in the parent thread safely..
This seems like it would be a fairly common task.. use threads to pull a bunch of information from hundereds of files/databases and populate a parent table/structure with info...
I'm not looking to be spoon fed, just a little guidance.
Can anyone point me in the right direction?
Please help.
Thanks!
-
-
staceyw
Before C# there was darkness...
CodeGuru123 wrote:Here's some psuedo code...
int[] ids = new int[1000];
string[] names = new string[1000]
for (int i = 0 ; i < 1000 ; i++)
{
// open customer001txt through customer999.txt
// update the id and name array avariables
}
This is a VERY simplified version.. the real project consists of opening up hundereds of encrypted database files that must be decrypted, parsed, and then arrays in the parent class/thread need to be updated with info...
The problem is the single thread version now works fine, but takes forever.. most of the burden is decrypting and unpacking the database files.. if I could multi-thread this I could cut down processing time big time I believe..
Right now I am trying to use QueueUserWorkItem() and just load up all the file names using the object state var.. but because my worker function is static, I cant figure out how to update the arrays in the parent thread safely..
This seems like it would be a fairly common task.. use threads to pull a bunch of information from hundereds of files/databases and populate a parent table/structure with info...
I'm not looking to be spoon fed, just a little guidance.
Can anyone point me in the right direction?
Please help.
Thanks!
This requires a lock (normally) to update shared state from other threads.
Something like below (not compiled):
int[] ids = new int[1000];
string[] names = new string[1000];
object sync = new object();
for (int i = 0 ; i < 1000 ; i++)
{
ThreadPool.QueueUserWorkItem(Func, state);
}
public static void Func(object state)
{
// Do work here.
// Update shared state here.
lock(sync)
{
//update ids array.
}
} -
oh excellent, thank you so much!
I was using lock(this) before but was getting compiler errors.. the lock(sync) works very well..
In my prelim test it seems to have shaved off 20% time and does now utilize all cpu cores..
Thanks!
-
evildictaitor
strcpy(malloc(4), "Oh noes!");
Are all of the statements in the loop independent? If so, I'd seriously advise that you get yourself a copy of the Parallel extentions library (which is currently CTPing) and have a look at that. You can parallelise code in a way that's reasonably easy to understand, and often faster than the thread pool queue.
Download from Microsoft
C9 Video
And a screencast to get you started. -
He established that there is shared state involved (arrays in the parent thread), hence the need for a lock. I would be interested to see how the ParallelFx folks handle thread-safe reduction operations with non-transitive operations though.
-
PerfectPhase
"This is not war, this is pest control!" - Dalek to Cyberman
JChung2006 wrote:He established that there is shared state involved (arrays in the parent thread), hence the need for a lock. I would be interested to see how the ParallelFx folks handle thread-safe reduction operations with non-transitive operations though.
Basing this off the example, and assuming that names[n] contains the 'names' for customer id stored in ids[n] then you wouldn't need a lock would you not? Step outside this narrow view then yes you would.
-
That's a good point. I assumed that if he ran into problems updating shared state with a thread pool-based implementation that his implementation was more involved in that but maybe not.
-
perhaps names and id arrays were a little generic..
each database file loaded modifies existing arrays in the parent thread, and in some cases rebuilding them..
The issue I was running into yesterday was lock (this), but lock (sync) where i define sync = new object(); seems to have gotten me past that hurdle..
Its 9am, time to start at it again and see what kind of results I achieve.
Thanks!
Thread Closed
This thread is kinda stale and has been closed but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.