Tech Off Thread

5 posts

Queue - Processing In Order

Back to Forum: Tech Off
  • User profile image
    MetaGunny

    Okay.  I have a question about the method that I was going to use to process items in a queue in order.

    Let's say I have a queue, with a bunch of records, doesn't matt of what.  The main point is that each queue, let's say, belongs to a certain account, so let's say we have an AccountID with each queue item.

    It may look like this:

    Account ID, Record
    543, 1
    123, 2
    543, 3
    543, 4

    Okay, now, the main criteria here is that I don't want to process record 4 for account id 543 before record 1 is processed for account id 543.

    Basically, to keep it simple, let's just say I'm grabbing all the records and then throwing them in a thread pool.

    Here is how I was going to do the processing:

    1) The first method I was going to do is when I'm looping through the records and throwing them in the thread pool, is on each iteration only throw in unique account id's.  So, on the first iterationof the above data, I would only throw records 1 and 2 in.  The idea is that if I grab 1000 records at a time, hopefully this will help prevent locking with the second method I use.

    2).  The second method I use is when I go to process the record, I was going to use SyncLock thread locking method to grab an object for that specific account id.  So, when the the next thread comes to process that account id, it would wait until that object is released.

    What do you guys think?  Or, what is a better method?

  • User profile image
    DoomBringer

    I'd make a collection of collections, sorting each record into a sorted list based on the UID.  You could use the UID as a hash to lookup the collection for the records.

  • User profile image
    zhuo

    I am not sure if I've understood your question.

    If you have a queue and presumably that queue is built up chronologically, why would record 1 ever get processed after record 4?

  • User profile image
    DoomBringer

    zhuo wrote:
    

    I am not sure if I've understood your question.

    If you have a queue and presumably that queue is built up chronologically, why would record 1 ever get processed after record 4?


    I think he wants to separate the records by user ID so a single thread can handle a single user ID, and he wants to preserve the ordering. 

  • User profile image
    MetaGunny

    Thanks Doom.  I'll give that a try.  The only problem some what is I don't know the user id in the beginning but have to do some processing as well to get that.

    They could process out of order because I'm not doing the processing on a single thread, sequentially.   I process through each and throw it into a thread pool

Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.