Blog Post

TPL Dataflow Tour

Sign in to queue

Description

This 15-minute video provides a high-level tour through some of the functionality available in the new TPL Dataflow library (System.Threading.Tasks.Dataflow.dll), an early preview of which was released as part of the Visual Studio Async CTP, available for download at http://msdn.com/vstudio/async.

Embed

Download

Download this episode

The Discussion

  • User profile image
    felix9

    DataFlow is truly useful. hope its flexible to write some custom blocks based on builtin ones with additional policies.

    And I hope its adaptable enough for large amout of relatively small data packets, and on 'not so many' cores, or its the wrong tool for that kinds of works ?

  • User profile image
    Toub

    Hi Felix-

    I'm glad to hear you find it useful.  And it is possible to build custom blocks; the system is primarily based on two interfaces, ISourceBlock<TOutput> and ITargetBlock<TInput>, which the built-in blocks implement and which you can implement as well.  Then you can link up your blocks with others, just as is done for the custom blocks, all of the extension methods that are built in and that work with these interfaces will apply to your block as well, etc.  You can build custom blocks in a few ways, and there's actually a short section on doing so in the document at https://www.microsoft.com/downloads/en/details.aspx?FamilyID=d5b3e1f8-c672-48e8-baf8-94f05b431f5c&displaylang=en, which you may find useful.

    Regarding handles large numbers of small data packets, it should work well for that situation.  If you find otherwise, please do let us know (note that we're still doing a fair amount of optimization to further decrease overheads and allocations, so this should also get better in the future).

  • User profile image
    Doncwan

    Hi,

    With regards to the ConcurrentExclusiveSchedulerPair; you mention we can have a single ActionBlock on the ExclusiveScheduler modifying shared state and others on the ConcurrentProcessor reading this state. Surely we should be passing the state around by pipelining it through the ActionBlocks? If not then If we do not do this then presumably we still have the requirement to use memory fences?

  • User profile image
    Gustavo

    Hi guys, excellent work! It's looking promising, I really liked the last example where you don't need lock's to read or write some shared state.
    However when trying to digest this a little and as I read the documentation, I could not help to notice that there's some functionality already offered with Rx. Do you guys feel that way too? How do you guys compare these two products?

  • User profile image
    aL_

    awsome stuff Smiley

    i wonder how it relates to Rx.. it doesnt feel exactly the same but still quite similar in a lot of ways Smiley

  • User profile image
    Jeno

    It would be great if we could use this library as a wrapper around the Microsoft HPC Pack 2008 R2 SDK (Microsoft.Hpc.*) somehow. The API concepts seem to be very similar. I would just love to use one API and don't have to care about much whether I am scheduling tasks on one machine or on a cluster. Some APIs, languages (erlang) etc already try to blur the boundaries between parallel computing and distributed computing so the idea is not very new. I think TPL Dataflow will be totally awesome and I really liked the presentation. The only reason why I mention HPC because at the end of the video suggestions/feature wish list, etc were encouraged. Smiley

Add Your 2 Cents