Stephen Toub: Inside TPL Dataflow

Download this episode

Download Video

Description

TPL Dataflow (TDF), System.Threading.Tasks.Dataflow, builds upon the foundational layer for asynchronous and concurrent programming using Tasks provided in TPL in .NET 4. TDF is a complementary set of primitives to those delivered in TPL in .NET 4, addressing additional scenarios beyond those directly and easily supported by the original APIs. TPL Dataflow utilizes tasks, concurrent collections, tuples, and other features introduced in .NET 4 to bring support for parallel dataflow-based programming into the .NET Framework. It also directly integrates with new language support for tasks and asynchrony provided by both C# and Visual Basic, and with existing language support in .NET 4 for tasks provided by F#.

Several of you probably wonder how TDF differs from Rx in terms of core capabilities and specific usage scenarios. Well, Stephen talks about this right here.

As usual, Stephen goes deep and clearly articulates many technical details of TDF. Another great Toub session!

Please watch the great screencast Stephen posted to C9 a while back covering some simple examples of programming with TDF.

Make sure to ask techincal questions over in the TPL Dataflow forums, where the TDF engineers spend time and look for questions to answer: http://social.msdn.microsoft.com/Forums/en-US/tpldataflow/threads


Tune in. Enjoy. Learn.

 

Embed

Format

Available formats for this video:

Actual format may change based on video formats available and browser capability.

    The Discussion

    • User profile image
      LukePuplett

      Hello - this is probably a big ask, but since starting a new contract the internet connection is a shocker and I'm missing out on all this fabulous information. I'm fast becoming obsolete.

      Can you run off a super lightweight mobile encoding so I can synch with my WP7 and catch up on the train home?

      Oh go on. I'll be your best friend.

    • User profile image
      felix9

      Mmmm.... I just saw a proposed session in the TechEd session preference survey with an interesting title:

      Parallel Extensions in .NET 4.5

      We will explore what's new in the Framework for parallel programming in 4.5.  Specifically, we'll dive into the new TPL Dataflow library and the various improvements around partitioning, cancellation, etc.

    • User profile image
      staceyw

      Much coolness.  Could you standardize on Add/Take instead of mixing Post/Receive/Add/Take in all blocks? Everything else is ready to go! JK Smiley  Will have to spend some more time with dataflow now.

    • User profile image
      Charles
    • User profile image
      staceyw

      Shouldn't DeclinePermanently() instead be something like PostLast(params T[] lastItems) ? At some point, I am going to be the last poster. But doing this is not atomic:

      Post(1);
      // Not atomic.
      DeclinePermanently();

      Another producer could (and eventually will) sneak in between the calls. The only way to avoid that is code in another sync structure (i.e. lock), which your trying to avoid. Thoughts?

    • User profile image
      Toub

      Thanks for the question, staceyw.  I've responded to it over on the TPL Dataflow forum at http://social.msdn.microsoft.com/Forums/en-US/tpldataflow/thread/2c5840ad-75d8-4c80-8a63-3782f18ccbf5.

    • User profile image
      James Alexander

      Great video! I was curious how you'd project out of a block a type different then the input type and was pleased to see it in the last ten minutes. I'm very interested to see the relationship be Dataflow and Rx develop.

    • User profile image
      aL_

      thanks for the clarification about the relationship with rx Smiley i havent watched the whole video yet but it just struck me when you where talking about the coordination stuff in the beginning, TDF also feels a bit like the CCR.. its been a while since we saw George chrysanthakopoulos on c9 Smiley

      more TDF screencasts would also be awsome Smiley Perhaps custom TaskSchedulers based on TDF? seems like a very good fit

      @felix9:

      i knew it.... ron jacobs aaalmost slipped the name 4.5 in his wf talk at pdc Smiley hopefully that will include that xaml editor goodness they showed for silverlight [assuming its linked to a vs service pack]

      hopefully they'll start talking about 4.5 soon

    • User profile image
      Toub

      Hi aL_

      Good eye. TPL Dataflow is in part based on and inspired by concepts from CCR, along with concepts from Axum and Visual C++ 10's Asynchronous Agents library, so you'll see a lot of similarities in terms of the kinds of problems you can solve.  The APIs were redesigned to fit in well with the rest of the .NET Framework and to take advantage of what the Task Parallel Library and other .NET goodies have to offer, as well as redesigned to incorporate some more scenarios and patterns we felt were important.

      Regarding TaskSchedulers, sure, we'll see what we can pull together.  Note that these dataflow blocks can be targetted to run on any TaskScheduler instance, so you could configure a block to run on the thread pool, or on the UI, or in a concurrent/exclusive fashion, or whatever underlying semantics you want to achieve by plugging in a custom scheduler.  It sounds like you're interested in and asking about the other direction, implementing a TaskScheduler with a dataflow block (like ActionBlock)... you could certainly do that, too.

    • User profile image
      Brandon

      I'd love to see another video on this stuff.  One that focusus on how you solve specific problems using TDF would be best.  Like Rx, it seems like good stuff, but knowning when to use it isn't always obvious.

    • User profile image
      aL_

      @Toub:

      its great to see elements of the CCR getting pulled closer to the framework, i always felt the ccr got underrated or otherwise didnt enter the developer mainstream.

      you're right, i was referring to implementing TaskSchedulers using TDF, maybe like a throttling ui scheduler that ensures the ui thread doesnt get flooded with updates. Just a simple scheduler that process one task at the time [just an actionblock basically] is a good practical sample too i think. Really, all the custom schedulers in the parallel programming samples are good candidates Smiley

       

    • User profile image
      Frank Hileman

      I am curious as to why the blocks in the dataflow library are mutable. In a recent project, we designed a dataflow library in .net for processing energy usage data. Early on we decided to make the dataflow graph objects (blocks) immutable. In this library, the enumerators used to get data out of the dataflow graph are mutable. Enumerators are created on demand by block nodes when you begin to iterate over the dataflow graph (the enumerators form a parallel type of graph).
      Immutable block objects had advantages. We could share them between threads, and we could share blocks as input components in multiple larger graphs. By computing an immutable hash code on a block on construction, using the hash code of input blocks in the computation, we could compare block graphs for equality, and use them as keys in dictionaries that memoized graph outputs (as long as inputs were immutable, which was the case for historical data).
      In the TPL Dataflow library, blocks are both graph nodes and enumerators.

    • User profile image
      Charles

      @Frank Hileman: Frank, create an account on 9 so you get the benefits of text formatting... Smiley

      C

    • User profile image
      Frank Hileman

      Hi Charles,

      I did not notice I was not logged in. Maybe previously I had to log in to comment at all. Thanks.

    • User profile image
      Charles

      @Frank Hileman: Cool Smiley
      C

    • User profile image
      Gustavo

      Hi guys, great video, keep 'em coming!
      @Al_: you could have shared some credit with me on the Rx comments :P http://channel9.msdn.com/posts/TPL-Dataflow-Tour#c634317373800000000
      Cheers!

    Comments closed

    Comments have been closed since this content was published more than 30 days ago, but if you'd like to send us feedback you can Contact Us.