Stephen Toub: Inside TPL Dataflow

Hello - this is probably a big ask, but since starting a new contract the internet connection is a shocker and I'm missing out on all this fabulous information. I'm fast becoming obsolete.

Can you run off a super lightweight mobile encoding so I can synch with my WP7 and catch up on the train home?

Oh go on. I'll be your best friend.

Mmmm.... I just saw a proposed session in the TechEd session preference survey with an interesting title:

Parallel Extensions in .NET 4.5

We will explore what's new in the Framework for parallel programming in 4.5.  Specifically, we'll dive into the new TPL Dataflow library and the various improvements around partitioning, cancellation, etc.

Much coolness.  Could you standardize on Add/Take instead of mixing Post/Receive/Add/Take in all blocks? Everything else is ready to go! JK   Will have to spend some more time with dataflow now.

Shouldn't DeclinePermanently() instead be something like PostLast(params T[] lastItems) ? At some point, I am going to be the last poster. But doing this is not atomic:

Post(1);
// Not atomic.
DeclinePermanently();

Another producer could (and eventually will) sneak in between the calls. The only way to avoid that is code in another sync structure (i.e. lock), which your trying to avoid. Thoughts?

Great video! I was curious how you'd project out of a block a type different then the input type and was pleased to see it in the last ten minutes. I'm very interested to see the relationship be Dataflow and Rx develop.

thanks for the clarification about the relationship with rx  i havent watched the whole video yet but it just struck me when you where talking about the coordination stuff in the beginning, TDF also feels a bit like the CCR.. its been a while since we saw George chrysanthakopoulos on c9

more TDF screencasts would also be awsome Perhaps custom TaskSchedulers based on TDF? seems like a very good fit

@felix9:

i knew it.... ron jacobs aaalmost slipped the name 4.5 in his wf talk at pdc hopefully that will include that xaml editor goodness they showed for silverlight [assuming its linked to a vs service pack]

hopefully they'll start talking about 4.5 soon

I'd love to see another video on this stuff.  One that focusus on how you solve specific problems using TDF would be best.  Like Rx, it seems like good stuff, but knowning when to use it isn't always obvious.

@Toub:

its great to see elements of the CCR getting pulled closer to the framework, i always felt the ccr got underrated or otherwise didnt enter the developer mainstream.

you're right, i was referring to implementing TaskSchedulers using TDF, maybe like a throttling ui scheduler that ensures the ui thread doesnt get flooded with updates. Just a simple scheduler that process one task at the time [just an actionblock basically] is a good practical sample too i think. Really, all the custom schedulers in the parallel programming samples are good candidates

I am curious as to why the blocks in the dataflow library are mutable. In a recent project, we designed a dataflow library in .net for processing energy usage data. Early on we decided to make the dataflow graph objects (blocks) immutable. In this library, the enumerators used to get data out of the dataflow graph are mutable. Enumerators are created on demand by block nodes when you begin to iterate over the dataflow graph (the enumerators form a parallel type of graph).
Immutable block objects had advantages. We could share them between threads, and we could share blocks as input components in multiple larger graphs. By computing an immutable hash code on a block on construction, using the hash code of input blocks in the computation, we could compare block graphs for equality, and use them as keys in dictionaries that memoized graph outputs (as long as inputs were immutable, which was the case for historical data).
In the TPL Dataflow library, blocks are both graph nodes and enumerators.

Hi Charles,

I did not notice I was not logged in. Maybe previously I had to log in to comment at all. Thanks.

Hi guys, great video, keep 'em coming!
@Al_: you could have shared some credit with me on the Rx comments :P http://channel9.msdn.com/posts/TPL-Dataflow-Tour#c634317373800000000
Cheers!