Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ

Posted By: Charles | Jul 16th @ 9:43 AM | 52,594 Views | 12 Comments

Microsoft Research recently announced the availability, under Academic Licensing, of Dryad, an infrastructure which allows a programmer to use the resources of a computer cluster or a data center for running data-parallel programs. A Dryad programmer can use thousands of machines, each of them with multiple processors or cores, without knowing anything about concurrent programming.
 
DryadLINQ is the managed high level programming abstraction used to compose Dryad vertex topology graphs that the Dryad infrastructure uses to partition parallel computations. Here, Erik Meijer and Dryad team member Roger Barga discuss Drayad and DryadLINQ at a high level so that most of us can understand the implications, history and future of Dryad. This is an introductory piece. Erik and I will dive deep into Dryad with one of the scientists behind it in the second part of this Expert to Expert mini series on Dryad. UPDATE: The Going Deep episode on Dryad is now live.

Enjoy! This is incredible and important technology for simplifying the inherent complexity of distributed computation in the cloud. In essence, DryadLINQ enables a sequential programming experience over what will execute across potentially thousands of machines (depending upon the computational complexity of the program) concurrently.  Much to learn here. Channel 9 will help teach.

Editorial note: When we discuss native code and the implementation of Dryad, the focus is on DryadLINQ not the Dryad infrastructure and low level vertex APIs, which are written in C++. Just to be clear...

Useful links:

Connect site: http://connect.microsoft.com/site/sitehome.aspx?SiteID=891

ER Website on Academic Use: http://research.microsoft.com/en-us/collaboration/tools/dryad.aspx

MSR Info: http://research.microsoft.com/en-us/projects/dryadlinq/

Rating:
0
0
#Jul 16th @ 10:57 AM
aL_
aL_
Rx ftw

ive never heard of dryad but it seems very cool Smiley

i have a question though, does it require windows hpc? it would be really awsome if you could run a client as a service on a bunch of desktops that just sit idle at night anyway and leverage those for cluster computing needs Smiley

 

im downloading now to check it out but it seemed like win hpc was required :/

 

--edit--

ah, i see the clients are just ordinary clients (i dont know how win hpc works at all Smiley ) but from what ive gathered so far the hoc server is still required to coordinate all the clients, is that correct?

 

i was hoping to do some test on the home network but maybe thats not possible..

cant wait to hear more about this stuff though Smiley

#Jul 16th @ 11:25 AM
Charles
Charles
Welcome Change

Yes. A Windows HPC Server 2008 cluster is required (cluster size = 2 to n).

C

#Jul 16th @ 11:55 AM
Bass
Bass
www.s​preadfirefox.c​om/5years/

I've been looking for something like this for .NET for awhile. It's looks very similar to Google MapReduce, and Apache Hadoop, except it integrates with .NET and even Linq which is pretty damn cool.

 

But seriously the license you guys put it under is lame as hell. Try Ms-PL next time, you'll like it. Smiley

#Jul 16th @ 12:03 PM
Charles
Charles
Welcome Change

We do talk about the differences between MapReduce and Dryad in this interview. In fact, I ask this exact question....

 

There are significant differences, in fact, beyond the higher level programming abstractions (like DryadLINQ). Also, as mentioned in the post description text, if you are a native developer (C++) you can muck around with the vertex algorithms and, topology map rewriting, etc. We mischaracterized this in the interview.

 

C

#Jul 16th @ 12:37 PM
Bass
Bass
www.s​preadfirefox.c​om/5years/

It was a somewhat interesting video.

 

MapReduce is an in fact a fairly high level programming abstraction. If you are programmer you only have to write two functions: map and reduce. Smiley  From what I understand Dyrad is even higher level in that it can describe a graph of operations that might take multiple maps and reduces otherwise. Basically it helps you write something more concise that might require a larger program with MapReduce. I think that's just the nature of Linq in action. Linq can represent a graph of operations in a single statement, to so easy to do with standard code. I don't think this signifies some mathematical difference between the two at the low level (I didn't get any hint that Dyrad could do parallelize things that MapReduce can't).

 

But DyradLINQ might be the most awesome thing since slice bread, but right now the only significant difference to me about something like Hadoop or this DyradLINQ is I can not legally use DyradLINQ. I don't care if you told me DyradLINQ was worse the Hadoop, the fact that it is designed for .NET means I would use it.

 

So you can sell it to me but quite frankly it's useless unless you license it better. Hadoop is fully open source and you can use it in commercial products. Why is DyradLINQ so restricted?

 

#Jul 16th @ 12:42 PM
Charles
Charles
Welcome Change

Restriction at this point is a function of the state of the technology. They are not ready to make it freely available. In some sense, this is more of a restricted preview at this point, but will this always be the case? Unlikely... That said, the Dryad People are the only ones who can speak to this accurately.

 

C

 

PS: Hopefully you'll find the E2E with Michael Isard (a scientist on the team) more than somewhat interesting (I think the interview with Roger does a great job introducing the technology and rationale that will be broadly understood by non-developers)...

#Jul 16th @ 1:27 PM
joechung
joechung

Terrific video!  Looking forward to the Deep Dive!

#Jul 16th @ 9:06 PM
Sven Groot
Sven Groot
My name has 9 letters. Coincidence? I think not...

Charles, thanks for listening to my request for this video! I'm definitely looking forward to the next one.

#Jul 17th @ 8:51 AM
Ribald
Ribald

None of the links listed in the post for Dryad/DryadLinq/Academic licensing work ( show  runtime error-custom errors) . BTW if you bing those the pages show the same runtime error.

Microsoft Communities