Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ
- Posted: Jul 16, 2009 at 9:43 AM
- 56,335 Views
- 12 Comments
Download
How do I download the videos?
- To download, right click the file type you would like and pick “Save target as…” or “Save link as…”
Why should I download videos from Channel9?
- It's an easy way to save the videos you like locally.
- You can save the videos in order to watch them offline.
- If all you want is to hear the audio, you can download the MP3!
Which version should I choose?
- If you want to view the video on your PC, Xbox or Media Center, download the High Quality WMV file (this is the highest quality version we have available).
- If you'd like a lower bitrate version, to reduce the download time or cost, then choose the Medium Quality WMV file.
- If you have a Zune, WP7, iPhone, iPad, or iPod device, choose the low or medium MP4 file.
- If you just want to hear the audio of the video, choose the MP3 file.
Right click “Save as…”
- High Quality WMV (PC, Xbox, MCE)
- MP3 (Audio only)
- MP4 (iPod, Zune HD)
- Mid Quality WMV (Lo-band, Mobile)
- WMV (WMV Video)
Microsoft Research recently announced the availability, under
Academic Licensing, of
Dryad, an infrastructure which allows a programmer to use the resources of a computer cluster or a data center for running data-parallel programs. A Dryad programmer can use thousands of machines, each of them with multiple processors or cores, without
knowing anything about concurrent programming.
DryadLINQ is the managed high level programming abstraction used to compose Dryad vertex topology graphs that the Dryad infrastructure
uses to partition parallel computations. Here, Erik Meijer and Dryad team member Roger Barga discuss Drayad and DryadLINQ at a high level so that most of us can understand the implications, history and future of Dryad. This is an introductory piece. Erik and
I will dive deep into Dryad with one of the scientists behind it in the second part of this Expert to Expert mini series on Dryad. UPDATE: The
Going Deep episode on Dryad is now live.
Enjoy! This is incredible and important technology for simplifying the inherent complexity of distributed computation in the cloud. In essence, DryadLINQ enables a sequential programming experience over what will execute across potentially thousands of machines
(depending upon the computational complexity of the program) concurrently. Much to learn here. Channel 9 will help teach.
Editorial note: When we discuss native code and the implementation of Dryad, the focus is on DryadLINQ
not the Dryad infrastructure and low level vertex APIs, which are written in C++. Just to be clear...
Useful links:
Connect site:
http://connect.microsoft.com/site/sitehome.aspx?SiteID=891
ER Website on Academic Use: http://research.microsoft.com/en-us/collaboration/tools/dryad.aspx
MSR Info: http://research.microsoft.com/en-us/projects/dryadlinq/
Comments Closed
Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation,
please create a new thread in our Forums,
or
Contact Us and let us know.
Follow the Discussion
ive never heard of dryad but it seems very cool
i have a question though, does it require windows hpc? it would be really awsome if you could run a client as a service on a bunch of desktops that just sit idle at night anyway and leverage those for cluster computing needs
im downloading now to check it out but it seemed like win hpc was required :/
--edit--
ah, i see the clients are just ordinary clients (i dont know how win hpc works at all
) but from what ive gathered so far the hoc server is still required to coordinate all the clients, is that correct?
i was hoping to do some test on the home network but maybe thats not possible..
cant wait to hear more about this stuff though
Yes. A Windows HPC Server 2008 cluster is required (cluster size = 2 to n).
C
I've been looking for something like this for .NET for awhile. It's looks very similar to Google MapReduce, and Apache Hadoop, except it integrates with .NET and even Linq which is pretty damn cool.
But seriously the license you guys put it under is lame as hell. Try Ms-PL next time, you'll like it.
We do talk about the differences between MapReduce and Dryad in this interview. In fact, I ask this exact question....
There are significant differences, in fact, beyond the higher level programming abstractions (like DryadLINQ). Also, as mentioned in the post description text, if you are a native developer (C++) you can muck around with the vertex algorithms and, topology map rewriting, etc. We mischaracterized this in the interview.
C
It was a somewhat interesting video.
MapReduce is an in fact a fairly high level programming abstraction. If you are programmer you only have to write two functions: map and reduce.
From what I understand Dyrad is even higher level in that it can describe a graph of operations that might
take multiple maps and reduces otherwise. Basically it helps you write something more concise that might require a larger program with MapReduce. I think that's just the nature of Linq in action. Linq can represent a graph of operations in a single statement,
to so easy to do with standard code. I don't think this signifies some mathematical difference between the two at the low level (I didn't get any hint that Dyrad could do parallelize things that MapReduce can't).
But DyradLINQ might be the most awesome thing since slice bread, but right now the only significant difference to me about something like Hadoop or this DyradLINQ is I can not legally use DyradLINQ. I don't care if you told me DyradLINQ was worse the Hadoop, the fact that it is designed for .NET means I would use it.
So you can sell it to me but quite frankly it's useless unless you license it better. Hadoop is fully open source and you can use it in commercial products. Why is DyradLINQ so restricted?
Restriction at this point is a function of the state of the technology. They are not ready to make it freely available. In some sense, this is more of a restricted preview at this point, but will this always be the case? Unlikely... That said, the Dryad People are the only ones who can speak to this accurately.
C
PS: Hopefully you'll find the E2E with Michael Isard (a scientist on the team) more than somewhat interesting (I think the interview with Roger does a great job introducing the technology and rationale that will be broadly understood by non-developers)...
Terrific video! Looking forward to the Deep Dive!
Charles, thanks for listening to my request for this video! I'm definitely looking forward to the next one.
None of the links listed in the post for Dryad/DryadLinq/Academic licensing work ( show runtime error-custom errors) . BTW if you bing those the pages show the same runtime error.
I'm not seeing this problem. Could you elaborate?
C
Seems to be working properly now.
No problem! I think Dryad is amazing and has a bright future (unfortunately, we can't speak to what that means exactly just yet...). I will release the E2E with Erik and Michael Isard next week. Look for it then.
C
Remove this comment
Remove this thread
close