Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ

Download

Right click “Save as…”

  • High Quality WMV (PC)
  • MP3 (Audio only)
  • MP4 (iPhone, Android)
  • Mid Quality WMV (Lo-band, Mobile)
  • WMV (WMV Video)

Microsoft Research recently announced the availability, under Academic Licensing, of Dryad, an infrastructure which allows a programmer to use the resources of a computer cluster or a data center for running data-parallel programs. A Dryad programmer can use thousands of machines, each of them with multiple processors or cores, without knowing anything about concurrent programming.
 
DryadLINQ is the managed high level programming abstraction used to compose Dryad vertex topology graphs that the Dryad infrastructure uses to partition parallel computations. Here, Erik Meijer and Dryad team member Roger Barga discuss Drayad and DryadLINQ at a high level so that most of us can understand the implications, history and future of Dryad. This is an introductory piece. Erik and I will dive deep into Dryad with one of the scientists behind it in the second part of this Expert to Expert mini series on Dryad. UPDATE: The Going Deep episode on Dryad is now live.

Enjoy! This is incredible and important technology for simplifying the inherent complexity of distributed computation in the cloud. In essence, DryadLINQ enables a sequential programming experience over what will execute across potentially thousands of machines (depending upon the computational complexity of the program) concurrently.  Much to learn here. Channel 9 will help teach.

Editorial note: When we discuss native code and the implementation of Dryad, the focus is on DryadLINQ not the Dryad infrastructure and low level vertex APIs, which are written in C++. Just to be clear...

Useful links:

Connect site: http://connect.microsoft.com/site/sitehome.aspx?SiteID=891

ER Website on Academic Use: http://research.microsoft.com/en-us/collaboration/tools/dryad.aspx

MSR Info: http://research.microsoft.com/en-us/projects/dryadlinq/

Tags:

Follow the Discussion

  • Allan LindqvistaL_ Kinect ftw

    ive never heard of dryad but it seems very cool Smiley

    i have a question though, does it require windows hpc? it would be really awsome if you could run a client as a service on a bunch of desktops that just sit idle at night anyway and leverage those for cluster computing needs Smiley

     

    im downloading now to check it out but it seemed like win hpc was required :/

     

    --edit--

    ah, i see the clients are just ordinary clients (i dont know how win hpc works at all Smiley ) but from what ive gathered so far the hoc server is still required to coordinate all the clients, is that correct?

     

    i was hoping to do some test on the home network but maybe thats not possible..

    cant wait to hear more about this stuff though Smiley

  • CharlesCharles Welcome Change

    Yes. A Windows HPC Server 2008 cluster is required (cluster size = 2 to n).

    C

  • BassBass Knows the way the wind is flowing.

    I've been looking for something like this for .NET for awhile. It's looks very similar to Google MapReduce, and Apache Hadoop, except it integrates with .NET and even Linq which is pretty damn cool.

     

    But seriously the license you guys put it under is lame as hell. Try Ms-PL next time, you'll like it. Smiley

  • CharlesCharles Welcome Change

    We do talk about the differences between MapReduce and Dryad in this interview. In fact, I ask this exact question....

     

    There are significant differences, in fact, beyond the higher level programming abstractions (like DryadLINQ). Also, as mentioned in the post description text, if you are a native developer (C++) you can muck around with the vertex algorithms and, topology map rewriting, etc. We mischaracterized this in the interview.

     

    C

  • BassBass Knows the way the wind is flowing.

    It was a somewhat interesting video.

     

    MapReduce is an in fact a fairly high level programming abstraction. If you are programmer you only have to write two functions: map and reduce. Smiley  From what I understand Dyrad is even higher level in that it can describe a graph of operations that might take multiple maps and reduces otherwise. Basically it helps you write something more concise that might require a larger program with MapReduce. I think that's just the nature of Linq in action. Linq can represent a graph of operations in a single statement, to so easy to do with standard code. I don't think this signifies some mathematical difference between the two at the low level (I didn't get any hint that Dyrad could do parallelize things that MapReduce can't).

     

    But DyradLINQ might be the most awesome thing since slice bread, but right now the only significant difference to me about something like Hadoop or this DyradLINQ is I can not legally use DyradLINQ. I don't care if you told me DyradLINQ was worse the Hadoop, the fact that it is designed for .NET means I would use it.

     

    So you can sell it to me but quite frankly it's useless unless you license it better. Hadoop is fully open source and you can use it in commercial products. Why is DyradLINQ so restricted?

     

  • CharlesCharles Welcome Change

    Restriction at this point is a function of the state of the technology. They are not ready to make it freely available. In some sense, this is more of a restricted preview at this point, but will this always be the case? Unlikely... That said, the Dryad People are the only ones who can speak to this accurately.

     

    C

     

    PS: Hopefully you'll find the E2E with Michael Isard (a scientist on the team) more than somewhat interesting (I think the interview with Roger does a great job introducing the technology and rationale that will be broadly understood by non-developers)...

  • Terrific video!  Looking forward to the Deep Dive!

  • Sven GrootSven Groot Don't worry... I'm a doctor.

    Charles, thanks for listening to my request for this video! I'm definitely looking forward to the next one.

  • None of the links listed in the post for Dryad/DryadLinq/Academic licensing work ( show  runtime error-custom errors) . BTW if you bing those the pages show the same runtime error.

  • CharlesCharles Welcome Change

    I'm not seeing this problem. Could you elaborate?
    C

  • Seems to be working properly now.

  • CharlesCharles Welcome Change

    No problem! I think Dryad is amazing and has a bright future (unfortunately, we can't speak to what that means exactly just yet...). I will release the E2E with Erik and Michael Isard next week. Look for it then.

    C

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.