Comment Feed for Channel 9 - Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ http://ecn.channel9.msdn.com/o9/previewImages/100/478816_100x75.jpg Channel 9 - Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ Microsoft Research recently announced the availability, under Academic Licensing, of Dryad, an infrastructure which allows a programmer to use the resources of a computer cluster or a data center for running data-parallel programs. A Dryad programmer can use thousands of machines, each of them with multiple processors or cores, without knowing anything about concurrent programming. DryadLINQ is the managed high level programming abstraction used to compose Dryad vertex topology graphs that the Dryad infrastructure uses to partition parallel computations. Here, Erik Meijer and Dryad team member Roger Barga discuss Drayad and DryadLINQ at a high level so that most of us can understand the implications, history and future of Dryad. This is an introductory piece. Erik and I will dive deep into Dryad with one of the scientists behind it in the second part of this Expert to Expert mini series on Dryad. UPDATE: The Going Deep episode on Dryad is now live.Enjoy! This is incredible and important technology for simplifying the inherent complexity of distributed computation in the cloud. In essence, DryadLINQ enables a sequential programming experience over what will execute across potentially thousands of machines (depending upon the computational complexity of the program) concurrently.  Much to learn here. Channel 9 will help teach.Editorial note: When we discuss native code and the implementation of Dryad, the focus is on DryadLINQ not the Dryad infrastructure and low level vertex APIs, which are written in C++. Just to be clear...Useful links:Connect site: http://connect.microsoft.com/site/sitehome.aspx?SiteID=891 ER Website on Academic Use: http://research.microsoft.com/en-us/collaboration/tools/dryad.aspx MSR Info: http://research.microsoft.com/en-us/projects/dryadlinq/ en Tue, 21 May 2013 09:48:03 GMT Tue, 21 May 2013 09:48:03 GMT Rev9 Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ ive never heard of dryad but it seems very cool Smiley

i have a question though, does it require windows hpc? it would be really awsome if you could run a client as a service on a bunch of desktops that just sit idle at night anyway and leverage those for cluster computing needs Smiley

 

im downloading now to check it out but it seemed like win hpc was required :/

 

--edit--

ah, i see the clients are just ordinary clients (i dont know how win hpc works at all Smiley ) but from what ive gathered so far the hoc server is still required to coordinate all the clients, is that correct?

 

i was hoping to do some test on the home network but maybe thats not possible..

cant wait to hear more about this stuff though Smiley

posted by aL_

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833638630000000 Thu, 16 Jul 2009 17:57:43 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833638630000000 aL_ Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ Yes. A Windows HPC Server 2008 cluster is required (cluster size = 2 to n).

C

posted by Charles

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833655360000000 Thu, 16 Jul 2009 18:25:36 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833655360000000 Charles Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ I've been looking for something like this for .NET for awhile. It's looks very similar to Google MapReduce, and Apache Hadoop, except it integrates with .NET and even Linq which is pretty damn cool.

 

But seriously the license you guys put it under is lame as hell. Try Ms-PL next time, you'll like it. Smiley

posted by Bass

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833673050000000 Thu, 16 Jul 2009 18:55:05 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833673050000000 Bass Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ We do talk about the differences between MapReduce and Dryad in this interview. In fact, I ask this exact question....

 

There are significant differences, in fact, beyond the higher level programming abstractions (like DryadLINQ). Also, as mentioned in the post description text, if you are a native developer (C++) you can muck around with the vertex algorithms and, topology map rewriting, etc. We mischaracterized this in the interview.

 

C

posted by Charles

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833678030000000 Thu, 16 Jul 2009 19:03:23 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833678030000000 Charles Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ It was a somewhat interesting video.

 

MapReduce is an in fact a fairly high level programming abstraction. If you are programmer you only have to write two functions: map and reduce. Smiley  From what I understand Dyrad is even higher level in that it can describe a graph of operations that might take multiple maps and reduces otherwise. Basically it helps you write something more concise that might require a larger program with MapReduce. I think that's just the nature of Linq in action. Linq can represent a graph of operations in a single statement, to so easy to do with standard code. I don't think this signifies some mathematical difference between the two at the low level (I didn't get any hint that Dyrad could do parallelize things that MapReduce can't).

 

But DyradLINQ might be the most awesome thing since slice bread, but right now the only significant difference to me about something like Hadoop or this DyradLINQ is I can not legally use DyradLINQ. I don't care if you told me DyradLINQ was worse the Hadoop, the fact that it is designed for .NET means I would use it.

 

So you can sell it to me but quite frankly it's useless unless you license it better. Hadoop is fully open source and you can use it in commercial products. Why is DyradLINQ so restricted?

 

posted by Bass

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833698450000000 Thu, 16 Jul 2009 19:37:25 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833698450000000 Bass Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ Restriction at this point is a function of the state of the technology. They are not ready to make it freely available. In some sense, this is more of a restricted preview at this point, but will this always be the case? Unlikely... That said, the Dryad People are the only ones who can speak to this accurately.

 

C

 

PS: Hopefully you'll find the E2E with Michael Isard (a scientist on the team) more than somewhat interesting (I think the interview with Roger does a great job introducing the technology and rationale that will be broadly understood by non-developers)...

posted by Charles

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833701660000000 Thu, 16 Jul 2009 19:42:46 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833701660000000 Charles Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ Terrific video!  Looking forward to the Deep Dive!

posted by joechung

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833728320000000 Thu, 16 Jul 2009 20:27:12 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633833728320000000 joechung Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ Charles, thanks for listening to my request for this video! I'm definitely looking forward to the next one.

posted by Sven Groot

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834003860000000 Fri, 17 Jul 2009 04:06:26 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834003860000000 Sven Groot Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ None of the links listed in the post for Dryad/DryadLinq/Academic licensing work ( show  runtime error-custom errors) . BTW if you bing those the pages show the same runtime error.

posted by Ribald

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834427070000000 Fri, 17 Jul 2009 15:51:47 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834427070000000 Ribald Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ I'm not seeing this problem. Could you elaborate?
C

posted by Charles

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834477570000000 Fri, 17 Jul 2009 17:15:57 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834477570000000 Charles Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ Seems to be working properly now.

posted by Ribald

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834542130000000 Fri, 17 Jul 2009 19:03:33 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834542130000000 Ribald Re: Expert to Expert: Erik Meijer and Roger Barga - Introduction to Dryad and DryadLINQ No problem! I think Dryad is amazing and has a bright future (unfortunately, we can't speak to what that means exactly just yet...). I will release the E2E with Erik and Michael Isard next week. Look for it then.

C

posted by Charles

]]> http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834661430000000 Fri, 17 Jul 2009 22:22:23 GMT http://channel9.msdn.com/Blogs/Charles/Expert-to-Expert-Erik-Roger-Barga-Introduction-to-Dryad-and-DryadLINQ#c633834661430000000 Charles