Expert to Expert: Erik Meijer and Michael Isard - Inside Dryad

Posted By: Charles | Jul 21st @ 8:55 AM | 44,587 Views | 15 Comments
Microsoft Research recently announced the availability, under Academic Licensing, of Dryad, an infrastructure which allows a programmer to use the resources of a computer cluster or a data center for running data-parallel programs.

A Dryad programmer can use thousands of machines, each of them with multiple processors or cores, without knowing anything about concurrent programming.

That's a pretty heady statement. What does Dryad do, exactly, to enable this level of abstraction, shielding programmers from the incredibly complex world of distributed parallel computing? Does the level of abstraction impact the degree to which sophisticated programmers can interact with and control some of the low level mechanisms of the Dryad runtime? What is it about LINQ that made it the no-brainer managed programming abstraction for Dryad?

Simply, how does Dryad work? This is the core question that Erik and I had after our conversation with Roger Barga (part one of this E2E mini-series on Dryad and DryadLINQ - perhaps we should focus just on DryadLINQ next time, but for now, all the information in this conversation is certain to keep you very busy and answer many questions you may have after learning about Dryad in part one...). 

Lots of whiteboarding here. Put on your thinking caps!

Enjoy.
Rating:
0
0
#Jul 21st @ 9:55 AM
SoftwareWarrior
SoftwareWarrior
Software Science

I really, really like the idea of Dryad. Any idea when we can get our hand on the bits? Not just the client but the whole thing. In academia were are starting to test Hadoop, Eucalyptus, and the like. I would love to add Dryad to the mix. Any chance of this happening in the near future?

#Jul 21st @ 11:16 AM
SoftwareWarrior
SoftwareWarrior
Software Science

Thank you Charles, this is going to be a lot of fun!

#Jul 21st @ 11:29 AM
SoftwareWarrior
SoftwareWarrior
Software Science

Quick question is "Windows HPC Server 2008" cluster the same as "Windows Server 2008" cluster? If no, can we use a "Windows Server 2008" cluster? For that matter, can we use any Windows machine?

#Jul 21st @ 12:12 PM
Charles
Charles
Welcome Change

The requirement is Windows HPC Server 2008. This is a framework and runtime that is for use in an HPC cluster. Good question in terms of whether or not a Windows Server cluster would suffice, but it won't.

C

#Jul 21st @ 1:00 PM
SoftwareWarrior
SoftwareWarrior
Software Science

So close and yet so far, implementation wise, vanilla Windows boxes would have more traction. Everyone has Windows boxes around. Thank you all the same.

#Jul 21st @ 6:52 PM
staceyw
staceyw
Before C# there was darkness...

@softwarewarrior.  "So close and yet so far, implementation wise, vanilla Windows boxes would have more traction. Everyone has Windows boxes around." 

 

I was thinking the same thing.  Installing multiple HPCs is a pretty high bar to kick the tires or even for research purposes.  Not even sure why that is required.  Why does it have a dependancy on hpc?  IMO, it should be able to work on any win system (xp and above) by just installing a listener/worker process on a system (even dynamically via rexec.exe) and wait for directions.  The high road solution would be to enlist any/all windows computers in your org and bring them up and down as needed creating  virtual-compute farms.

#Jul 21st @ 7:41 PM
Bass
Bass
Channel 9, best used in moderation

Damn right and license it for non-academia while you are at it. I really think it's incredibly lame to discriminate like that. Fabulous job there on completely alienating commercial developers who need clustering and would love access to something like this. You know, the kind of people that actually make you money in the form of increased Windows sales.

 

Yes it does piss me off. What reaction do you expect to do when you show me something interesting and make it illegal for me to use it. Gratification? Hell no. This is really so lame.

 

I knew I should I sticked with Java. I'd actually have a real clustering solution today (Hadoop). Really .NET can be so much better if you all stopped with your overzealous software hoarding mindsets. But Java is looking better every day.

#Jul 21st @ 8:32 PM
Charles
Charles
Welcome Change

Good questions, all. You'll soon hear from the people who can best answer them.

C

Microsoft Communities