Maestro: A Managed Domain Specific Language For Concurrent Programming (Channel 9)

Josh Phillips(PM), Niklas Gustafsson(Architect), and Artur Laksberg(Developer) of the Parallel Computing Platform Team spend some time with me to discuss a managed (.NET-based) DSL (Domain Specific Language) for concurrent programming, Maestro. Maestro incorporates well-entrenched language patterns (imperative, OO, C style syntax, etc) and language constructs (channels, agents, domains) in a compelling way to make concurrent composition more accessible and familiar to the legions of sequential code composers.

Here we dig into the architecture and design of the Maestro language and compiler as well as discuss the philosophy behind this incubation project (at this point in time there are no plans to release Maestro as a product - it's a research project, an incubation...). Why create another language to help solve the Concurrency Problem? What's the advantage over implementing a library (this is .NET after all -> CLR + BCL = most of the power of the platform)? There's obviously good reasons for implementig Maestro as a language, but you'll need to watch and listen to find out.

Enjoy.

Note: After this interview was conducted and posted to Channel 9, the Maestro team has renamed their technology to Axum. So, they are now the Axum team and the managed DSL for concurrent programming they're incubating is called Axum. Smiley

Follow the Discussion

Follow
Oops, something didn't work.

Try again?

Sign In to be begin to follow these comments

What does this mean?
Following an item on Channel 9 allows you to watch for new content and comments that you are interested in. You need to be signed in to Channel 9 to use this feature.

Getting "follow" information

Stop following these comments

Start following these comments

What does this mean?
Following an item on Channel 9 allows you to watch for new content and comments that you are interested in and view them all on your notifications page.

Removing your "follow"

Setting up your "follow"

Did you know you can
sign up for email notifications?
- Subscribe to this comments

ilmar

Dec 22, 2008 at 1:10 PM

Super interesting stuff, keep this coming!
ZippyV Fired Up

Dec 22, 2008 at 6:06 PM

I think I get the concept of it but what are some of the applications that could benefit from this? A business application, a game, ... ?
Charles Welcome Change

Dec 22, 2008 at 10:34 PM

Any application with concurrent tendencies It's a DSL, but the specific domain is concurrency/parallelism and what it offers (at this point theoretically since you can't play with it yet) is a readily comprehensible way (providing sequential semantics that you're most likely very used to) to program multiple asychnronous domains that communicate via channels and the agents they possess. At least, this is my understanding after listening to the big brains explain their thinking.

C
mkoops

Dec 23, 2008 at 1:37 PM

Charles,
Brilliant stuff, one question though - Why is the MP4 version 500 MB v 160 odd for WMV?

Mark
Charles Welcome Change

Dec 23, 2008 at 1:40 PM

In the new year, this will be fixed (most likely with the addition of a low res MP4). Right now, MP4's are being encoded at a high bit rate (maybe too high). Sampy (Mike Sampson), the inventor of Magic Folder (the application that does all of the processing magic for our media files) is aware of this and has plans to address this.

Glad you liked the interview!

C
esoteric λ
Dec 23, 2008 at 4:20 PM
I have thoroughly enjoyed all the M and Parallelism talks. For parallelism there is enough to keep you busy for quite some time.

So now we know that Maestro is built on CCR.

CLR
- TPL
- CCR
- Maestro
- ?
C/C++
CRT
PPL
?

I would be interested in

Native vs managed integration? - CRT/PPL vs TPL/threadpool
TPL/CCR integration?
What scale do you expect what libraries to be able to manage?
What level of concurrency do what libraries efficiently enable?
How do what concurrency paradigms affect program complexity and predictability and how future proof will they be
In the past sequential programs became faster with faster processors; we cannot just expect parallel programs to continue to scale - exposing concurrency at some level may seem fine, but it will start to hurt at some level -- program flow prevents absolute parallelism; maximization how? - Niklas talks about speculative parallelism...
Dogfooding -

Where is Microsoft be dogfooding these libraries - applications and services; what is being done in Windows itself, going forward?
What will it dogfood where and why?
How deep into the OS will it dogfood and what will be used where?
What does concurrency mean for Windows?

Implicit concurrency

Is it feasible?
What languages?
What timescale?
On what scale?
Dynamic program optimization for needed parallelism (no. of cores) or static partitioning with "fixed-grained" tasks?

Parallelism concepts, how many are there both research-stage and applied
Will the asking ever stop?
nagunn

Dec 27, 2008 at 10:11 PM

It's interesting that the CCR is the machinery behind all this. It's an excellent piece of work and a shame (to my mind) that its adoption does not seem to have been wider. I think the actor model is certainly one of the better models for taming concurrency, and the CCR provides a strong foundation for it, amongst other things.

I think anything that Microsoft do to promote this model is going to be a good thing in the long run, and I look forward to a possible future release of Maestro in some form.

The deep isolation present in Erlang, which supports the actor model as a first class concept, is also available in Decentralized Software Services, which builds atop of the CCR and is available to C# developers now.

For those interested in the functional perspective, there is a non-blocking asynchronous message passing implementation available within the F# CTP and I've been toying with using the CCR from within F# computation expressions to simplify the syntax somewhat.
Charles Welcome Change

Dec 29, 2008 at 2:05 PM

What makes you think that CCR adoption is not wide? Consider that many people are using CCR and in some cases very novel (and unexpected ways, like MySpace...).

C
nagunn

Dec 30, 2008 at 12:12 AM

Charles hi,

I don't know how wide the adoption has been of course. I did say that it *seemed* not to have been wide and of course I could be wrong - wouldn't be the first time - but using the only benchmarks I have, namely (1) the number of articles in the blogosphere on CCR and (2) the amount of activity on the CCR/DSS forums, it's not an entirely invalid assumption.

Of course it could be that not many think the CCR is worth blogging about, or that it is such an obviously intuitive model that not many need the forums, but I think neither of these is true and your own comments in the video suggest that some developers may consider the programming model something of a mental leap from where they currently are.

I personally think the CCR is a really powerful model and *well worth* the investment. We have commercial licences for it at my place of work and maybe one day I'll be able to talk about that.

On the other hand, it could be that by helping solve issues both of concurrent systems design and of scalability, everyone is using it, but keeping quiet, hoping that no-one else will. But I doubt it...

Happy New Year!
Charles Welcome Change

Dec 30, 2008 at 4:25 PM

CCR is not as popular as it could be due to how it requires a shift in "concpetual programming". There are many teams inside of Microsoft that have implemented some concurrent system using CCR (generally in incubation stage or experimentation). Anybody who does anything with Microsoft Robotics is using the CCR (albeit implicitly). I think I understand your general question: "Why isn't the CCR more poplular than it appears to be?"

I will make sure to ask the creator of the CCR this question at our next interview. I know that George will probably disagree with the notion of a lack of CCR adoption, but he'll have more concrete evidence than I can come up with off the top of my head.

Thanks for the discussion!

C
staceyw Before C# there was darkness...

Jan 06, 2009 at 12:45 PM

Very cool. I remember a thread me and evildictator had on this very idea that was similar. Grab patterns from Erlang and CCR and wrap it in a language/combiler pair that will enforce correct-by-construction design. I also like that the patterns itself is a simpler version of STM. Essencially the R/W agents are a localized mini-STM without needing to implement a full STM. And that you can even replace it with STM under the covers as an implementation detail and the syntax does not change much and the model remains the same. And think you still have the flex of using PTL libraries "inside" your agent. It smells like this is the right approach and abstraction going forward IMHO. Nice work, hope to use some ctp bits in future.

Also thinking... Suppose the windows UI thread loop was *replaced with a single Agent. Now we don't have to worry about IsInvokeRequired() or BeginInvoke() or even hanging the UI thread inside callback method. We have a clear seperation and threads don't matter in terms of UI correctness. This would also mean we can forget about what thread we got a callback on (i.e. timer event) as it will not matter. The UI agent can then be free to have any type of parellelize it can correctly handle inside itself and all is hidden from programmer. It would mean we would have to think about explicitly blocking/pausing other UI actions if you don't want people clicking same button twice before last action completes for example. A light begins to appear...
NiklasGustafson

Jan 07, 2009 at 1:39 PM

I think it's a good idea to explore how an agent-isolated model can be used to simplify expressing "ownership" of the UI. It certainly seems worth exploring in some depth.

Another interesting angle is to look at how Maestro contracts can be used to express correct ordering of UI events, to control enabling/disabling of UI elements and force some formalism on UI interactions. Don't know whether that's actually going to prove useful, but it seems interesting to explore.