Maestro: A Managed Domain Specific Language For Concurrent Programming

The Discussion

• 

  Super interesting stuff, keep this coming!

• 

  I think I get the concept of it but what are some of the applications that could benefit from this? A business application, a game, ... ?

• 

  Any application with concurrent tendencies It's a DSL, but the specific domain is concurrency/parallelism and what it offers (at this point theoretically since you can't play with it yet) is a readily comprehensible way (providing sequential semantics that you're most likely very used to) to program multiple asychnronous domains that communicate via channels and the agents they possess. At least, this is my understanding after listening to the big brains explain their thinking.
  
  C

• 

  Charles,
  Brilliant stuff, one question though -  Why is the MP4 version 500 MB v 160 odd for WMV?

  Mark

• 

  In the new year, this will be fixed (most likely with the addition of a low res MP4). Right now, MP4's are being encoded at a high bit rate (maybe too high). Sampy (Mike Sampson), the inventor of Magic Folder (the application that does all of the processing magic for our media files) is aware of this and has plans to address this.
  
  Glad you liked the interview!
  
  C

• 

  I have thoroughly enjoyed all the M and Parallelism talks. For parallelism there is enough to keep you busy for quite some time.

  
  

  So now we know that Maestro is built on CCR.

  
  

  CLR

  • TPL
  • CCR
  • Maestro
  • ?

  C/C++

  • CRT
  • PPL
  • ?

  I would be interested in

  • Native vs managed integration? - CRT/PPL vs TPL/threadpool
  • TPL/CCR integration?
  • What scale do you expect what libraries to be able to manage?
  • What level of concurrency do what libraries efficiently enable?
  • How do what concurrency paradigms affect program complexity and predictability and how future proof will they be
  • In the past sequential programs became faster with faster processors; we cannot just expect parallel programs to continue to scale - exposing concurrency at some level may seem fine, but it will start to hurt at some level -- program flow prevents absolute parallelism; maximization how? - Niklas talks about speculative parallelism...
  • Dogfooding -
    • Where is Microsoft be dogfooding these libraries - applications and services; what is being done in Windows itself, going forward?
    • What will it dogfood where and why?
    • How deep into the OS will it dogfood and what will be used where?
    • What does concurrency mean for Windows?
  • Implicit concurrency
    • Is it feasible?
    • What languages?
    • What timescale?
    • On what scale?
    • Dynamic program optimization for needed parallelism (no. of cores) or static partitioning with "fixed-grained" tasks?
  • Parallelism concepts, how many are there both research-stage and applied
  • Will the asking ever stop?

  
  

• 

  It's interesting that the CCR is the machinery behind all this. It's an excellent piece of work and a shame (to my mind) that its adoption does not seem to have been wider. I think the actor model is certainly one of the better models for taming concurrency, and the CCR provides a strong foundation for it, amongst other things.
  
  I think anything that Microsoft do to promote this model is going to be a good thing in the long run, and I look forward to a possible future release of Maestro in some form.
  
  The deep isolation present in Erlang, which supports the actor model as a first class concept, is also available in Decentralized Software Services, which builds atop of the CCR and is available to C# developers now.
  
  For those interested in the functional perspective, there is a non-blocking asynchronous message passing implementation available within the F# CTP and I've been toying with using the CCR from within F# computation expressions to simplify the syntax somewhat.
  

• 

  What makes you think that CCR adoption is not wide? Consider that many people are using CCR and in some cases very novel (and unexpected ways, like MySpace...).
  
  C

• 

  Charles hi,
  
  I don't know how wide the adoption has been of course. I did say that it *seemed* not to have been wide and of course I could be wrong - wouldn't be the first time - but using the only benchmarks I have, namely (1) the number of articles in the blogosphere on CCR and (2) the amount of activity on the CCR/DSS forums, it's not an entirely invalid assumption.
  
  Of course it could be that not many think the CCR is worth blogging about, or that it is such an obviously intuitive model that not many need the forums, but I think neither of these is true and your own comments in the video suggest that some developers may consider the programming model something of a mental leap from where they currently are.
  
  I personally think the CCR is a really powerful model and *well worth* the investment. We have commercial licences for it at my place of work and maybe one day I'll be able to talk about that.
  
  On the other hand, it could be that by helping solve issues both of concurrent systems design and of scalability, everyone is using it, but keeping quiet, hoping that no-one else will. But I doubt it...
  
  Happy New Year!
  

• 

  CCR is not as popular as it could be due to how it requires a shift in "concpetual programming". There are many teams inside of Microsoft that have implemented some concurrent system using CCR (generally in incubation stage or experimentation). Anybody who does anything with Microsoft Robotics is using the CCR (albeit implicitly). I think I understand your general question: "Why isn't the CCR more poplular than it appears to be?"
  
  I will make sure to ask the creator of the CCR this question at our next interview. I know that George will probably disagree with the notion of a lack of CCR adoption, but he'll have more concrete evidence than I can come up with off the top of my head.
  
  Thanks for the discussion!
  
  C 

• 

  Very cool.  I remember a thread me and evildictator had on this very idea that was similar.  Grab patterns from Erlang and CCR and wrap it in a language/combiler pair that will enforce correct-by-construction design.  I also like that the patterns itself is a simpler version of STM.  Essencially the R/W agents are a localized mini-STM without needing to implement a full STM.  And that you can even replace it with STM under the covers as an implementation detail and the syntax does not change much and the model remains the same.   And think you still have the flex of using PTL libraries "inside" your agent.   It smells like this is the right approach and abstraction going forward IMHO.  Nice work, hope to use some ctp bits in future.
  
  Also thinking...  Suppose the windows UI thread loop was *replaced with a single Agent.  Now we don't have to worry about IsInvokeRequired() or BeginInvoke() or even hanging the UI thread inside callback method.  We have a clear seperation and threads don't matter in terms of UI correctness.  This would also mean we can forget about what thread we got a callback on (i.e. timer event) as it will not matter.  The UI agent can then be free to have any type of parellelize it can correctly handle inside itself and all is hidden from programmer.  It would mean we would have to think about explicitly blocking/pausing other UI actions if you don't want people clicking same button twice before last action completes for example.  A light begins to appear...

• 

  I think it's a good idea to explore how an agent-isolated model can be used to simplify expressing "ownership" of the UI. It certainly seems worth exploring in some depth.
  
  Another interesting angle is to look at how Maestro contracts can be used to express correct ordering of UI events, to control enabling/disabling of UI elements and force some formalism on UI interactions. Don't know whether that's actually going to prove useful, but it seems interesting to explore.