Bart De Smet: Observations on IQbservable - The Dual of IQueryable

The Discussion

• 

  Here's an image representation of the three dimensional graph of Rx's computational fabric:

   

  

• 

  I'm quite interested in using expression trees to create optimized IL code as Bart briefly mentions. Is there anything in the works for that? In essence an expression tree optimizer (LINQ to MSIL?).

   

  PS. Sum could be implemented such that it still returns an IMonad<T>, i.e. with a return type of

   

      I{Enumerable,Queryable,Observable,Qbservable}<T>

   

  First() would then really exit the monad.

• 

  Nice video and nice t-shirt.

   

  (That t-shirt reminded me I still need to finish reading my copy of TaPL lying on my book shelf)

• 

  > PS. Sum could be implemented such that it still returns an IMonad<T>

   

  As discussed in the video, IO<T>.Sum() does return an IO<T>. Now for IE<T>, the problem is backwards compatibility. Changing IE<T>.Sum() to return an IE<T> instead of a T breaks existing code.

• 

  Of course. I just wanted to mention that it could be implemented in such a way (call if Sum2 if you will).

   

  That said, Bart's explanation for staying in the monad in the asynchronous observable monad is nice. cancellation and composability.

   

  The Wmi demo is also cool. Must play with it.

• 

  Excellent video! Whenever I have to open 4 Wikipedia pages just to keep up with the conversation it is destine to be a very good topic! 

   

  Keep up the great work!

• 

  Wow am I seeing LINQ to PowerShell in the diagram. Very cool.

• 

  "Bart's explanation for staying in the monad in the asynchronous observable monad is nice. cancellation and composability."

   

  Good observation ( or qbservation ) => Everything about Rx, from it's foundation to the API, is related to composable system design and development (aka software compositionality or composability). Monads are the pieces in a truly composable software puzzle. Rx abstracts the monad from your mind and subsequently your fingers tapping the keyboard. You get to focus on the task at hand: building reliable asynchronous systems, orchestrating concurrency, as Bart said.

   

  Wait a minute, perhaps one need only think of monads as things that reliably encapsulate compositionality, composable pieces of computational fabric. When you leave a monad during execution, then you leave the comfortable, reliable white sand beaches of the composable world for one rife with unexpected or random behavior.

   

  Where's Dr. Beckman, the Wizard of Monads. Ahh. That's right: Monad, monad, monad!

   

  C

• 

  IQbservable hurts my eyes. Very similar to IObervable. I sure violates a few naming conventions.

• 

  Wonderful video, great to have Bart back on camera again. I have to agree about "IQbservable" being visually just too close to IObservable. Has "IQbservable" been run past the Framework Design Guidelines team at Microsoft? Well, the remaining FDG team (given that some of them have gone to work at Google...). Aside from the naming issue, wonderful, deep stuff. Keep it coming!

• 

  Thanks for watching and for the feedback. We're open to all sorts of discussions on the Rx forums, including the naming ones. This said, we tend to try getting the semantics right first, since alpha conversion is a widely understood technique. Not trying to downplay the importance of API design in the large, Wadler's Law seems to capture exactly what we're talking about .

• 

  Great stuff! Is this an old video? Bart seems to be reverse-aging.

• 

  This video was shot on Friday, May 14, 2010 in the late afternoon in Redmond, WA, USA.
  C

• 

  IQbservable will never make it into the base class library. That will not happen. I am absolutely sure of it. In this vid Kim goes through some of the naming conventions the Rx team had in various API inceptions, and what the BCL changed them to after significant deliberation, introspection, cogitation, ratiocination, disambiguation and no doubt disputation.

• 

  Thanks for the kind words, Jules.. It may be my new diet that's paying off .

   

  A few more pointers for the viewers:

  • Types and Programming Languages by Benjamin Pierce (UPenn), containing coverage of the T-shirt
  • Advanced Topics in Types and Programming Languages by the same author
  • Expert to Expert: Brian Beckman and Erik Meijer - Inside the .NET Reactive Framework (Rx), explaining our interface duality
  • Expression Trees on MSDN, as a good starting point to familiarize yourself with the tree APIs

  Things not mentioned in the video:

  • Some "imperative operators" using interfaces like IConnectableObservable, requiring code blocks (and hence statement trees in a fantasy homo-iconic world), are not available on Qbservable. There are just a handful of overloads that fall in this camp such as the parameterless Publish method (the lambda version is there though).
  • Qbservable also provides support for join patterns as expression trees. That is, the And and Then operators from System.Joins are available in an expression tree form as well. It's left to the reader to dream up the endless possibilities this enables for joining multiple data streams "in the cloud".
  • Constructors like FromAsync which return a Func<T1, ..., Tn, IObservable<TResult>> are supported too. They return a similar Func returning an IQbservable<TResult> instead. That IQbservable will contain an expression tree with an Invoke node upon invoking the returned function.
  • Parameters on Qbservable operators are only protected against null reference inputs. That is, no operator-specific enforcements are made. For example, one is allowed to write Take(-5) against an IQbservable, but not against an IObservable. It's up to the provider to decide what -5 means (take last 5?, don't allow?).
  • Observable has a static get-only Provider property that exposes the IQbservableProvider that can be used to create expression trees for "local" queries and can be used to get stuff like Return in an expression form, using Observable.Provider.Return(5), returning an IQbservable<int>. This is complementary to the AsQbservable operator.
  • Use of AsQbservable on a "local" observable sequence allows one to write expression trees against such a source. When calling Subscribe on the resulting IQbservable, which also implements IObservable per definition, the tree gets rewritten to target the Observable.* operators, including join patterns and whatnot.
  • For signatures Qbservable operators, notice that only a minimum set of parameters has been tweaked to be expression-friendly compared to the corresponding Observable operators. That is, things like Amb(IObservable, IObservable) only get the first parameter as IQbservable. This opens up for quite some flexibility; providers can type-check on the second argument.

  Play and amaze us with great observable providers .

• 

  IQbservable is very bad naming, imo

   

  How about IAnswerable?

   

  I bet if you let the niners brainstorm on it for about a day, you'd come up with something far more suitable.

   

  cheers!

• 

  Changing names is easy, especially in our world of incubating ideas. Changing semantics on the other hand is hard. So let's focus on the latter first and then get to the aesthetics of it all. De gustibus non est disputandum and The colour of the bikshed apply here.

   

  Oh, and the "visually close" argument raised earlier is rather weak, especially since homo-iconicity is all about that . Is x => x too much like ... x => x, while they may be radically different behind the scenes (delegate versus expression tree)? Also keep in mind the interface is not what users of providers see, at all.

   

  This said, feel free to discuss naming on a dedicated thread on the Rx forums.

• 

  Hey, great video - well worth downloading the hirez version for as well!

   

  I actually have an application that I will implement a IQbservable provider for - it's going to be awesome.

   

  btw.  Types and Programming Languages by Benjamin Pierce (UPenn), is totally on my reading queue - any book that's got a jacket quote from Robin Milner gets my vote.

• 

  I also think IQbservable is pretty horrible. Even IQObservable (eye queue observable) would be a lot better. But I think the full IQueryableObservable is best, especially since I'm almost never actually going to type the name, just like IQueryable and IEnumerable in LINQ context. The 'var' keyword takes care of most cases, and IntelliSense takes care of the rest. Consider that to write foo.AsQueryableObservable all I need to type into VS2010 is "foo.AQO", and that the clarity of the name is more important than a bit of extra typing.

   

  By the way, don't think that the name was the most interesting part of the vid, it's just the only part we have something to disagree on 

• 

  If I was to hook this up to something like MSMQ, how would transactions flow through RX? 

• 

   

  Bridging Rx with MSMQ is totally doable. At first sight, it doesn't look like there's a domain-specific queue query language you can apply to contents of a query, so the queryable approach seems redundant. Nonetheless, some of the System.Messaging APIs may benefit from mapping onto Rx equivalent operators. For example, the TimeSpan parameter used to specify a timeout on various Send/Receive operations is an ideal candidate for exposure through the Timeout operator of Rx.

   

  Concerning this bridging, simply creating operators that expose a message queue as an IObservable and vice versa would likely suffice. In fact, I've done such a thing in the past. Transactions could be totally part of this too. Basically, you'd check whether a MessageQueue object is marked as Transactional, and if so, you'd likely do something along those lines:

  • When importing an IObservable to MSMQ, you'd likely want a transactional send of many OnNext messages, rolling back the transaction on OnError and committing it on OnCompleted. This won't work well for infinite sequences of course (and likely you don't want to do a transactional "send just one message" kind of thing).
  • When importing an MSMQ queue as an IObservable, you'd likely want transactional receive to work on a per-message basis, only committing the transaction if the OnNext message you send out to the consumers gracefully returns. That is, you guarantee the user has seen the message successfully and could handle it.

  If this automatic transaction management doesn't work for you, a better option may be to have the user control the transaction himself, simply by use of MessageQueueTransaction objects. Not as sexy or smart as the above, but maybe necessary to accommodate for various more complicated scenarios.

   

  I'll leave it as an exercise to the reader to implement such bridging operators.

   

  (Edit: wanted to paste code, but it went terribly wrong; will try again later)

   

  Hope this helps. Happy hacking!

• 

  I will have to view that a couple more times.  Good show.

   

  Question.  What do I need to run that sample Bart did?  Just the new Rx?  I have vs2008 and 3.5.

• 

  While Sum in IE/IQ could return an IE/IQ, that would actually (for most people) be unexpected design.  Most people would rightly expect a summation operation to return a single item of the element type of the sequence (IE<T> -> T).  They would not expect it to return some sequence that just happens to contain a single element.  The design of Sum in the enumerable/pull/synchronous world is correctly chosen.

   

  The observable/push world is different in its asynchronicity, and that's the motivation for the design there.

   

  @Paulo:  I agree.. I've already complained to a couple folks on the team.  I've suggested "IObservableQuery" since they don't like "IQueryableObservable".

• 

  Oof.. the 'e' in "ToQueryable" ended up on a separate line on the otherwise beautiful diagram.

• 

  The sample shown in the video will be posted at a later point at either the Rx team blog or my blog. I'll keep you current on that one.

• 

  One versus many is a popular debate amongst language people. In fact, certain functional languages that will remain unnamed encode a lot of stuff using lists (e.g. "nullability" in terms of zero-length lists to represent "not present"), while others introduce whole separate language features to deal with 0 versus 1 cardinality (e.g. "option"). I'm much more in the former camp as many can encode one, but not vice-versa.

   

  Whether or not the IE/IQ design for aggregates it the right one is something that can be debated for a long time in a similar manner. If we're doing LINQ to SQL using it anyway, isn't a single result a single-column, single-row table? People have always used different methods on low-level APIs a la SqlCommand depending on the desired outcome. While you could use ExecuteReader to get a scalar value back, ExecuteScalar makes things simpler. But if LINQ generalizes operators over sequences, something can be said about it being closer to the ExecuteReader approach. With the sole argument of one more method call adding friction, xs.Sum().Single() isn't too bad. This said, I do buy the arguments made.

   

  The thing that's certain is that in an asynchronous world you don't want to leave the monad. There are multiple reasons for that. One is cancellation, but the more important one is to preserve composition. The former is definitely an IO thing, but the latter applies equally well in the world of IE. We may introduce similar aggregates in our Interactive Extensions (Ix) assembly for LINQ to Objects as well. Simply because the original LINQ design didn't account for further composition, doesn't mean it's not better. In particular, with operators like Amb and Timeout (amongst many others) introduced in Rx and Ix, composition for aggregates is incredibly useful. Also, a different return type outside IE requires people to write different code to do data binding to the result of an aggregation versus a grid-shaped result.

   

  Finally, in the IQ case, the fact you loose the expression tree breaks composition on that level in certain cases. Not if you're "lucky" and things like Sum() appear in a selector lambda expression. But if you want to translate computations involving multiple aggregates (agreed, not often done in today's world), you're left out in the dark as there's no way to get the tree representing Sum without triggering its execution (unless you manually reconstruct it using infoof(Queryable.Sum) in a MethodCallExpression). Btw, if aggregates stayed in IE, computation over multiple aggregates could be done easily using operators like Zip, which is just a "lifted binary operator" for single-element sequences.

• 

  infoof?  Don't toy with my emotions there, bud

   

  The point I was making is that the final implementation should match the characteristics of what would be built with it.  In Rx's case, Sum-as-sequence makes perfect case as you adequately explain.  In IE's case, Sum-as-scale fits well with what people almost certain end up doing.  In any event, a user-study would reveal this (albeit too late to change the established SQO specifications if I'm proven wrong).

• 

  FYI Bart.  In sample code #22 "MoreOnTime" has a cross-thread exception.  At least with vs2008 and 3.5

  Thanks for code.

• 

  There are a few samples where the form quits before event subscriptions are disposed; it may be that's the case you're hitting, which looks similar in terms of exceptions. I'll have a look. Thanks for letting me know.

• 

  Hi

  Nice video.

  Does someone know where I can download the code ?

  Thanks 

• 

  Hi folks,

   

  The sample of the LINQ to WQL provider can (finally) be downloaded. More information can be found on this post on the Rx forum. Please use the forum for further questions and/or discussions on the subject.

   

  Have fun,

  -Bart

• 

  will you guys make rx work across network boundary? e.g. push notification to subscribers (with wcf ) but hiding the implementation?