Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Bart De Smet: Observations on IQbservable - The Dual of IQueryable

Download

Right click “Save as…”

You first learned about Rx on C9. We've led you through the basic concepts of reactive programming to the deep mathematical foundation behind Rx (interface duality). By now, you should understand that IObservable is the dual of IEnumerable. Today, you will learn some new concepts (for many of you) in addition to the introduction of Rx's newest interface, IQbservable, the dual of IQueryable. In effect, the addition of IQbservable completes the interface puzzle within Rx. But what does this mean?

The great Bart De Smet takes us through the fundamentals and specifics behind this new interface, which ships in the latest version of Rx. Most of the time is spent at the whiteboard. There's also a short demo at the end of the conversation. Here's the flow:

Whiteboarding:

  • Expression trees recap (lambdas convertible to either anonymous methods or expression trees)
  • How LINQ to Objects versus LINQ to SQL works, introducing IQueryable<T>
  • Look at the Queryable extension methods and how they stitch expression trees together
  • Differences between IQbservable<T> and IQueryable<T> (mainly simplification)
  • Extended role of IQbservableProvider compared to IQueryableProvider
  • Synergy between IQueryable and IQbservable (ToEnumerable/ToObservable "sideways" conversion)
  • What operators are available (answer: 99% - explain why that 1% is omitted)

Demo:

  • Sample observable LINQ provider (LINQ to WQL)

Put your thinking caps on, turn up the volume, sit back, and learn. Erik Meijer and team are innovating at a level we haven't seen in a while around here. Rx is profoundly evolving and taking LINQ along for the ride. Incredible work!

Enjoy this latest episode of Going Deep. Ask questions. Bart et al. will answer them here and on the Rx forums.

Tags:

Follow the Discussion

  • CharlesCharles Welcome Change

    Here's an image representation of the three dimensional graph of Rx's computational fabric:

     

    representation of the three dimensional graph of Rx's computational fabric

  • Bent Rasmussenexoteric stuck in a loop, for a while

    I'm quite interested in using expression trees to create optimized IL code as Bart briefly mentions. Is there anything in the works for that? In essence an expression tree optimizer (LINQ to MSIL?).

     

    PS. Sum could be implemented such that it still returns an IMonad<T>, i.e. with a return type of

     

        I{Enumerable,Queryable,Observable,Qbservable}<T>

     

    First() would then really exit the monad.

  • Nice video and nice t-shirt. Smiley

     

    (That t-shirt reminded me I still need to finish reading my copy of TaPL lying on my book shelf)

  • > PS. Sum could be implemented such that it still returns an IMonad<T>

     

    As discussed in the video, IO<T>.Sum() does return an IO<T>. Now for IE<T>, the problem is backwards compatibility. Changing IE<T>.Sum() to return an IE<T> instead of a T breaks existing code.

  • Bent Rasmussenexoteric stuck in a loop, for a while

    Of course. I just wanted to mention that it could be implemented in such a way (call if Sum2 if you will).

     

    That said, Bart's explanation for staying in the monad in the asynchronous observable monad is nice. cancellation and composability.

     

    The Wmi demo is also cool. Must play with it.

  • Greg Braygbrayut Check out PhraseMeme Scanner for Windows Phone at ​PhraseMeme.​com

    Excellent video! Whenever I have to open 4 Wikipedia pages just to keep up with the conversation it is destine to be a very good topic! 

     

    Keep up the great work!

  • Wow am I seeing LINQ to PowerShell in the diagram. Very cool.

  • CharlesCharles Welcome Change

    "Bart's explanation for staying in the monad in the asynchronous observable monad is nice. cancellation and composability."

     

    Good observation ( or qbservation Wink ) => Everything about Rx, from it's foundation to the API, is related to composable system design and development (aka software compositionality or composability). Monads are the pieces in a truly composable software puzzle. Rx abstracts the monad from your mind and subsequently your fingers tapping the keyboard. You get to focus on the task at hand: building reliable asynchronous systems, orchestrating concurrency, as Bart said.

     

    Wait a minute, perhaps one need only think of monads as things that reliably encapsulate compositionality, composable pieces of computational fabric. When you leave a monad during execution, then you leave the comfortable, reliable white sand beaches of the composable world for one rife with unexpected or random behavior.

     

    Where's Dr. Beckman, the Wizard of Monads. Ahh. That's right: Monad, monad, monad!

     

    C

  • Paulo Morgadopajocomo Paulo Morgado

    IQbservable hurts my eyes. Very similar to IObervable. I sure violates a few naming conventions.

  • Wonderful video, great to have Bart back on camera again. I have to agree about "IQbservable" being visually just too close to IObservable. Has "IQbservable" been run past the Framework Design Guidelines team at Microsoft? Well, the remaining FDG team (given that some of them have gone to work at Google...). Aside from the naming issue, wonderful, deep stuff. Keep it coming!

  • Bart De Smetbdesmet Bart De Smet ​[MSFT::SQL::​Cloud​Programmabi​lity::Rx]

    Thanks for watching and for the feedback. We're open to all sorts of discussions on the Rx forums, including the naming ones. This said, we tend to try getting the semantics right first, since alpha conversion is a widely understood technique. Not trying to downplay the importance of API design in the large, Wadler's Law seems to capture exactly what we're talking about Smiley.

  • Great stuff! Is this an old video? Bart seems to be reverse-aging.

  • CharlesCharles Welcome Change

    This video was shot on Friday, May 14, 2010 in the late afternoon in Redmond, WA, USA.
    C

  • Vesuviusvesuvius Count Orlock

    IQbservable will never make it into the base class library. That will not happen. I am absolutely sure of it. In this vid Kim goes through some of the naming conventions the Rx team had in various API inceptions, and what the BCL changed them to after significant deliberation, introspection, cogitation, ratiocination, disambiguation and no doubt disputation.

  • Bart De Smetbdesmet Bart De Smet ​[MSFT::SQL::​Cloud​Programmabi​lity::Rx]

    Thanks for the kind words, Jules.. It may be my new diet that's paying off Smiley.

     

    A few more pointers for the viewers:

    Things not mentioned in the video:

    • Some "imperative operators" using interfaces like IConnectableObservable, requiring code blocks (and hence statement trees in a fantasy homo-iconic world), are not available on Qbservable. There are just a handful of overloads that fall in this camp such as the parameterless Publish method (the lambda version is there though).
    • Qbservable also provides support for join patterns as expression trees. That is, the And and Then operators from System.Joins are available in an expression tree form as well. It's left to the reader to dream up the endless possibilities this enables for joining multiple data streams "in the cloud".
    • Constructors like FromAsync which return a Func<T1, ..., Tn, IObservable<TResult>> are supported too. They return a similar Func returning an IQbservable<TResult> instead. That IQbservable will contain an expression tree with an Invoke node upon invoking the returned function.
    • Parameters on Qbservable operators are only protected against null reference inputs. That is, no operator-specific enforcements are made. For example, one is allowed to write Take(-5) against an IQbservable, but not against an IObservable. It's up to the provider to decide what -5 means (take last 5?, don't allow?).
    • Observable has a static get-only Provider property that exposes the IQbservableProvider that can be used to create expression trees for "local" queries and can be used to get stuff like Return in an expression form, using Observable.Provider.Return(5), returning an IQbservable<int>. This is complementary to the AsQbservable operator.
    • Use of AsQbservable on a "local" observable sequence allows one to write expression trees against such a source. When calling Subscribe on the resulting IQbservable, which also implements IObservable per definition, the tree gets rewritten to target the Observable.* operators, including join patterns and whatnot.
    • For signatures Qbservable operators, notice that only a minimum set of parameters has been tweaked to be expression-friendly compared to the corresponding Observable operators. That is, things like Amb(IObservable, IObservable) only get the first parameter as IQbservable. This opens up for quite some flexibility; providers can type-check on the second argument.

    Play and amaze us with great observable providers Smiley.

  • bryaneddsbryanedds An ​individuali​st is he who is saving himself from all those who are saving the world.

    IQbservable is very bad naming, imo Smiley

     

    How about IAnswerable?

     

    I bet if you let the niners brainstorm on it for about a day, you'd come up with something far more suitable. Wink

     

    cheers!

  • Bart De Smetbdesmet Bart De Smet ​[MSFT::SQL::​Cloud​Programmabi​lity::Rx]

    Changing names is easy, especially in our world of incubating ideas. Changing semantics on the other hand is hard. So let's focus on the latter first and then get to the aesthetics of it all. De gustibus non est disputandum and The colour of the bikshed apply here.

     

    Oh, and the "visually close" argument raised earlier is rather weak, especially since homo-iconicity is all about that Smiley. Is x => x too much like ... x => x, while they may be radically different behind the scenes (delegate versus expression tree)? Also keep in mind the interface is not what users of providers see, at all.

     

    This said, feel free to discuss naming on a dedicated thread on the Rx forums.

  • rhmrhm

    Hey, great video - well worth downloading the hirez version for as well!

     

    I actually have an application that I will implement a IQbservable provider for - it's going to be awesome.

     

    btw.  Types and Programming Languages by Benjamin Pierce (UPenn), is totally on my reading queue - any book that's got a jacket quote from Robin Milner gets my vote.

  • I also think IQbservable is pretty horrible. Even IQObservable (eye queue observable) would be a lot better. But I think the full IQueryableObservable is best, especially since I'm almost never actually going to type the name, just like IQueryable and IEnumerable in LINQ context. The 'var' keyword takes care of most cases, and IntelliSense takes care of the rest. Consider that to write foo.AsQueryableObservable all I need to type into VS2010 is "foo.AQO", and that the clarity of the name is more important than a bit of extra typing.

     

    By the way, don't think that the name was the most interesting part of the vid, it's just the only part we have something to disagree on  Angel

  • PerfectPhasePerfectPhase "This is not war, this is pest control!" - Dalek to Cyberman

    If I was to hook this up to something like MSMQ, how would transactions flow through RX? 

  • Bart De Smetbdesmet Bart De Smet ​[MSFT::SQL::​Cloud​Programmabi​lity::Rx]

     

    Bridging Rx with MSMQ is totally doable. At first sight, it doesn't look like there's a domain-specific queue query language you can apply to contents of a query, so the queryable approach seems redundant. Nonetheless, some of the System.Messaging APIs may benefit from mapping onto Rx equivalent operators. For example, the TimeSpan parameter used to specify a timeout on various Send/Receive operations is an ideal candidate for exposure through the Timeout operator of Rx.

     

    Concerning this bridging, simply creating operators that expose a message queue as an IObservable and vice versa would likely suffice. In fact, I've done such a thing in the past. Transactions could be totally part of this too. Basically, you'd check whether a MessageQueue object is marked as Transactional, and if so, you'd likely do something along those lines:

    • When importing an IObservable to MSMQ, you'd likely want a transactional send of many OnNext messages, rolling back the transaction on OnError and committing it on OnCompleted. This won't work well for infinite sequences of course (and likely you don't want to do a transactional "send just one message" kind of thing).
    • When importing an MSMQ queue as an IObservable, you'd likely want transactional receive to work on a per-message basis, only committing the transaction if the OnNext message you send out to the consumers gracefully returns. That is, you guarantee the user has seen the message successfully and could handle it.

    If this automatic transaction management doesn't work for you, a better option may be to have the user control the transaction himself, simply by use of MessageQueueTransaction objects. Not as sexy or smart as the above, but maybe necessary to accommodate for various more complicated scenarios.

     

    I'll leave it as an exercise to the reader to implement such bridging operators.

     

    (Edit: wanted to paste code, but it went terribly wrong; will try again later)

     

    Hope this helps. Happy hacking!

  • William Staceystaceyw Before C# there was darkness...

    I will have to view that a couple more times.  Good show.

     

    Question.  What do I need to run that sample Bart did?  Just the new Rx?  I have vs2008 and 3.5.

  • Keith J. FarmerKeith J. Farmer My bike

    While Sum in IE/IQ could return an IE/IQ, that would actually (for most people) be unexpected design.  Most people would rightly expect a summation operation to return a single item of the element type of the sequence (IE<T> -> T).  They would not expect it to return some sequence that just happens to contain a single element.  The design of Sum in the enumerable/pull/synchronous world is correctly chosen.

     

    The observable/push world is different in its asynchronicity, and that's the motivation for the design there.

     

    @Paulo:  I agree.. I've already complained to a couple folks on the team.  I've suggested "IObservableQuery" since they don't like "IQueryableObservable".

  • Keith J. FarmerKeith J. Farmer My bike

    Oof.. the 'e' in "ToQueryable" ended up on a separate line on the otherwise beautiful diagram.

  • Bart De Smetbdesmet Bart De Smet ​[MSFT::SQL::​Cloud​Programmabi​lity::Rx]

    The sample shown in the video will be posted at a later point at either the Rx team blog or my blog. I'll keep you current on that one.

  • Bart De Smetbdesmet Bart De Smet ​[MSFT::SQL::​Cloud​Programmabi​lity::Rx]

    One versus many is a popular debate amongst language people. In fact, certain functional languages that will remain unnamed encode a lot of stuff using lists (e.g. "nullability" in terms of zero-length lists to represent "not present"), while others introduce whole separate language features to deal with 0 versus 1 cardinality (e.g. "option"). I'm much more in the former camp as many can encode one, but not vice-versa.

     

    Whether or not the IE/IQ design for aggregates it the right one is something that can be debated for a long time in a similar manner. If we're doing LINQ to SQL using it anyway, isn't a single result a single-column, single-row table? People have always used different methods on low-level APIs a la SqlCommand depending on the desired outcome. While you could use ExecuteReader to get a scalar value back, ExecuteScalar makes things simpler. But if LINQ generalizes operators over sequences, something can be said about it being closer to the ExecuteReader approach. With the sole argument of one more method call adding friction, xs.Sum().Single() isn't too bad. This said, I do buy the arguments made.

     

    The thing that's certain is that in an asynchronous world you don't want to leave the monad. There are multiple reasons for that. One is cancellation, but the more important one is to preserve composition. The former is definitely an IO thing, but the latter applies equally well in the world of IE. We may introduce similar aggregates in our Interactive Extensions (Ix) assembly for LINQ to Objects as well. Simply because the original LINQ design didn't account for further composition, doesn't mean it's not better. In particular, with operators like Amb and Timeout (amongst many others) introduced in Rx and Ix, composition for aggregates is incredibly useful. Also, a different return type outside IE requires people to write different code to do data binding to the result of an aggregation versus a grid-shaped result.

     

    Finally, in the IQ case, the fact you loose the expression tree breaks composition on that level in certain cases. Not if you're "lucky" and things like Sum() appear in a selector lambda expression. But if you want to translate computations involving multiple aggregates (agreed, not often done in today's world), you're left out in the dark as there's no way to get the tree representing Sum without triggering its execution (unless you manually reconstruct it using infoof(Queryable.Sum) in a MethodCallExpression). Btw, if aggregates stayed in IE, computation over multiple aggregates could be done easily using operators like Zip, which is just a "lifted binary operator" for single-element sequences.

  • Keith J. FarmerKeith J. Farmer My bike

    infoof?  Don't toy with my emotions there, bud Smiley

     

    The point I was making is that the final implementation should match the characteristics of what would be built with it.  In Rx's case, Sum-as-sequence makes perfect case as you adequately explain.  In IE's case, Sum-as-scale fits well with what people almost certain end up doing.  In any event, a user-study would reveal this (albeit too late to change the established SQO specifications if I'm proven wrong).

  • William Staceystaceyw Before C# there was darkness...

    FYI Bart.  In sample code #22 "MoreOnTime" has a cross-thread exception.  At least with vs2008 and 3.5

    Thanks for code.

  • Bart De Smetbdesmet Bart De Smet ​[MSFT::SQL::​Cloud​Programmabi​lity::Rx]

    There are a few samples where the form quits before event subscriptions are disposed; it may be that's the case you're hitting, which looks similar in terms of exceptions. I'll have a look. Thanks for letting me know.

  • plbplb

    Hi

    Nice video.

    Does someone know where I can download the code ?

    Thanks 

  • Bart De Smetbdesmet Bart De Smet ​[MSFT::SQL::​Cloud​Programmabi​lity::Rx]

    Hi folks,

     

    The sample of the LINQ to WQL provider can (finally) be downloaded. More information can be found on this post on the Rx forum. Please use the forum for further questions and/or discussions on the subject.

     

    Have fun,

    -Bart

  • kevinkevin

    will you guys make rx work across network boundary? e.g. push notification to subscribers (with wcf ) but hiding the implementation?

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.