Kim Hamilton and Wes Dyer: Inside .NET Rx and IObservable/IObserver in the BCL (VS 2010)

Sign in to queue

Description

You recently learned about Erik Meijer's latest innovation, Rx, here on Channel 9. Clearly, judging by the views and comments on that post, it piqued your interest and curiosity. Wes Dyer, a rock star developer on Erik's team of rock stars, implemented a great deal of Rx and helped to design it along with Erik.

Visual Studio 2010 will ship with with two new types, IObserver and IObservable. Rock star developer Kim Hamiliton (you've met her on C9) implemented these two types in the BCL and worked closely with Erik and Wes to insure that Rx in the BCL is designed and implemented to meet the strict requirements for new types in .NET's robust general purpose library. As you've learned previously, IObservable is the mathematical dual of IEnumerable. We of course talk about this here, but from the developer perspective. You've already learned about the math and continuation monad behind this with Erik and Brian Beckman.

Here, Kim and Wes dig into the implementation of IObservable and IObserver in the 2010 version of the BCL. How did Erik and team work with the BCL folks? What were the design decisions that led to the final implementation of the two Rx types in the BCL? What do these two new types enable for .NET developers? This is a great example of how innovation comes to market: incubation teams come up with a brilliant idea, refine it by working with multiple teams and some researchers in MSR, pass it along to a product group, they go back and forth on implementation details and design requirements and finally the new stuff is added to the shipping code base. Great stuff!!

Make sure to watch this all the way through. You never know what kind of magic can happen if you know how to summon a wizard.

Enjoy!

Embed

Download

Download this episode

The Discussion

  • User profile image
    vesuvius

    So I take it this is not in VS 2010 beta 1, but may be in beta 2? Downloading...

  • User profile image
    Charles

    It will be in VS 2010 Smiley Not sure of the exact ship stage (Beta 2, etc). It will certainly be in RTM...

    C

  • User profile image
    tomkirbygre​en

    A.W.E.S.O.M.E. Smiley I don't suppose there's any rough estimate of Beta 2's arrival Charles? Is breath holding advised?

  • User profile image
    Charles

    I'm excited too. This is awesome stuff and very deep in implications for distributed reactive programming on the .NET stack. To be clear, I do not know the exact ship vehicle (meaning, I do not know if beta 2 is the target). All I know is that these two new types will ship as part of VS 2010. The VS People own the timing plan. Me, I'm just a lawnmower. You can tell me by the way I walk.

     

    C

  • User profile image
    RobertL
  • User profile image
    Charles

    We are actually thinking about this Smiley

    C

  • User profile image
    gdesroches

    In your first C# sample...

    var o = Observable.Return(1); 
    o.Subscribe( x => Console.WriteLine("OnNext({0})", x), ex => Console.WriteLine("Error()"), () => Console.WriteLine("Completed()") );
    


    I see you are creating an IObservable object, and you are subscribing to it, but what is telling the observable object to begin calling the OnNext method on its subscribers?

  • User profile image
    Bass

    When you say 2010 BCL, you mean .NET 4 right?

  • User profile image
    Charles

    Yes. .NET 4 is the product marketing terminology. Sorry for the confusion.

    C

  • User profile image
    epentangelo

    I guess the implementation of the Observable.Subscribe method that is coming out of Observable.Return just calls OnNext.

    So in fact by subscribing you're calling yourself back with the value 1. 

     

    Observable.Return<T>(T value) = new IObservable { Subscribe(observer) = observer.OnNext(value) }

  • User profile image
    Judah

    Very cool.

     

    Wes really is a rockstar developer.

    Wes, please resume blogging, we miss you.

  • User profile image
    exoteric

    Nice. And what a funny surprise with Beckman, trained in the art {M[onad],F[unction],C[omposition]}-fu to join in Smiley Now to wait for the arrival of the next beta.

     

    I'm curious how GUI programming will evolve with the arrival of Rx.

  • User profile image
    cdwatkins

    Well the Obserable.Return function doesnt just return any old IObserable.  The one it returns calls OnNext the moment after someone subscribes (as it doesnt need to wait, it already knows what it should return).

  • User profile image
    kgooding

    Interesting stuff, yet another reason to look forward to the RTM release!

    Out of curiosity, like you mentioned developers tend to use IEnumerable over arrays, do you think developers will have the tendancy to use this over events in C# .NET? Will any of the controls we use today be extended with IObservable implementations?

  • User profile image
    Charles

    Yes. And yes again.

     

    C

     

     

    PS: Stay tuned to the BCL team blog (or Soma's blog or the VS blog) to find out when IObserver and IObservable will be available for your programming enjoyment. I can't wait!!

  • User profile image
    robertboeck​el

    Interesting talk! But why you guys call it OnError() ? I think OnException() would be more appropriate as there's quite a different meaning to both words.

  • User profile image
    cdwatkins

    I was thinking about the new IObservable interface.  At first I thought great, brilliant, amazing, revolutionary.  But then I thought about how you could take an IEnumerable and make it an IObservable, and the reverse.  For instance you can just block the MoveNext until the next event comes in, or you can just call getnext and then IObservable.OnNext(Current) to switch it the other way.  So in the end what makes IObservable any better or different?  Well the IEnumerable has a slight assumption that when you do a GetNext/Current it will be fairly fast(as it is synchronous), the IObservable makes no such assumption.  If you want to continue computation while you wait for the GetNext to finish you better do that on a different thread, so there is thread overhead.  In the end I would say that IObservable is STRICTLY better than IEnumerable.  Anything IEnumerable can do IObservable can do.  For instance: foreach(type a in IEnumerable){ write(a)} is the same as bool end=true;IObservable.Subscribe(a=>write(a),a=>throw a,()=>end=false;);while(end){};  I bet you could even have the same syntactic suger of the foreach work for IObservable.  The only difference is going the other way where the MoveNext blocks would need to be done on a separate thread so IEnumerable is never better (and sometimes worse) then IObserable.

     

    In the end why don’t you just convert ALL IEnumerable’s over to the IObservable pattern (as well as all events)? (I know that you have backwords compatibility issues with this, but other then that)

  • User profile image
    aL_

    cool stuff Smiley (shorter than usual though) The brian cameo in the end was epic Wink

     

    events since .net2 has been reffered to in the docs as "first class citizens" but they cant be used like the  IEvent<T> mentioned in the video. do you consider the events that exsist in .net today first class still? using Rx with exsisting event is a bit of a hassle precisly because you cant pass them around, is that something you might change in .net4?

     

    the relation between tasks and iobservable is kinda interseting because they sort of overlap.. to bad will tasks wont implement IObservable  Sad seems like a good fit imo.. there are some problems i guess, does it call OnNext when the progress changed or when its done for example. extension methods might be a good way to go but they relly beg to be integrated well Smiley an interview with wes and joe duffy would be really interesting Smiley waiting for dev11 seems a bit harsh though :/

     

    also, why is AnonomusObserver internal? are the Subscribe overloads/Select the replacement for having it public? it seems like it would be a very useful thing to have when writing integration with Task or my own stuff...

    and why is Combine internal? i think there is a BinaryObservable in there somewhere thats also internal.. please dont repeat what you did with TreeVisitorBase (i think its called), the base class for tree visitors used in linq (that is finaly public in .net4, yay Smiley ). Having to rewrite that stuff in every project is such a pain.. im not saying thats what you've done but please, please dont make stuff internal lightly Smiley

     

    Kim mentioned that only the interfaces are in the bcl, not the extension methods? is this really correct or did i misunderstand? Rx is great but what makes is super powerful are those extension methods :O surely the entire System.Reactive.dll will be in .net4?

  • User profile image
    aL_

    well stricktly speaking you are right Smiley all IEnumerables can be expressed as IObservables and vice versa. i think thats part of the mathematical dual thing. (atleast i think so, i dont have any formal proof on hand Smiley )

     

    IObservable really shines when you have a bunch of things you want to coordinate. consider a keyboard shortcut. to implement that you'd typically have a keyDown event handler and then store the keys that are currently down and look into a list of your keyboard shortcuts. this however means shared state (= generally speaking, bad ) with Rx however you can write a linq statement that manages this.

     

    also, with Rx you get a new "listener" for your keyboard shortcut, something you dont get with just events. with events you have to manage the dispatch of when your shortcut is pressed, with Rx you get an object that you can subscribe to Smiley

    check out these blog posts for more info (also where i snagged this example)

     

    http://themechanicalbride.blogspot.com/2009/07/developing-with-rx-part-1-extension.html

    http://themechanicalbride.blogspot.com/2009/07/developing-with-rx-part-2-converting.html

     

    i dont think Ienumerable is dead though, its more a question of what makes most sense in each case, sometimes we're pulling data and sometimes we're getting it thrown at us, sometimes we want to block while waiting on data/computations and sometimes we dont Smiley

  • User profile image
    aL_

    you can actually start playing with Rx right now Smiley check out this blog for details:

    http://themechanicalbride.blogspot.com/2009/07/introducing-rx-linq-to-events.html

    also contains lots of good info on why Rx is useful in general

  • User profile image
    Charles

    Yep. Thanks for the link. The new types discussed in this interview aren't in the .NET 4 BCL yet, however...

    C

  • User profile image
    cdwatkins

    "sometimes we want to block while waiting on data/computations and sometimes we dont"

     

    well IEnumerable is basicly the creator of the object expressing you should (or must if you dont split off a diffrent thread), block while you wait for the result.  IObservable says you "may" block or not block as you wish.  The nice thing about IEnumerable is it expresses that the author doesnt expect it to take too long to get the next value, although even this kind of assumption can be broken slightly.  Look at Linq to Entities or Linq to SQL, both of these take a long time to get all thier values the first time.  Why should the program have to block while it waits for these to finish?  I think IEnumerable will still probably exist for some "in-memory" very quick things, that blocking doesnt really matter on, but I wish there was a better way for authors to express that they dont expect it to take long to get the values then use a totaly diffrent type.

     

    Basicly IEnumerable is sync with a method call to get the result, and IObserable is async with a callback to get the result.

  • User profile image
    aL_

    "IObservable says you "may" block or not block as you wish"

     

    well yeah but if you want to block waiting for an observable, thats something you have to sort out your self. there isnt really anything to block on since you dont call the IObserver to get the values, it calls you.

     

    what you say about IEnumberable taking a long time to get the values is true in some cases but in some sort of general way its not because IEnumerable is lazy. IObservable is sort of the opposite, its hyperactive Tongue Out it keeps throwing values at you at its own pace. this could potentially mean that some work is done in vain, your program might throw away the data because its not ready to handle it or the data have become outdated. the lazyness of IEnumerable limits that behavior

     

    "why should the program wait"

     

    i dont think its all that diffrent comparing IEnumerables and IObserables that way though.. neither are have any threading baked in. Even if IObservables are used the program have to wait for the database at some point Smiley 

     

    i still think it comes down to preference and convenience in the end Smiley you can really use either one, in some cases is really hard to use one or the other and in most cases the devide isnt as large.

     

    IEnumerables long lost twin has finally made it into .net, making it twice as powerful  Smiley (sans the extension methods it seems :/)

  • User profile image
    JeffVa_MS

    If you're interested in a nice challenge with Rx, go over to my blog and try solving the tripple-click puzzle: 

     

    http://blogs.msdn.com/jeffva/archive/2009/08/11/fun-system-reactive-puzzle.aspx

  • User profile image
    Tom Lokhorst

    This reply is a bit late perhaps, but I don't quite understand this.

     

    Does Return really call OnNext after someone subscribes, or calls it OnNext during the call to Subscribe? If I call Subscribe, I get back an IDisposable, but do I get the change to call Dispose before the Observable starts pushing values?

     

    In other words, what if I want this method:

    IObservable<T> ToObservable<T>(this IEnumerable<T> xs) { return new IObservable<T>() { IDisposable Subscribe(IObserver<T> obs) { foreach (var x in xs) obs.OnNext(x); obs.OnDone(); return someDisposable(); // Don't know how to implement
     } } }

     

    Can I unsubscribe from the IObservable before or during the series of OnNext calls?

    If not, how would this be implemented in a way that does allow for me to unsubscribe, should I explicitly spawn a new thread?

  • User profile image
    gt

    Yes, you can unsubscribe. You don't need to use another thread, but given the ToObservable() method you do.

    public static IObservable<T> ToObservable<T>(this IEnumerable<T> xs) {
     return new Observable<T> () { Enumerable = xs };
    }

    Here's the internal class needed to handle subscription:

    internal sealed class Observable<T> : IObservable<T> {
    
    private readonly UnSubscribe _unsubscribe = new UnSubscribe (); 
    public IEnumerable<T> Enumerable { get; set; } 
    public IObserver<T> Observer { get; set; } 
    public IDisposable Subscribe(IObserver<T> o) { 
    Observer = o; 
    ThreadPool.QueueUserWorkItem(Worker); 
    return _unsubscribe; 
    } 
    private void Worker (object state) { 
    foreach (T item in Enumerable) {
     if (_unsubscribe.Cancel)
     return; 
    Observer.OnNext (item); 
    }
     Observer.OnCompleted ();
     }
    }

    And here's the IDisposable internal class that flags when the subscriber is no longer interested:

    internal sealed class UnSubscribe : IDisposable {
     public bool Cancel { get; set; }
     public void Dispose() {
     Cancel = true;
     } 
    }

  • User profile image
    Tom Lokhorst

    I guess I'm not as familiar with the .NET BCL as I thought, I didn't know about the ThreadPool class.

    But from what I've read, it might still schedule the work on another thread (or the current thread if it becomes idle first). So I guess I'm implicitly using another thread instead of explicitly creating my own.

     

    Since the Observer.Return method doesn't use the ThreadPool class (at least not in the Silverlight binary), I guess that means that, that one isn't "unsubscripable".

     

  • User profile image
    ericis

    Because I already wrote my own implementation of IObservable with what appears to be a very different solution, I'd be curious to know what I can do to merge it with the Reactive Framework. My solution was developed because I was frustrated with managing weak event references as well as the implementation of events in WPF and now Silverlight using INotifyPropertyChanged versus DependencyProperty. Let me know...

  • User profile image
    RainerHilmer

    I like Rx very much but before I invest 41 minutes of my time let me please ask this:
    Is this video still up to date? Rx has undergone some changes since the production.

Add Your 2 Cents