E2E: Erik Meijer and Wes Dyer - Reactive Framework (Rx) Under the Hood 2 of 2

Still watching but I must say already that the IO<IO<T>> has something extremely satisfying about it. Very simple (well, superficially so).

Let's be honest, wouldn't you just love to be in the room during one of these white board sessions! I know envys not a good thing per say - but you folks who get to work along side Erik et al... Wow.

Thanks much C, Erik, and Wes.  Another great 2 part.  Naturally, I am thinking away about how you might bring this into ADO.Net Data Services (et al).  Say I wanted to get notified each time someone does an update or delete on specific condition, etc.  I can see how you may do this on the service end inside a method, but am wondering about the comm channel.  Because of firewalls and the http request/reply pattern, you still end up with essentially a client-side pull model instead of a pure push model.  Maybe that is just something we have to live with in a distributed environment.  That said, any ideas on this subject?  You can abstract the server side Push by having some kind of timer on the client side that pulls stuff down from a server's observable queue.   However, things like background timer tasks and what is the correct amount of time between pulls, message ordering starts to get messy.  Any new ideas in the network side of this Beauty? It is really cool to find the ONE.  Because now you can standardize on this for all async Push - Sql Notifications based on Linq queries, Ado.Net Data Services notifications, all .Net services, Windows Services, devices, .Net MicroFramework, .Net POS, etc.    Great work. 

Second question.  Do you think there will be a way to remote the observer query to the server, so the server processes any Where clause to reduce unneeded traffic on the wire?  Example, send me X where Y <= 2 || Y >1000

Well I think there would be some restrictions (on types that can be used and method calls), that could be passed to the server.  But look at what ..net ria services, what it does is it takes the IQueryable expression tree and serializes it directly and sends it off to the server.  IQueryable is just expression trees that havnt been compiled into a function that returns an IEnumerable yet.  I am sure they could have a simlar thing in IObservable that would have interfaces like  IRequestable Select<source,result>(this IRequestable , Expression<Func<source,result>> ).  I dont know quite what they would call it though.  IQueryable unforchantly derives from IEnumerable, but otherwise it would be identical in its signatures.  IQueryable assumes that something is going to serialize it to pass it to something else for proccessing.  In the case of observables you would be assuming something is going to serialize it to do the proccessing (filtering or other proccessing) remotely.  Maybe something like you can pass it up to the server and have it tell you when some specific row of a database table changes, or off to active directory to let you know when some user gets locked out.  All the server does is provide you with access to everything you have access to as one giant IObservable, and then the client can select (and filter) the type of events it wishes to listen to, and only those are sent down.

Also intresting to think about this in terms of a UI framework.  Where the framework itself has one giant IObserable which sends a new value every time the mouses moves or keyboard is pressed, and it hands off to all its children .Where(...) verions of the giant IObserable that filter for just its selected area of the screen or currently selected child (which then wraps its children in another .Where).  In the end you dont need to even test do any hitbox testing or anything, everything just flows down stream.  I spose most UI frameworks like WPF already do this with thier visual tree's just invoking methods on a child of a known type (like UIElement), but when you look at the 100's of On... overrideable method calls, you got to wonder if filtering only what you need might be a better option.  Where they dont have to keep extending the base class to add additional method calls, and instead add new derived event types that return more specific infromation if you want that extra info.  Or say your a team like Microsoft Surface that has some specific type of "new" interaction events you want to pass to your children based on specific proprietary hardware.  In the current WPF, this is very hard/impossible (I think they derived thier own UIElement type and force all thier children had to derive from that to get the new events).

Charles, do you think it possible to get perhaps a type proliferation overview, maybe in interview format, talking about where you plan to retrofit these types as well as possibly in what order. I imagine this will be a big undertaking as it basically amounts to every single instance of where you have an event today.

I wonder if it wouldn't make sense to systematically and automatically retrofit all of .Net with these types, that is, using some naming convention, simply inject one IObservable property for every event. If I remember correctly, you're just using an event and holding a subscriber delegate to the event and on dispose you just remove the same delegate. So it should be a mechanical process to retrofit. This way we can also have this retrofitted to good old WF.

Also, it would seem like this would have some interesting deeper implications for concurrency and coordination if we have parallel events going on and we coordinate them with these query combinators, hmm...

My guess unfortunately is they aren’t going to retrofit anything, at least initially.  We would be lucky to get the IObservable interface in the BCL (and not just as an extension), although this might happen in .NET 4.0.  If they actually went converted events to be IObservable’s they would break everyones code that depended on those events.  Even if they rewrote the compiler somehow so the event keyword now MEANS IObservable, a lot of code would break (think of the IL changes on a different language).  Ah well, the good thing is that at least all events can easily be converted into IObservable’s.

No no no, you misread - or I used the word retrofit wrongly. The idea is to *add* an IObservable property to all classes where there is already an event. This would bring the meaning of Reactive Extensions to a whole new level, meaning Reactive Extensions *everywhere*.

Charles: Maybe you should just rename the posts to Reactive Extensions for .Net: ...... to defuse confusion. Or I should just get to work now heh...

Rx is really really cool it will have as big an impact as linq-to-objects (or linq to enumerables really) had..

what is crucial though is interop with exsisting stuff.. tasks for example, or the event based calls in silverlight (not a big fan of those) or begin/end invoke, or events..

there are stories for most of them but some are better than others, converting an event to an iobservable is somewhat clunky and involves magic strings (although there is type safe way of doing it to)

i think what we'll see is a bunch of extension method libraries for getting iobservable versions of various events.. its to bad they didnt have this stuff when wpf was created

On the other hand, if syntax for creating observables from events is sufficiently easy, this may be overkill, hmm...

Maybe I'm confused about your points regarding events, but the reactive framework adds extensions to wrap events, events are reactive anyway, the reason you would want 'iobservable' events is for the composition advantage they give.

edit: the points regarding let at the end were interesting, but perhaps not really discussed properly, ie.. no practical example.

Great video two parter tho, good ending .

You guys never got to Until (mentioned in part 1). Any follow up details?

I also wish you'd talk more about Let. How is that implemented? Does it give you lazy computation? E.g. Haskell lets your write:

  zero = let x = x in 0

And you won't get an infinite loop. Does Let allow somethign similar?

Good stuff!

Not sure who you are addressing here but if me, then the point was to make the use of Rx have as low a syntactic overhead as possible. This is an example use of Rx I found,

IObservable<Event<MouseEventArgs>> mouseMoveEventObservable = Observable.FromEvent<MouseEventArgs>(myControl, “MouseMove”); mouseMoveEventObservable.Subscribe(...);

So looking at this it's not the sweetest syntax you could arrive at, but here the lacking use of type-inference is a culpit as well, so we make it a bit sweeter

var mouseMoveEventObservable = Observable.FromEvent<MouseEventArgs>(myControl, “MouseMove”); mouseMoveEventObservable.Subscribe(...);

Then finally the solution we would have wanted from the outset

myControl.MouseMove.Subscribe(...);

Clearly there is some syntactic overhead which will repreat itself in each and every program. Whether one actually cares about something like this is a different matter.

This looks very good.

I have a question: you mention that SelectMany "merges" event streams, and you have to use until make sure that you stop listening when a new event comes in (in the dictionary suggest example). I cannot think of an example where you'd NOT want to use until. So why isn't this the default behavior for SelectMany? Can you give an example where you'd want to have SelectMany to merge the event streams instead of switching to the most current one?

I see this example on the "unfold" blog that is probably the approach you recommend - and it looks quite forgivable

public static IObservable<Event<MouseButtonEventArgs>> GetMouseLeftButtonDown(this UIElement that) { return Observable.FromEvent<MouseButtonEventArgs>(that, "MouseLeftButtonDown"); } 

The compositional nature of these things ("IE" & "IO") is very inspiring. Looking forward to the beta 2 drop, hopefully with Rx.

It'll also be interesting to see the extension method proliferation in support of these new types. At least if a legacy API does not support it, one can always add support oneself.

Thanks Erik.  That was a good read.  I see Birrell (of MSR and high priest) was referenced in that (RPC) paper.  It would be cool if you could at some point do an expert-to-expert on this subject with him - as he also seems to have some background in distributed systems and concurrency (e.g. Automatic Mutual Exclusion project).  Someone may enjoy following paper before bed - automatic mutual exclusion, multithreading, atomic fragments, transactions, c#, o my... :

http://research.microsoft.com/en-us/projects/ame/ame-yielding-montanari.pdf

http://community.bartdesmet.net/blogs/bart/archive/2009/09/12/taming-your-sequence-s-side-effects-through-ienumerable-let.aspx

Thinking about Erik's Fundamentalist Functional Programming talk at OSCON reminds me: 

DateTime.Now should be an IObservable<DateTime>!

Charles,

Is there an expert-to-expert episode out there where CCR (Concurrency and Coordination Runtime) is compared to Rx?

I would like to know of a specific similarities. May be Erik, from his functional and Rx perspective, can lead CCR discussion?

I've heard CCR is very fast and I would like to know how Rx will fit into this concurency picture.

Can I apply both CCR and Rx? What would the best use case for CCR and Rx? Where one is better/worse than the other for concurrent apps?

I watched how MySpace implemented CCR into their code, it was very interesting.

This seems very exciting stuff.

Coming from the videogames industry I got fascinated by functional reactive programming, and started learning Haskell and FRP a couple of years ago.

C# is still my favorite language for writing production software, and it's great to see that my studies of Haskell & FRP will now help me write better reactive C# code. I did not expect this FRP hobby of mine to actually be useful in C#! So many thanks guys!

Now the FRP systems I looked at (Fran, Yampa, Fruit, Reactive) are side effect free; it will be interesting to see how easy it is to master Rx in the presence of side effects.

One thing that seems unclear to me is if Rx uses times at which events happen, and how they are sorted. For example, in Conal Elliott's Reactive library you have a merge function that merges two event streams, ordering them by occurrence time, left-biased if two events happen at exactly the same time.  Not sure how Rx handles that; I guess because it embraces side effects, it doesn't really care, since it is fully push based, what is pushed first is handled first?

Really looking forward to play with it!

I am trying to work out what advantages this has over classic tightly coupled event handling or asynchronous WCF service calls. The only thing I can think of is the merging of events via linq however, is this not achievable via a 'normal' messaging model such as msmq where by the queue is doing the merging of events/messages? An uglier example of this is having a session state aware WCF service. Am I missing the point and is this just another way of doing it with the advantage of being easier to implement so developers can concentrate on discovering the needs of a system rather implementing solutions to the problems the needs of the system produces?

Regarding the problem with receiving non linear events and solving this with the until, how does this work when the observers are in distributed system like a cloud environment where the until would need to be able to block/dispose of processes on other physical/virtual machines? Unless the cloud is already doing the thread merging and I just haven't found out yet.

WOW ... TKX Charles .... this was simple an amazing video to watch. And YES Erik Rx is both elegant and mathematically correct (just like LINQ) BUT PLEASEEEE do not retire. Wes Dyer dude u rock too bro ... and I am jealous u get to work with Erik

THANKS again Charles/Erik for putting togather the E2E series and allowing us, "the App Develpers" really learn more about language design and how stuff is actually implemented behind the scenes.

p.s.  I introduced these videos to my hard core Java developers and they are slowly but surely getting warmed up to the beauty of C# and .NET in general just love it.