Bart De Smet

Back to Profile: bdesmet


  • Bart De Smet: MinLINQ - The Essence of LINQ

    @Gregory81: Perfect answer. Materialize - Where (or SelectMany of course...) - Dematerialize is a roundtrip scheme that can be used to implement exception handling for sequences. T is turned into Notifcation<T> and back.

  • Bart De Smet: MinLINQ - The Essence of LINQ

    @AdamSpeight2008: To define First, think of an Aggregate whose accumulation object tracks the first element and whether it has seen it already. However, keep in mind this is not the best definition of First by any means, because Aggregate will drain the entire sequence before it can produce the result. This doesn't work for side-effecting iterators (in LINQ to Objects, you won't see the effects for anything but enumeration to the first element), and infinite sequences. Notice how First is one of the only operators in Rx that leaves the monad; aggregates in the Ix world are a bit of a cata* (fill in the blanks yourself) from a monadic point of view. In Rx, aggregates return single values inside the monad, which allows for cancellation, and a Cata (in the sense used in the interview) has less value in this world.

  • Bart De Smet: MinLINQ - The Essence of LINQ

    @ShinNoNoir: Thanks for watching in 2011 too! The Amb roundtrip is indeed a beautiful one.

    There are plenty of other ones, so here's one exercise for the viewers: assume your team has only the budget to use one try...catch block (assume each block costs as $1,000 and the licensing model of the C# compiler is to pay per try...catch used) in the entire codebase of your LINQ to Objects implementation of the Standard Query Operators (i.e. IEnumerable<T> based). How'd you spend it (*) such that you can easily implement error-handling operators in the future? Examples include operators such as Catch :: IE<T> -> (Exception -> IE<T>) -> IE<T>, but also Retry, etc.

    Tip: you want to realize Catch, Retry, etc. using a roundtrip scheme of IE<T> -> ??<T> -> (wow, error handling got so much easier here thanks to the $1,000 well-spent) -> ??<T> -> IE<T>. Try to find what I've highlighted in bold and you may be onto something (nested tip: we got this in Rx and Ix).

    (*) Constraint: use the try...catch inside a proper operator/combinator that deals with sequences. I.e. simply wrapping try...catch in a generic helper method such as the one shown below is not an option:

    static void Catch<E>(Action action, Action<E> handler) where E : Exception
    catch (E e)

  • Bart De Smet: MinLINQ - The Essence of LINQ

    @exoteric: Sure, there's real-world significance, ranging from setting a design philosophy for a query language or processor with proper layering of operators and their semantics, where compositionality rules the world, onto implementations itself. I won't yet comment on the latter category at this stage, rest assured we can keep ourselves more than busy on our team.

    For now, it should be clear that the MinLINQ implementation is truly meant as an academic exercise, in particular when it comes to using the implementation for real-world scenarios, due to the various shortcuts that have been taken (no disposable nature, error handling omitted, etc.).

    Exercise for the viewers: what'd be a good generalization of the set of GroupBy operators? How about other kinds of partitioning of an input sequence in chunks (e.g. Buffer* operators in Rx)?

  • DevCamp 2010 Keynote - Rx: Curing your asynchronous programming blues

    There are indeed similarities between the two, but enough differences to warrant their coexistence. Let's talk about this now.

    First of all, from an interface point of view, IObservable<T> and Task<T> share the same mode of data acquisition, with data being pushed at you through a continuation (cf. IObserver<T> and the Task<T> ContinueWith method). One difference lies in the arity of the data. While IObservable<T> can describe the acquisition of multiple values of type T (cf. OnNext), a Task<T> can only signal a single value. So, IObservable<T> is more general than Task<T> from a one-versus-many data-centric point of view. In other words, IObservable<T> is great to describe push-based streams of data (such as for event processing) while Task<T> covers the single-valued case. No surprise we have conversions from Task<T> into IObservable<T> in the Rx libraries. You could do the other way around as well, for example collecting an IObservable<T>'s data into an IList<T> which is wrapped in a Task<T>. Here you're loosing the timing information of inidividually acquired values though (read on).

    Now on to the Async CTP. What the new async features in C# and VB are all about is sequential composition for asynchronous programming. It's all about writing code that runs in response to an asynchronous task's (task used in the broadest sense here, not just Task<T>) sequentially with respect to the code that triggered the operation. In reality, the compiler turns the sequential-looking code into a continuation on the asynchronous task:

    var res = await op();

    This really turns into something along the following lines (in reality much more complex using a state machine much alike iterators, and with a check for BeginAwait's Boolean return value):

    var aw = op().GetAwaiter();
    aw.BeginAwait(() => {
        var res = aw.EndWait();

    The implementation of GetAwaiter depends on the return type of the "op" call above. In case it's a Task<T>, the awaiter represents an entry-point to establish a ContinueWith operation passed to the BeginAwait method. However, the compiler is by no means tied to any particular implementation of the "await pattern" and will choose whatever is available through regular method resolution (instance methods first, then extension methods, all regular rules apply). In particular, the Rx library implements the await pattern for IObservable<T>:

    var res = await Observable.Range(0, 10);
    Console.WriteLine(string.Join(", ", res));

    The inferred type for res will be IList<int>, collecting all the values from the IObservable<int> sequence (here the range with values 0 through 9) that were signaled before OnCompleted. If an error is signaled through OnError, the EndAwait method call will surface it. Notice the use of an IList<T> for the await on IObservable<T> because await is about single values, while IObservable<T> is about sequences or streams of data.

    To conclude, let's give an other observation (pun intended) here. Rx and async/await are totally complementary. I think we all agree that both technologies deal with asynchronous programming. One difference is in the arity of data, another is in the way you perform composition. We've seen exactly this coexistence situation for synchronous programming as well. Here are the two language features I'm talking about:

    • ; - No, not a typo: semi-colon. Semi-colon is the way to sequence synchronous code operations in imperative style languages like C#. It's also your way to flow data out of an operation, into a variable and propagate it to the next operation.
    • Query expressions, aka LINQ - This allows for composition over sequences of data by chaining (sequencing) together query operators through a fluent method invocation pattern. This is not your typically control flow, but totally complementary to the above.

    Today we're seeing this repeated for asynchronous programming. For the first bullet point, we now have the await keyword, which allows code-centric sequencing of asynchronous operators. For the second bullet point, LINQ can be used to operate on sequences of data (using a pipelining mechanism for sequencing) such as IObservable<T> collections (or IAsyncEnumerable<T>, search C9 for more information on that one's role). Both can nicely coexist.

    Hope this helps!

  • Bart De Smet: LINQ to Z3

    @aL_: Will see what we can do around this in the future. This said, expression trees are more of a tool than a destination. However, as more LINQ to Anything stuff materializes, I can see some good fit for a future episode that would benefit from further explanation of those animals.

  • Bart De Smet: LINQ to Z3

    @piersh: I'll figure out a way to get the bits to the community in the forseeable future. Likely some further exploration is desired to streamline the experience further and have more Z3 constructs exposed through the API. Stay tuned.

  • Bart De Smet: LINQ to Z3

    @Kyrae: Some interesting thoughts you have there Smiley.

  • Bart De Smet: LINQ to Z3

    @Batman: @Charles: @Novox: The Flemish part it is indeed. For a good introduction into the many sub-atomic (due to the size) dimensions (due to regions, governments, languages, etc.) of the country Belgium, have a look at (obviously they're wrong about the "Swiss chocolate" and "French fries" Tongue Out).

    I'm personally from the area around Ghent (visit using Bing maps:, so my accent is definitely Dutch, though I speak a bit of French and German as well.

  • Bart De Smet: LINQ to Z3

    @felix9: Standard Query Operators go only as far as what was once deemed "standard" and applicable to "queries". Monads are a superset of sequence-style query comprehensions.

    Let's elaborate a bit on those points. First, even within the domain of querying, different data models or acquisition methods can give rise to other kinds of operators. For example, the push-based nature of IObservable<T> makes it a very natural candidate for signal-based programming where an observable sequence acts as a signal wire (e.g. TakeUntil). While it'd be very natural to expose such capabilities using language-integrated constructs, that may be a bit of a stretch as it goes beyond general purposing querying (which, till further notice, LINQ is about). Similarly, time-centric operators open up for a whole series of imaginary additions as well.

    Personally, I do like the query comprehension syntax for the things it's good at. At some point, you'll for sure run short on the provided keywords (with varying mileage in C# versus VB, the latter of which has more keywords added for LINQ), but you can quite gradually "ease into" the dotting style of operator chaining (unless you have to insert between existing clauses, bad luck if you don't like some wholesale refactoring of the query expression).

    Stricly speaking, you'd only need SelectMany (aka "bind") support ("do"-notation in Haskell, multiple from clauses in C#), to have all the power of monads at your service, assuming there are ways to "return" into the monad (e.g. new[] { value } for enumerables, Observable.Return for observables, etc.). Obviously that's too minimalistic (search for my project called MinLINQ to see this extremism in practice).

    Finally, on the CaaS front, it's not at all sure at this point whether language syntax will be provided as an extensibility point, unless you can go in and extend the lexer and parser itself (causing you to create a new "dialect" of C#/VB altogether). Doing so definitely would limit the movements the benevolent dictators of the language design can take going forward ("oops, we may be clashing here with the 'bar' keyword introduced by customer 'foo' for the 'baz' dialect of C#"). The fluent nature of extension methods and the (relative, that is, without the support for "sectioning") conciseness of lambda expressions makes C# already a fairly good host for internal DSLs that follow an API-centric design. Popular patterns can always be lifted into proper language constructs, e.g. LINQ and the new "await" stuff.

  • Bart De Smet: LINQ to Z3

    @aL_: Your idea of using let-bindings in query expressions is definitely worth looking at. In fact, I've done something along those lines before (I won't elaborate on the broader context there just yet, in order to whet the appetite of the Niners Wink). One drawback is its use of Select underneath (through compile-time erasure using transparent identifiers), which makes it indistinguishable from proper select clauses. Hence, we open the door for more expressive queries written by the user, for better or for worse. This said, some tricks can be played.

  • Bart De Smet: LINQ to Z3

    @exoteric: A lot of real-world technologies already use Z3, cf. their portofolio mentioned in various presentations (see for some links, e.g. to the Slides for WING 2009, cf. slide number 4). But creative minds can of course go crazy using Z3, subject to its academic license that is (no, I'm not an attorney Wink).

    Concerning Z3 versus Solver Foundation, I think the versus approach used for technology comparison is a little misplaced. Sure, there's some overlap, but Solver Foundation has its own strength and positioning in the market. Referring to it by its codename, "Optima", you should get some good insights in the kinds of problems it specifically aims to tackle. The target audience is also much further away from low-level software plumbers, more into the realm of business analysist (cf. its plug-in with Excel).

    Once we make progress on (still hypothetical) Solver Foundation bridges using LINQ, I'll provide you with an update through the known channels Wink.