Loading user information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading user information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Where's the ESB?

48 minutes, 51 seconds


Right click “Save as…”

This talk, which I recorded this week here at Microsoft campus in Redmond, is a slightly shortened version of a talk I did a few weeks ago at a conference, and which is a follow up to a blog post I wrote in January titled "Utopia ESB".

You can find the slide deck on SkyDrive.

The motivation for doing this talk is fairly straightforward. We see the "ESB" term and the architectural notion behind the ESB showing up in many customer engagements and also in RFPs. And we see them showing up in contexts where the goal is scale and availability. The reality is that you can't have both a central control model that the ESB theory promises and yet run a scalable and robust federation of services. The goal of this talk is to highlight the paradox and to present an alternative that's proven in the architecture of large distributed systems.

This talk is obviously another controversial one, so I'm looking forward to comments here or on Twitter under @clemensv

 [Sorry for the somewhat dim light and the ambient A/C noise in the background. That's a daylight conference room in Redmond under Northwest Spring skies and our humming building 24]


Follow the discussion

  • Oops, something didn't work.

    Getting subscription
    Subscribe to this conversation
  • Hi Clemens, 

    Great talk, I'm struggling to come up with a good question, the essence of your talk reminds me of a couple of overly grand xSP projects that failed due to tying  too many services together. (not just on the systems side but also on the sales side).

    My thoughts for both at the time were can we build a rack of service A and sell that first, I failed to explain this well enough I guess. Plenty of experience tells me your right.

    Kind regards,


    P.S. I recommended you to MVA recently, you've got me to blame if they come asking you about doing a course

  • Phil SandlerPhil Sandler

    Fantastic. Thanks very much for going this talk. I've some of these points made here and there, but this puts it all together in one coherent argument.

    I'm guessing you won't be getting any invitations to lunch from the Biztalk team? :)

  • Hi Clemens,

    what do you think about NServiceBus ?

  • NServiceBus is a commercial messaging framework that layers over the actual messaging infrastructure. My understanding is that it favors a peer-to-peer model over centralization.
  • codputercodputer Don't you know what a cod is?

    I get it that centralized cannot scale, but what if the ESB is actually built on top of Azure, leveraging all that Azure Service Bus has to offer.  Leveraging the Message class, messages of any contract could be routed anywhere, as well as inspected.  It would then also be able to scale up and out as needed.  With some creativity in the Message headers (which I recall you have leveraged in the past), a messaging route strategy could clearly be implemented.

    Going the other way, and only having Gateways, the subscriber is coupled to the publisher.  Would it not be better if the subscriber (endpoint that need data) would publish their requirement (on a true bus), and the publisher (who has the data) listen for those "events" on the same bus and simply responds?  Would this not move us to push data services, and do more in stream (StreamInsight/RX-hot observable) implementations on both ends?

  • @codputer This isn't primarily a scalability issue. This is an ownership issue. Who owns that universal pub/sub bus abstraction? If I have a service and that service spews out financial market data, it's the service I go to to subscribe to that data. Yes, I could make a universal pub/sub bus so that I could subscribe to any other kind of market data as well, but now you've moved the problem of making multi-source market data uniformly consumable to the universal bus. A service that's the clearly chartered owner of a particular business domain is a lot better at that.

  • Scott BanwartScott Banwart

    @ClemensVasters So what is Microsoft's story for building gateways? If I need additional mediation capabilities (EDI, transformation, etc.) today I would still need to purchase BizTalk licenses to get those capabilities in the Microsoft stack. Either you build a shared BizTalk system that all the gateways share which introduces the ESB disadvantages you outlined in the video, or you put a BizTalk server at each gateway and incur astronomical licensing costs.

    Is Microsoft planning to introduce anything to assist with building gateways?

  • You need BizTalk's capabilities for certain paths, not all paths. Yes, if you want to handle EDI interchanges inbound or outbound you will still need a right-sized BizTalk installation for that particular path and scaled to the particular load (which usually isn't consumer-scale) that makes up part of a service's gateway, but it's not "the" gateway.

    Multiples service can obviously share that same installation of that kind of traffic if there are clear rules about who owns what and the paths are distinct in order to minimize cross-service dependencies.

    That said, I'm working on some pieces that aim to help with gluing things together for gateways. All the technology you'll need exists in the stack, but there's a gap on pulling it all together. You'll hear about this in the next few weeks and it's going to be an open initiative.

  • Scott BanwartScott Banwart

    I didn't mean to imply that BizTalk would be the gateway, only that some of it's capabilities aren't available in other parts of the Microsoft stack.

    I'll be looking forward to see what it coming for gateways. Right now the story is "Here is a bunch of components that you have to wire together yourself," and that isn't a very compelling story to tell to prospective clients. Especially with the ESB vendors pushing their everything-in-one-box angle.

  • BizTalk really is a premium-feature toolbox. If you need EDI or SWIFT or HL7 support or the adapters then you'll have to license that toolbox according to what you need. The "free" parts of .NET are licensed through Windows Server or through Windows Azure, these premium features are licensed with that toolbox. The BizTalk Branch edition already covers a lot of ground http://www.microsoft.com/biztalk/en/us/pricing-licensing.aspx

  • markmark

    dude, that was awesome. well done. the golf comment especially struck home :-)

  • Ashith RajAshith Raj Wanna be a DotNet Geek

    Thanks for Demystifying the ESB concept. You rock.  One curious question I have is about BizTalk ESB Toolkit - why is ESB Toolkit is proposed by MSFT? what are the scenarios we should use it within the enterprise

Remove this comment

Remove this thread


Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.