ARCast - The Evolution of Architecture with Martin Fowler

Download this episode


Change is happening all around us, new technologies, new methodologies, but how are these changes affecting the ways in which systems are architected and how do recent developments like patterns and refactoring cause us to think differently about architecture? These are the questions that I had in my mind as I sat down to chat with Martin Fowler and Ian Robinson of Thoughtworks on this episode.






Start Time


Welcome friends, welcome back to ARCast. I am your host Ron Jacobs bringing you yet another episode from the Architecture Insight Conference in Newport, Wales, in the UK. Well, I was, well I was just really happy to have the opportunity to have somebody like Martin Fowler and Ian Robinson from ThoughtWorks come on the show to talk to me about their thoughts on which the ways that we architect systems are changing. There is evolution happening, new superior methods are evolving out of the old crud… the slime that architecture has crawled up out of and well, I just wanted to know what these things are and who better to ask than Martin Fowler and Ian Robinson. Lets welcome them to ARCast.[Claps]


Ron Jacobs

Well, welcome back. I am Ron Jacobs here at the Architecture insight conference, here at the beautiful Celtic Manor Resort where I am joined today by Martin Fowler and Ian Robinson. Welcome guys to ARCast. [Martin: Thank you.][Ian: Thank you] And so today, we are going to talk about the evolution of architecture and where this is going and so how do you see architecture changing in the last few years? Where do you think it is going? Martin!


Martin Fowler 

Umm, when I think of evolution of architecture and the focus that I am, I tend to put on to it is, what one of the some mistakes, I think that, we have tended to make, make in the industries is, to view that the good design is something that you build and then develop and then change is not something that one would expect. In fact, I think art of the underlying assumption is being that, if you design things right first time, you don’t have to change it, that’s when you know you’ve succeeded and increasingly people I think, have come to the conclusion that, well this is just very unrealistic position to be.


Martin Fowler

So, what we said we have to think about from beginning is how do you to constantly evolve and change the design or architecture of existing systems and how do you make that in a graceful way of doing that really is part of your basic way of working and that’s where, I see this evolution of architecture being…, being quite important and of course then there is a meta evolution of thinking about architecture as one of the evolutions is to realize that evolution is important, [laugh] which is I suppose the.. the..the meta question!


Ian Robinson

So, the architecture is one that’s gonna accommodate evolution.



Yeah, and thinking about changes to how you put your stuff together and coming up with technique that allow you to be able to make changes more easily.



Ah, OK, so this is, this is part of the push or wave agile has become a popular thing and really … because I think in the past we sort of assumed, that if we can perfectly understand the thing we needed to build upfront then we can perfectly gather requirements and perfectly build it, but in practice that wasn’t practical.



Yeah! I mean... It’s a... it’s a very basic level. A good example of this is refactoring, which obviously something I’ve been long interested in. For a long time people really weren’t interest in how you would change an existing code base! Um… Change with code base was a kind of seen as a necessary evil, um... preferably to be minimized and ALWAYS one would expect it would make things worse.



And I very aggressively combated that when I subtitled my book on refactoring, I talked about improving the design of existing code that actually change can be a beneficial thing to the overall design. And that’s had quite a significant impact. And so now we are now beginning to see refactoring tools & here I am not going to take credit because that’s the work by people who did the tools. But I think we have changed the expectations of what people say good. Increasingly I see now that more able development teams are absolutely expecting refactoring to be a fundamental part of how they work.



Well that’s an interesting thought because it’s sort of a necessary aspect of maintaining the code in the long run. [Ian: Yes.] I can recall working on a product in silicon valley in the mid 90s, where a the original portions of this product were written under Windows 1.0, and they had never bothered to refactor anything and so the code by the time it got to 95-96 was absolutely hideous, I mean, it was almost un-maintainable, because for that very reason that they had never thought to maintain it or to refactor.



I think they are pretty good at accommodating change during software design and development, but we’re less good at thinking about the overall lifetime of a system and change coming up once something has been deployed in production and how do we accommodate change and how do we help support operations and those people have to maintain and support systems. And so lot of our interests is in coming up with practices, designs, architectures to support change and evolution, not just during the initial design and development but for rather entire lifetime of the system.



This is something that is particularly important at, I mean at this conference like so many of us are getting a lot of the service oriented architecture buzz. Well! Surprise! Surprise! Services and their interfaces are gonna have to change and it’s important to make that part of your way of thinking and that’s now one of the things that Ian is particularly interested in do you think about interfaces that are kind of changeable in that context?



Well, in there is a tension there, isn’t there? Because for years we have been telling people that interfaces should be immutable, that they should be fixed and once people start using them you can’t change them. How do you accommodate change with interfaces?



Right! I don’t think that’s necessarily true any more when we talking about the community of services, service providers and consumers. So I know that there is lots of interest at the moment in how we evolve and version services, I know patterns & practices group has an interest in that.



We work with clients to develop communities of service providers and consumers, where we know that the provider needs to evolve independently of those consumers and typically we find that despite people at hearing to those four tenets of SOA, none the less they end up strongly coupling their consumers to their providers and very often by reflecting the entire contract inside that consumer. So they use XSD validations and a kind of naïve serialization. Which means, whenever the provider wants to change and modify what is its interface, make extensively breaking changes then all of those consumers have to jump at the same time.



As we have been working with clients, we have to introduce these ideas around doing just enough validation so consumers really should be consuming only those bits of a provider they really want and intend to process rather than naively reflecting the entire service within each individual consumer. And if we can bring to the surface some of these real contracts that exist between service providers and consumers, how a service is actually been consumed here and now; and if we have insight into that process then we are better positioned to plan and mange the overall evolution of that system, so what we are talking about really just enough validation but also introducing a form of contract that really allows the consumers to express what is they really want of a system, what is this functionality they want to exploit and what they expect those contracts that the service providers exposes, what they expect those contracts consisted and once we can have insight into these contracts then I think we are better placed to evolve the provider independently from the consumer.



Ok, I am not sure I understand! How could a service… I guess, consumer of a service consumes it without taking too much, I don’t know how to even express this, [Ian: OK. All right] too much dependency?



So if you think first of all about the messages that we might want to exchange between a provider and a consumer. [Ron: Yeah] I might provide, might expose a complex message structure, it might be a pretty large message that I am sending to a no. of consumers.



Now, each of those consumers might only be interested in small amount of information in that message. Typically, if they did XSD validation and they doing all or nothing validation of that entire document. But if the consumer had some way of telling the provider, “Well I am interested in these particular fields, that’s the nature of my contract with you as a provider, please guarantee that in the future whatever you do, as you evolve, you maintain that very particular contract and that contractual interest, or expectations that I have with you.”

And then if the provider conveyed informed of each of those set of expectations, that come from different consumers, then they really have a total set of expectations that they have to continue to fulfill, but everything else that falls out side of that, outside that set of expectations is up for grab, they can change it willy-nilly.



And good example of how that works, if you think of messages being transported around those XML documents a good way to make things break with every change is to use some system with automatically generates a class of an XSD and something of that kind when anytime you can change the document, booms. You got to rebuild the class in order to be able to consume it. So you got real version issues.



Even after three fields, what it’s much better to do is to write something that says, “OK, here is three bits of XPath to tell me the three bits I want to get. And I’ll take that, that and that.” And then you got much more flexible approach. If you remove fields that you are not interested in or add extra fields, the things still work. Because you are just slicing into the document and grabbing what you need as opposed to trying to process the whole thing.



What’s interesting about this is, it sort of turns the model of contracts on its head. Right! I mean, because emphasis has always been on the contract from the servers or the service providers point of view, saying “Here is my contract” and you are suggesting that it ought to be possible for the clients to say, ”Well! I know you have that message but I only care about this part.”



Yeah, and we call that consumer driven contracts. [Ron: Ah!] So to my mind services are of no use to business unless they are being consumed. So, by trying to focus on those consumers and allowing them to express, what functionality they want to their service, what business value they want to exploit, then we start really getting closer to the way in which your system is realizing some business value.



But, now the interesting thing from services point of view there would be that, the parts of my contract that matter essentially become the union of the elements that consumers care about and anything that no consumer cared about doesn’t really matter.



Exactly, and so, I may very well publish an interface that several methods, several different messages, documents whatever, those bits of that interface or that contract that nobody is using, well, if I got rid of that, who would notice. So that’s the stuff about interfaces being immutable, I think, there are ways of expressing our insights into the relationship between providers and consumers that the change or attitude to interfaces.



Ok, I understand what you are saying but I am not sure, I see the value in thinking about it this way. Maybe so, I guess there is value in the sense that if I wanted to add something new to my contract that no consumer knows about, therefore they don’t care about it, therefore it shouldn’t really matter, it shouldn’t break them.



And that’s, that’s exactly the point, a very simple change to interface is you want to add a new field, [Ron: Yeah] maybe because this one consumer here wants to have field, now that shouldn’t affect 50 other consumers what I have currently got. They should just carry on working, there should be no change.



[Ron: Aha] Now as I said, if we define our contracts in terms of XSD now we got to have either up the new version number or create a whole new service just in order to accommodate with one freaking little field that one guy wants . But if you think of your contracts in this consumer oriented way, then nobody cares. You just add what you want for one guy.



But this kind of works against the whole flow of the way the tooling around all this stuff works though. Doesn’t it? Because all the tooling is like pointing at your contract or generate a proxy in a very fixed on that contract, but if you change that contract everything is gonna blow up.



Yeah, but I would say that a lot of the Microsoft tools WSE…, Indigo… even ASMX web services allows us to inject different kinds of validators into the pipeline. So we can do just enough validation rather than consuming an entire service. All or the nothing consumptions of service. So I think we do have a lot of flexibility at the level of their technologies we are using.



Well it depends on how advanced you want to get. Right! And so the very simplest way is to do these things are always going to be very tightly bounded to the contract or at least they are today.



Yeah, that’s true. [Ron: Yeah.] That’s fine in many many circumstances but well, we know we have a community of services where we want some of them to evolve or we know that some of them are going to evolve at different rates, then I think at that point its worthwhile considering some of these other issues and thinking about consumer driven contracts.



And how does the consumer or how does this knowledge of the consumer cares about flow back to the service provider?



OK, So, we have expressed this in terms of a pattern and we have tried to remain relatively implementation agnostic. So you could do an entire out of band. You know, you should get team sitting next to one another. They could actually turn to one, one team member turn to another. And say, you know, these are the things we are really interested in or we are really using but you can obviously work your way up the stack and start actually passing messages out of bound. And there is provision for that in many ways, metadata exchange and stuff like that give us some model for being able to exchange lots of additional information about the way in which services are being consumed. [Ron: Interesting.]



Another good way to think of it in terms of example approach in one that particularly uses tests, if I require these three fields to be present on a message while I can write test cases that express that and then send those test cases back to the provider and basically say, “As long as those test cases run, I am gonna be OK.” And so then the provider effectively has to look at the union of test cases of the consumers. So those kinds of techniques can also be used to try and do this kind of thing.



Ok. This is one way in which we are supporting the requirements for evolution of the services by maybe not being so tightly bound to the message contract from the consumer side. I think some other way you guys are thinking about architecture evolving to support this sort of rapid change?



Another area that, that could, in particular, a colleague of us has been working in is of evolving databases. Because databases have traditionally been considered to be one of the things that you just got to get right from beginning. And Pramod Sadlage, is the name of my colleague, who is doing this kind of stuff since about 1999 where we worked on the project in States.



And just last week he and Scott Ambler released a book in my series perhaps on refactoring database. Which is all about how do you make these concern changes on databases and a lot of this kind of stuff is very similar. You want to be able to do things like add a column to a table and that’s obviously because of the nature of SQL itself is usually a non destructive change because if you add a column it doesn’t affect any of your SQL calls because you either a name what you have or you take it all that with *, and that has that same property but you are not breaking that was built in SQL.



But obviously one of the complication with database is you also have to migrate the data. So refactoring of a database also has to include data migration as a part of that. And these techniques are not been terribly well known, but we been steadily using them up on to our projects so over the last 4 or 5 years and that to me is another very significant shift in enterprise systems is for increasing classes of application we can deal with refactoring the database.



Now one of the big issue lies when database itself is being used as an integration mechanism. I mean this is where, again I think the service oriented architecture being actually works kind of nicely. Because at least one of the common assumption between service oriented architecture is you should hide the database stuff behind a different kind of middle message oriented whatever and API and then you don’t use a shared database as an integration mechanism, use some kind of messaging techniques. So that fits in well with that.



Well, I think one of the enlightening things Ian and you mention that you were involved in the integration patterns work and when I was in patterns and practices, we use that. It was a wonderful work, highly recommended. One of the enlightening things to me about, that I learnt from that project as, as people have moved more towards service oriented integration as the key way, the prime way they integrates system often they found that sometimes the sheer volume of data that must be integrated sort of overwhelms the technology stacks that supports service oriented integration. Right now that, so there are at least two different styles of integration such as shared database.



Yeah, See you end up with several different styles within any one system. I think that’s fine. You know we deal all the time with heterogeneous system and that’s no problem but if we need to have a certain awareness of what we are doing and insight into it rather than having the technology lead us. So, yeah, it’s a matter of evolving the practices and the insights so that we have more confidence as we moving into production and supporting and operating systems.



Well, you find that someone will come up and say, “Oh! I am trying to integrate these two systems and it worked pretty well in test environment but once we started moving, you know, millions of rows a day, the day we begin to fall away pretty quickly.” And that, you know, this is when you realize that the expense of this serialization to XML and back is very, very significant, when it comes to the sheer volume of data.



And, that’s, that’s probably going to become increasing problem, I think we are going to see more and more instrumentation injected into those service as we are exchanging documents and messages. And so there is inevitably going to be an overhead not just about serializing data but inspecting it as we receive it or as we send it. And so, yeah, we need to start thinking about those issues as well.



OK. So we talked about a sort of refactoring services and refactoring databases. What else is evolving in architecture?



[Ian: Come on]Those of the two, I guess, are very much in the front of my mind. But I think more with it, we just have to look at that as a constant statement we have to look at. Whenever we are coming up with the design about anything, we have to say “OK, what’s version 2 and version 3 gonna look like and how we gonna get there in a gracefulest way we can do that.” And that’s one of the thing I like about iterative development because it kind of forces you to evolve all the time even in your development stage



And one of the nice things about evolutionary of the database stuff was Pramod would do this every single iteration. It evolved the database. It migrated all the test data. He will do all of this every, months before things went live and what happened of course is the, when system went live and he had to evolve live database, he’d already figured out how to do it. With me, it was already a done thing.



So in many ways iterative development forces you into this constant evolution notion and yeah, it causes a lot of pain because you have to solve all the problems about evolve… evolution early on in the project, but early pain is cause of benefits of agile development as far as I am concerned. [laugh]



Well, the point is that we are going to have pain anyway; we might as well get over with it early on. I mean, you know, you talk to people about evolving a database, especially a production database; the pain is, in that data migration. Just simply adding a column or whatever, database might turn for a couple of days while it migrates over the data. So you do have to have a specific plan about how you are going to accomplish that.



[Ian: Embrace pain.][laugh] Yeah, what, they can get pain early, while you’re still able to deal with it. The worse thing to do is to have it a when you are in rush and you got a hell lot of things to do, and also do little bits of pain at a time so it gradually builds up and you learns techniques, you can apply those techniques steadily and then you’ll figure out how to cope and it’s just the same is with integration. Integration used to be a big pain for lots and lots, and it still is of course for lots and lots, of projects.



I mean for us at ThoughtWorks integration is been a non event for many years because we’ve learnt to integrate all the time, multiple times a day. We had a hand obviously in developing Cruise Control and Cruise Control.Net and lots of project all over the world now use these tools and do continuous integration. And the big pain of integration is you anticipate so many facts now, as a result of that and I think that’s a that’s an another great example of, “If it painful do it more often.”



[laughs] What I can recall early in my development career, I worked on a very distributed team and so we would oft be, you know, theses guys over here doing there little bit, those guys doing there little bit, we would all have our thing working and then every couple of weeks we go through this very, very painful process of everybody trying to put all the changes together and suddenly nothing would work. For 2, 3, 5 days, you know, nothing is working and then we get going again. That was, that was an extraordinarily unproductive.



Everything stops, and even at the end of 5 days you’re not entirely sure, where you stand, things are working but does it support the level of functionality you expected it to be before you try to integrate those two systems or have you introduce other problems that are still to pop up further down the line.



Yeah, and, and regressions were always occurring because one side will get the fix in but other side wouldn’t, it in the integration. One... the good set of change would get thrown away and they will always come back and, lots of pain will definitely evolve around that.



Martin has already mentioned Cruise Control and continuous integration but another colleague here Graham Tackley has been doing lot of work around deploying into virtual server environments, so every day, in addition to continuous integration, we will also be deploying into environments that try to reproduce the production environments. We can tear them down and redeploy into them over and over again. So again, we trying to avoid being surprised by changes between development and test environments and then production environments



Well, and I think that’s an absolutely one of the best thing that’s happen at that long time is ability to virtualize the production environment. Especially when you are involving other parties in your development process. Because they can actually reproduce the production environment in their location, which is something that was virtually impossible before and you would never know everything is going to work until you go in real environment.



Yeah. So, let me ask a little bit about patterns, because this is one of the things you are well known for. And I am always grasping about how are patterns really, really helping people, architects in the real world, I mean, other than that, we know a lot the people have bought the books and they have them seen on their shelf, but when do they become… how do you make them actually helpful in your daily work?



Of course, I am not the right person to ask that question to. Since I am the person who writes about them and Ian maybe is the better person, since he actually has to use the stuff.[laugh]



Well, I think firstly good patterns are really after the fact, they are expression of experience. So I am always weary when somebody comes up with new pattern that probably hasn’t been tried and tested. So good patterns are reflection of what we’ve been doing for long time, we, you know, software development community, and so distilled experience, distilled knowledge and we end up with way of describing systems that don’t necessarily have to go all the way down to the technical level, to the implementation detail.



But if something that helps us to bridge the gap between business and the functionality we are trying to provide to that business, I think it introduces middle ground into the picture and a way of talking about systems that can involve a broader range of stakeholders. So there is a language and a vocabulary that day to day helps development teams, talk to all the other stakeholders in the program, you know, there is, a lot of a, very specific patterns that for the raising but a lot of integration patterns in particular help us describe what is the way anticipating doing as we connect systems, and they do make sense of what we are doing to the business.



Ok, so you know, a pattern is putting a name on something that we’ve already been doing anyway but, but to many people who haven’t been doing that thing mean the hope, I think, is that they can read about it and learn about it and, and then suddenly they are benefiting from the collective experience of those who did it.



That certainly been the intention is to try and say, “Here is what we currently understand about how this pattern works and when you should use it.” And that’s something that people can use as a starting point, so they don’t begin from blank sheet of the paper. They can actually begin from some degree of guidances as to how to do things.



It definitely will never cover hundred percent of circumstances because there is so much stuff in the localized things. So one of my taglines is, “Patterns are all half baked you have to finish them off by yourself.” But the point is you are starting at least a half way along, you got some idea what you got to worry about, some sense of some of the issues to deal with and how you can deal with them and it will you help you to get there a little bit quicker and, and I am not the someone who looks the huge silver bullet victories, those tiny little victories please me. So that’s why I tend to be comfortable with patterns because it’s lot of little victories.



Yeah, well and you know the thing we always struggled with when I was in patterns and practices team is thinking about, “OK! We created a book, like integration patterns, but how do we help drive that experience into the day to day work of the people who have to get these things done?” and probably the easiest way is to take a pattern, patterns, the patterns and bake them into the framework that the people can just apply but that quickly becomes difficult because the frameworks begins to grow exponentially and become very complex beasts on their own. And we were after the silver bullet victories like you said, and sometimes they are very difficult to achieve. Whereas, you know, just taking a small pattern, understanding and applying it is really the goal.



Plus, patterns is simply invitation to experience the pain earlier. [laugh] You get excited.



Everybody is after pain free development but we haven’t gotten there yet aren’t we.[laugh]



Oh! We never will! Software development is tough and I don’t think anything is going to make it easier. I mean certain thing will become easier, what, memory management for instance but memory managed system make that much easier, but then, we got whole new problems to deal with as well, because we always wanted to do more with software



So I think as a fundamental activity, software development is always going to be hard and always going to benefit most of having talented people working on it. What we can do is try and accelerate the process as much as we can, make the whole thing as rapid and have new and different kinds of pain instead of the old ones.



I am very happy to no longer have to manage my own memory. Thank you very much. That’s great. But it doesn’t mean that there is lack of things to do. [Ron: Yeah]



Well, Martin and Ian thank you so much for joining me today on ARCast [Ian: Thank you.] [Martin: Thank you!]



Martin Fowler and Ian Robinson, ladies and gentlemen. Hey! Wow! Thinking about evolution of architecture. You know these guys are some really smart thinkers and I love the work they’ve done with patterns. And, you know, I’ve encountered ThoughtWorks through my time with patterns and practices and we collaborated with them around  many projects and they were just terrific and you can tell they really know their stuff and are really thinking out on the edge, where things are going and this is just fantastic stuff. So, you know, I encourage you to go to take a look at their site on They have some interesting papers and some patterns. Ian gave me a paper of some other patterns that he has done and that’s great and that’s why we are here. Here on ARCast.




Available formats for this video:

Actual format may change based on video formats available and browser capability.

    The Discussion

    • User profile image

      One question I have about iterative development or constant refactoring/evolution is:

      How do you ensure that the codebase remains in a maintainable (readible / tidy) state and that it doesn't suffer from the constant churn.

      Otherwise I agree with all the sentiments expressed in this podcast, especially taking the pain early or "go ugly early" as it is known here.

      I would be very interested in futher reading about the evolution of service interfaces and Just Enough Validation which was discussed in the early part of the podcast.

      Any suggestions?

    Comments closed

    Comments have been closed since this content was published more than 30 days ago, but if you'd like to send us feedback you can Contact Us.