Announcer: It is Wednesday April 25, 2007 and you are listening to ARCast.
Ron Jacobs: Hey, welcome back to ARCast friends, this is your host Ron Jacobs. Today we are going to Israel. That's right, in Tel Aviv. I was there in January where I spoke at their developer academy. What a blast that was. They had it at a movie theater complex; they actually made little movie trailers with some of the speakers. It was hilarious.

Now what we have for you today is a recording of an architecture panel discussion. I don't have the whole thing. We had too few microphones and so there was a lot of passing around of microphones. So you are going to hear a lot of little noise of people handing microphones around. I tried to minimize that as much as possible.

I did excerpts of highlights of some of the best things that were said. So if you feel like there is some disconnected thought there, that is the reason why. But there are some real gems in what was said, so I thought I would share it with you. So listen up! Here is the architecture panel.

All right, welcome everybody to this architect panel. I am Ron Jacobs. I am the host of the ARCcast show, which is a podcast for architects.

We have more chairs over here if you guys want to come in and make yourself comfortable.

So, today we have assembled a distinguished panel of architects--they look very distinguished anyway. So we were trying to decide what kind of topic would be interesting to talk about with a bunch of architects. I always like to hear about the mistakes people make.

[laughter]

I think when I talk to people about what makes a great architect, the number one thing I hear from people is "Experience." Well, what makes great experience but making big mistakes and learning from them, right? What is even better is to learn from the mistakes of others. So I want to maybe not focus so much on the mistakes but the lessons that we learned and the things that you would do differently going forward.

OK, who wants to go first? I want to hear about some experience, some lessons learned.
Man 1: I will tell you about the time when I learned the hard way the difference between tiers and layers.
Ron: Ok.
Man 1: I think it was about, it was a long time ago. It was probably my first distributed system ever. I was very proud of myself at the time because I designed this very nice thing about the UI and the separation of layers. It was kind of envy. See, I did not know how to term it then.

I was very, very proud because the events just flew. No code was under any control; everything went to another layer and the data was another layer. I was very happy.

Then we kind of, you know, tried to add more clients. I said, "It is no problem! We will just take this tier, this layer, and we will put it on our computer and clients will connect to that and everything will be fine."

But it wasn't actually. It went really badly. It worked for the first client, it worked for the second client; but when you added more and more clients, everything just went downhill from there.
Ron: You mean the performance was too slow? What was going wrong?
Man 1: Right, you need a lot of performance. You have got a lot of data flowing on the network. We didn't really account for security. We did not account for a lot of things.

It took me some time before I learned about something called "The fallacies of distributed computing." There are eight fallacies right now; they were drafted by people from Sun--sorry about that.
Ron: [laughs]

[laughter]
Man 1: It was done in the '80s I think. It started in the '80s.

The network is there, you cannot just disregard it and don't think about it. You have to know that it is not always reliable. There is latency. You have to take care of security. The network is not heterogeneous; there is no one administrator, and stuff like that. And you really have to think about it.

So tiers, you should actually consider tiers as boundaries, something that you have to cross. It is a trust boundary. You have to know who you are talking with. You have to know if you want it to interact with you, would you allow it to do it, and things. You have to think about the data that is going on between these tiers.

You don't want to have... Well, bandwidth today is relatively easy to accommodate; you can always add it. It costs more, but you can usually figure that out. But you still have to account for latency. Latency didn't get that better over time; you still have relatively large latencies, especially if you are going overseas. So you just can't disregard it.

So, as a rule, I would say that a layer is a logical boundary but a tier is a physical one. And it is really a different thing and you have to think about it. I think that is the lesson..
Ron: Yeah, so it is sort of a nave distribution of logic.
Man 1: It doesn't work.
Ron: OK. So I am just curious, anybody else have a similar experience when distributed computing? Yes?
Man 2: I am talking about computing with [indecipherable] regarding a performance requirement from an architectural point of view.
Ron: OK.
Man 2: And I think there is a connection between it. And the thing that we, as architects, do not pay enough attention to is performance requirements. And the thing that we are not paying enough attention is not that we are not aware about those performance requirements, but whether that we are not splitting those performance requirements into the layers that we are creating.

A project I was involved in was, for example, a smart-client project. A WinForms application on the client side, and some services on the service side where each service is a standalone process, and the communication between the processes was with MSMQ--it was the technology that they chose.

And the client performance requirement was that each process could handle 100 messages per second. And what I mean by handle, is just suppose we have a queue with 100 messages, then they receive from the queue the analysis of the message, then creating another message and sending to another process should be in ten milliseconds. OK, 100 messages per second so it is ten milliseconds per message.

So, taking these requirements into account, it is not saying: Well, OK, when MSMQ can handle it. Because it is not just MSMQ, it is not just a transport layer. Because we, as an architect, design the layers; and we should set up the requirement that the transport layer, the very basic software layer should handle on top just two or three milliseconds per message.

And then when I am choosing my technology I should not ask myself whenever MSMQ can handle 100 messages per second, or ten milliseconds per message, but whether it could be two milliseconds per message. And then I should ask myself: What is the average message size that I am dealing with? Because it is different between 500 bytes I am sending and 50, 000 bytes I am sending.

And just after analyzing a set of the requirements between taking the bigger quantity, and split it into the layer I am creating; just after that I can make a smart architect decision regarding whatever transport layer I will use and what would be the requirements.

In a project I was involved in there was a problem with the amount of data that we send. In order to analyze what were the performance problems we wrote some application performance counter about how many received messages we can deal with and how many sent messages we can deal.

So I think that we, as an architect, should pay more attention not just to the big performance requirements from the client point of view but rather to split it and set up requirements for each of the layers that we are building.
Ron: In this project, I take it then from what you said that after you built it you found that the performance wasn't as good as you had hoped--maybe not acceptable. Then you had to rework the architecture or something to make it work?
Man 2: No. Probably the problem was with something else. But for us, as an architect, this is what we should do. Then, when the client says there is a performance bottleneck, we can check each layer because we created the application performance counters.

We can say this layer is very good because it can handle two messages per second. By writing, for example, application performance counter, we can check each layer.
Ron: Oh, OK.
Man 2: So It is very easy in.NET to write application performance counters.
Ron: This really comes down to instrumenting the application in a way where you can identify where the bottleneck is.
Man 2: Yes.
Ron: OK. That's a very useful thing. Sometimes we see performance is bad and then people go crazy trying to fix things that don't matter and that kind of stuff.
Man 2: Yeah. I think if I want to tell just one thing--just analyze it, split it to the tiers, and write application performance counter. It will really help you.
Ron: I am just thinking about from the proactively planning for and managing performance risk. So what you are talking about is a risk to the project that the performance is not going to be acceptable.

One of the things I found when I was younger, was that I was not as super-confident, so if the manager said, "Oh what are you guys wasting time for." I would be like, "OK, well you know what you're doing. I'll listen to you. Let's stop wasting time."

I wasn't confident enough to say, "No. This is really important. We have to do this, and here is why." I think sometimes it is difficult to learn to stand up against the business owners or the people who are paying the checks for the project and say, "No! We have to do this." Have you had to learn to be more confident and able to stand up against that?
Udi: The way that I probably explain it is -- you don't necessarily have to stand up to them, it's not an us versus them thing. Rather, you have to make it clear what are the ramifications of such a decision.

For instance, if the manager says, "You should stop wasting time and start developing."

Say, "If we were to make such a decision we run the risk at a later point in the project of not meeting functional requirements and having to throw a year's worth of work away and starting it over again. If that is a risk you are willing to take as a business owner, I'm willing to go for that but it's your decision. It's my job to give you the implications of your decision, because you may not have the whole picture."

I don't really view my job as standing up next to them, fight the good fight, but rather making sure that the decision makers have the proper data that they need in order so that they can make the appropriate decisions.

They might say: "Well, look. From what you know this is a one-year project. But what we are talking about is a five-year project. In order to finance the next four years, we need to get something done!

If it doesn't handle 300,000 messages a second, that's OK. It has to look good. That's the point of this year's project. Next year, if we have to throw it away once we got financed for the next four years, that's fine."

So, we may not have the whole business picture. It's not really our job to fight the fight but rather to give the implications, technical and project, of certain decisions.
Man 1: I think that, summarizing our jobs, we're not just... When you're saying you are an architect, you are not just dealing with how to do the code, how to architect your system. A lot of times you have to deal with the politics of dealing with the decision-makers. I'm strongly agreeing with Udi that you need to lay the facts in front of them and let them make the right decision.

So, if I'm saying I create a risk matrix in your architecture, put numbers there that are visible to them. If the decisions-makers are bothered by money, put numbers that say what risk costs how much money. If the decision-maker doesn't care about money but cares about the labor hours, put the risk in labor hours so he will know and has the right factors to make the right decision.
Ron: Yes. That's an important point. When you're speaking to the decision-makers you have to speak in their language. You don't say: "This will take us some refactoring and lots of new lines of code and more modules." They go: "What? What are you talking about?" But, if you're saying: "This is going to cost us another $100,000" - "Oh, OK! Now I understand!"
Man 1: So, no!

[Laughter]
Ron: "No, we're not doing that."

Ok, all right, well what other mistakes made? Udi, you haven't given us one yet. What do you think?
Man 3: He doesn't do mistakes.

[Laughter]
Udi: I'd have to say that one of my earlier mistakes was, I guess you could call it "being in love with technology."

Coming from a development background (some architects come from a more business background) I loved technology. I loved writing code and doing things in five lines of code that other people could do in 100 lines of code--that just made my day.

Every new technology that came out I'd just jump all over it.

The problems occurred when the projects started becoming larger. The architecture and all the code in the system was tightly tied to the technology decisions that we made.

In that project, it was a BizTalk 2002 implementation, where a lot of our code was tightly coupled, to say the least, to BizTalk. And when it turned out, in terms of performance, that BizTalk could not handle that load, we weren't able to salvage any of the applicative logic that we had, because it was also tied to the specific technology that we used. That ended off being a total write-off; we had to start all over again.

The lesson learned for me was to not skip the phase of creating a technology-independent architecture, as much as possible, dealing with the logic entities. And only after doing that step, doing the technology mapping; and trying to keep as minimal coupling as possible to the specific technology. Because you just might end up having to throw it all away.

And the question is: are you just throwing away this talk or are you throwing away 20 million dollars of investment? That was a big one for me.
Ron: That's an important point because I've always found, from my work on products for Microsoft, when we would design a product you always have a certain use in mind. This is the way people are going to use this product, and you design it a certain way.

And then you talk to people who say, "Well, could it do this?" And you say, "Well, yeah I guess you could make it do that but we really didn't intend for it." "But that's what I wanted to do." "Yeah OK, well, I don't know if that's going to really work."

[laughter]

Or you'll get, somebody will try this and they'll get a strange behavior and then they'll want to file a bug or want a hotfix or something. And we go, "Well yeah, I guess that's a bug but we really never intended for somebody to do that so we're probably not going to do a hotfix on that one." And they get angry about it.

But I think it's an important point to look at. Any piece of product or technology, platform infrastructure, it has an intended use. When you're architecting, even if you do a technology independent model, when you're mapping it you have to say, "Does the way I intend to use that thing fall within its intended use? " It's a critical piece there.
Man 1: You should be concerned about it as an architect because the position of an architect is not that well defined in many organizations. You have something like a fire and forget architect.

[laughter]

Goes through a lot of projects. Pulls out a design, something he came up with during the last night, all sweating, and then he says, "OK, this is the solution, it's like a physics theory, this is the solution, this is how things work."
Ron: Yeah. Go build it!
Man 1: See you in one year.

[laughter]

So you have to be, I believe, married to the project from the beginning to the end and make sure that not only, of course, of your design, because your designs probably will change during the course of the project as you understand more about the requirements when they are built.

And you should make sure that not only of the high level design. Because an architect, I believe, deals with both form--which is the -ilties, the performance, the manageability, usability, etcetera--and the content, the design, the actual design. I don't believe in an architect that only draws a bunch of boxes and says this is the performance.

[laughter]

Yeah. Come from MCS. You should be able to delve into the detail in many aspects of the project and see that the actual design in this hot spot is written correctly or understood by the development team as you understand it.

So there's a lot of interaction that should be going on. Not just the creativity part but the interaction with development teams. So stay with the project from the beginning up until the end, and that will help the project succeed and your work being done.
Ron: Being sort of technology agnostic it reminds of one of the biggest mistakes that I was involved with. I was working for an enterprise software manufacturer in the late '90s and they felt kind of burned by Microsoft's ever-changing data access strategy.

[laughter]
Man 2: Evolving, right.
Ron: Yeah, it was evolving. So when they brought me in they gave me a team and they said "We want you to create a data access layer that will isolate us from Microsoft. So that if Microsoft changes their mind again it won't matter, because we are going to use your layer and the rest of the products will use this layer, and we won't ever have to change this again."

So we went through and learned the intricate details of OLE-DB, which is really painful.

[laughter]

And wrote a layer in C++ that was very high performance; very, very good. And we finished it in late '98. About a year later Microsoft said: We are going to do.NET.

[laughter]

So all the investment that we put into that layer, in reality never ever was used. It didn't pay off. So it occurred to me that if we are thinking that someday we are going to completely abstract away the platform infrastructure by writing these isolation layers, that kind of bet is dangerous, it might never pay off.

So, that was just my tale there.

Somebody who has worked on technologies from Microsoft, you always want to take the technology forward. But you struggle with the impact of moving it forward because there is always an impact on all of the things that depend on it.

So you think hard about backwards-compatibility and trying to ensure that the behavior is exactly the same as it was in a previous release. And then you find, in many cases, that you can't move the technology where it needs to go without breaking some eggs, you know, along the way.

So there is always this tension between moving things forward to a better place and breaking the applications that rely on it.

And, you know, even you mentioned Enterprise Library. And I thought a lot about that because I was involved with the initial version of Enterprise Library, and so then you watch it evolve. And knowing the guys on the team and knowing how hard they worked to try to think, "Well, is there some way we can move this forward without breaking it?" You know, it is a very, very tough balancing act.

I agree with Udi that dealing with change is part of what we have got to do as an architect. And maybe part of the challenge is to just resist the urge to always change.

I can't believe I am saying that. But we are always hearing Microsoft going like: "Hey!.NET three is great! Everybody should move to.NET 3!" And from time to time I will run into a person who will say, "Well, I have my app on.NET 1.1 and it is working just fine. Maybe I could just leave it there." [laughs] And we say, "Well, you are obviously very slow and behind the times."

But maybe that is a smart thing for an architect to say. You know, in some cases maybe we ought to resist the urge to push things forward for the sake of doing so.

[music]

Oh, I can't believe I said that. I mean, OK, you should move to.NET 3.0 [laughs]. Just kidding. No, do the right thing. Do the right thing for your project. You know, we want you to do the right thing. Look, if.NET 3.0 isn't compelling enough for you to move to it, don't move.

I think eventually people will move on to new things when they find value in them. And if they don't find value then we are not really doing our job of making a compelling product, are we? And that should be a lesson for us as well.

Hey, I hope you enjoyed that.

Please forgive all of the rattling and bumping of microphones getting moved around, I apologize. We had more panelists than microphones, but we did the best we could. I almost didn't use this, but then I listened to it and I thought "Man, there are some really good parts in there, and if I just kind of slice it up I think I can make it work."

I hope you enjoyed it, and we will see you next time on ARCast.
Announcer: ARCast Radio is a production of the Microsoft Architecture Strategy Team. www.arcast.net.