ARCast.TV - PlentyOfFish.com How one man beat the big guys

Ron Jacobs: OK, road trip. Hey, this is ARCast on the road, on the way to Vancouver, BC, where we're going to see PlentyOfFish.com. But we have to drive through and get some burgers on the way. Do you want anything? Oh, OK.

[laughter]
Ron: I'm here with Brenton.
Brenton Webster: How you doing?
Ron: Yeah. And do you want anything, Brenton?
Brenton: No, I'm good. Thanks, man.
Ron: OK. All right. We'll see you there.
Announcer: It's Thursday, August 2nd, 2007, and you're watching ARCast TV.

[music]
Ron: Hey, welcome back to ARCast TV. This is your host, Ron Jacobs. And a few months ago, back in April, I went on a road trip with Brenton Webster, who was one of my colleagues here at Microsoft. He's since moved on to kind of doing his own thing. Good luck, Brenton, with that.

But we had fun on this road trip up to Vancouver, British Columbia, which is not far from us here in Seattle. It was just a couple hour drive, so we made a little jaunt up there to visit Markus Frind, who is just a really amazing guy. He built this website called PlentyOfFish.com, and I was blown away.

This just little one-man operation that's kind of taken over the world. And his competitors are huge! They have hundreds of employees, and they're doing amazing, big stuff. And here's one guy -- just one guy -- with.NET and IIS and SQL Server, who is literally taking on the world: 30 million hits a day. Check it out. Vancouver, BC.

[audio cuts]
Ron: Hey, we're crossing the border into Canada. The road trip for ARCast continues. We'll be in Vancouver, BC very soon.

[music]
Ron: Hey, this is Ron Jacobs, and I'm here in Vancouver, BC--as we say, British Columbia. That's Canada, for those of you out there that don't know. And I'm joined today by Markus Frind. And Markus, you are the guy behind PlentyOfFish.com.
Markus Frind: I guess you could say that.
Ron: Yeah?
Markus: I'm Mr. Everybody.

[laughter]
Ron: So you're everything from the CEO to the receptionist to the janitor. The whole thing, right?
Markus: Yeah.
Ron: OK. So, PlentyOfFish.com, tell me about it. What is it?
Markus: It's a free online dating site, and it's used by millions of people. It's written in all Microsoft technologies, so.NET and Windows Server.
Ron: So, I know a lot of will people go like, "OK, big deal. I can go and create a little site." How is it working? Is it doing well?
Markus: We hit 33 and a half million page views yesterday on just the site itself, and another million or so in the forums, so it's probably one of the biggest sites in the world.
Ron: Wow. OK, now, just to compare, some of the other big dating sites that are out there, they have a lot more staff than you do.
Markus: From what I know, Match and Lavalife and a few of the others, they have around 3-400 employees each.
Ron: Yeah.
Markus: And they have hundreds of servers. I've just got one web server and a couple of database servers.
Ron: [laughs] Now, that's pretty incredible when you think about it. You built a business, over 30 million page views a day. One web server and a couple of database servers? That's pretty incredible. How'd you pull that off?
Markus: It's not as difficult as it sounds.
Ron: Yeah?
Markus: If you break down 33 million page views a day at peak, you're only looking at about 5-600 page views a second.
Ron: Yeah.
Markus: You have servers that can do hundreds of millions of instructions a second. It's not that difficult to pull off a couple hundred page views a second.
Ron: OK, OK. I like that. That's optimistic, OK. So how did you get into this?
Markus: Well, in Vancouver, in 2000, there were a couple hundred dot com companies. And then, over the course of the next three years, there was only a handful left, and I just jumped from one to the next.
Ron: Yeah.
Markus: And I only knew ASP, and I had to learn ASP.NET. And I was like, "Oh, crap. I can't read books. Every time I read books, I never remember anything I read and I forget about it." So I just said, "Hey, I can build a dating site. I can do it better than all these other ones. And I can do it for free."
Ron: [laughs]
Markus: And that's how it started.
Ron: Yeah? And then, how did you attract all this traffic to your site?
Markus: Well, it was just basically word of mouth. And it just went nuts in Canada.
Ron: Yeah.
Markus: And then it started spreading to the UK, Australia, and the United States.
Ron: So it's a worldwide site, now?
Markus: Oh yes, it's absolutely huge. I think the US is only fifth, in terms of market penetration, and we're already in the top four or five sites.
Ron: Amazing. Now, most people would say, "Oh, look, if you want to have a globally scaled web application, you've got to have multiple data centers and big contracts with content providers." You don't have any of that.
Markus: I use Akamai for the images.
Ron: OK.
Markus: Simply because of the load time from Australia. If you have eight images that need to be loaded on a page, and it takes 100 milliseconds, go eight times 100 milliseconds, you're looking at a second just for the image requests.
Ron: Right.
Markus: So you distribute the images. The ASP and so on, it's not a big hit. You can't tell the difference, really.
Ron: Oh, OK. OK. Very cool. But then you're able to do this with just you as being the sole architect and developer and tester and everything, huh?
Markus: Yeah, just me myself and I.
Ron: [laughs]
Markus: It's one web server. [laughs]
Ron: Yeah, yeah. Well, this is just fascinating to think about. Now, though, most dating sites, you have to pay a fee, get a membership. But yours is free. How does that work?
Markus: I just make money off advertising. So, although I probably slightly have more traffic than Match now, but they're doing $3-400 million off subscriptions a year.
Ron: Yeah.
Markus: So, because I'm just myself, I don't have hundreds of employees, I can make a decent living, just me myself and I, and slop up some advertising.
Ron: Now, most people wonder about this. Does it really pay that much for a few ads on the side of a page? Is it enough to build a business on?
Markus: Well, when you have 30 million page views a day, you can make money doing anything.
Ron: [laughs]
Markus: There's just so much traffic that...
Ron: Yeah.
Markus: You could sell dog food and make money.
Ron: [laughs] OK, yeah. You make a fraction of a cent on each page view, you're good to go.
Markus: Yeah. A dollar a CPM would be 30 grand a day, so you're just looking at 5, 10 cents a CPM, you still make a lot of money.
Ron: Wow. OK, all right. [laughs]
Markus: [laughs]
Ron: And of course, here, we're at the luxurious world headquarters of PlentyOfFish.com, overlooking the harbor here in Vancouver. Yeah, not a bad place that you've got here.
Markus: When I wake up at 11 or 12, I like to come out here and work a little bit.

[laughter]
Ron: Yeah, get on and check the stats, see how many hits today, right? Start counting the money. [laughs]

OK. Well, I think I'd love to hear more about how the site is built and some of the lessons you learned along the way, because scaling a site to this many page views... And I know you say it's not hard, but a lot of people are going, "That's hard for us. We have a hard time getting decent performance at a lot less." So, would be curious to know what some of the lessons were learned. So let's go inside where it's not quite so noisy, and then we'll chat some more about that.
Markus: OK.

[audio cuts]
Ron: Inside, in the lovely Dr. Evil chair here.
Markus: [laughs]
Ron: It makes me want to do this. [laughs] So Marcus, now, we talked a little bit outside about this amazing site you built and all these page hits. Now, a lot of people will say, "That must take tons of servers," but yet, you said just one. So, what have you learned?

When you began this thing, obviously, it wasn't getting you that kind of traffic as you went. So you were talking earlier to me about you made fixes over time as you learned what kind of things happen when you get this sort of traffic. So, rewind back in time, to when you were first starting. What do you recall about those days?
Markus: Necessity is the mother of all change. So, basically, when you grow quickly, but you don't grow super quickly, you have time to adapt, change, modify things, make them work better. So the most important thing, number one thing, is RAM. So, the more RAM you have, that solves all your problems.
Ron: [laughs] OK.
Markus: And then after that, you just get bigger machines. But when you're starting up and you're just starting the site, I found that it's best if you go as simply as possible. Don't use any built-in components from ASP.NET.
Ron: Oh, really?
Markus: That's what I found.
Ron: Oh.
Markus: I just wrote everything myself. Everything is super simple, nothing more complex than an if-for-while loop. No built-in controls, nothing. The hardest thing is database access.
Ron: Oh, OK.
Markus: And as long as you keep database access fast, then you have no issues.
Ron: Well, first of all, how do you kind of monitor these things? You're watching for response times and that sort of thing? What'd you do to see how it was working?
Markus: Well, I was using the--what's the thing called where you launch the CPU and page requests per second? There was that, but a much easier way was just using the task monitor and seeing the network bandwidth used. And if it's a flatline, you're good; if it goes like this, you're screwed. [laughs]
Ron: [laughing] Oh, OK.
Markus: So that was my one second check. And basically, if I started getting spikes like this, I'd go investigate to see what was happening, and I'd try and optimize it. And it's usually blocking in the database. It's all database related. I've never had issues with ASP.NET.
Ron: Oh, OK.
Markus: Well, actually, I did, in the first version of ASP.NET. It was just concurrency issues. When there was too many people online at the same time, it would just slow down and cause a lot of problems. But ASP.NET version two fixed all that, and then it went away.
Ron: Oh, OK. So you don't have to do all this magical tuning and config files with super secret numbers or anything like that, just pretty much, straight up, just works.
Markus: Straight database optimization is all it is.
Ron: Yeah, yeah.
Markus: I had built a lot of applications, even outside of this, and rarely, if ever, is your presentation layer a problem. If you do stupid things, like call the database 20 times on a page view, then you're going to be screwed no matter what you do.
Ron: Yeah.
Markus: The ASP.NET stuff is really trivial. I mean, what are you going to do? You're going to get information from the database and display it. Where can you screw that up?
Ron: [laughs] I don't know. I've seen some people do a very good job on that.
Markus: [laughs]
Ron: You were talking about making the database access optimal. And a lot of people I've talked to who run very large-scale websites say the same thing, that it's all about the data, getting the access to the database right. So let's go to that layer. What kind of lessons did you learn there?
Markus: Just separate the reads from the writes. It's basically just common sense stuff. Try and make a read-only database if you can. De-normalize your data, so if you need to fetch stuff from like 20 different tables, try and create one table that's just used for reading. But basically, just common sense stuff. Keep it really, really, really simple.
Ron: Yeah?
Markus: And there's no magic bullet. It's just a whole bunch of little things. And oftentimes, you'll find that you add a new feature or functionality and it screws up everything else, and you've got to go back and fix it. You've just got to play around and see what works. One day, it'll work. When your database doubles in size, it's not going to work anymore.
Ron: Yeah.
Markus: Because things change.
Ron: Well, let's talk about some of those. Now, you said that you have two databases on your site now, right?
Markus: Up to three now.
Ron: Up to three? OK. So, one of them is read-only?
Markus: The one is the main database, and then the other two databases are just for searches.
Ron: Oh, OK.
Markus: Because the radius-based searches take up a lot of resources.
Ron: OK. So you kind of replicate from the main one to the search ones...
Markus: Yeah.
Ron: And support searches on that. OK. And then, when searches are made on your site, then you route the search to either one. They're basically both doing the exact same thing?
Markus: Yeah. There are slight differences in-between them. But basically, if I want to do searches on the search results page, I go to one server. If I want a little bar of images across everyone's profiles, which is also a radius-based search, I go to another one. And I load balance it between the two.
Ron: Oh, OK. OK. Now, about optimizing the reads versus writes. I know this seems really basic to you, but a lot of people are just learning this stuff. What would you say that's important?
Markus: Because if you don't have a lot of RAM, and you do reads and writes at the same time, you might get the swap file involved and then the whole system just hangs for a few seconds.

And, if you need a read, or a write to complete before you can do a read, otherwise you get all these kinds of issues. It's basically, if you are doing only one thing in the system, it'll work really, really well.

If you have one system, and you are just doing writes, you're good. If you have one system and you are just doing reads, you're good. But as soon as you start mixing the two, you've got a lot of problems -- like locking, and blocking and all these kinds of issues.
Ron: Oh, OK. And so, RAM, when you were talking about adding about lots of RAM, it's not just the web server that you were talking about, it's the database server as well.
Markus: Oh the database, definitely. If you can hold the whole database into RAM, do it.
Ron: Ah! OK.
Markus: That's... your first bottleneck, and your only major bottleneck will always be RAM. If you're maxed on CPU, you are doing something wrong. You've just got to really, really optimize it.
Ron: Yeah, yeah. OK. So, all right. So, as you've grown you kind of learned these lessons. One of the things that somebody who looks at this might go like, well, you don't have any product managers telling you what your marketplace wants, you don't have any architects designing this stuff... Do you sit around the day and think: "What new features should I add to my site today?" Or,how do you come up with ideas for how to improve your site?
Markus: Well, I usually I just, come up with one and I'll write it up within 24 hours, throw it up, patchwork most of the time, and then wait and see what the user response is. If it's a great, well then improve it a bit more, if it sucks, well, then you take it down.
Ron: Now, by user response you don't mean like people sending you mails saying like: "Oh I love that new thing you added!" You're looking at the statistics of the way the site is working.
Markus: Yeah, I see what the users are actually doing on the site. I can add anything, I'm going to get a whole bunch of users that whine, you're going to have a lot bunch of users that love it.
Ron: Ah!
Markus: You can't listen to user responses. That's only a fraction of a tiny minority that's going to complain about something. Doesn't matter what I do, someone will complain.
Ron: Well especially with 30 million page views. You couldn't possibly hear from enough people to get a representative sampling.
Markus: Exactly, so I just look, you know, do the messages per user increase, do the session time increase? That kind of stuff.
Ron: Ah, OK. So, you're just very agile about this, get an idea, try it out, does it help?, Not? Yeah, I can take it out. OK. And now, when you think about another big requirement of a site like this, of course is just, strong availability numbers. So you don't have a big test team doing load testing on this stuff or anything like that. Or, do you have site outages? What do you do?
Markus: If MySpace went down for a couple of hours, I mean, it doesn't hurt, though, it's not going to hurt mine. I mean, in my site it does never go down, if it goes down it's like a minute or so.

But, I've never had any issues, never really. The biggest issues have been DNS-related worries. I have had some weird ISP, something that says that "your site isn't available today." So, that's the biggest issue. But people, it's a free site, people expect downtime, even if there isn't any, it is still expected, and you look at MySpace, I mean it goes down for hours...
Ron: Yeah!
Markus: And no one cares. If you are selling something, and, you know, the people are paying a bunch of money every month, then... then you need high availability numbers, because each minute of downtime cost you a lot of money.
Ron: Right, right. OK. So, basically you...
Markus: As a free site, it is not that important.
Ron: You created the business model where you can make money but lower the expectations!

[laughter]
Markus: OK. But it works, it works. Like you said, even if it goes down, users are going to think it's their own problem. It's like: Oh it's my ISP, or my service's down, or something. It's a common response. People may even not know that the site is down.
Ron: Ah! OK. All right. Wow! Well, so... it's growing nicely, it's scaling out, you've learned a lot, where do you go from here?
Markus: Well, you know, scaling from 30 million to 60 million pages is not that difficult, because it's just doubling in size. So, as you get, you know, going from a million to 12 million pages, that's huge, a day. But now, I see everything once out, I could scale the 60 million pages with no problem, maybe a second web server, do round-robin DNS, and other than that, no major issues coming out, everything is sort of you can see stuff sort of miles ahead now.
Ron: Yeah! And, you know, but you don't have any kind of business analyst, you know, you are not looking out at analyzing the market or anything like that. Basically, people like to go out on dates, they like to meet each other, they're lonely, there is a growing number, there's good demographic there, right?

[laughter]
Ron: I guess that behavior is not that difficult to anticipate, right?
Markus: I don't need to hire a bunch of people to tell me common sense.

[laughter]
Markus: It's blindingly obvious when you look at the site what's going on. Or, if that isn't obvious, if you just go to other competitors and see what they're doing.

I mean, chances are that these couple of a hundred employees that they've hired, they must be doing something right. So, if they're doing something right, I just go at it and look and "Oh, it looks like a good feature, and maybe I should have that!"

I mean, the industry isn't terribly innovative. I mean, you are hooking up people, how innovative can you get?

[laughter]
Ron: Wow! OK, well, you know, this is pretty simple, pretty basic, but it's working for you, and congratulations on making a great business, just by yourself! It's terrific! Well, thanks so much, Marcus!
Markus: Well, thank you for having me.

[music]
Ron: Shock that one up to "I should have done that." You know, I can't believe, 30 million hits a day. Well, let just say he's making some money. And it doesn't sound like it was that hard, I mean, he makes it sound like, "Hey this is trivial" But running a website like that I know it's not that easy. I mean Marcus is a smart guy. It's easy for him but, hey, you could do it too. And ASP.NET has everything you need to make this possible.

And on top of that we're adding a bunch of very cool new stuff, I just wet to some training on SilverLight and WPF and XBAP applications. I'm telling you, the future in this stuff is wide, wide open, so stick around, keep an eye up for it, and maybe you could build the next PlentyOfFish.com. We'll see you next time on ARCast TV.
Announcer: ARCast TV is a production of the Microsoft Architecture Strategy Team, www.arcast.tv.