Microsoft embraces the open source elephant

http://www.wired.com/wiredenterprise/2011/10/microsoft-and-hadoop/

I recall bitching about 3 years ago when I was mostly a Windows developer that there is no equivalent of Hadoop that works with Microsoft technology. Hadoop is truly a MASSIVELY important project, maybe some of you don't realise it because you don't know what you are missing if you only live in Microsoft la la land. Almost everywhere where you need to crunch boadloads of data (think terabytes or pentabytes, not that uncommon if you are a bank or government agency or pretty much anyone with a lot of customers), you are probably running analytics on a Hadoop cluster (Bing is no exception, apparently).

Well it looks like Microsoft is committing to bring Hadoop to Windows. About f**king time really. Who knows, they might even port it to .NET/WinRT, or at least WinRT compatible execution contexts.

Anyway I think this is a pretty big deal, up there with Microsoft embracing JavaScript.

13 minutes ago, Bass wrote

... maybe some of you don't realise it because you don't know what you are missing if you only live in Microsoft la la land...

well cool, this means i can still stay in Microsoft la la land. Kudos for MS.

What exactly does "porting it to run on Windows" mean, really? It already runs on Windows, you just need a bash shell (like Cygwin) to run the scripts that you need to interact with it. So all it takes is rewriting those scripts in PowerShell or whatever, and you're good.

Of course, Hadoop isn't optimized to run on Windows, and from the description it sounds like they want to integrate it with SQL Azure etc. which would be considerably more work, of course.

Then again, Hadoop isn't really optimized to run on anything. It has astronomical overhead for certain scenarios, because a lot of the implementation just isn't very efficient. That's why there's guys like MapR who are completely replacing major pieces like HDFS with their own stuff just to get around the inherent inefficiency of the thing.

Once again the ugly head of the Google PR machine pops up.

Microsoft doesn't use Hadoop for anything (at least anything outside of curiosity), but has some pretty big databases like the Bing and hotmail backends. In fact, for every massive database that Google has running on custom hardware/software, there's a similarly sized database at Microsoft running on normal Microsoft programs.

In fact, that's one of the big selling points of SQL server. It pretty much doesn't matter who you are, you probably don't care about uptime/security/data integrity as much as hotmail does, and you probably don't have as much data/updates/searching as Bing does, so if SQL can do those, it can do your applications too.

@evildictaitor: So you imply that Hotmail runs on SQL? This isn't my field, but from what I understand that would be near impossible/impractical. Did you miss the NoSQL buzzword?

@fanbaby: No, it looks like it really does run on SQL Server ...

Herbie

1 minute ago, Dr Herbie wrote

@fanbaby: No, it looks like it really does run on SQL Server ...

Herbie

Which proves, if nothing else, that Microsoft does actually eat its own dog food.

And why are folk always surprised that MS is involved in free software? It's in their best interests to be, surely.

Didn't Microsoft come up with the protocol that enables Ajax?

@Dr Herbie: As I said, it's not my field, but this is quite shocking! It would be interesting to see a comparison between hotmail and gmail (or yahoo, i assume they too use nosql).

52 minutes ago, Ray7 wrote

*snip*

 

Which proves, if nothing else, that Microsoft does actually eat its own dog food.

And why are folk always surprised that MS is involved in free software? It's in their best interests to be, surely.

Didn't Microsoft come up with the protocol that enables Ajax?

Well, I wouldn't call it a "protocol". They came up with two key technologies for IE, the XmlHttpRequest object and iframe element, which are essential components of Ajax and cross-site scripting. These technologies were originally designed to make Outlook Web Access possible, but they've since become web standards and now power hugely-popular web services such as Google Maps, Gmail, and Google Docs.

2 hours ago, fanbaby wrote

@Dr Herbie: As I said, it's not my field, but this is quite shocking! It would be interesting to see a comparison between hotmail and gmail (or yahoo, i assume they too use nosql).

The fact they do use it doesn't necessarily mean getting it to work effectively isn't difficult or potentially expensive in terms of hardware requirements. Sometimes you have to take things pretty much to the limit just to find out what those limits are, and SQL Server scales to some pretty amazing sizes.

Much the same was true of Kerberos, which it was assumed could scale pretty much infinitely by the designers. After implementing Active Directory internally at Microsoft (which has one of the largest internal IT infrastructures in the world), there were comments that although it still worked when you got near that size it begins to 'creak' in places. Which was pretty much also the focus for a much larger investment behind PKI technologies at Microsoft.

34 minutes ago, Geoffreyk wrote

*snip*

Sorry for being picky, but this is not correct. Ajax has nothing to do with cross-site scripting.

Also, iframes have very little, approaching nothing, to do with ajax. There was a time, way back in web history that iframes were all we had, so we used them to do asynchronous calls back to the server. But there is no need to use them today, and to lump them in with ajax makes no sense. The XmlHttpRequest object was created specifically for the purpose that we now call 'ajax'. Iframes were never intended for that purpose.

And to only call out 3 Google products as examples of ajax seems a bit one sided. Practically every "hugely-popular web service" has some sort of ajax somewhere.

Did you notice that I used the word "AND" between the terms "Ajax" and "cross-site scripting"? They are technically different things, yes, and you can have Ajax without the other. However, the real power of Ajax is in the ability to create mash-ups using multiple web services across domains, and being able to not only GET the data from those services, but to also POST across domains to those services. That, IMO, is the quintessential Web 2.0., and there's no way to achieve that without iframes.

And I mentioned those three Google services specifically, because Google gets practically all the credit for "inventing" Ajax. BTW, you might also notice that practically every "hughly-popular web service" with "some sort of Ajax somewhere" also uses iframes, so don't be so quick to discount the importance of it.

It's a niche tool for a niche market.  It's only been very recently that terrabyte and higher databases have been affordable to lots of entities and at that, it's taken a few years for those entities to start purchasing hardware of that size.  It's only now that it makes sense to get into this market. 

10 minutes ago, Geoffreyk wrote

*snip*

No, the real power of ajax is to refresh the content of a web page without having to do a complete server round trip and complete page redraw.

Going across domains is not ajax. That would be a mashup. You are confusing the two.

Go back and re-read my posts. Slowly this time. I did not once confuse Ajax and cross-site scripting.

Web 2.0 != ajax. It would be correct to say that web 2.0 is a use of ajax.

No sh1t, Sherlock. I said:

However, the real power of Ajax is in the ability to create mash-ups using multiple web services across domains, and being able to not only GET the data from those services, but to also POST across domains to those services. That, IMO, is the quintessential Web 2.0.

Translation: Web 2.0 is the goal. Ajax is a method to achieve it.

Using Ajax just to eliminate full-page re-rendering is yesterday's news. Being able combine experiences from multiple web services into something completely seamless is the next step, and just getting data from the different web services is yesterday's news too. Being able to POST and share data across domains is what makes Web 2.0 special.

Google get credit for being the first, large, successful use of ajax. Not for "inventing" it. I have never, ever, heard anyone say that.

The use of iframes != ajax. Even in the modern web context, iframes are used mainly for advertising. They are only used to POST data when that data can not be transmitted in a GET. And this technique has only really caught on because the iframe itself has been abstracted away in libraries like jquery.

Dude, you're so set on trying to find some technical inaccuracies in what I posted, that you're seeing claims that I never made. My point from the start is that Microsoft came up with two technologies, XmlHttpRequest and iframe, which are essential for Ajax and cross-site scripting.

Translation: If you want to have both Ajax AND cross-site scripting, both XmlHttpRequest and iframe are essential.

How do you get "Ajax = cross-site scripting" from that?

Beyond that, you're just getting into a pissing match about the importance of the iframe.

So what if it's been abstracted away? So has XmlHttpRequest by libraries like jQuery. What's your point?

Sure iframes are used considerably for ads, but that's not the only and most important use case. Take a look at any site that uses DISQUS for a commenting system. DISQUS uses a toolkit called easyXDM that enables cross-site posting of data. You enter your comments from a site like Engadget, the comments get posted on DISQUS' web service, and the posted comments render on the originating site. easyXDM uses iframes and cross-origin resource sharing (CORS) to achieve cross-domain posting. Does it seem like somewhat of a hack? Damn straight, it does. So what? If a service like DISQUS depends on it, it's good enough for me.

@PaoloM:

Did I hit a nerve PalaoM? It's not meant to be insulting.

If you live and work with Microsoft products exclusively, it's fair to say probably wouldn't encounter Hadoop. Considering it's not a Microsoft product, and it (mostly) runs on Linux and Java based environments, how the hell would someone who only uses Windows and .NET encounter it exactly?

Since it's going to be integrated into Azure and SQL Server, that may no longer be true. Therefore, this is news for people who live and work with Microsoft products exclusively. This is exciting news to me. It could be exciting news for people who don't run or don't want to run Linux and Java, because Hadoop will start to become available and better supported for people who live and work with Microsoft products exclusively.

@Sven Groot:

Cygwin is a piss * way to run software on Windows. I'm assuming Microsoft plans to port it _natively_ to Windows, as well as integrate it better with SQL Server and Windows in general.

Getting Hadoop to run at optimum performance and reliability has spawned an entire army of consultants. Obviously there is a lot of people doing work on this.