Help Wanted: Big Data and the Talent Gap
- Posted: Jun 18, 2012 at 1:07PM
There is a lot of energy and discussion these days around "big data," and most of it seems to focus on the underlying technology enablers and what you can do with them ... real-time analytics, noSQL stores, map|reduce capability, complex event processing, and a host of other technologies, architectures, and approaches to deal with what Bret Swanson described in his seminal 2007 Wall Street Journal piece, "The Coming Exaflood." The megatrends driving this are pretty well-documented at this point ... cheap devices on the front end means you can capture and collect staggering volumes of data, while cheap storage on the back end means you can save all of it, opening the door to the slicing & dicing of massive datasets for everything from simple insights to machine learning to fielding a better baseball team. All goodness. But another pivot on this that's worth considering is the emerging talent gap that's opened up, and poised to only get wider as the pool of qualified math geeks, statistics quants, and analytically-skilled college grads fails to keep pace with the petabytes and zettabytes of data filling up the world's datacenters as we speak.
I may actually be underselling the amount of attention being put on this problem, but that attention has only shown up recently, and now it seems to be everywhere. The talent situation as it relates to data is getting a lot of punditry now, with lots of smart people trying to figure out the point in time at which the talent supply will catch up with demand and reach equilibrium, as the future career opportunity becomes more apparent to aspiring data quants in their formative years, and as universities load up on courses offered to prepare them for those careers. We've seen this movie before ... COBOL programmers were in big demand long before universities even had a thing called "the computer science department", and during the dot-com boom in the late 90's, even a flimsy grasp of basic HTML was enough to get you contract work churning out websites.
But we should probably talk about the role of software, too . The answer to people shortages in the face of economic growth opportunity has always been automation, which was probably the single most impactful thing that fueled the industrial revolution. So how do you automate the statistician's proverbial "ah-ha" moment while wallowing in a large dataset? This is where the McKinsey study hits the nail on the head in its May 2011 big data report, which calls out one of the key ways big data delivers value as the ability to "use automated algorithms to replace and support human decision making." This is, of course, easier said than done, but there are examples if this all around us. Bing Travel is one of the best examples I know of ... originally launched as Farecast, the service does massive scale analytics on some 200+ billion pieces of historical airfare data to build predictive algorithms for future airfare price movements. No one is suggesting you hire a Harvard mathematician to help you decide whether or not to buy a plane ticket, but the leap from a consumer scenario like travel to business scenarios like drug discovery or natural resource exploration is an easy one to make ... it's all about data-driven decision-making on a course of action with an expected outcome, and given the state of the analytical talent pool now and in the future, it will be increasingly software-driven.
As for the people who *are* in the talent pool, there are some math geeks out there who are going to get rich. Both my friend & self-described data geek Peter O'Kelly, and my colleague Dave Campbell have been telling me for nearly 10 years now, "Mark my words ... the data guys are going to win in the end." They may have been onto something. Meanwhile, my kids are teenagers, and they're getting all this heat from school counselors to make all sorts of monumental career choices at an age where you can barely think beyond next week. I just tell them there's this thing called 'supply & demand', and if they know math and statistics, they'll never have to worry about a job, ever.