Analyzing "big data" with LINQ to HPC
"Big data" refers to unstructured data sets so large that they cannot be analyzed using traditional database tools. Today, big data are becoming more common; it is prevalent not just in Web traffic, but also in industries like oil & gas, finance and manufacturing. Based on Microsoft Research’s Dryad project, LINQ to HPC is a programming model and distributed runtime for building analysis solutions for big data. It goes beyond MapReduce and leverages the LINQ programming model and HPC scheduler to execute optimized query graphs across a cluster of machines. In this session, you will learn how to use LINQ to HPC on both Windows Azure and an on-premise Windows cluster to build analytic apps that deal with big data. These apps will be able to scale out to hundreds of machines without having to deal with scheduling, data replication and node failure complexities generally associated with programming a large, distributed data-parallel system.