Introducing U-SQL - A new language for Massive Data processing

Sign in to queue

The Discussion

  • User profile image
    pserranne

    Good session to get a quick overview about U-SQL. Glad to find out a whole series of these has been made available on channel9. Keep them coming... :D

  • User profile image
    sokhaty

    Nice intro! A few questions:

    * how soon are you planning to add support for aggregate/windowing UDFs? On-line docs talk only about row-wise UDOs - user defined operators. Since choice of windowing functions is currently limited, you should at least allow developers writing their own

    * are there plans to provide "local" mode for U-SQL dev and test scenarios (similar to local mode for Hadoop, dev simulators for Netezza and so on)? Perhaps you can add it as part of the Azure SDK with a local emulator.

    * what's the feedback channel for U-SQL?

  • User profile image
    M.Rys

    @pserranne: I am glad you like it! Please let us know if you want us to drill deeper into any specific aspects.

    @sokhaty: User-defined Aggregators are available now (that part of the documentation needs still to be written, thanks for reminding me :)). However they cannot be used in the context of windowing expressions. Please go to

    https://feedback.azure.com/forums/327234-azure-data-lake

    to file a request. Do you need user-defined aggregators with windowing expression? User-defined ranking or user-defined analytics functions? Some of them are on our backlog but at a lower priority right now.

    We are currently working on a local development experience inside VisualStudio (for now) that will provide local running of U-SQL, local debugging of server failures and more. However I would not call it an emulator (emulator implies some heavy weight installation requirement that we want to avoid) :).

  • User profile image
    m1nkeh

     

    oops, wrong vid

  • User profile image
    Rowland

    u-sql looks like a knock off of Pig in some ways. Does it have some part of the roots there?

  • User profile image
    M.Rys

    @Rowland: U-SQL is based on SCOPE (https://academic.research.microsoft.com/Paper/4439987.aspx). It has some similar concepts as Pig such as the global expression composition and thus the ability to optimize a script globally. I wouldn't call it a knock off of Pig though. The language is otherwise much more SQL and C# based.

Add Your 2 Cents