How to expedite container startup with Project Teleport and Azure Container Registry
Build with an Azure free account. Get USD200 credit for 30 days and 12 months of free services.
Start free todayHow to prepare data using wrangling data flows in Azure Data Factory
Play How to prepare data using wrangling data flows in Azure Data Factory
Description
Gaurav Malhotra joins Scott Hanselman to show how wrangling data flows in Azure Data Factory empower you with a code-free, serverless environment that simplifies data preparation in the cloud and scales to any data size with no infrastructure management required. It uses the industry-leading Power Query data preparation technology (also used in Power Platform dataflows, Excel, and Power BI) to prepare and shape the data. Built to handle all the complexities and scale challenges of big data integration, wrangling data flows enable use Apache Spark execution to help you easily prepare data at scale.
[0:02:00] - Demo
- Wrangling data flows docs
- Power Query
- Azure Data Factory overview
- Azure Data Factory docs
- Create a free account (Azure)
Download
The Discussion
-
Sad The "Done" button after creating Merge Queries does nothing. Upon debugging receiving Sink dataset filepaths cannot contain a file name. Please remove the file name from 'WranglingFile1'. Pretty much in preview stage as it says :)
-
Andre Why is it called "wrangling data flow"? To wrangle means to quarrel or collect.
-
parask @Sad:
The "Done" button is just an indicator for UX to know that user is done with doing the transformation. I agree that this is confusing and we are working on improving the experience.
Regarding the sink dataset, as the error message says, the sink dataset can only contain the folder (and no file). The idea is once the transformations are run on source dataset, the ADF will create the output files in the folder specified in the sink dataset. Please remove the file name from sink dataset. -
Mr TA This is poor design. Now I have to create 2 datasets for each blob: one without a filename for the sinks, and one with a filename for everything else. What were you thinking when you did this, Microsoft?
-
Siva There is no support for on Prem datasources such as Oracle warehouse,SQL Server ?
I see bunch of cloud sources only
More episodes in this series
How to keep your apps healthy with Genie and Navigator in App Service Diagnostics
Related episodes
How to monitor your Azure Data Factory pipelines visually using Gantt views
How to execute Azure Machine Learning service pipelines in Azure Data Factory
Create dependent pipelines in your Azure Data Factory
Build ETL pipelines collaboratively using Git integration in Azure Data Factory
Code-free modern data warehouse using Azure SQL DW and Data Factory
Rerun activities inside your Azure Data Factory pipelines
Quickly build data integration pipelines using templates in Azure Data Factory
Code-free data transformation at scale using Azure Data Factory
Monitor your Azure Data Factory pipelines proactively with alerts
Run Azure Functions from Azure Data Factory pipelines
