(Part 1 of 3) - Intro to CHEF and Azure with Steven Murawski and Blain Barton - Florida…
Play Exploratory IoT Analysis with R
Description
[00:00] - Introduction
[04:41] - Create Virtual Machine
[07:19] - Install R Studio
[09:14] - Create New Script
[10:11] - Install Packages
[11:12] - Connect to SQL Azure
[16:08] - Statistical Summaries
[19:20] - Variance & Standard Deviation
[21:00] - Filters & Dot Plots
[22:32] - Box Plot
[22:57] - Removing Outliers
[26:18] - Time Series
[29:41] - Density Plots
[32:21] - Conclusion
Abstract:
In this video we will do an initial exploratory analysis on a water flow data set that came from a prototype that I built. The prototype consists of a water pump, a valve and a flow meter. The data set exists in SQL Azure. We will use R and R Studio to perform the analysis from an Azure virtual machine.
The Code:
install.packages("RODBC")
require("RODBC")
d <- odbcDriverConnect(connection = "Driver={SQL Server Native Client 11.0};Server=tcp:qvr9aar3i6.database.windows.net,1433;Database=TelemetryDB;Uid=drcrook@qvr9aar3i6;Pwd=David!2345;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;")
sqlColumns(d, "WaterFlows")
flows <- as.data.frame(sqlFetch(d, "WaterFlows"))
summary(flows)
flows
flows["WaterFlow"]
summary(flows["WaterFlow"])
require("RODBC")
d <- odbcDriverConnect(connection = "Driver={SQL Server Native Client 11.0};Server=tcp:qvr9aar3i6.database.windows.net,1433;Database=TelemetryDB;Uid=drcrook@qvr9aar3i6;Pwd=David!2345;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;")
sqlColumns(d, "WaterFlows")
flows <- as.data.frame(sqlFetch(d, "WaterFlows"))
summary(flows)
flows
flows["WaterFlow"]
summary(flows["WaterFlow"])
var(flows["WaterFlow"])
sd(flows$WaterFlow)
sd(flows$WaterFlow)
nrow(flows["WaterFlow"])
plot(flows["WaterFlow"])
nrow(flows[flows$WaterFlow < 9, ])
plot(flows[flows$WaterFlow < 9, ]["WaterFlow"])
plot(flows["WaterFlow"])
nrow(flows[flows$WaterFlow < 9, ])
plot(flows[flows$WaterFlow < 9, ]["WaterFlow"])
#initial boxplot showing outliers
boxplot(
x = flows$WaterFlow,
xlab = "Flow/Minute",
horizontal = TRUE
)
#calculate outlier threshold
outlierThreshold <- sd(flows$WaterFlow) * 1.5
#calculate mean
m <- mean(flows$WaterFlow)
#filter out anything under threshold
filtFlows <- flows[flows$WaterFlow > (m - outlierThreshold),]
#filter out anything over threshold
filtFlows <- filtFlows[filtFlows$WaterFlow < (m + outlierThreshold),]
summary(flows$WaterFlow)
summary(filtFlows$WaterFlow)
#new box plot, no outliers
boxplot(
x = filtFlows$WaterFlow,
xlab = "Flow/Minute",
horizontal = TRUE
)
#line graph w/outliers
plot(
y = flows$WaterFlow,
x = flows$CollectionTime,
type = "l"
)
#line graph no outliers
plot(
y = filtFlows$WaterFlow,
x = filtFlows$CollectionTime,
type = "l"
)
boxplot(
x = flows$WaterFlow,
xlab = "Flow/Minute",
horizontal = TRUE
)
#calculate outlier threshold
outlierThreshold <- sd(flows$WaterFlow) * 1.5
#calculate mean
m <- mean(flows$WaterFlow)
#filter out anything under threshold
filtFlows <- flows[flows$WaterFlow > (m - outlierThreshold),]
#filter out anything over threshold
filtFlows <- filtFlows[filtFlows$WaterFlow < (m + outlierThreshold),]
summary(flows$WaterFlow)
summary(filtFlows$WaterFlow)
#new box plot, no outliers
boxplot(
x = filtFlows$WaterFlow,
xlab = "Flow/Minute",
horizontal = TRUE
)
#line graph w/outliers
plot(
y = flows$WaterFlow,
x = flows$CollectionTime,
type = "l"
)
#line graph no outliers
plot(
y = filtFlows$WaterFlow,
x = filtFlows$CollectionTime,
type = "l"
)
t <- ts(flows$WaterFlow, frequency = 6307200)
plot(t)
plot(t)
plot(density(flows$WaterFlow))
plot(density(filtFlows$WaterFlow))
Download
Right click or Alt+Enter to download this episode
- MP3 (30.3 MB)
- Low Quality MP4 (76.4 MB)
- Mid Quality MP4 (237.4 MB)
The Discussion
-
MostafaElzoghbi Amazing IoT demo with cloud capabilities :D
-
wildgrok Wow David I just got home after attending the conference you gave on this on Carnival in Miami today (1/21/2016). Very inspirational, well presented. Looking forward for the next one.
For Diana the organizer, the SQL Diva: thanks for bringing David!
Jorge Sr. from Miami Nice
-
Diana Betancourt Thanks for the nice feedback!!!
-
wildgrok -
wildgrok Bring David again for a conference on how to make our stuffed animals
-
Rob Casey A very nice introduction to using Azure and R and very well explained all the way through what your thought process was.
More episodes in this series
Interactive Music in the Browser - An interview with JavaScript Developer Yotam Mann
Related episodes
SNR - Episode 5- Data Manipulation with R
PASS Summit 2015! Join the world's largest gathering of SQL Server and BI Professionals!
TechNet Radio: Data in the Cloud (Part 3) - Incorporating Azure analytics services
Data Science For Beginners - 5 Questions Data Science Answers
Episode 2 - R Jump Start - Introduction to Microsoft R Open
Backyard Data Science: An Inside Look at What You Can Do with Azure Machine Learning and…
Episode 1 - Show Introduction
#ImagineThis: Episode #003 - Technology Solves Problems
Creating a Database Project for Artificial Intelligence
TechNet Radio: (Part 21) Building Your Hybrid Cloud - Protecting Data with Encryption -…
