Kinect Studio and a Harmonica Player
Today we have two from Taulty, Mike Taulty, a long time Friend of the Blog (six+ years!) continues to share his Kinect for Windows v2 finds...
One is a Kinect powered Harmonica Player the other a great look at the Kinect Studio
Two recent times he's been highlighted;
Returning back to this series of posts, I wanted to continue experimenting with the Kinect for Windows V2 SDK and to try out how I would go about collecting audio data from the sensor.
I tried to come up with something “interesting” or “semi-real” to do with the audio data rather than just to draw the angle that the Kinect thinks the beam is coming from based upon its array of microphones and the best that I could come up with was to experiment by linking the Kinect up with another little hobby of mine where I’ve been trying (not very successfully) to learn to play the diatonic harmonica.
I’m still very much a beginner with the harmonica and one of the areas where I’m still struggling is with getting decent, clean, bent notes.
If you’ve not tried a harmonica then essentially the diatonic has 10 holes with each producing 2 notes (one for blow, one for suck) but a number of holes can also produce additional notes by altering the air flow over the reeds inside the device and causing a note to “bend”.
So, I figured that I’d try and put together a simple prototype app which could help me identify whether I was producing the right note from the harmonica as I’m playing it. In doing so I wanted the app to figure out;
Is there anyone standing in front of the Kinect sensor?
Are they alone? If so my hope is that the Kinect will use its automatic capability on focusing on the sound coming from them rather than from anywhere else although I could have taken a manual approach to directing the sound input – the device supports that.
Do they look like they are playing a harmonica? For this, I went with the simple idea of them having their hands near their head. That isn’t always going to work and especially not if they are e.g. playing a guitar at the same time but it works for what I need. I can’t play the harmonica on its own, never mind combined with a guitar.
What note are they playing? Can the note be displayed and can some guidance be given as to whether the note is accurate or whether it needs tweaking a bit.
and I wanted all of that to work across the user entering/leaving the scene, putting the harmonica down, unplugging the sensor and plugging it back in and so on.
Here’s a little video of where I’ve got to so far in terms of the app that I knocked together. Please excuse the harmonica notes and you might notice that the highest frequency notes don’t work – I’m still trying to see if there’s anything that I can do on that
The only other thing going on in that code is that the Pitch Tracking algorithm gives me a –50 to +50 confidence value as to how well the note has been hit (midiCents in the code).
I use this to update a couple of simple progress bars as a way of trying to indicate the accuracy with which the note is being played.
For now, that’s pretty much it – there are at least a couple of problems with what I have right now;
The upper notes aren’t registering. When I play the high C and A, I find that they are being identified by the algorithm as C6, A6 whereas I think they are C7, A7.
The algorithm is a bit “jittery” in that when playing notes the display flickers a little. I think I could do something with that from an Rx point of view by maybe windowing some of the data.
I’m not sure if the algorithm will report any of the overblow bends because I don’t know how to play those so I can’t test it out
This only attempts to deal with a C harmonica. There are many other keys available
But I had some fun playing with it and I may well revisit it. If the code’s of interest to you then it’s here for download (licensed under Ms-PL as above).
Project Information URL: http://mtaulty.com/CommunityServer/blogs/mike_taultys_blog/archive/2014/10/01/kinect-for-windows-v2-hello-audio-world-for-the-net-windows-app-developer-amp-harmonica-player.aspx
Project Source URL: http://mtaulty.com/downloads/KinectHarmonica.zip
Continuing this series of posts there’s something that I’ve been using as a tool for Kinect development which I hadn’t really mentioned on the blog here so I thought I’d bring it up and that’s “Kinect Studio”.
There’s coverage of Kinect Studio here in the Channel 9 video;
but I hadn’t really appreciated how much Kinect Studio was giving me until I’d started doing some actual development against the Kinect sensor and also until I’d carried a Kinect sensor around in my bag and started to ask the question of whether that was really necessary.
In terms of working with a sensor like this there are going be times when;
- You don’t have a sensor available. Examples;
- You might be working on a 10-person team where not everyone has a sensor.
- You might be running code in some kind of test environment where it’s not practical to be plugging in a sensor.
- You might be travelling and getting out a Kinect sensor and plugging it in on the train/plane is going to (at best) get you some very funny looks from other passengers.
- You have a sensor with you but you can’t use it. This might apply to any of the situations above but, specifically, if you are doing solo development then debugging certain scenarios can be really quite tricky. Example;
- You are doing anything which involves standing in front of the sensor. Standing in front of the sensor while debugging on your laptop can be really tricky.
Example 1 – Body Data
In this post I was attempting to write some WPF code which drew skeletal data in 3D and debugging that can be really tricky if you’re working along and your body is at your keyboard rather than half way across the room standing in front of the Kinect sensor.
Example 2 – Audio Data
In this post I was trying to do something a little more advanced in that I wanted to;
- Detect the sensor.
- Detect a single user in front of the sensor.
- Detect that the user has their hands near their face.
- Attempt to detect the pitch of the note coming from a harmonica that they are playing.
Having not come from a “Kinect for Windows V1” world, I’m not sure how much of what the Kinect Studio offers would be “expected” to a developer who’s already been doing work there although I know that there have been a bunch of improvements to the way that it works. As a newcomer with the V2 sensor and SDK I think it’s really a case of a tool that you’d have to think about building if it didn’t already exist – I tend to hit points where I realize that I need to stop getting up in front of the sensor and start recording data via the Kinect Studio if I’m going to make any process.
It’s also the mechanism via which you can record data for custom gestures, but that’s another story…