Coffeehouse Thread

12 posts

Forum Read Only

This forum has been made read only by the site admins. No new threads or comments can be added.

Fun multitouch audio app

Back to Forum: Coffeehouse
  • User profile image
    dentaku

    Some goood ideas for Windows Phone 7 audio app interfaces.

    I think that video uses Usine
    http://www.youtube.com/watch?v=K-foi4bm4-I
    http://www.sensomusic.com/usine/
    Of course it's better on a tablet which is bigger so some of this stuff just wouldn't be useful on a tiny phone.

    I'm more interested in MID controller apps that send MIDI or OSC to the computer and have the more powerful machine actually create the sound but it's good to see how people are thinking about what touchscreen interfaces should look like for audio manipulation.

  • User profile image
    itsnotabug

    i think latency is going to be a big problem if you want to do any real-time audio apps, but it could work for offline beatbox/pattern arrangement type apps. android also suffers from this... it's the nature of garbage collected managed code. this is one area where iphone has a considerable lead... the next iOS is actually including native midi i/o support!

    i'm working on a quirky pattern-based sequencer type app now that uses nAudio library. i'd love to get it to work on wp7, but not sure if it will.

  • User profile image
    dentaku

    Yup, latency is definitely a problem when it comes to phones but Windows Phone 7 does seem to be pretty good for this kind of stuff. Like I said, using the phone as a MID controller that just sends data to a computer which handles all the audio is also very inetersting.

     Ian Walker has been working on sequencing apps. http://rd3d2.wordpress.com/ if you haven't seen them yet.

    , itsnotabug wrote

    i think latency is going to be a big problem if you want to do any real-time audio apps, but it could work for offline beatbox/pattern arrangement type apps. android also suffers from this... it's the nature of garbage collected managed code. this is one area where iphone has a considerable lead... the next iOS is actually including native midi i/o support!

    i'm working on a quirky pattern-based sequencer type app now that uses nAudio library. i'd love to get it to work on wp7, but not sure if it will.

  • User profile image
    BitFlipper

    I think part of the problem is the WP7 API. Last time I checked, the audio buffer size could only be set to a minimum of 100ms. That is way too much latency for "realtime" interactive audio apps. They have to fix this asap.

    I don't think this is an issue with managed vs native. Games are usually much more CPU intensive and they are pretty responsive. The only time managed vs native comes into play is when you need to do custom DSP that is too slow with managed code.

  • User profile image
    dentaku

    Would XNA apps perform better than regular Silverlight ones for this kind of stuff?

    You said "the audio buffer size could only be set to a minimum of 100ms". In professional applications on a computer I try and set my ASIO soundcard's buffer size to 256 samples which translates to Input Latency = 7.21ms, Output Latency 7.23ms.

    This is why I would love to see how this XNA app performs outside the emulator and ona real phone.

  • User profile image
    BitFlipper

    , dentaku wrote

    Would XNA apps perform better than regular Silverlight ones for this kind of stuff?

    You said "the audio buffer size could only be set to a minimum of 100ms". In professional applications on a computer I try and set my ASIO soundcard's buffer size to 256 samples which translates to Input Latency = 7.21ms, Output Latency 7.23ms.

    It has been a while since I played with the audio stuff in the emulator, but IIRC, you had to add a reference to the audio API that is actually part of XNA, so basically you are already using XNA in this case, even if your main app is SL.

    The other thing is that the latency I'm talking about was for input. I didn't play with audio output so hopefully that is different in that you can specify shorter buffers than 100ms (btw, who specifies audio buffers in ms? It should really be in samples). My app would take the audio input, calculate the instantaneous pitch, and display it on a scrolling graph overlaid on a musical scale in realtime. A singer can for instance use this to do more accurate vocal exercises, etc. The DSP performance of the app was more than acceptable, the part that sucked was the latency. How much of this was due to the emulator itself was unclear, but even if it was "as low as" 100ms on hardware (the minimum buffer size), it would still be too laggy.

    EDIT: One more thing... The way that the audio buffers in WP7 work is bizarre too. First of all, you cannot specify samplerate. Samplerate is fixed (at least it was for input in the emulator for me). So then you specify your "buffer size". Let's say you specifiy 100ms. If the actual samplerate is, say, 22KHz, then that is 2,200 samples for one buffer. So then for each buffer you get a notification that it is ready. From the time that the notification is sent until you retrieve the buffer, the actual number of samples would grow. So you might retrieve the buffer and then see that there are more than 2,200 samples. You have to loop and keep getting a new buffer until there are no more samples, all the while at the lower level, samples are added as they show up. Weird.

    The way it should work is that you specify a buffer size (in samples, please!), and when you get a notification that a buffer is ready, it should not matter exactly when you retrieve the buffer, as long as it is before the next notification (at which point you lost that last buffer and only get the next one). And when you retrieve it, it should be exactly the number of samples you specified.

    Because of this I didn't have a lot of faith in the people working on the audio APIs of WP7 to get this stuff right. I feel such bizarre audio implementations are just naive.

  • User profile image
    dentaku

    , BitFlipper wrote

    Because of this I didn't have a lot of faith in the people working on the audio APIs of WP7 to get this stuff right. I feel such bizarre audio implementations are just naive.

    @BitFlipper:Yup, audio stuff is very important for phones. It's not a niche market and people take their iOS audio apps very seriously for example. It's gotten to the point where people make music with completely virtual instruments on their computers (I've never used the inputs on my soundcard because I only use VSTs etc.) so they're not averse to using software on their mobile devices as more than just toys.

  • User profile image
    itsnotabug

    , BitFlipper wrote

    I don't think this is an issue with managed vs native. Games are usually much more CPU intensive and they are pretty responsive. The only time managed vs native comes into play is when you need to do custom DSP that is too slow with managed code.

    that's just it... almost anything interesting you'd want to do is "custom dsp". xna can do mixing, volume, panning, maybe a couple different types of filters, but i couldn't find any help with building a virtual synth, spectral shaping, custom effects or even an event model that supports midi. midi is 25+ years old! the "close to the metal" game loop and dsp in xna supports gaming and gaming only.

    since native code is not supported on wp7, i'm afraid we'll never see the type of compelling creative apps on it like the iphone. on my iphone i have nanostudio, sunvox, and a 4-track recorder... arguably more power than the beetles had, in my pocket.

  • User profile image
    dentaku

    , itsnotabug wrote

    * or even an event model that supports midi. midi is 25+ years old! the "close to the metal" game loop and dsp in xna supports gaming and gaming only.

    Being able to simply send MIDI data out of a phone wirelesly without having to completely "invent" the feature yourself would be amazing. The phone would send MIDI data to a tiny app on the computer that would then output it to a virtual MIDI port (like the great free MIDI-Yoke) and now any music software could use the phone as a controller. People do it all the time but you need to mess with OSC and all kinds of custom software, most of which are pretty much iOS and OSX only.

  • User profile image
    itsnotabug

    well you can send any message your want out of wp7 (assuming it can do at least udp over wifi), then your receiving app would do the heavy lifting, translating that to a midi stream. importing winmm.dll in your receiving app gives you access to multimedia functions including midi. you can enumerate your midi devices and send the midi stream out to your virtual midi port. so you'd still have to model it your own way on the device, but translate it on the workstation. of course all these layers of abstraction adds to latency on top of whatever latency your host instrument already has.

  • User profile image
    exoteric
  • User profile image
    dentaku

    @exoteric:Nice, and it's not an iPad app Smiley
    It actually runs on regular Windows machines.

    It's very similar to TouchAble and TouchOSC

Conversation locked

This conversation has been locked by the site admins. No new comments can be made.