Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Webcam Based Laser Tracking for Human-Computer Interaction

  In this article, we will put together a program which will allow us to move the mouse cursor on our computers with a laser pointer, and even generate mouse clicks using only a webcam for computer vision. One really cool use of this would be with a projector. With a projector, you could use this application and turn its projected image into an interactive screen, and even open files/menus by just pointing at them with a laser pointer!
http://ashishrd.blogspot.com

Difficulty: Intermediate
Time Required: 1-3 hours
Cost: Free
Software: Visual C# Express Editions,
Hardware: Webcam, Laser Pointer
Download:

I will be making use of Andrew Kirillov's motion detection code for image acquisition, and the AForge.NET framework, for certain image processing tasks. I'll start off by showing you how the image processing works for this program, and then I'll walk you through the important parts of the code. So, let's begin!

Image processing and laser tracking...

The laser tracking code works by finding the brightest pixel in the webcam's field of view. It is pretty much similar to the code I had written for my previous article - Laser Tracking Camera. However, since quick image processing is necessary for this program to work in real-time, I have modified the code a bit, and now it's much faster than before. The two image processing techniques I have used for optimization are - image cropping and image resizing.

Image cropping is basically just trimming off the edges of an image to keep unwanted things outside the camera's view. Image cropping also reduces the number of pixels in an image, and makes image processing fast. On the other hand, resizing images reduces the pixel resolution of an image. The pixel resolution of an image describes the resolution of an image with the set of two positive integer numbers, where the first number is the number of pixel columns (width), and the second is the number of pixel rows (height). (Examples: 320x 40, 640x480, 1024x768). An image from Wikipedia illustrates:

clip_image002[4]

The picture on the left is sampled at 20x20, and the image on the right is sampled at a higher pixel resolution, 100x100. Obviously, the higher resolution image has more detail. However, in image processing, while increased resolution results in greater information and detail, it comes at the price of memory and computing speed.
For cropping the video feed from a webcam, you can click-and-drag a selection rectangle around the area you would like to keep, and the program would remove everything else.
clip_image003[4]

The selection rectangle is a semi-transparent rubber band rectangle (picture above). A rubber band rectangle (aka focus rectangle), is a reversible rectangle that tracks with the mouse pointer while you move it around. This makes it easy to select the area you want to work on. If you are working with a projector, you can easily select its projected area, and the program would map its coordinates to the coordinates on your computer screen, and remove everything else.

Here's how you can make a rubber-band rectangle between two points (say x1,y1 and x2, y2):

//Compute width and height of rectangle
int width = Math.Abs(x2 - x1);
int height = Math.Abs(y2 - y1);                
//Decide x and y of rectangle
int x = 0, y = 0;
if (x1 < x2)
    x = x1;
else if (x1 > x2)
    x = x2;
if (y1 < y2)
    y = y1;
else if (y1 > y2)
    y = y2;
//Draw the rectangle
Rectangle rect = new Rectangle(x, y, width, height);
Graphics g = Graphics.FromImage(image);
g.FillRectangle(new SolidBrush(Color.FromArgb(60, 184, 184, 0)), rect); //Draws a semi-transparent rectangle using alpha blending
g.Dispose();


You will find this code in the ProcessFrame event in MotionDetector1.cs. Here, x1 and y1 are updated in the MouseDown event of the camera window, whereas x2 and y2 are updated in the MouseMove event.
After we know how big the selected area is, we can trim the edges by using the Bitmap.Clone method:

Bitmap tmpImage0 = image.Clone(new Rectangle(x, y, width, height), 
System.Drawing.Imaging.PixelFormat.Format24bppRgb);


For resizing the video feed by a certain factor, adjust the resize factor numeric up/down control. If you take a resize factor, say 2, then the program would divide the length and width of the webcam images by 2, and the resulting images would actually be 2^2 = 4x smaller. For example, if your webcam captures images of size 320 by 240, the total number of pixels in each image would be 320 x 240 = 76800.  If we resize these images by a factor of 2, the resulting 160 by 120 size images would contain only 160 x 120 = 19200 pixels (which is 4x less than 76800).

clip_image004[4]

Here's how we can resize an image by a given factor using the Resize filter in the AForge.Imaging library:

AForge.Imaging.Filters.Resize resize = new Resize(image.Width / resizeFactor, image.Height / resizeFactor, InterpolationMethod.NearestNeighbor);

Bitmap tmpImage1 = resize.Apply(tmpImage0);

Just keep in mind that the lesser you resize your images, the more tracking resolution you'd get while tracking lasers. However, since tracking resolution comes at the cost of memory and computing speed, pick a resize factor which works best for your computer and webcam. A resize factor of 2 or 3 usually works fine for me.
After we're done with cropping and resizing, the program can go through the pixels on the image and find the brightest pixel (I'm assuming this would be the laser dot):

for (int y = 0; y < 240; y++)
{
    for (int x = 0; x < 320; x++) 
    {
        byte red, green, blue;

        red = uBitmap.GetPixel(x, y).red;
        green = uBitmap.GetPixel(x, y).green;
        blue = uBitmap.GetPixel(x, y).blue;
        
        float brightness = (299 * red + 587 * green + 114 * blue) / 1000; 

        if (brightness > certainValue)
            // Do something
    }
}


Here, uBitmap, is an instance of the UnsafeBitmap class I had discussed in my previous article.

NOTE:
If you find brightness tracking unreliable in certain lighting conditions, change the above code to perform color detection instead of brightness detection by writing something like "if ((red > 220) && (green < 170) && (blue < 170))" instead of writing "if (brightness > certainValue)".
After our program has the x and y coordinates of the laser dot, it can compute its position on the screen:

//Get screen width and height
int screenWidth = Screen.GetBounds(this).Width;
int screenHeight = Screen.GetBounds(this).Height;
float cursorX, cursorY;
//Compute location of laser dot into camera coordinates cursorX = ((float)screenWidth / imageWidth) * x; cursorY = ((float)screenHeight / imageHeight) * y;
//Set cursor position Cursor.Position = new Point((int)cursorX, (int)cursorY);

Now that you know how the code works, lets move on to the next section, in which I will tell you how to use it.

Some notes on using the software

  • The program searches for the brightest pixel in the camera's field of view, so the lighting conditions of your room can affect its performance. Adjust the brightness threshold and lighting conditions so that nothing (except the laser) exceeds the brightness threshold. If this doesn't work, change the brightness detection code to perform color detection instead (as described earlier).

clip_image005[4]

  • If you want to reset video cropping to normal, just right-click anywhere in the camera window.
  • For clicking on something with a laser pointer, just turn off the laser when the mouse cursor is over the item you want to click on. Turning the laser off simulates a left button, single-click. You might also want to change your Folder Options to allow files/folders to be opened with single-clicks.
  • While selecting a resize factor, remember that computing speed would come at the cost of tracking resolution. Say, for example, if I use a resize factor of 1, I'd get pretty good tracking resolution, but my program would slow down to a crawl. On the other hand, if I pick a resize factor greater than 4 or 5, I'd get a decent frame rate, but horrible tracking resolution. These values could be different for you (depending on your webcam and computer). For my computer, a resize factor of 2 or 3 works fine and I get about 27-30 frames/second.


Conclusion


We have reached the end of this article, and I hope you enjoyed reading it. Now, here's some homework for you: Try implementing features like double clicking, right clicking and click-drag. If you've watched the video for this article, you must have noticed that I was drawing stuff on MS-Paint using this program. For accomplishing this, I had to modify my code to perform click-drag. If you can think of an easier, laser-based way to switch between different clicking modes, I think that would be very cool. So, use your ideas, and if you end up doing something cool, I'd love to hear about it. Smiley Have fun!

About the Author

Ashish Derhgawen is an IT student, currently living in New Delhi, India. He has been coding since fourth grade. Some of his other interests are harmonica playing, wildlife and cricket. When he’s not at school, he spends his time working on unusual projects related to robotics, webcams, and electronics besides others. You can reach Ashish through his blog at http://ashishrd.blogspot.com.

Follow the Discussion

  • JoeJoe

    Could you please make it as .exe file to work under Vista. Thank you very much!

  • Clint RutkasClint I'm a "developer"

    @Joe, download Visual Studio Express and try it out ofr yourself!  Just hit F5 to compile it.

  • NickNick

    Ashish, your project is great indeed!

    When I tested it I found some details with my webcam (A4 Tech).

    If I set via wmcap.exe the exposure manually to low value (-12 as minimum) I get fps (~17).

    If the exposure is at maximum (-10) - fps is approximatelly 6-7.

    In auto exposure mode I can reach all 30.5 fps if I do capture near very bright surfice (for example monitor screen) but for all that the webcam seems to be very hot by touch.

    Could you explain this behavior?

    Regards,

    Nick

  • JPF321JPF321

    @Coding4Fun .. thanks for the quick response.  Thanks!

  • Clint RutkasClint I'm a "developer"

    @JPF321 Ashwin's code could do this with an ink control in .net but after PDC (Nov 17 to 19th, 2009) we'll have an article being released that can do this too but with XNA and WPF written by Jeremiah Morrill.

  • JPF321JPF321

    Ashish, thanks for the wonderful work and blog!

    I'm wondering what mods/tweaks were needed to this codebase to write "Hello World" as you did in your demo. It seems that, as is, it only makes mouse-click when laser goes OFF. So how did you get to click & drag lift pen and then click & drag again?

    I'm a newbie, but I think my question is a good one

    Thanks,

    jpf

  • Clint RutkasClint I'm a "developer"

    @Mike  http://laserinteraction.codeplex.com/SourceControl/ListDownloadableCommits.aspx  click the download link on the right side.

  • MikeMike

    The download link isn't working Sad

    There is no current recommended release for this Project

    Please help

  • Clint RutkasClint I'm a "developer"

    @Ian, just tried, codeplex loaded.

  • IANIAN

    the download link isn't working a got an error message

    "Sorry, the page you requested was not found"

    Sad

  • Clint RutkasClint I'm a "developer"

    @builtinengineer

    did a quick search and this is a C++ solution it seems like.  I did not however test or verify the code.  We have c# examples on this.  http://www.hytherion.com/beattidp/comput/pport.htm

  • builtinengineerbuiltinengi​neer

    can anyone help me to access parallel port using c?

  • shobikashobika

    please send the source code to shobikapearl@gmail.com

  • Clint RutkasClint I'm a "developer"

    @shobika download link is labeled "Download"  Smiley  Can be found over at http://www.codeplex.com/laserinteraction

  • Andy LuiAndy Lui

    Great project!

    I'm not a programer..

    How can I download the software and try with my fast camera and laser?

  • Clint RutkasClint I'm a "developer"

    @Andy Lui, this project you have to download the source and compile.  Download Visual Studio Express c# 2008 (http://www.microsoft.com/express), download the source code, open up the motion.sln file (refered to as the solution file), and just hit F5 or the play button.

  • Clint RutkasClint I'm a "developer"

    @Anna Glief Yup, actually just compiling the application with F5 will make it an executable!

    For a possible performance gain, switch to Release Mode.

    In the folder with the project, you'll see a BIN directory.  You'll have a DEBUG and RELEASE folder in there.  Everything needed to run the application is in that folder pending external dependancies you may have referenced that would need an installer.

    If you need additional help, contact us via the link at the top of the page!

  • Anna GliefAnna Glief

    It looks great in debug mode.

    But is it possible to create an exe?

    If Yes: how???

    (you are right: i'm a complete c#-newbee)

    I think this could be very helpfull for my friend, who has heavy disabillities (he can not move his hands/arms)

  • Clint RutkasClint I'm a "developer"

    @/ yup, go nuts, the application is open source Smiley

  • //

    Is it ok if i use these concepts in my project?

  • Clint RutkasClint I'm a "developer"

    @/ you still need to follow the license that the source code is released under however.  Different open source licenses have different rules when using them

  • //

    Gracias...dere were a lot of 'terms and conditions of use' & 'EULA' files! hence the question! sry fr d prev 2 comments, net prob!

  • Clint RutkasClint I'm a "developer"

    @Jordan Gray, go to the directory with the source code, you should see a BIN directory.  Depending on what you compiled it as, (I'll bet DEBUG) you'll see 2 folders.  so laserSource\bin\debug will have all the files you need to run the application.  You'll find the EXE in there.

  • Jordan GrayJordan Gray

    I just downloaded it and ran it, seems like it will work well (I just need to add some kind of filter to my camera, or dim my screen). However, I have a stupid question. I have no experience with Microsoft Visual Whatever I Just Downloaded, and so I was wondering if there was a way to permanently create this program (i.e., make a .exe that I can run when I want to run this program, as opposed to open Microsoft Visual, opening the "motion.sln", and then hitting F5).

    Sorry if that sentence ran on too long. Wink

    - Jordan

  • Jordan GrayJordan Gray

    I think the program is a great idea, I'm about to try and compile it.

    BTW, I like how you encourage people to compile the program for themselves, rather than you doing all the work for them. That's awesome Smiley

    - Jordan

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.