The feature that will be discussed in this article.

The project uses the following attributes:
•    Wiimote: Nintendo RVL-CNT-01
•    Bluetooth USB dongle
•    BlueSoleil bluetooth stack (using version 2.3.0)
•    Software library ‘WiimotLib', (courtesy Brian Peek)
•    Software development: MS-Visual C# 2005

The Hard Stuff

The soul of the Wiimote machine is a device called MEMS.

What are MEMS? These tiny devices, sizing micro to nanometers, are small integrated devices, combining electrical and mechanical components, hence the naming of Micro Electro Mechanical Systems.

For those interested in the history of the MEMS, please read the next article: “MicroELectroMechanical Systems (MEMS) by Salvatore A. Vittorio – web link: http://www.csa.com/discoveryguides/mems/overview.php”

By making all those fantastic movements possible, the MEMS device has found its way into the mass (gaming, a.o.) market.

The following is an excerpt from “memsnet.org”, explaining the MEMS in a nutshell:

“Micro-Electro-Mechanical Systems (MEMS) is the integration of mechanical elements, sensors, actuators, and electronics on a common silicon substrate through microfabrication technology. While the electronics are fabricated using integrated circuit (IC) process sequences, the micromechanical components are fabricated using compatible "micromachining" processes that selectively etch away parts of the silicon wafer or add new structural layers to form the mechanical and electromechanical devices.”

The following is a view of the ‘internals' of a MEMS device: (courtesy of www.silicondesigns.com)

Nintendo's Wiimote motion detection is sensed and controlled by the ADXL330 MEMS accelerometer device from Analog Devices:

(iMEMS: Integrated Microelectromechanical System)

Having covered the hardware stuff, we will elaborate on the soft part:

Browsing the Internet learns that there are several Wiimote applications floating around – the one we needed, a basic PC mouse functionality also exists, it's called ‘WiinRemote', hosted at the site http://onakasuita.org/wii/index-e.html., and is built with the Borland Delphi 6 Pro development framework.

Alas, living in a non-Delphi habitat, it forces us to either convert the WiinRemote to, or start from scratch with VisualStudio. Both are unfavorable. A middle way was chosen: use the WiinRemote as a reference and build the mouse application based on Brian Peek's Wiimotelib library.

That's how it all started ... .

Implementation

First off, we are confronted with the mechanical behavior of the Wiimote, in particular how the human complex movements translates to MEMS data, got converted in the chip into voltages and ultimately arrives at our VisualStudio development screen as hex values.

Then there is the choice between using the Wii IR LED bar array or not; using this method has the advantage of an easier interface algorithm. The drawback is: yet another device, plus its power supply, to incorporate in the scene. So we decided not to use it and go for the Wiimote only.

The Wiimote outputs, as described in the ADXL330 data sheet, three voltages, representing the x,y and z axis, covering 6-DOF (Degrees Of Freedom).

These accelerometer data, can then be worked out in a Motion Control system, which can be depicted as follows:

Human Visualization:

Figure 1 depicts the trivial situation where a user wants to move the cursor from point A to point T, by manipulating the Wiimote.

While the user is moving the Wiimote around in a kind of intuitive manner, the system should response with a dynamic behavior: starting swiftly, moving smoothly and a full-stop when the user ceases the activity.

The start-stop functions contains a kind of contradictio-in-termino: it has to start immediately and come to a full stop at once, which in fact is ‘not possible' because a user's hand is never completely at rest, it'll always have some minor, high frequency, ‘movement' (a.k.a vibration).

To solve this problem we have to enter the Motion Control domain.

A setup of the Motion Control (I-type gain) is depicted in Figure 2.

In here the human factor is included in the closed loop; by doing so, the unwanted ‘jitter' (resulting in a ‘nervous' cursor behavior) can be taken into account and will be suppressed as much as possible.

The Wiimote as such, is used as a gravity sensor, because it is highly influenced by the gravity force; the data sent out to the receiver is leading for its current angular state in space. Therefore only the X and Y components are of relevance.

Explaining the control loop:

Technical Visualization:

In the feed forward path we see, from left to right, the human brain block as the initiator of all activities: it triggers the cursor movement (from point A to target T) by igniting neurons, sending muscle movement commands to the hand holding the Wiimote.

This will produce the user-activity, ‘ua' input signal for the Bluetooth control block, which comprises of: the Wiimote Controller → Bluetooth USB Dongle → Bluetooth Driver Software.

At this stage we are actually in-the-PC; signal ‘b' will be processed by our Control Function, outputting the ‘c' signal which in turn will be passed through an Integrator.

The Control Function is a non-linear function called ‘Signed Square', with one variable.

Why this particular function?

The human intuitive movement consists of a uniformly accelerated and decelerated motion; for this type of movements we have a quadratic mechanical relation between distance and speed.

The quadratic function is to achieve a swift, dynamic behavior of the Wiimote cursor at ‘large' distances (e.g. when starting the motion), and a sluggish one when approaching the target spot.

The Integrator enables us to realize the relative positioning on the screen.

Another benefit of the SignedSquare function is the existence of the so-called ‘dead zone', around the target position (see graph, Figure 3). This dead zone is particularly necessary because the human hand is never steady, completely at rest. It will always show some kind of ‘vibration' which will trigger the Wiimote event, resulting in a nervous, unsteady cursor positioning behavior.

Within this region the cursor isn't allowed to move (it acts like the differences between the target and actual positions and speed are zero), beyond this region it should react immediately.

The Signed Square function in its general form:

f(x) = a.x²

A complicating issue is the fact that the Wiimote delivers positive as well as negative values while performing its duty in the real world; we have to take this into account in the formula and maintain its current working quadrant space signature. To achieve this, the ‘a'-term should carry the right sign, accordingly to the Wiimote data values.

Mathematical derivation of the sign function:

sign = √ x² / x {1}

f(x) = sign × x² {2}

(Remark: in the logical C# solution, the sign factor will be taken care of by the built-in ‘Math.Sign' method. Also, the possibility of a division by zero is covered as well. )

Putting {1} and {2} together gives us the wanted result:

f(x) = x² × ( √ x² / x)

f(x) = x × √ x²

which gives the following results for the positive and negative values of x:

for x › 0 then f(x) = x²

for x ‹ 0 then f(x) = -x × √ (-x)² = -x × √ x²

f(x) = - x²

The integrator accumulates the ∆X, ∆Y movements; the output ‘y', is formally the wanted X,Y coordinates of the targeted position – however, it has to pass the ‘Screen Limitation' block to avoid excessive off-screen coordinate values which otherwise will result in an annoying cursor-hysteresis behavior (: cursor goes off-screen and ‘takes-a-while' before coming into vision when the Wiimote movement direction is reversed).

The Screen Limitations block outputs the ‘ny' cursor position signal which is then feedbacked into the control loop, closing the loop.

As can be seen from Figure 3, the transfer function inhibits any movement around the ‘dead-zone' (around zero area), thus preventing any cursor vibrations to occur when at rest or at a full-stop moment.

The Coding Stuff

As said in the introduction, the main scheme of this project is not completely new; it refers for a great deal onto the ‘WiinRemote' application. Differences exists, for example in the number of Timer threads used, and mainly the motion control approach of the project (although I'm not sure whether WiinRemote also utilized it – the site doesn't carry any documentation, takes too much time to reverse-engineer/debug it).

The Code structure:

In C# syntax:

Retrieve the button mapping between the Wiimote and the (impersonated-) mouse by reading the XML configuration file – App.config:

   1: <?xml version="1.0" encoding="utf-8" ?>

   2: <configuration>

   3:   <appSettings>

   4:     <!--These are the WiiMote to Mouse button assignments:-->

   5:     <add key="WiiMote_A" value="MOUSE_LEFT"/>

   6:     <add key="WiiMote_B" value="MOUSE_RIGHT"/>

   7:   </appSettings>

   8:   

   9: </configuration>

  10:  

And assign it to the relevant variables:

   1: #region ReadButtonAssignment

   2: // Here we ask for the button mapping of the WiiMote wrt the Mouse functions

   3: // Get the info from the XML-configuration file (App.Config):

   4:  

   5: private void ReadButtonAssignment()

   6: {

   7:     strButtonA = ConfigurationManager.AppSettings["WiiMote_A"];

   8:     strButtonB = ConfigurationManager.AppSettings["WiiMote_B"];

   9: }

  10: #endregion ReadButtonAssignment

We retrieve the Wiimote data through the event scheme – therefore we need an event handler to react on such an occasion.

The wm_OnWiimoteChanged event handler is responsible for handling the Wiimote events, occurring when it is moved.

Executing Brian's method asynchronously, we receives the Wiimote's raw acceleration data for the x, y and z directions.

   1: #region wm_OnWiimoteChanged

   2: void wm_OnWiimoteChanged(object sender, WiimoteChangedEventArgs args)

   3: {

   4:     m.WaitOne();

   5:     WiimoteState ws = args.WiimoteState;

   6:  

   7:     // Display the WiiMote energy state:

   8:     BeginInvoke((MethodInvoker)delegate()

   9:     {pbBatteryLevel.Value = (ws.Battery > 0xC8 ? 0xC8 : (int)ws.Battery); });

  10:      float f = ((float)(ws.Battery / 192.0f) * 100.0f);

  11:     BeginInvoke((MethodInvoker)delegate()

  12:     { lblBatteryLevel.Text = "Energy Level: "+f.ToString("F")+ " %"; });

  13:  

  14:     // Information needed for Mouse movements ....:

  15:     BeginInvoke((MethodInvoker)delegate()

  16:     { lblAx.Text = ws.AccelState.X.ToString(); });

  17:     BeginInvoke((MethodInvoker)delegate()

  18:     { lblAy.Text = ws.AccelState.Y.ToString(); });

  19:     BeginInvoke((MethodInvoker)delegate()

  20:     { lblAz.Text = ws.AccelState.Z.ToString(); });

  21:  

  22:     AnalyzeButton(ws);//Read + interpret WiiMote data, acquired via Bluetooth

  23:  

  24:     m.ReleaseMutex();

  25: }

  26: #endregion wm_OnWiimoteChanged

Knowing what the button is doing, we proceed and perform the event:

   1: #region ButtonEvent

   2: // ButtonEvent prepares the actual event performing, providing the event

   3: // method with the necessary input: which_button, what_event_type and whether

   4: // it's a click or dragging event.

   5:  

   6: private void ButtonEvent(int Button, int EventType, ref ButtonState

   7:                          ButtonState)

   8: {

   9:     if (((EventType == nWII_EVENT_UP) && ButtonState.PushFlag)||

  10:        ((EventType == nWII_EVENT_DOWN)&& !(ButtonState.PushFlag)))

  11:     {

  12:         switch (Button)             // Which button are we dealing with ...

  13:         {

  14:             case (nWII_BUTTON_A):

  15:                 {

  16:                     bRepeatFlag = false;

  17:                     PerformEvent(strButtonA,EventType,bRepeatFlag);

  18:                     break;

  19:                 }

  20:             case (nWII_BUTTON_B):

  21:                 {

  22:                     bRepeatFlag = false;

  23:                     PerformEvent(strButtonB, EventType, bRepeatFlag);

  24:                     break;

  25:                 }

  26:             default: break;

  27:         }

  28:     }

  29:     // Administer the PushFlag:

  30:     if (EventType == nWII_EVENT_DOWN) ButtonState.PushFlag = true;

  31:     if (EventType == nWII_EVENT_UP) ButtonState.PushFlag = false;

  32: }

  33: #endregion ButtonEvent

The actual event-performing code:

Due to lack of HID (HUMAN INTERFACE DEVICE) support in this .NET framework, we have to ask for assistance from the good old WIN32 API and reach it through the P/Invoke, set available by the Runtime.InteropServices.

   1: #region PerformEvent

   2: // Method performing the Mouse activities, based on the WIN32-API calls

   3: // (P/Invoke):

   4:  

   5: private void PerformEvent(string ButtonAssignType, int EventType,

   6:                           bool RepeatFlag)

   7: {

   8:     if ((EventType == nWII_EVENT_DOWN)&&(RepeatFlag == false))

   9:                                     nLastPushTime = DateTime.Now; ;

  10:  

  11:     // Do the Mouse Event ...:

  12:     if (ButtonAssignType == "MOUSE_LEFT")

  13:         switch (EventType)

  14:         {

  15:             case nWII_EVENT_DOWN:

  16:                 Win32.mouse_event(Win32.MouseEventType.MOUSEEVENTF_LEFTDOWN,

  17:                                  0,0,0,0);

  18:                 break;

  19:             case nWII_EVENT_UP:

  20:                 Win32.mouse_event(Win32.MouseEventType.MOUSEEVENTF_LEFTUP,

  21:                                  0,0,0,0);

  22:                 break;

  23:             default: break;

  24:         }

  25:  

  26:     if (ButtonAssignType == "MOUSE_RIGHT")

  27:         switch (EventType)

  28:         {

  29:             case nWII_EVENT_DOWN:

  30:                 Win32.mouse_event(Win32.MouseEventType.MOUSEEVENTF_RIGHTDOWN,

  31:                                  0, 0, 0, 0);

  32:                 break;

  33:             case nWII_EVENT_UP:

  34:                 Win32.mouse_event(Win32.MouseEventType.MOUSEEVENTF_RIGHTUP,

  35:                                  0, 0, 0, 0);

  36:                 break;

  37:             default: break;

  38:         }

  39: }

  40: #endregion PerformEvent

All the stuff above is useless if the event-catching and notifying mechanism isn't functioning ... .

This is the responsibility of the two timer threads, started up at program loading.

The CursorMotionTimer task is to trigger the cursor movement; it periodically (default = 10 msec) fetches the X and Y acceleration data from the Wiimote and process it according to the motion control scheme.

   1: #region CursorMotionTimer_Tick

   2: private void CursorMotionTimer_Tick(object sender, EventArgs e)

   3: {

   4:   // Some information: the WiiMote utilize the ADXL330 (3-axis, +/- 3g MEMS Accelerometer)

   5:   // from Analog Devices. It is a complete 3-axis accelerometer with signal conditioned

   6:   // voltage outputs, measuring position, motion, tilt, shock and vibration.

   7:   // Operating voltage: 1.8 to 3.6V

   8:   //

   9:   // The Motion Control is currently based on a Singular Signed Quadratic form with one

  10:   // variable, in its general form: F(x) = ax^2. However, since we have to deal with the

  11:   // WiiMote which delivers positive and negative values, the formula has to be adapted,

  12:   // otherwise the negative values will be lost due to the squaring.

  13:   // Ultimately the working implentation form is:

  14:   // sign = sqrt(x^2) / x     (1)  (the code will use the built-in Math.Sign function)

  15:   // f(x) = sign * x^2        (2)

  16:   // putting (1) and (2) together gives the working algorithm implemented in this version:

  17:   // f(x) = x * sqrt(x^2)

   1: //---------- POSITION_X:------------------------------------------

   2: dWiiMoteSetSpeedX=Convert.ToDouble(lblAx.Text)-dWiiOffsetX;//CURRENT position X from WiiMote

   3:  

   4: // Using the Signed Square function:

   5: nSign = Math.Sign(dWiiMoteSetSpeedX);    // Mind the Wiimote directions ...

   6: dWiiMoteSSXQuadratic = Math.Pow(dWiiMoteSetSpeedX,2.0) * nSign;

   7:  

   8: dWiiMotePosX=(dWiiMotePosX+(dWiiMoteSSXQuadratic*dSamplePeriod)*nSpeedGain); // INTEGRATOR

   9:  

  10: // Do some limitations ...

  11: if (dWiiMotePosX >= 1) dWiiMotePosX = 1;

  12: if (dWiiMotePosX <= -1) dWiiMotePosX = -1;

  13: dMousePosX = dWiiMotePosX;

  14:  

  15: MousePosXFiltered = (MaxScreenX / 2)*(dMousePosX + 1);   // Mid screen

   1: //---------- POSITION_Y:------------------------------------------

   2: dWiiMoteSetSpeedY=Convert.ToDouble(lblAy.Text)-dWiiOffsetY;// CURRENT position Y from WiiMote

   3:  

   4: // Using the Signed Square function:

   5: nSign = Math.Sign(dWiiMoteSetSpeedX);    // Mind the Wiimote directions ...

   6: dWiiMoteSSYQuadratic = Math.Pow(dWiiMoteSetSpeedY,2.0) * nSign;

   7:  

   8: dWiiMotePosY=(dWiiMotePosY+(dWiiMoteSSYQuadratic*dSamplePeriod)*nSpeedGain);   // INTEGRATOR

   9:  

  10: // Do some limitations ...

  11: if (dWiiMotePosY >= 1) dWiiMotePosY = 1;

  12: if (dWiiMotePosY <= -1) dWiiMotePosY = -1;

  13: dMousePosY = dWiiMotePosY;

  14:  

  15: MousePosYFiltered = (MaxScreenY / 2) * (dMousePosY + 1);   // Mid screen

Having sorted out all those stuff, we can now address the cursor:

   1: //----------------- MOVING THE CURSOR: ----------------------------

   2:     Win32.SetCursorPos((int)MousePosXFiltered, (int)MousePosYFiltered);

   3: }

   4: #endregion CursorMotionTimer_Tick

Our cursor should now arrive at the targeted position.

The DragMouseTimer method:

Moving the mouse is one activity, click on something and drag it is another thing. We will cope with this action variety in the following manner:

If it senses that the button is being held down for a certain period of time -in this case more than 800 msec (value from WiinRemote)- the event will be set to the EVENT_DOWN type, hence if the Wiimote is moved, it will drag the object it is clicked on.

   1: #region DragMouseTimer_Tick

   2: // Method performing the Mouse dragging while the Left or Right button is held down.

   3: // It has a 'pondering' window of 800 msec - beyond this, the RepeatFlag will signal

   4: // the ButtonEvent method that it shouldn't execute a subsequent mouse event,

   5: // instead maintain the current one, while the user is still holding down the key.

   6:  

   7: private void DragMouseTimer_Tick(object sender, EventArgs e)

   8: {

   9:     TimeSpan Duration = DateTime.Now - nLastPushTime;

  10:     if ( myButtons.ButtonA.PushFlag && Duration.TotalMilliseconds>800)

  11:     {

  12:         bRepeatFlag = true;

  13:         PerformEvent(strButtonA,nWII_EVENT_DOWN,bRepeatFlag);

  14:     }

  15:  

  16:     if (myButtons.ButtonB.PushFlag && Duration.TotalMilliseconds > 800)

  17:     {

  18:         bRepeatFlag = true;

  19:         PerformEvent(strButtonB, nWII_EVENT_DOWN, bRepeatFlag);

  20:     }

  21: }

  22: #endregion DragMouseTimer_Tick

How it works:

Running the executable will produce a minimized WinForm on the Taskbar; the PC cursor is by now taken over by the Wiimote. You can use it with the A-button as a Left mouse button and the B-button as t he Right mouse button. To return the cursor control to the standard mouse, you have to kill the application.

Epilogue:

That's all there is, for the moment; the movement control is still under investigation and is far from a finished state!

If time permits and enlightened ideas pop-ups, we'll surely improve the device movement behavior (or somebody else can jump in the code).

DISCLAIMER:

The software is in a “work-in-progress” state – not fully tested – contains flaws – subject to changes.

Use it with care.

Wiet's Bio:

Wiet's experience covers both the hard and software framework. Started as an electrotechnical engineer designing computer boards and completing it with software programming (from Assembly via PL/M to C#) to let the hardware to come alive. Hard- and software interaction is his favorite working area.

Currently he is working for Philips Healthcare, a Dutch founded international company, carrying out his contribution to the firm from within the X-Ray Predevelopment department.

You can reach Wiet via: wietlie@hotmail.com

Chris' Bio:

Chris is an enthusiastic puzzler on real world problems. His knowledge is control theory and interfacing all kinds of contrasting areas of technology, such as analog-digital, hardware-software, sensor-actuator, human-machine. His fascination is the art of problem conversion to other domains, where they can be solved, followed by the back translation to its original domain to implement the solution. His programming experience covers various languages like LabView, Java(script), C#, HTML.

Currently he works in the same predevelopment department as Wiet.

Chris can be reached at: space4chris@hotmail.com