Headtreacking met Wiimote

From Fontys VR-Wiki
Jump to: navigation, search


After seeing Johnny Chung Lee's youtube video [1]of his DesktopVR application, I was wondering if this type of headtracking could be used in a more generic way. I looked at the VRPN website and noticed that the Wii was not in the list of supported trackers yet. For those who don't know VRPN, here's a brief explanation from their website:

"The Virtual-Reality Peripheral Network (VRPN) is a set of classes within a library and a set of servers that are designed to implement a network-transparent interface between application programs and the set of physical devices (tracker, etc.) used in a virtual-reality (VR) system."

In other words, it provides a generic way to interact with typical VR input devices, such as professional tracking systems. VRPN can be used by a number of VR/3D software packages, such as Virtools, OpenSceneGraph or VR Juggler. Therefore implementing Wii headtracking in a VRPN driver seemed like a logical choice.

VR Juggler is a VR middleware platform. It provides an abstraction layer between virtual reality hardware and software. It allows you to code an application in C++ and OpenGL and use it on a wide variety of VR systems (e.g. multi-display configurations with (head)tracking) by simply changing a configuration file.


I work at Fontys University (Eindhoven, The Netherlands), where I'm researching various possible applications for a soon-to-be-installed 4-sided virtual reality CAVE. Development for this is mainly done using VR Juggler. I've implemented Wii headtracking support as an experiment and a possible way to interact with our applications without the need of the actual CAVE. Instead of using it for headtracking, it can also be used to simulate the position of a wand (3d-mouse). This might be more convenient than simulating the position of such a device with keyboard input.

How does it work?

As can be seen on Johnny Chung Lee's youtube video, and on his website, the wiimote is positioned underneath or above your screen. It is pointed at a slight angle to allow for a better field of view. The wiimote contains an infrared camera with 1024x768 resolution, which can see the infrared leds of the wii sensor bar (or any other infrared leds for that matter). Based on the position of the leds in the camera's image and the distance between them, an algorithm developed by Johnny Chung Lee determines the position of the sensor bar relative to the wiimote. If we know the position of the sensor bar in free space, that means we can use it as a tracking device.

Johnny Chung Lee's application uses Brian Peek's managed wiimote library. Since VRPN uses C++ I decided it was too much of a hassle to try and make a wrapper for it, so I looked for a C++ wiimote library. I stumbled across the WiiYourself! Library, which happened to be based on Brian Peek's C# library. So far so good. The disadvantage is that VRPN and VR Juggler are cross-platform, while WiiYourself! is Windows only. I did not solve this problem, instead I just made a windows-only VRPN server for the wiimote.

I ported some of Johnny Chung Lee's C# code to C++ to get the position of the sensorbar in the real world. This data is fed to VR Juggler through VRPN. Based on this information, VR Juggler takes care of determining and rendering the correct perspective.

Software used

  • VRPN 07.15 [2]
  • WiiYourself! 1.0 [3]
  • VR Juggler 2.2.1 [4]
  • Windows Driver Development Kit (needed to compile WiiYourself!) [5]
  • BlueSoleil Bluetooth driver (might or might not be needed, depends on your Bluetooth hardware ) [6]
  • Visual Studio C++ 2005

Hardware used

  • Wiimote
  • Wireless sensorbar
  • Bluetooth receiver

Technical details

  • VRPN part
  • Added: vrpn_Tracker_Wii.h & vrpn_Tracker_Wii.C (compiled as C++)
  • Class inherits from vrpn_Tracker
  • Additional dependencies: WiiYourself!.lib
  • Additional includes: wiimote.h

Download sources here[7]

In addition to the tracker class, there has to be a VRPN server implementation that uses this class. The visual studio project of this server can be downloaded here [8].

'q' quits the server. Any other key corrects the perspective. Hold the sensorbar in front of the middle of the screen before doing so.

If you want to skip the hassle of compiling vrpn with the wii driver, you can download the binary version of this VRPN server here [9]. Compiled for win32. You might need the .NET framework to run this.

VR Juggler part

In the VR Juggler configuration file, you have to specify a display surface of the size of your computer display. Based on this and the tracker's position, VR Juggler will be able to render the correct perspective. Now you let VR Juggler use a VRPN server for either the head- or wand position. More information on how to do this can be found on the VR Juggler website.

The demo I made tries to mimic the look of Johnny's DesktopVR application a bit. It displays 4 targets, which are placed on different positions on the X,Y & Z-axis. It also displays a wireframe 3d grid that seems to be attached to the screen, just like in the DesktopVR demo.

Download VR Juggler Visual Studio project here.[10]



The setup running the demo application.


VR simulator viewport, the position of the head relative to the screen surface and 3D objects can be seen here.

Video[11] showing the setup running the demo application. For some reason my wiimote keeps losing track of the infrared leds, so the image is a bit jumpy. This also happens when I try Johnny Chung Lee's DesktopVR app.

The software is provided as is.

Wouter Wognum 14-4-2008