We present a remote flashlight that lights your way in a dark room. Don't wake up sleeping friends in the room, grab your flashlight virtually and just use your arm to control it freely. You don't need any gadgets in your pocket, simply use the interactive room!


Motivation and Results


You go inside the room at night. It's all dark and you cannot find your way. You want to turn on the light but you have a friend or significant other napping/sleeping in the room. You don't want to wake them up by turning on the main light. You could have used a flashlight in your pocket but *GASP*, you haven't one! To solve this real-life problem, let's instrument the room with a remote light that you can easily control with gestures!

Our prototype system does just that. Here, take a look at our video!




Gestures for Light Control


Our gesture system is based on the movement of the arm relative to the body. Our aim is to achieve a natural interaction with 2D orientation control, as if you are actually holding a flashlight with your right hand. Notice that we do not aim to have a complete 3D calibration with the environment. In other words, we are not lighting where the user points to in the room. By making the gesture relative to the body, it makes a much simpler setup, and allows the user to re-orient themselves in the room easily without changing the direction of the light.

Orienting the light

The orientation of the light is based on the orientation of your right hand relative to your body. We take the right-shoulder as the main point of reference, as shown in Figure 1. We construct a plane on the user's torso (with left-right shoulder and spine joints). The projection of the right arm direction (elbow to wrist) to this plane gives us the 2D orientation of the flashlight (blue circle shown in figure 2). S_r-S_l is used as one of the orthogonal basis (left-right) for the control and we construct the other orthogonal basis (S_r-d) on the plane. The recognition of the users arm movement only happens when the flashlight is turned on by the user, with the simple gesture below.

Turning light on/off

Similar to orientation, we control on/off state by using our arms. This time we use the left-arm to avoid confusion with our orientation gestures. Basically, you raise your upper-left arm parallel to your shoulder-line to activate/de-activate the flashlight. The dot product of left/elbow-left/shoulder and left/shoulder - right/shoulder lines (normalized) is computed to find how parallel the lines are. On Figure 1, the related lines are S_l-R_l and S_l-E_l. Once the light state is changed, a message is sent to the micro-controller (Arduino) and tracking this state is disabled for 3 seconds to avoid any unwanted fast switches between on/off states.

kinect_light_annotated.jpg
Figure 1: Red dots denote the joints that are used to control the light. S_x denote shoulders, E_x denote elbows, W_r is the wrist, and S is the spine joint. The blue circle is how the E_r-W_r line is projected onto the plane defined by S_r-S_l-S points.


Hardware Setup

Kinect for Windows
20121119_184355.jpg
We use it to track the user's body and extract the skeleton form. It can even see you in dark!
Arduino Uno
20121119_184525.jpg
It's the micro-controller we use to control the remote flashlight.
2 Servo's
9nq2usq6.jpg
Our servos give us the ability to change the orientation of the remote flashlight.
Flashlight
(head)
iqb6y3bc.jpg
We took an off-the-shelf flashlight and removed the battery casing to make it smaller and easy to move. The power for the light is fed through a circuit independent of the Arduino and switch via a relay.
Various Lego Parts
kinect_lego_parts.jpg
To construct the base platform to hold flashlight, what is more playful and colorful than Legos?
Batteries
batteries_kinect.jpg
In case you need more power than what your micro-controller than provide to your flashlight, batteries come to the rescue.
Relay
relay.jpeg
The relay allows for the Arduino to switch the battery power on and off to the flashlight.
Laptop/PC
laptop.jpg
You need this to process the data from Kinect, recognize the user, and map the user movement to the remote flashlight.

Micro-controller / Servo / Light Setup


Our micro-controller setup basically includes two parts:
  • Servo control : The 2 pins of the servos are appropriately connected to Arduino board. No other electrical components are needed.
  • Flashlight control (on/off): A relay is wired up to a battery source. The separate circuit is logically switched by the Arduino.
kinect_light_board_back.jpg
Our circuit diagram is as follows:

Light My Way Sketch_bb.png



20121119_184619.jpg
Base Servo (left-right rotation)
vf65l4zn.jpg
Upper Servo (up-down rotation)


Software Implementation


Software for micro-controller:

We implemented an Arduino program which reads in the commands from computer and controls the servo and flashlight circuits through its output pins. We use Servo library of Arduino to control respective output pins on the micro-controller (pins 9-10). We parse the message from the PC on a per-byte basis. Signalling is sent to the relay via pin 11. The complete source code is available here.

Software for PC:

Our implementation is based on SkeletonViewer example found in Windows Kinect SDK. The source code is written in C++ following the existing example project. We extended this sample by adding our gesture detection as the skeleton data is extracted by the application and we follow the approach described above. We also add a serial communication handle on startup, and use this handle to send updates over serial port appropriately.

Communication between PC and micro-controller:

WE developed a protocol consisting of 4 message types: 2 for controlling x/y angles, and 2 for turning the light on/off. Note that data is only sent from computer to Arduino (one-way). We used the serial port for communication between our C++ application and the Arduino. We used the basic Windows API functions (CreateFile) to open the port, and Write() function to write the message string to the serial port handle. This example code has been useful to get the basics of serial port handling. Arduino uses its simple serial API to read data from the port one byte at a time. We then parse the input message to extract the target angle of the servo or light action as needed.


Retrospective and Challenges

  • Serial Communication / COM#: We could not communicate with one of our Arduino boards using serial port communication on Windows. The main difference was that the arduinos was attached to different COM ports on the PC when plugged in, and the one assigned to COM10 could not be connected. It's uncertain why this Uno would not work when it was set up to interface with our C++ code. Both Arduino worked well with the Arduino interface software. We decided to use the the one Arduino that typically attached to COM6. Another solution might have been to follow this article.
  • Relay / Flashlight Control: We also had some issues with the wiring of the relay component. Our initial observation was that we could need higher voltage than Arduino can provide, and thus wanted to use a relay circuit to power the flashlight. We found a relay which required less voltage in this process (3.3V), and we still kept the relay circuit as part of the project since it was already available. Another issue can be the maximum current that Arduino can provide, and whether it would be enough to power both the flashlight and all the servos dynamically.
  • Serial Communication / Timing:We continue to lose our serial connection from C++ application to Arduino after a few seconds. We observe a timeout occuring on occasion upon serial writes, leading to the device being locked. We modified timeout setting and tried to account for this but have been unable to do so. To be able to access Arduino again, we needed to:
    • Re-plug the Arduino to computer.
    • Open up serial communication monitor on arduino software (& close it to free up the COM resource for the application)
    • Run the application again. This was somehow unpleasant when we were losing the communication between frequently, and re-running thing took more than a few easy steps.
  • Designing the light base: We tried to come up with an easy and effective design and improve it along the way. One of the ideas we had was to use pulleys to extend the servo range, if needed. The photo below illustrates this idea. However the pulley on the servo was smaller than the lego one. This in turn reduce the rnage of motion significantly as the servo only had 180 degrees of motion. pully_design.jpg

What's in the future?


Pairing the Kinect with a light as a physical output device can lead to further interesting installations and projects. We have previously seen lasers that tracks the users body for interaction ( Laser interaction prototype - Jaysonh.com / http://vimeo.com/44904580). Another approach in controlling a remote light source would be to have a completely calibrated environment, where the light follows a user's hand (projects on the user's hand) or projects to the direction that the user points. This, however requires knowing the environment and projecting the users hand line on the the environment, figuring out the 3D position, and moving the servos appropriately.
.
Hardware