Kinect - And then there was light!


Lee Stearns, PhD student, Department of Computer Science
Preeti Bhargava, PhD student, Department of Computer Science



INTRODUCTION AND MOTIVATION

Many of the labs on our campus use motion sensors to conserve power by turning off the lights when there is no one in the room. However, these sensors have a limited detection range and so they can frequently turn off the lights even if there is someone working in the room. The MIND Lab in the A.V. Williams building has one of these sensors, and the students working in the lab sit in cubicles hidden from the sensor. If no one gets up from their desk for twenty minutes, the lights will turn off and someone has to walk up to the sensor to wave at it so that it turns them back on. This is rare during the middle of the day because there are enough people in the lab that someone will usually enter or leave the room before the motion sensor times out. But other times such as nights and weekends, when there are only a couple of people in the room, it can quickly become annoying. Imagine a poor graduate student huddled in one corner of the room at night finishing up an assignment or a paper that is due that very night and the motion sensor turns the lights off to plunge the room into darkness. He or she then has to keep walking back and forth to turn the lights back on every 20 minutes.

We propose a solution to this problem. Our system detects how many people are in the room by tracking them as they walk in and out, and uses an arm attached to a servo motor to activate the motion sensor and turn the lights back on.

OVERVIEW
systemoverview.jpg
Figure 1 Overview diagram of the system

As shown in Figure 1, the system consists of a Microsoft Kinect and an Arduino Uno with a servo motor connected to it. A cardboard arm is attached to the servo. When people walk in and out of the room, the Kinect tracks them and keeps a count of number of people currently present in the room. Whenever the motion sensor turns the lights off, if the number of people in the room is greater than zero, the system sends a 'move' command to the Arduino which causes it to move the servo, along with the attached arm, in a sweeping motion imitating a hand wave. We track the number of people in the room because we do not want the servo to move when there is no one in the room, since that would defeat the whole purpose of using a motion sensor to conserve power.

HOW IT WORKS

1. TRACKING AND COUNTING PEOPLE
kinect.jpg
Figure 2 Microsoft Kinect

The Microsoft Kinect (Figure 2) has a standard video camera and an infra-red camera that work together to provide a depth map of the 3D scene. The Kinect has been trained as a learning system to recognize body parts. This can then be used to identify body parts in the incoming data stream using the GPU on a per pixel basis. The classifications are then pooled across pixels to produce hypothetical 3D body joint positions used by a skeletal tracking algorithm. [2]

people screencapture.jpg
Figure 3 Our system tracking and counting people

Via our GUI, users can specify where the exit of the room is, As soon as a person comes in the room, our code uses the pixel depth values within each person blob and averages them to find the 3D centroid for that person. It then tracks the movement of the person across the room. Depending upon the direction of movement - away from exit or towards exit, the system increments or decrements a counter tracking the number of people currently in the room. Figure 3 shows a screen shot of the Kinect GUI when several people are walking into the room.

2. DETECTING IF THE LIGHTS ARE OFF

The Kinect measures the intensity of the ambient light in the room by averaging the grayscale pixel values from the RGB camera. If it falls below a certain threshold, then the Kinect concludes that the lights have been turned off and decides whether or not to activate the servo motor depending on the number of people currently in the room.

3. HANDWAVE

arduinohand.jpg
Figure 4 Arduino connected to the servo + arm setup

Figure 4 shows the servo and arm setup. The initial position of the arm is at 90 degrees, or vertical. As soon as the servo receives the command 'move', it rotates the arm twice from 45 degrees to 135 degrees. The arm finally comes to a stop at 90 degrees. This simulated handwave movement is detected by the motion sensor and it turns the lights back on.

sweep_BB.png
Figure 5 The arduino connected to the servo motor (Figure borrowed from [1])

Figure 5 shows the arduino + servo circuit. Servo motors have three wires: power, ground, and signal. The power wire is red, and is connected to the 5V pin on the Arduino board. The ground wire is black and is connected to GND pin on the Arduino board. The signal pin is yellow and is connected to pin 0 on the Arduino board.

TECHNICAL CHALLENGES:

1. If two people walk in side by side, the Kinect can detect only one of them as the other one is hidden from view. Hence, the system works best if everyone coming into the room enters separately or in single file.
2. The Kinect can track at most 6 people who are in its field of view. Thus, this system is not suitable for a crowded area where a number of people are walking in and out of the room, often simultaneously.
3. The Kinect occasionally has trouble tracking people who are walking away from the camera, or with unusual motion patterns (such as with their arms in their pockets).

FUTURE WORK

The limitations listed in the previous section could be addressed by using a network of two or more Kinects from different angles. However, this would introduce additional challenges, such as matching up people that are tracked in multiple views. This could be potentially be addressed by registering the depth blobs visible in the cameras, or by matching the clothing colors from the RBG cameras. Additionally, by processing the depth values directly rather than using the Kinect's built in segmentation it would be possible to track more people at once. Unfortunately due to the time limitations for this assignment, we didn't have a chance to test any of these potential improvements.

SUPPLEMENTARY MATERIALS:

Github
Youtube

REFERENCES:

[1]. http://arduino.cc/en/Tutorial/Sweep
[2]. http://www.i-programmer.info/news/105-artificial-intelligence/1562-how-kinect-tracks-people.html