Motion Tracking in Processing
In this example you will be able to see a basic computer vision application that uses a live camera feed to track the motion that is created. Using this information it is possible to create further automations and trigger other media such as video files, animations, or sounds. The program also uses OSC, a protocol that enables communication between local or remote computer systems.
Computer vision is a research field in computer science and Artificial Intelligence that uses image processing and complex computation to enable computers – or robotic systems – to identify specific properties of an environment using a camera as a sensing device. Here, we are going to see how to implement a simple but nevertheless efficient algorithm that identifies the motion that is created by the user. This method is very popular in many fields such as music, dance/performances, VJing, video gaming, and more.
In order to be able to open the source code file, you need to install in your computer Processing – a cross-platform (and free) programming environment based on Java. After you download and install Processing in your operating system, open the MotionTracking.pde file that exists inside the zipped folder that is included in this page.
Here follows an explanation of the code (for novice users, please go through the official Processing documentation). In the setup function of the program we set basic properties of the program such as window size, framerate, and fonts used for the text that appears on the screen window. We also initialize the OpenSoundControl communication using the oscP5 library (library is included in the download). In these settings we set the listening port to 7401 (although at the moment there is no function in the program for doing anything if it listens something, but can be used in the future if you want), and also the send IP address and port number of the computer we would like to send the tracked information. At the moment the address “127.0.0.1” is used, which basically says that the information can be sent within the same machine that the program runs (local communication). To send this information to another machine, you would have to make a client that listens to the OSC prefix text, find its IP address, and then use the information that is sent from this patch. However, it is possible to use the OSC protocol in many different programs and for many different reasons (music, dance/performance, automation, security, etc). Currently the information you can send is (1) the motion amount in float numbers (2) the higher threshold trigger in boolean (i.e. true or false), and (3) the lower threshold trigger in boolean as well.
In the setup function we can set the camera that is going to be used for the tracking. Using the command Capture.list(), the program identifies all the connected cameras of your system and displays them on the console window using the println() function. Therefore, we can find out the array number a particular device has been stored, and use it in the Capture constructor. Also, in the constructor we can set the size of the camera capture (lower sizes result in less detailed https://soniconlab.com/images but faster processing) – here I set it to 160×120 pixels. Finally, we create a second video frame (prevFrame) that is going to be used later for the image processing, and set the smoothing values for the processing.
After the program has been initialized, we can call the draw() function. Here, we read frame by frame the camera contents and create a copy of them in the prevFrame buffer. Then, we use a for loop to identify each pixel of the video and compare it with its previous one (stored in the prevFrame). This difference basically determines the amount of pixels that changed from a frame before; the less the pixels changed, the lower this number will be (the least is number 0, meaning that there is no motion tracked at all), and the more the pixels changed, the higher the number (maximum is the number of the video resolution we are tracking, meaning that there is full movement tracked on the camera).
After we find out about the motion amount, we average this number so that it spans across many different frames and not just a particular one. This is more useful and pragmatic, however the smoothing amount depends on your applications. Nevertheless, you can define the smoothing you want, and set it from inside the code – setup() function, smoothValue variable here set to 50. After we process and receive the smoothed value of the motion, we can use it to track specific things; for example in this program there are two thresholds that can trigger different processes based on the tracked amount – a low threshold that is set to true if the motion drops below a certain point (lowThresholdValue), and a high threshold that is set to true if the motion rises above a certain point (highThresholdValue). Finally, in the draw() function we set the screen interface to display important information, and also set the OSC messages for communication.
The following function is responsible for displaying all the information on the final screen window such as text, smoothing value, and low/high thresholds.
Next function essentially formats the OSC messages that are going to be sent to the client computer. For each stream of information we need to set a specific prefix that is going to be used from the client system as well (i.e. “/MotionAmount”). Thus, each OSC message contains this prefix with the value currently identified from the program.
And last but not least, the following class in used to smooth the values out, creating an array of the last tracked values and averaging them altogether. To change the array, we need to pass a different argument from the program initialization. Larger values result in more smoothing but less responsive changes.
The final screen window appears in the following screenshot. For any further information, questions, or suggestions please email us or comment below.
