Week 5 -- The World Beyond

Intersection with Physical Computing

Things to Remember About Serial Input & Processing

only one program can use the serial port at time!

use HyperTerminal (PC) or zTerm (Mac) as a debugging tool

start serial in "setup": beginSerial(dataRate);

Before you run your program, choose the serial port you'll be using from the "Sketch" menu.

The callback handler for incoming serial is called serialEvent() and the current byte being read is stored in a variable named 'serial'.

Up until now, our ability to deal with input in the processing environment has been limited to the mouse and keyboard. This week, however, we move beyond these devices and examine how to process input from a microcontroller or any other device that might send information via the PC's serial port.

Read Tom Igoe's page: Serial to the Desktop
Read Tom Igoe's page: More on Serial Communication.

Examples

In Class Serial Example -- getting one value.
In Class Serial Example -- getting three values.
PIC Code for these examples.

Sound

Things to Remember About Using Sonia

sound sample files go in the "data" directory

sonia files go in the "code" directory

start Sonia in "setup": Sonia.start(this);

always include a "stop" function:

         public void stop(){
             Sonia.stop();
             super.stop();
         } Processing can also handle different outputs, such as sound. Created by Amit Pitaru, Sonia is an external Library (API) for the Processing platform. that provides advanced audio capabilities. Features include sample playback, realtime sound synthesis, realtime FFT (frequency) analysis of the microphone input, and writing .wav files from samples.

Sonia uses the JSyn plugin by Phil Burk. To use Sonia, you need to install the JSyn browser plugin.

Click here to install the JSyn plugin.

To develop Processing projects with Sonia, visit Amit's web site to download the necessary files:
http://www.pitaru.com/sonia/.

Simple Sample Playback Example.

For more examples, visit Amit's site:
http://www.pitaru.com/sonia/examples.htm.

Working with live video

In processing, we can deal treat a live video image the same way we treat a BImage object. We can draw a frame to the screen (multiple times / at varying scales, etc.) and we can access its pixels.

There are a few things we'll need to do in order to get set-up for video. First, you'll need to have a video camera (duh), connect it to your machine, and install the proper drivers. If you're on a mac, you should be all set. On a PC, however, you'll need to make sure you have quicktime installed (http://www.apple.com/quicktime/). PCs also require a vdig to allow quicktime to take over capturing -- you can get a free one here: http://www.vdig.com/WinVDIG/.

To start the video stream, you'll want to call the beginVideo function. beginVideo takes 3 parameters, the width and height of the captured video and desired frames per second. Note that these are fixed values that cannot be changed as the program runs. Reference links to Processing Site
beginVideo()
endVideo()
videoEvent()
video

  beginVideo(320, 240, 15);

Once you've started capturing, you can then draw the video to the screen just as you would with a BImage. The video variable always contains the most recently captured video frame.


  image(video, 0, 0);

This first example draws the video to the screen at point (0,0) with a width and height determined by mouseX and mouseY. Note the use of "videoEvent" -- while this is not required, this example shows that we are only drawing a new frame when one is available.

source code

This third example shows how we could use the "tint" command to average frames together, creating a motion blur effect.

source code

Computer Vision

Computer vision refers to the the field of research that studies ways computers can gather and interpret visual information (usually via a digital video camera.) We're interested in how techniques employed in computer vision algorithms can be used to enhance screen (and non-screen) based interactive systems. You'll find some resources here:

http://www-2.cs.cmu.edu/~cil/vision.html
http://homepages.inf.ed.ac.uk/rbf/CVonline/
http://www.cs.hmc.edu/~fleck/computer-vision-handbook/

In working with processing, you might also make use of Josh Nimoy's "WebCamXtra", a software library for Processing, Java, and Macromedia Director (Windows and Mac OS X.) WebCamXtra has simple-to-use advanced features, such as color tracking, glob detection, etc. For more information, please visit:

http://webcamxtra.sourceforge.net/

For our purpose, we will look at a two very basic examples of how to track a given color in a video image. Our methods won't be as robust or sophisticated as advance CV programs, but the underlying principle will be the name:

open video stream

use a nested for loop to analyze all pixels of each frame

develop an algorithm/test to evaluate the pixels based on a set of conditions (such as locating the pixel with the brightest color)

Tracking Brightness

This first example checks the brightness of each pixel of each frame, saving the location for the brightest pixel in a variable. It then draws a rectangle at the location of the brightest pixel.

source code

The next example employs the same method as above, however, instead of tracking the brightness of a pixel, it tracks how close any given pixel is to a certain color (the color is determined by the user clicking the mouse on a certain pixel.) To calculate the difference between two colors, we use the following formula (assume two variables of type color, color1 and color2):

float r1 = red(color1); float g1 = green(color1); float b1 = blue(color1);
float r2 = red(color2); float g2 = green(color2); float b2 = blue(color2);
float diff = sqrt(sq(r1 - r2) + sq(g1 - g2) + sq(b1 - b2));

source code

Assignment

Do something with novel input or output using more than keyboard, mouse, screen. Come prepared to talk about what your next steps will be, ie midterm.

back to syllabus