Category Archives: General

General

Night #5: thread() in Processing

Posted on by
3

I just updated the Thread tutorial on the Processing wiki to include a little known function in Processing called thread(). It’s undocumented as of now, but will be a part of the 2.0 documentation. Here’s how it works.

(This following is excerpted from the tutorial).

You are likely familiar with the idea of writing a program that follows a specific sequence of steps — setup() first then draw() over and over and over again! A Thread is also a series of steps with a beginning, a middle, and an end. A Processing sketch is a single thread, often referred to as the “Animation” thread. Other threads sequences, however, can run independently of the main “Processing” sketch. In fact, you can launch any number of threads at one time and they will all run concurrently.

Processing does this all the time, whenever you write an event callback, such as serialEvent(), or captureEvent(), etc. these functions are triggered by a different thread running behind the scenes, and they alert Processing whenever they have something to report. This is useful whenever you need to perform a task that takes too long and would slow down the main animation’s frame rate, such as grabbing data from the network (XML, database, etc.) If a separate thread gets stuck or has an error, the entire program won’t grind to a halt, since the error only stops that individual thread. To create independent, asynchronous threads, you can use the thread() function built into Processing.

void setup() {
  size(200,200);
  // This will tell someFunction() to execute now as a separate thread
  thread("someFunction");
}
 
void draw() {
 
}
 
void someFunction() {
  // This function will run as a thread when called via
  // thread("someFunction") as it was in setup!
}

The thread() function receives a String as an argument. The String should match the name of the function you want to run as a thread. While using the thread() function is a very simple way of getting an independent thread, it is somewhat limited. Being able to make a thread object is a great deal more powerful, and this can be done by extending java’s Thread class.

For more about how to do that, take a look at the full tutorial.

Following is an example draws a loading bar in the “animation” thread that reports progress on another thread(). This is a nice demonstration, however, it would not be necessary in a sketch where you wanted to load data in the background and hide this from the user, allowing the draw() loop to simply continue.

Threads.zip

Night #4: Sorting the Vertices of a Polygon

Posted on by
1

The following problem came up in my ICM course this year. A student was working on a sketch that involved tiling polygons arbitrarily drawn by a user. Allowing a user to set the vertices of a polygon should be easy enough, right? But what if the user does not happen to draw them in a nice clock-wise (or counter clock-wise) order?

Imagine for a moment, you have an ArrayList of PVectors called “vertices.” When the user clicks, the mouse you could add that mouse location to that ArrayList.

void mousePressed() {
  vertices.add(new PVector(mouseX,mouseY));
}

You could then draw that list as a polygon using beginShape() and endShape().

void draw() {
  beginShape();
  for (PVector v : vertices) {
    vertex(v.x, v.y);
  } 
  endShape(CLOSE);
}

But depending on how the user set the vertices, you might end up with:

when what you really want is the following:

One solution for solving this problem is to always sort all of the vertices according to their relative angle from the center. Let’s say you calculate the center of the polygon as the average location of all vertices.

  PVector centroid = new PVector();
  for (PVector v : vertices) {
    centroid.add(v);
  } 
  centroid.div(vertices.size());

You can then make a vector that points from the center to each vertex and get its direction (using PVector’s heading2D() method).

  for (PVector v : vertices) {
    PVector dir = PVector.sub(v, centroid);
    float a = dir.heading2D() + PI;
  }

Note how we add PI to the angle. The heading2D() function will return an angle between -PI and PI and it’ll be easier to sort if we just have an angle between 0 and TWO_PI. One way to sort an ArrayList is called a Selection Sort. In the example below, you’ll find a variation of the selection sort. The code iterates through the ArrayList, finds the element with the highest angle, removes that element and sticks it at the end of a new ArrayList. It does this again and again until the original ArrayList is empty. The result is a new ArrayList in sorted order.

Following is that example which implements all of the above into a class. If you are looking for an exercise, try allowing the user to move or delete existing vertices. Also, how you would make a system of these polygons so that the user can draw more than one?

Source: PolygonVertexSorting.zip

Night #3: Regular Expressions in Processing

Posted on by
1

Several years ago I became somewhat obsessed with regular expressions while reading Jeffrey Friedl’s Mastering Regular Expressions. At the time, I wrote a short tutorial about regular expressions for my course Programming from A to Z. The sad truth is that if you’ve ever done regular expressions in Java, it’s pretty darn awkward compared to, say, python or perl. The good news is there are some nice regex helper functions in Processing that can make it a bit easier. Before we get to that let’s start with the Java API:

  • Pattern — a compiled representation of a regular expression.
  • Matcher — an engine that performs match operations on a character sequence (or String) by interpreting a Pattern.

An example of Pattern and Matcher in Java (which you can write directly into Processing) looks like the following:

String inputtext = "This is a test of regular expressions.";  // Input text
String regex = "test";              // The regular expression to match
Pattern p = Pattern.compile(regex); // Making a Pattern object with the regex 
Matcher m = p.matcher(inputtext);   // Matching that regex in the input string
if (m.find()) {
  System.out.println(m.group());     // Here's the match!
} else {
  System.out.println("No match!");   // No match!
}

Of course, in most cases, you want to do something more sophisticated where you iterate over many matches.

// Regex that matches double words
// Ugh, look at all these double back slashes!!
String regex = "\\\\b(\\\\w+)\\\\b\\\\W+\\\\1";   
 
Pattern p = Pattern.compile(regex);     // Compile Regex
Matcher m = p.matcher(content);         // Create Matcher
while (m.find()) {
  System.out.println(m.group());
}

Processing provides some regex helper functions that wrap all of this Java Pattern/Matcher stuff. They are match() and matchAll().

The match() function is used to apply a regular expression to a piece of text, and return matching groups (elements found inside parentheses) as a String array. If there is no match, the function will return null. If no groups are specified in the regular expression, but the sequence matches, an array of length one (with the matched text as the first element of the array) will be returned.

Here’s an example (this is straight from the reference page).

String s = "Inside a tag, you will find <tag>content</tag>.";
String[] m = match(s, "<tag>(.*?)</tag>");
println("Found '" + m[1] + "' inside the tag.");
// Prints to the console:
// "Found 'content' inside the tag."

The matchAll() function is at first a bit confusing because it returns a two dimensional array. But if you look right back to how match() works, it’s pretty simple. match() assumes you want just one match, and gives you an array, a list of all the groups for that single match. matchAll() assumes you want all the matches, so it gives you a bunch of those arrays, one for every match. What’s an array of an array? A two dimensional array! The first dimension is the match itself, and the second dimension is the group for that match, i.e.

String s = "Some tags! <tag>one</tag> <tag>two</tag> <tag>three</tag>.";
 
// Match this regular expression
String[][] m = matchAll(s, "<tag>(.*?)</tag>");
// Loop through all the matches     
for (int i = 0; i < m.length; i++) {
  // Print out group 1 for each match                
  println("Found '" + m[i][1] + "' inside a tag."); 
}

This new example uses a regex that matches anything inside an HTML href tag and draws it the screen.

Regex.zip

Night #2: Fade Sound In and Out Using Minim

Posted on by
1

I made this example earlier in the semester after seeing countless projects with the following functionality: turning a sound on when a sensor reading reaches a given level, turning the sound off when a sensor reading falls below a certain level. Most the projects used Minim for sound playback and pause() and play() to start and stop a sound, along with rewind() to send the sound back to the beginning. While this does work, I find a more effective strategy is to fade a sound in and out using shiftGain().

Now this is a fairly simple problem if you can pinpoint the moment a sound should fade in. For example, let’s say you want it to happen when the mouse is clicked.

void mousePressed() {
  player.shiftGain(-80, 13, 1000); 
}

The above line of code will fade the sound over 1,000 milliseconds (i.e. 1 second) from a decibel level of -80 (inaudible) to 13 (some vaguely loud level.)

If you put this code in draw(), however, you’ve got a problem.

void draw() {
  if (sensorVal > threshold) {
    player.shiftGain(-80, 13, 1000); 
  }
}

You can’t call shiftGain() over and over again! You want this to happen just once, the moment the sensor value reaches that threshold. Introducing a boolean is a quick way to solve this problem. If you set the boolean to true when the sound is fading and only call shiftGain() when the boolean is set to false, you’ve now got only one call to the function.

boolean fade = false;
 
void draw() {
  if (sensorVal > threshold && !fade) {
    player.shiftGain(-80, 13, 1000); 
    fade = true;
  }
}

The question remains: when does fade get set back to false? One likely solution is to reset the boolean when the sensor value falls below the threshold, i.e.

  if (sensorVal > threshold && !fade) {
    player.shiftGain(-80, 13, 1000); 
    fade = true;
  } else if (sensorVal < threshold) {
    fade = false;
  }

Following is an example that does this with a double threshold (fading up above a value and fading down below a value). In addition, it offers some other improvements (such as having the sound fade from its current gain level). The mouseX location is the stand-in for a sensor value.

Source: SoundFade.zip

Night #1: Zoom and Pan in 2D

Posted on by
8

This came up in my course “Introduction to Computational Media” this year. How does one pan and/or zoom in a 2D Processing world? We could certainly introduce P3D into the mix, but there is a nice, elegant way we can create the effect of panning and zooming and still live in 2D. Here’s how it works.

First, you need to make sure you translate to the center of your window.

  translate(width/2, height/2);

Then you can use the scale() function to scale the world according to a percentage (i.e. 2.0 is 200%, 0.5 is 50%). We’ll use a variable called zoom.

  float zoom = 1.5;  // 150%
  scale(zoom);

Then we can translate additionally to pan according to some offset.

  float offsetX = 100;  // Some arbitrary offset
  float offsetY = 0;
  translate(offsetX,offsetY);

Here is an example (running in processing.js) that allows the user to pan around a design by dragging the mouse, and zoom in and out using key presses.

Source: PanZoom2D.zip

Hanukkah: 8 Nights of Processing

Posted on by
1

Hello world.

I have completely failed to remember I have a blog. Well, I suppose it wasn’t a failure of memory, rather it just fell out of my routine to keep things up-to-date. When you have one of these jobs at an academic institution, one thing they require you to do every so often is put together a big pile of paper that provides evidence you are contributing member of society. The way I’ve always managed to do that in the past is by scanning through all my old posts to find out if I have in fact done anything. According to this blog, I have not existed since April 2011. I’m going to try to remedy that. The first thing I’m going to do is post a list of things I’ve been up to in the last seven or eight months (this is for me and anyone reading should really just stop reading). The second thing I’m going to do (starting tonight) is simultaneously attempt to revive this blog and celebrate the festival of lights by posting eight nights of Processing examples. I’ve made a few new ones for the upcoming Processing 2.0 release as well as helped out students on a number of different problems over the course of the fall semester. So hopefully this can be a place where I can document and share some of these examples in the hopes that they will help others.

So first, the part everyone should feel free to ignore. . .

  • April 2011. I visited my good friend Chris Ault at The College of New Jersey and gave a presentation about the Nature of Code and Most Pixels Ever and saw several demos of student work.
  • June 2011. I attended Seb Lee-Delisle’s Creative JS course in NYC. I made this: fibonacci particles
  • June 17-20 2011. I participated in the Processing 2.0 summit. Ben Fry, Casey Reas, Andres Colubri, Jer Thorp, Patrick Hebron, and I worked together to define the 2.0 release. (Thanks to Casey for the photos below.)

    • Processing 2.0
    Processing 2.0

  • Summer 2011. I participated in Google Summer of Code as a mentor for Processing.
  • September 2011. Launched the “Magic Book” project with Rune Madsen and Miguel Bermudez at ITP (stay tuned for more about this).
  • October 2011. I attended IndieCade in Los Angeles and participated in a panel discussion with Zach Gage, Robert Hodgin, and Casey Reas, and Daniel Shiffman. We discussed projects which use code as a generator of form, and explore the various ways in which procedurally generated elements can contribute to the mechanics, dynamics, and/or aesthetics of video games. (video of the panel).
  • October 2011. Attended Art && Code 3D and was proud to be Greg Borenstein’s helper for his workshop Using the Kinect with Processing. Also really enjoyed Stephen Howell’s Scratch + Kinect workshop.
  • November 2011. Began development on a CakeMix Processing library. Source on github.
  • Fall 2011. Technical edited Greg Borenstein’s upcoming book: Making Things See.
  • December 2011. Gave a presentation at the Neuberger Museum of Art as part of the New Media Lecture Series
  • December 2011. Completing work on Processing 2.0. New examples, new tutorials. Stay tuned for further updates

    • .

Next up, 8 Processing examples!

OpenCV Matching Faces Over Time

Posted on by
12

One of the most common questions I get regarding blob tracking is “memory.” How do I know which blob is which over time? Computer vision libraries, for the most part, simply pass you a list of blobs (with x, y, width, and height properties) for any given moment in time. But the blobs themselves represent only a snapshot of that particular moment and contain no information related to whether the blobs existed before this very moment. This may seem absurd given that as human beings it’s so easy for us to watch a rectangle moving across a screen and understand conceptually that it is the same rectangle. But without additional information (such as color matching, an AR marker, etc.) there’s no way for an algorithm that analyzes one frame of video to know anything about a previous frame. And so we need to apply the same intuitions our brain uses (it was there a moment ago, it’s probably still there now) to our algorithms.

To illustrate one solution to this problem, I’ve created an example that tags an OpenCV face “rectangle” with an ID number and attempts to track that face over time, matching new faces that OpenCV finds with earlier ones. This example is somewhat of an oversimplification whose purpose is to demonstrate a particular technique — a new face is the same as the previous one that was closest to it. But there are certainly additional and more sophisticated ways that the match could be made. In addition, it’s likely useful to add some interpolation to the face’s movement and size changes so that it appears less jittery.

First, we need to establish our own Face class. OpenCV just gives us a new array of Rectangle objects every frame so we need our own Face object that persists (in an ArrayList).

class Face {
 
  // A Rectangle
  Rectangle r;
 
  // Am I available to be matched?
  boolean available;
 
  // Should I be deleted?
  boolean delete;
 
  // How long should I live if I have disappeared?
  int timer = 127;
 
  // Assign a number to each face
  int id;

Our main program then needs an ArrayList to keep track of the Face objects that currently exist:

ArrayList faceList;

Finally, in draw(), OpenCV gives us an array of Rectangle objects, the faces it currently sees.

  Rectangle[] faces = opencv.detect();

It’s our job to match these with any Face objects we have in our ArrayList. The way I see it, there are three scenarios.

1) We have nothing in our Face ArrayList. In this case, we add a new Face object for every single Rectangle in the faces array, i.e.

  if (faceList.isEmpty()) {
    // Just make a Face object for every face Rectangle
    for (int i = 0; i < faces.length; i++) {
      faceList.add(new Face(faces[i].x,faces[i].y,faces[i].width,faces[i].height));
    }

2) OpenCV found more faces than we currently have in our list. In this case, we need to match our current Face objects with an OpenCV Rectangle and then add new Face objects for any remaining Rectangles.

} else if (faceList.size() < = faces.length) {
    boolean[] used = new boolean[faces.length];
    // Match existing Face objects with a Rectangle
    for (Face f : faceList) {
       // Find faces[index] that is closest to face f
       // set used[index] to true so that it can't be used twice
       float record = 50000;
       int index = -1;
       for (int i = 0; i < faces.length; i++) {
         float d = dist(faces[i].x,faces[i].y,f.r.x,f.r.y);
         if (d < record && !used[i]) {
           record = d;
           index = i;
         } 
       }
       // Update Face object location
       used[index] = true;
       f.update(faces[index]);
    }

Notice how in the above code boolean variables are used to keep track of which Rectangles have already been matched. We don't want two Face objects to think they are the same face!

3) Finally, the third scenario is that we have more Face objects than OpenCV has found. In this case, we need to match our existing Face objects and then mark any leftover ones for deletion.

  } else {
    // All Face objects start out as available
    for (Face f : faceList) {
      f.available = true;
    } 
    // Match Rectangle with a Face object
    for (int i = 0; i < faces.length; i++) {
      // Find face object closest to faces[i] Rectangle
      // set available to false
       float record = 50000;
       int index = -1;
       for (int j = 0; j < faceList.size(); j++) {
         Face f = faceList.get(j);
         float d = dist(faces[i].x,faces[i].y,f.r.x,f.r.y);
         if (d < record && f.available) {
           record = d;
           index = j;
         } 
       }
       // Update Face object location
       Face f = faceList.get(index);
       f.available = false;
       f.update(faces[i]);
    }

Full source is here: whichface.zip

Rotate a Vector (Processing.js!)

Posted on by
5

The question of how to rotate a PVector object (the data of the vector itself, I’m not talking about rotating while drawing) came up in my nature of code course yesterday. To do this, you’ll need to convert the vector to polar coordinates (radius + angle), adjust the angle, and the convert it back to cartesian to solve for the components (x and y). A function would look something like:

// Rotate a vector in 2D
void rotate2D(PVector v, float theta) {
  // What's the magnitude?
  float m = v.mag();
  // What's the angle?
  float a = v.heading2D();
 
  // Change the angle
  a += theta;
 
  // Polar to cartesian for the new xy components
  v.x = m * cos(a);
  v.y = m * sin(a);
}

(Update thanks to Vilhelm’s comment below: You can also use a 2D rotation matrix!)

void rotate2D(PVector v, float theta) {
  float xTemp = v.x;
  v.x = v.x*cos(theta) - v.y*sin(theta);
  v.y = xTemp*sin(theta) + v.y*cos(theta);
}

Now, this is really just a ruse. A big excuse for me to figure out how to get a Processing example in a wordpress post using processing.js! I was able to do this fairly quickly with three quick steps:

1) Download processing.js 1.0.
2) Install wordpress processing.js plug-in.
3) Update the plug-in to use processing.js 1.0 (it hasn’t been yet). Follow these instructions.

And here it is!

And now the source:

// Rotate2D function
// Daniel Shiffman
 
void setup() {
  size(200,200);
  smooth();
}
 
void draw() {
  background(100);
  PVector xaxis = new PVector(75,0);
  float theta = map(mouseX,0,width,0,TWO_PI);
  rotate2D(xaxis,theta);
  drawVector(xaxis,width/2,height/2,1.0);
}
 
// Rotate a vector in 2D
void rotate2D(PVector v, float theta) {
  // What's the magnitude?
  float m = v.mag();
  // What's the angle?
  float a = v.heading2D();
 
  // Change the angle
  a += theta;
 
  // Polar to cartesian for the new xy components
  v.x = m * cos(a);
  v.y = m * sin(a);
}
 
// Renders a vector object 'v' as an arrow at a location xy
void drawVector(PVector v, float x, float y, float scayl) {
  pushMatrix();
  float arrowsize = 4;
  // Translate to location to render vector
  translate(x,y);
  stroke(255);
  // Vector heading to get direction (pointing up is a heading of 0)
  rotate(v.heading2D());
  // Scale it to be bigger or smaller if necessary
  float len = v.mag()*scayl;
  // Draw three lines to make an arrow
  line(0,0,len,0);
  line(len,0,len-arrowsize,+arrowsize/2);
  line(len,0,len-arrowsize,-arrowsize/2);
  popMatrix();
}