Kinect and Processing

Kinect Processing Library in Action

This is all very preliminary, but here is a first pass as a Processing Kinect library:

http://www.shiffman.net/p5/kinect.zip
(Mac OSX only for now, sorry!)
UPDATE (12/18/10): New version of the library can be downloaded from github:

openkinect.zip.

Source: https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

None of this would have been possible without the heroic efforts of Hector Martin, the OpenKinect project, and various members of the openFrameworks community. There’s a great thread with discussion and code here:

http://www.openframeworks.cc/forum/viewtopic.php?f=14&t=4947

Video in action here:

Processing code looks like:

 
import shiffman.kinect.*;
 
PImage img;
PImage depth;
 
 void setup() {
  size(640,240);
  NativeKinect.init();
  img = createImage(640,480,RGB);
  depth = createImage(640,480,RGB);
}
 
 void draw() {
  NativeKinect.update();
  img.pixels = NativeKinect.getPixels();
  img.updatePixels();
 
  depth.pixels = NativeKinect.getDepthMap();
  depth.updatePixels();
 
  image(img,0,0,320,240);
  image(depth,320,0,320,240);
}
  • http://web.ics.purdue.edu/~jkapusta/profile/index.html Jkap

    is there any possible way this works for windows Vista? im a huge processing nerd and want to begin hacking my kinect asap, but can’t find a good way to get the data from my kinect into my processing program, any help or advice would be appreciated. What is NativeKinect? is that the gray image?

  • http://www.shiffman.net Daniel

    I would love help porting to windows! All the source is on github

    https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

  • Meirion

    Also would love to see Kinect running within Processing. Has there been progress with a Windows version?

  • romit

    Hello,

    great library here. any reason why you didn’t implement the tilt control in the library.
    I see the ability there in the Device class in the source.

  • http://www.shiffman.net Daniel

    @romit, check out the new version, it’s in there now!

  • romit

    yep got the new version! it’s great. What do you think about getting more processed values from kinect? It can do some interesting calculations from what I read and it can understand voice commands. I would love to try and make these capabilities available on processing but I don’t know how to compile a processing library. I believe it has to be done in a certain specific way.

  • http://www.shiffman.net Daniel

    I’m definitely looking into adding more features, stay tuned!

  • Chris

    The kinect supposadly returns an 11-bit depth image. I know this is more unwieldy to display, but it would be useful extra resolution. Perhaps you can return a uint16 array as one of the possible calls?

  • http://www.shiffman.net Daniel

    Hi Chris, check out the new version of the library. It has a function called getRawDepth() that gives you the 11-bit data.

    http://www.shiffman.net/p5/kinect

  • jhdgkss

    Hi there iam really looking forward to this. Has there been any further development for a windows version of this lib. And will there be any function to collect points cloud data?

  • http://www.shiffman.net Daniel

    Sadly I need someone else to take on the windows version, don’t have time right now. Point cloud example is included in the new download!

    http://www.shiffman.net/p5/kinect

  • http://drewskillman.com Drew

    Just throwing in a big plus one for interest in a windows version! Thanks!

  • Macv

    Here’s the windows version – use Kinect’s camera as a webcam:
    http://www.e2esoft.cn/kinect/

  • http://drewskillman.com Drew

    I’m a little confused – that looks like the windows version of the kinect drivers, (which i have working!), but I’m not quite sure how to get a version of the openKinect processing library for windows. That doesn’t seem to have been ported over yet as far as I can tell. Maybe i’m missing something?

  • http://drewskillman.com Drew

    I should add that I’m also interested in trying to capture the kinect as a normal webcam, which I see is what you might have been suggesting Macv. Only trick is I’m not sure how to capture that feed in processing. I would normally use opencv.capture(), but I get “Error while starting capture : device 0″ that way.

  • http://www.jankokol.com jan kokol

    it’s a very cool thing.

    a small hack creates a cube cloud and transfers data to a DXF file. open in rhino and have your 3d scan as a model for further processing.

    jan

    ———————————————————–

    // Daniel Shiffman
    // Kinect Point Cloud example
    // http://www.shiffman.net
    // https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

    // Simple DXF Export
    // by Simon Greenwold

    // PeasyCam
    // by Jonathan Feinberg

    // kinect to rhino import via DXF (as box_cloud)
    // by jan kokol

    import org.openkinect.*;
    import org.openkinect.processing.*;
    import peasy.*;

    import processing.dxf.*;
    boolean record= false;

    PeasyCam cam;

    // Kinect Library object
    Kinect kinect;

    float a = 0;

    // Size of kinect image
    int w = 640;
    int h = 480;

    // We’ll use a lookup table so that we don’t have to repeat the math over and over
    float[] depthLookUp = new float[2048];

    void setup() {
    size(screen.width,screen.height,P3D);
    kinect = new Kinect(this);
    kinect.start();
    kinect.enableDepth(true);
    // We don’t need the grayscale image in this example
    // so this makes it more efficient
    kinect.processDepthImage(false);

    cam=new PeasyCam (this,100);
    cam.setMinimumDistance(50);
    cam.setMaximumDistance(500);

    // Lookup table for all possible depth values (0 – 2047)
    for (int i = 0; i < depthLookUp.length; i ) {
    depthLookUp[i] = rawDepthToMeters(i);
    }
    }

    void draw() {

    background(0);
    fill(255);
    textMode(SCREEN);
    text("Kinect FR: " (int)kinect.getDepthFPS() "\nProcessing FR: " (int)frameRate,10,16);

    if (record == true) {
    beginRaw(DXF, "output.dxf");
    }

    if(key=='s'){
    save ("image.jpg");
    }

    // Get the raw depth as array of integers
    int[] depth = kinect.getRawDepth();

    // We're just going to calculate and draw every 4th pixel (equivalent of 160×120)
    int skip = 4;

    // Translate and rotate
    //translate(width/2,height/2,-50);
    //rotateY(a);

    for(int x=0; x<w; x =skip) {
    for(int y=0; y<h; y =skip) {
    int offset = x y*w;

    // Convert kinect data to world xyz coordinate
    int rawDepth = depth[offset];
    PVector v = depthToWorld(x,y,rawDepth);

    stroke(255);
    pushMatrix();
    // Scale up by 200
    float factor = 200;
    translate(v.x*factor,v.y*factor,factor-v.z*factor);
    // Draw a box
    box(1,1,1);
    popMatrix();
    }
    }

    if (record == true) {
    endRaw();
    record= false;
    }

    // Rotate
    //a = 0.015f;
    }

    // These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
    float rawDepthToMeters(int depthValue) {
    if (depthValue < 2047) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 3.3309495161));
    }
    return 0.0f;
    }

    PVector depthToWorld(int x, int y, int depthValue) {

    final double fx_d = 1.0 / 5.9421434211923247e 02;
    final double fy_d = 1.0 / 5.9104053696870778e 02;
    final double cx_d = 3.3930780975300314e 02;
    final double cy_d = 2.4273913761751615e 02;

    PVector result = new PVector();
    double depth = depthLookUp[depthValue];//rawDepthToMeters(depthValue);
    result.x = (float)((x – cx_d) * depth * fx_d);
    result.y = (float)((y – cy_d) * depth * fy_d);
    result.z = (float)(depth);
    return result;
    }

    void stop() {
    kinect.quit();
    super.stop();
    }

    // press 'r' for dxf export
    void keyPressed() {
    if (key == 'r') {
    record = true;
    }
    }

  • EngineerArinG

    I have developed a Java wrapper around PrimeSense’s kinect driver:

    http://sourceforge.net/projects/jkinect/

  • Cris

    I have openkinect library installed, library s examples works but the above (that stars with import shiffman.kinect.*; )  doesn t!
    I get an error      No library found for shiffman.kinect
                              As of release 1.0, libraries must be installed in a folder named ‘libraries’ inside the ‘sketchbook’ folder.
    the package Shiffman does not exist. You might be missing a library

    What is this..?
     Could somebody explain me the difference between    
    import shiffman.kinect.*; and
    import org.openkinect.*;import org.openkinect.processing.*;I m really new with processing, so if this is a stone to your head..please stay calm!

    Cris._

  • JeanPaul

    Hey Daniel, trying the User Example of the SimpleOpenNI Library I found out that if I try to paint an ellipse with (for example) RIGHT_HAND coordinates, it doesn’t paint where the RIGHT hand is, I see and I found out it´s because skeleton’s coordinates aren’t from the same (0,0,0) coordinates Processing has. How can I solve it? 

  • Anonymous

    Check out Greg Borenstein’s examples! They demonstrate how to do this.

    https://github.com/atduskgreg/Making-Things-See-Examples

  • http://www.facebook.com/Tabriz314 Carlos A. Iriarte Martinez

    I have a problem when I try to run te examples. http://imageshack.us/photo/my-images/189/sinttulo3sjf.png/

  • Anonymous

    The library is mac only, unfortunately.  Try SimpleOpenNI.  http://code.google.com/p/simple-openni/

  • http://www.facebook.com/Tabriz314 Carlos A. Iriarte Martinez

    And how I use SimpleOpenNI with the Kinect? Any good tutorial?

  • Anonymous

    Making Things See!  http://shop.oreilly.com/product/0636920020684.do

  • http://twitter.com/Patmcc19 Patrick McCracken

    When I try to run this I get an error saying there is nothing named NativeKinect, and it highlights the NativeKinect.init(): line. Any reason this would be?

  • Anonymous

    That’s the old library.  Take a look at this page instead:

    http://www.shiffman.net/p5/kinect/