Kinect and Processing

Kinect Processing Library in Action

This is all very preliminary, but here is a first pass as a Processing Kinect library:

http://www.shiffman.net/p5/kinect.zip
(Mac OSX only for now, sorry!)
UPDATE (12/18/10): New version of the library can be downloaded from github:

openkinect.zip.

Source: https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

None of this would have been possible without the heroic efforts of Hector Martin, the OpenKinect project, and various members of the openFrameworks community. There’s a great thread with discussion and code here:

http://www.openframeworks.cc/forum/viewtopic.php?f=14&t=4947

Video in action here:

Processing code looks like:

 
import shiffman.kinect.*;
 
PImage img;
PImage depth;
 
 void setup() {
  size(640,240);
  NativeKinect.init();
  img = createImage(640,480,RGB);
  depth = createImage(640,480,RGB);
}
 
 void draw() {
  NativeKinect.update();
  img.pixels = NativeKinect.getPixels();
  img.updatePixels();
 
  depth.pixels = NativeKinect.getDepthMap();
  depth.updatePixels();
 
  image(img,0,0,320,240);
  image(depth,320,0,320,240);
}
  • http://web.ics.purdue.edu/~jkapusta/profile/index.html Jkap

    is there any possible way this works for windows Vista? im a huge processing nerd and want to begin hacking my kinect asap, but can’t find a good way to get the data from my kinect into my processing program, any help or advice would be appreciated. What is NativeKinect? is that the gray image?

  • http://www.shiffman.net Daniel

    I would love help porting to windows! All the source is on github

    https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

  • Meirion

    Also would love to see Kinect running within Processing. Has there been progress with a Windows version?

  • romit

    Hello,

    great library here. any reason why you didn’t implement the tilt control in the library.
    I see the ability there in the Device class in the source.

  • http://www.shiffman.net Daniel

    @romit, check out the new version, it’s in there now!

  • romit

    yep got the new version! it’s great. What do you think about getting more processed values from kinect? It can do some interesting calculations from what I read and it can understand voice commands. I would love to try and make these capabilities available on processing but I don’t know how to compile a processing library. I believe it has to be done in a certain specific way.

  • http://www.shiffman.net Daniel

    I’m definitely looking into adding more features, stay tuned!

  • Chris

    The kinect supposadly returns an 11-bit depth image. I know this is more unwieldy to display, but it would be useful extra resolution. Perhaps you can return a uint16 array as one of the possible calls?

  • http://www.shiffman.net Daniel

    Hi Chris, check out the new version of the library. It has a function called getRawDepth() that gives you the 11-bit data.

    http://www.shiffman.net/p5/kinect

  • jhdgkss

    Hi there iam really looking forward to this. Has there been any further development for a windows version of this lib. And will there be any function to collect points cloud data?

  • http://www.shiffman.net Daniel

    Sadly I need someone else to take on the windows version, don’t have time right now. Point cloud example is included in the new download!

    http://www.shiffman.net/p5/kinect

  • http://drewskillman.com Drew

    Just throwing in a big plus one for interest in a windows version! Thanks!

  • Macv

    Here’s the windows version – use Kinect’s camera as a webcam:
    http://www.e2esoft.cn/kinect/

  • http://drewskillman.com Drew

    I’m a little confused – that looks like the windows version of the kinect drivers, (which i have working!), but I’m not quite sure how to get a version of the openKinect processing library for windows. That doesn’t seem to have been ported over yet as far as I can tell. Maybe i’m missing something?

  • http://drewskillman.com Drew

    I should add that I’m also interested in trying to capture the kinect as a normal webcam, which I see is what you might have been suggesting Macv. Only trick is I’m not sure how to capture that feed in processing. I would normally use opencv.capture(), but I get “Error while starting capture : device 0″ that way.

  • http://www.jankokol.com jan kokol

    it’s a very cool thing.

    a small hack creates a cube cloud and transfers data to a DXF file. open in rhino and have your 3d scan as a model for further processing.

    jan

    ———————————————————–

    // Daniel Shiffman
    // Kinect Point Cloud example
    // http://www.shiffman.net
    // https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

    // Simple DXF Export
    // by Simon Greenwold

    // PeasyCam
    // by Jonathan Feinberg

    // kinect to rhino import via DXF (as box_cloud)
    // by jan kokol

    import org.openkinect.*;
    import org.openkinect.processing.*;
    import peasy.*;

    import processing.dxf.*;
    boolean record= false;

    PeasyCam cam;

    // Kinect Library object
    Kinect kinect;

    float a = 0;

    // Size of kinect image
    int w = 640;
    int h = 480;

    // We’ll use a lookup table so that we don’t have to repeat the math over and over
    float[] depthLookUp = new float[2048];

    void setup() {
    size(screen.width,screen.height,P3D);
    kinect = new Kinect(this);
    kinect.start();
    kinect.enableDepth(true);
    // We don’t need the grayscale image in this example
    // so this makes it more efficient
    kinect.processDepthImage(false);

    cam=new PeasyCam (this,100);
    cam.setMinimumDistance(50);
    cam.setMaximumDistance(500);

    // Lookup table for all possible depth values (0 – 2047)
    for (int i = 0; i < depthLookUp.length; i ) {
    depthLookUp[i] = rawDepthToMeters(i);
    }
    }

    void draw() {

    background(0);
    fill(255);
    textMode(SCREEN);
    text("Kinect FR: " (int)kinect.getDepthFPS() "\nProcessing FR: " (int)frameRate,10,16);

    if (record == true) {
    beginRaw(DXF, "output.dxf");
    }

    if(key=='s'){
    save ("image.jpg");
    }

    // Get the raw depth as array of integers
    int[] depth = kinect.getRawDepth();

    // We're just going to calculate and draw every 4th pixel (equivalent of 160×120)
    int skip = 4;

    // Translate and rotate
    //translate(width/2,height/2,-50);
    //rotateY(a);

    for(int x=0; x<w; x =skip) {
    for(int y=0; y<h; y =skip) {
    int offset = x y*w;

    // Convert kinect data to world xyz coordinate
    int rawDepth = depth[offset];
    PVector v = depthToWorld(x,y,rawDepth);

    stroke(255);
    pushMatrix();
    // Scale up by 200
    float factor = 200;
    translate(v.x*factor,v.y*factor,factor-v.z*factor);
    // Draw a box
    box(1,1,1);
    popMatrix();
    }
    }

    if (record == true) {
    endRaw();
    record= false;
    }

    // Rotate
    //a = 0.015f;
    }

    // These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
    float rawDepthToMeters(int depthValue) {
    if (depthValue < 2047) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 3.3309495161));
    }
    return 0.0f;
    }

    PVector depthToWorld(int x, int y, int depthValue) {

    final double fx_d = 1.0 / 5.9421434211923247e 02;
    final double fy_d = 1.0 / 5.9104053696870778e 02;
    final double cx_d = 3.3930780975300314e 02;
    final double cy_d = 2.4273913761751615e 02;

    PVector result = new PVector();
    double depth = depthLookUp[depthValue];//rawDepthToMeters(depthValue);
    result.x = (float)((x – cx_d) * depth * fx_d);
    result.y = (float)((y – cy_d) * depth * fy_d);
    result.z = (float)(depth);
    return result;
    }

    void stop() {
    kinect.quit();
    super.stop();
    }

    // press 'r' for dxf export
    void keyPressed() {
    if (key == 'r') {
    record = true;
    }
    }

  • EngineerArinG

    I have developed a Java wrapper around PrimeSense’s kinect driver:

    http://sourceforge.net/projects/jkinect/

  • Cris

    I have openkinect library installed, library s examples works but the above (that stars with import shiffman.kinect.*; )  doesn t!
    I get an error      No library found for shiffman.kinect
                              As of release 1.0, libraries must be installed in a folder named ‘libraries’ inside the ‘sketchbook’ folder.
    the package Shiffman does not exist. You might be missing a library

    What is this..?
     Could somebody explain me the difference between    
    import shiffman.kinect.*; and
    import org.openkinect.*;import org.openkinect.processing.*;I m really new with processing, so if this is a stone to your head..please stay calm!

    Cris._

  • JeanPaul

    Hey Daniel, trying the User Example of the SimpleOpenNI Library I found out that if I try to paint an ellipse with (for example) RIGHT_HAND coordinates, it doesn’t paint where the RIGHT hand is, I see and I found out it´s because skeleton’s coordinates aren’t from the same (0,0,0) coordinates Processing has. How can I solve it? 

  • Anonymous

    Check out Greg Borenstein’s examples! They demonstrate how to do this.

    https://github.com/atduskgreg/Making-Things-See-Examples

  • http://www.facebook.com/Tabriz314 Carlos A. Iriarte Martinez

    I have a problem when I try to run te examples. http://imageshack.us/photo/my-images/189/sinttulo3sjf.png/

  • Anonymous

    The library is mac only, unfortunately.  Try SimpleOpenNI.  http://code.google.com/p/simple-openni/

  • http://www.facebook.com/Tabriz314 Carlos A. Iriarte Martinez

    And how I use SimpleOpenNI with the Kinect? Any good tutorial?

  • Anonymous

    Making Things See!  http://shop.oreilly.com/product/0636920020684.do

  • http://twitter.com/Patmcc19 Patrick McCracken

    When I try to run this I get an error saying there is nothing named NativeKinect, and it highlights the NativeKinect.init(): line. Any reason this would be?

  • Anonymous

    That’s the old library.  Take a look at this page instead:

    http://www.shiffman.net/p5/kinect/

  • Jewe

    Thanks a lot for that. It works great. Is there a way to support the “Kinect for Windows” Hardware? Greetings.

  • Alexandre Rangel

    Hi Daniel, can the library work in 64 bits mode, on Processing 2?

  • shiffman

    shoot, that’s a great question. I might have to rebuild it. I am swamped with a million projects, I’d really like to return to this and fix up the library but haven’t had time. If anyone would like to help, let me know! It’s on my list. . .

  • Alexandre Rangel

    Great to know there might be a way! I’ve been using your library and would like to push the projects further, using Processing 2 at 64 bits. Comparing to SimpleOpenNI, one thing that works better for me with your library is the average point tracking, like on this work: http://www.youtube.com/watch?v=JYIjt7Wf3Lc

  • Marco

    Hi Daniel. I was wondering if you know how to set the resolution for the Kinect to 1280×1024? I believe this can be done in RGB video mode in processing but i can’t seem to figure out how to code for this.

    Any help would be much appreciated. Thank you in advance

  • shiffman

    The resolution that comes out is 640×480, you can use the image() function to draw it bigger, i.e.

    image(img,0,0,1280,1024);

    or you can use copy() to actually sample up the pixels.