Updated Kinect Library for Processing

Posted on by

A quick post to announce an updated Kinect library for Processing on github. This new library uses the most recent libfreenect drivers (mac os x only still) and builds off of the existing JNI Java Wrapper. Links:

Download library and example:
openkinect.zip

Source:
https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

My tests show 30 FPS no problem for either the depth or RGB image, if you request both images, however, I’m only currently getting about 20 FPS. The update fixes a few bugs, exposes the raw depth data and includes a point cloud demo (see above video).

I have a list of “to-do” items here:

https://github.com/shiffman/libfreenect/issues
(a shortened version of the very long list in my head)

  • http://akamediasystem.com AKA

    Thanks for the library! I was having a weird issue when running the point cloud demo in which I could run it once successfully, but every subsequent attempt to run would fail at:
    int[] depth = kinect.getRawDepth();

    After a little tinkering, I realized that the cause might be that we’re asking for something that’s not ready yet. Adding
    delay(1000);
    to the end of the setup() routine seems to fix the problem.

    Thanks again for making the library – it’s awesome, as is your work in general!

    - AKA

  • http://akamediasystem.com AKA

    Uh, never mind, my suggestion does not seem to remedy the issue.

    Sorry to bug you!

  • http://thunderheadxpler.blogspot.com/ Mansour

    Created a libraries folder in my sketchbook folder and unziped the openkinect.zip
    I can see the RGBDepthTest sample and can load it. However, when I run it I get:
    processing.app.debug.RunnerException: The function enableDepthImage(boolean) does not exist.
    at processing.app.Sketch.placeException(Sketch.java:1543)
    at processing.app.debug.Compiler.compile(Compiler.java:149)
    at processing.app.Sketch.build(Sketch.java:1573)
    at processing.app.Sketch.build(Sketch.java:1553)
    at processing.app.Editor$DefaultRunHandler.run(Editor.java:1485)
    at java.lang.Thread.run(Thread.java:680)

    Any clues ?
    Thanks

  • http://www.shiffman.net Daniel

    @AKA. Try unplugging your kinect and plugging it back in. This fixed it for me (I just updated the library with proper disposal methods so this error shouldn’t happen again.)

    @Mansoour I changed the names of some of the functions. The new library has the corrected code (but it’s just enableDepth() and enableRGB()).

  • Filipe Simões

    Will this be ported to windows?
    Thank you very much. Great job, by the way!

  • Peter

    @AKA

    I don’t know why, but you should add a new command in setup() routine to fix the problem.

    kinect.enableRGB(true);

  • http://fiesefalle.de Bernd

    wow, thanks a lot for this!

    the only problem I have with the sketch using the point cloud example is that I get a “NullPointer Exception” at 80% of times pointing to the line of “int rawDepth = depth[offset];”.

    So I have to restart the sketch a lot to actually get it running. Without this it would be perfect. I would be happy to know if there is anything I can do to address this. Anyhow this is great. Thanks!

  • http://straathof.acadnet.ca/ Gerry Straathof

    Hey! Thanks for sharing. I can’t wait to get started on abusing this system with generative programs and such…

  • http://caseyd.badubadu.com CaseyD

    I found that I resolved AKA’s problem by brutally tossing in

    // added to allow the camera to have a chance to ‘wake up’
    try{
    Thread.currentThread().sleep(2000);
    } catch (InterruptedException ie)
    {
    // uncaught exception how rankerous!
    }

    to setup() in the processing examples.

    I’m very new to Processing, these examples help.

  • Filipe Simões

    I have a strange problem with both the point cloud and the RGBDepthTest. They only work for a few seconds.

    I’m using Mac OS X 10.6.5 in VMWare.

    Any idea what migh be happening?

    Thank you very much.

  • Filipe Simões

    Sorry for bothering you, but just to add to my previous post that I’m having Isochronous transfer error. Can you please point me out what does that mean? Is there a way to prevent it from happening? Thank you very much.

  • Filipe Simões

    I’ve encountered many people complaining about the problem i described in previous posts. The examples only work for a few seconds. I tracked the problem in many sites and it appears to be linked to the usb drivers, mainly in the isochronous properties of it. As suggested somewhere, I changed the usb_libusb10.c.

    FN_WARNING(“Isochronous transfer error: %d\n”, xfer->status);
    libusb_submit_transfer(xfer);
    //strm->dead_xfers ;

    The strange thing about this change is that it works for the glview examples, but it has no change in the processing examples. Anyone can figure out why?
    I was able to use the processing examples but with a ‘not that efficient’ solution (to say the least!) – I use the following sequence at the start of each draw():
    kinect.enableDepth(false);
    kinect.enableDepth(true);

    The FPS drops to 4, but everything keeps going. Of course it is easy to make this sequence happen less often, which should increase FPS, but still, this isn’t the perfect solution. I hope someone comes to it soon. Thanks.

  • Filipe Simoes

    Sorry Shiffman for using your site for these posts, but know I just want to share my solution to the problems I described. I’m just trying to give in return a little of the enormous help I’ve been getting from many places on the web.
    I was using VMWare with 2 cores to run Mac OS X. I just switched the settings for a single core and it worked. Now I get no “Isochronous transfer error”. I hope this saves someone the huge amount of hours such a small detail made me loose.

    Again, thank you very much Shiffman for your work and for sharing.

  • http://www.shiffman.net Daniel

    Hi all, I’ve been out of town (got stranded b/c of the snowstorm) and am just getting back to things here in NYC. If anyone is still having trouble with the library, post an issue on Github and I will take a look!

    https://github.com/shiffman/libfreenect/issues

  • http://www.shiffman.net Daniel

    @AKA and @CaseyD I fixed the NullPointer bug now. It gives you an array of 0′s instead of null while it’s waiting for the camera to wake up.

  • http://www.3dinteractivo.com avillanueva

    Hi Daniel,

    I am going to try to compile your library to Windows 7 64 bits. I do not anticipate problems with the java part, but compiling libfreenect and the rest to create a JNI compatible DLL working under windows might be difficult. Can you please provide some guidance? Thanks

  • http://www.shiffman.net Daniel

    @avillanueva, I would love help with this! In theory, all you need to do is build libfreenect for Windows and reference the appropriate DLLs via JNI. I’ve now done this a few times on Mac, but haven’t recently on Windows. Here are some resources that might help?

    http://java.sun.com/docs/books/jni/html/jniTOC.html

    http://www.inonit.com/cygwin/jni/helloWorld/c.html

    Let me know if there’s anything more I can do to help!

    All my source is here:

    https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

  • Pacman

    Hi Daniel,

    First of all thanks and congratulations for your library. I’m trying to use the library with the examples you provide but each time i tried to execute it I’m getting:

    Exception in thread “Animation Thread” java.lang.UnsatisfiedLinkError: /Users/pacman/Downloads/openkinect2/examples/RGBDepthTest/code/libKinect.jnilib: no suitable image found. Did find: /Users/pacman/Downloads/openkinect2/examples/RGBDepthTest/code/libKinect.jnilib: mach-o, but wrong architecture
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1823)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)
    ….

    Any idea of why can this be happening. I’ve tried to compile the library from your sources but iI’m not able to do that. Any help please?

  • http://www.shiffman.net Daniel

    @pacman, what OS what version of Processing? Are you running it in Eclipse or the Processing IDE? (Try forcing 32-bit mode java in your Java preferences)

  • Damien

    @avillanueva, how are you going with the windows 7 port?

    Love to play with it.

    D

  • Pacman

    Hi Daniel,

    I had an old version of processing. I have updated to the last version and evrything is working now. Thanks for your help. :)

  • jon m

    Hi Daniel,

    I can get the library to work jsut fine in processing, but i am trying to get it to run in Eclipse, i’m forcing 32bit, but still no luck. i have this error….

    Exception in thread “Animation Thread” java.lang.UnsatisfiedLinkError: /Users/pb17/Documents/_Eclipse/Sketch_Kinect/libraries/libKinect.jnilib: no suitable image found. Did find: /Users/pb17/Documents/_Eclipse/Sketch_Kinect/libraries/libKinect.jnilib: mach-o, but wrong architecture

    in the referenced librares section, i have these files,
    libKinect.jnilib
    openkinect.jar

    core.jar
    controlP5.jar

    i think it’s narrowed down to a library linking problem.

    MBP 10.6.5 / processing 1.2.1

    thanks
    jon

  • http://www.shiffman.net Daniel

    Double check you have -d32 in the right run place for the right run configuration. That’s the error you’ll get if you are running in 64 bit!

  • http://www.namtes.com daniel

    Hi,

    I have downloaded the library but it dont work.
    I have a Mac book pro 2010 and I dont understand why
    processing show mw “NO KINECT ADVICES”.

    I would like to use this cool library,
    any help will be very appreciate.

    cheers

    Daniel

  • http://www.jackmerrell.co.uk Jack

    Hi Daniel,
    I have recently been looking at your Kinect lib for Processing, its great! thank you!
    However I was wondering if it would be possible, in the pointcloud sketch to change the sketch from drawing points to draw curve vertex or lines so that you could see contours around an object?
    Many Thanks

  • http://www.facebook.com/profile.php?id=757573572 Jon Mytyk

    yes, exactly the problem… (i was trying to run as an applet, and the config settings were for an application)

    new problem to sort…
    Invalid memory access of location 000000e4 eip=97356a74

  • Anonymous

    Does that problem happen only when you quit? Or is there a particular action performed when it happens (RGB image, IR image, depth data?)

  • http://www.absara.com Lionel

    Hello, mister Shiffman. It is a long chain of requests around the impassioning emergence notion that drives me to your blog. I’m switching a bit from the original angle of your post because this ‘To-do list’ you mention at the end of the article reminds me something I’ve read about organizing one’s time, based on the plasticity of the brain. The promoter of the model is Mark Forster, a British consultant. The reference is here: http://www.cactusforest.net/2011/02/why-mark-forsters-superfocus-is-so-much.html — It is of interest. Have a good day — Warm regards, Lionel

  • Vali

    Hello Daniel, thank you very much for your work. Is there any chance, to get the RGB image transferred to match with the depth image? I tried to look at the Math but I don’t have any clue about C++. This would be sooo helpful. Thank you a lot

  • Anonymous

    Unfortunately, this is not a feature of the library. I hope to work on more features come this summer. you can also check out simple-openni, not sure if this is built into the Processing library yet, but it is part of openni.

    https://code.google.com/p/simple-openni/

  • hanso

    hello daniel! thanks for your great scripts. I´m a pretty rookie right now, could you please explain me how i get things started in procession with you library? the kinect is connected to my mac, processing is opened, do i just need to drag & drop the 2 files ( library folder) into a new sketch and hit run?

  • hanso

    hello daniel! thanks for your great scripts. I´m a pretty rookie right now, could you please explain me how i get things started in procession with you library? the kinect is connected to my mac, processing is opened, do i just need to drag & drop the 2 files ( library folder) into a new sketch and hit run?

  • hanso

    oooups, ok, now im read, i´ve just found your getting started page, thank you!! :)

  • athena

    hello daniel ,.. thanks for your great job ,..

    Can I run this library on windows ???

  • http://twitter.com/shiffman Daniel Shiffman

    Not at the moment unfortunately, sorry!

  • Willharris200

    is there anyway to double out size of the output, or is it restricted to 640×480. I just mean upscaling the output.

  • Anonymous

    The kinect data is by definition 640×480 but you can always use copy() or some other method to upsample.

  • Someone

    I’d like to capture the kinect image and use with the openCV library. But I don’t know how to achieve that :|

  • Andarkmatter

    Hello Daniel,
    I would like to know if it is possible to read C4D files in Processing, I saw you did the opposite reading the Processing info. from the kinect to the C4D, any clue? 

    Thanks 

  • Anonymous

    Try posting on the Processing forum. I don’t have any experience with this, but I’m guessing someone else has tried!

  • Leaphighster

    Hi can the ASUS XTION PRO LIVE work with the code ???

  • http://twitter.com/JordvnKing JORDVN KING

    How do I record the pointcloud info??