ITP students Jeff Howard and Alex Vessels are working on a video piece for the upcoming Big Screens show. One of the challenges we’ve always had working on the IAC Video Wall is that the computers that actually drive the wall are locked up in a control room far away from the lobby itself. So any external devices (cameras, sensors, etc.) have to communicate with those machines over the network. For this year’s show, we’ve developed a new solution using UDP streaming to send imagery from a camera to the control room.
This example owes a ton to the Processing UDP Library by Stephane Cousot. I’m using the Java DataGram classes directly instead of the Processing library, but the code draws heavily on techniques from the UDP library.
A couple quick notes about the code.
First, whenever sending data over a network, it’s generally a good idea to try to minimize the amount of data. A 320×240 image has 76,800 pixels, each pixel having 3 bytes (red, green, and blue). That’s 230,400 bytes that need to be sent 30 times per second. It’s do-able, but if we could compress the image first (encoding it as a JPG, for example) before sending, this will save a ton. Encoding a JPG in Java is pretty easy since the ImageIO classes take care of that for you. Assuming you have a PImage variable “img” the code looks something like:
// We need a buffered image to do the JPG encoding int w = img.width; int h = img.height; BufferedImage b = new BufferedImage(w,h,BufferedImage.TYPE_INT_RGB); // Transfer pixels from localFrame to the BufferedImage img.loadPixels(); b.setRGB( 0, 0, w, h, img.pixels, 0, w); // Need these output streams to get image as bytes for UDP ByteArrayOutputStream baStream = new ByteArrayOutputStream(); BufferedOutputStream bos = new BufferedOutputStream(baStream); // JPG compression into BufferedOutputStream // Requires try/catch try { ImageIO.write(b, "jpg", bos); } catch (IOException e) { e.printStackTrace(); } |
You might also notice that the above code includes a try/catch statement around the encoding of the JPG. Certain “exceptions” (i.e. errors) in Java require special handling and try/catch is one way of fulfilling that requirement. You’ll see in the example code that try/catch is also required for the UDP communication. For more about exception handling in Processing check out: http://wiki.processing.org/w/Exceptions.
Finally, you’ll also notice in the examples that you are required to specify the IP address you are sending to and the port, i.e.
int clientPort = 9100; InetAddress client = InetAddress.getByName("localhost"); ds.send(new DatagramPacket(packet,packet.length,client,clientPort)); |
The port can be anything (as long as it doesn’t conflict with a port you are using for something else). Generally, 9000 and above is a safe bet. The IP address should be the IP of the machine receiving the video stream. For demonstration purposes, it’s simply “localhost”, i.e. we’re going to send and receive on the same machine.
Finally, the checkForNewImage() function in the VideoReceiver example is blocking, meaning if no data is coming in Processing will stop and wait. So really, it should exist in a separate thread. Stay tuned for an updated threaded version as well as simple thread tutorial!
Update: the thread example is in the download and more here:
http://www.shiffman.net/2010/11/13/processing-and-threads/
Download the code for sending and receiving: image_streaming.zip.