Jump to content

How to do Augmented Reality in Processing

+ 5
  odewahn1's Photo
Posted Apr 12 2010 01:33 PM

"Augmented Reality" (AR) describes a class of cloud-backed, sensor-based user interfaces. Google Goggles, for example, uses your phone's camera and location sensors to call the google search service. The results are then superimposed over the original image to augment the picture with additional metadata.

Superimposing metadata on top of a QR Code, a type of 2-dimensional bar code, is another common AR pattern. This Answer will show you how to use Processing to use the ZXing library to create AR programs that look something like this:



Before jumping into the code, though, a bit of background. A QR code has two basic parts -- the 3 positioning elements, which are the large square blocks at the three corners, and the data elements, which is everything else. The following image should give you the basic idea:

Attached Image

You can generate a QR code from a variety of sites, like the general purpose Kaywa or the more URL specific site goo.gl. Once you have the code, you need a program that can read it from an image file and decode its position and information content.

The XZing library provides this capability. While it's an external library and is a bit tricky to set up, it is super fast and accurate, and is well-worth the extra time. Here's how:

  • You'll need to download the ZIP file from ZXing project page.
  • Unpack it into some directory that I'll call ZXHOME.
  • Drop into a terminal, and then cd into the ZXHOME/core. Run ant to build the .jar file. Then change into ZXHOME/javase. Run ant again to build the javase.jar file.
  • Create a new Processing sketch. Use the "Sketch -> Add File" menu and add both core.jar and javase.jar files to the project. You'll need to add both these files to every project in which you use ZXing.


Once you've added the files, you can start writing the sketch. There are a few wrinkles to using it (for example, you have to add the full path name to the various Reader classes), but the following Sketch should give you the basic idea of how to use the library.

Basically, it uses the video library to capture the image from your system's default webcam, and then uses ZXing to check for the presence of a QR Code that has an encoded ISBN for an O'Reilly book. If it detects one, it will pull the books image from our cover library and superimpse it over Position Indicator 0. One other thing to note -- you'll need to create a font in order to see the position indicator numbers. To do this, select "Tools>Create Font." Choose SanSerif with a size of 48 and make sure the filename is "SansSerif-48" (it will automatically add the vlw extension.)

While the code is simple, it should give you most of what you need for your own experiments:

import processing.video.*;
import com.google.zxing.*;
import java.awt.image.BufferedImage;

Capture cam; //Set up the camera
com.google.zxing.Reader reader = new com.google.zxing.MultiFormatReader();

int WIDTH = 640;
int HEIGHT = 480;

PImage cover; //This will have the cover image
String lastISBNAcquired = ""; //This is the last ISBN we acquired

// Grabs the image file 

void setup() {
 size(640, 480);
 PFont metaBold;
 // The font "Meta-Bold.vlw" must be located in the 
 // current sketch's "data" directory to load successfully
 metaBold = loadFont("SansSerif-48.vlw");
 textFont(metaBold, 36); 

 cam = new Capture(this, WIDTH, HEIGHT);
}
 

void draw() {
 if (cam.available() == true) {
 cam.read(); 
 image(cam, 0,0);
 try {
 //Create a bufferedimage
 BufferedImage buf = new BufferedImage(WIDTH,HEIGHT, 1); // last arg (1) is the same as TYPE_INT_RGB
 buf.getGraphics().drawImage(cam.getImage(),0,0,null);
 // Now test to see if it has a QR code embedded in it
 LuminanceSource source = new BufferedImageLuminanceSource(buf);
 BinaryBitmap bitmap = new BinaryBitmap(new HybridBinarizer(source)); 
 Result result = reader.decode(bitmap); 
 //Once we get the results, we can do some display
 if (result.getText() != null) { 
 println(result.getText());
 ResultPoint[] points = result.getResultPoints();
 //Draw some ellipses on at the control points
 for (int i = 0; i < points.length; i++) {
 fill(#ff8c00);
 ellipse(points[i].getX(), points[i].getY(), 20,20);
 fill(#ff0000);
 text(i, points[i].getX(), points[i].getY());
 }
 //Now fetch the book cover, if it is found
 if (!result.getText().equals(lastISBNAcquired)) {
 String url = "http://covers.oreilly.com/images/" + result.getText() + "/cat.gif";
 try {
 cover = loadImage(url,"gif");
 lastISBNAcquired = result.getText();
 } catch (Exception e) {
 cover = null;
 }
 }
 //Superimpose the cover on the image
 if (cover != null) {
 image(cover, points[1].getX(), points[1].getY());
 }
 }
 } catch (Exception e) {
// println(e.toString()); 
 }
 }
}


Enjoy, and be sure to post any interesting projects you do back here. Also, we'd love to here if you'd use an AR-registration system for your books...

Getting Started with Processing

Learn more about this topic from Getting Started with Processing.

Learn computer programming the easy way with Processing, a simple language that lets you use code to create drawings, animation, and interactive graphics. Programming courses usually start with theory, but this book lets you jump right into creative and fun projects. It's ideal for anyone who wants to learn basic programming, and serves as a simple introduction to graphics for people with some programming skills.

See what you'll learn


5 Replies

 : Apr 12 2010 02:28 PM
This is great, thanks! I also had to do one more step:

In Processing, choose Tools>Create Font. Choose SanSerif with a size of 48 and make sure the filename is "SansSerif-48" (it will automatically add the vlw extension.
 : Apr 12 2010 02:58 PM
I think you might be able to speed things up slightly. It looks like cam.getImage() returns a BufferedImage, so you should be able to do away with creating a buffered image and instead do:

       LuminanceSource source = new BufferedImageLuminanceSource((BufferedImage)cam.getImage());
       BinaryBitmap bitmap = new BinaryBitmap(new HybridBinarizer(source));


Also, if you know you'll only be reading QR Codes, try initializing a QRCodeReader instead of a MultiFormatReader:

com.google.zxing.Reader reader = new com.google.zxing.qrcode.QRCodeReader();

+ 1
  odewahn1's Photo
Posted Apr 13 2010 05:46 AM

Thanks, Brian. These are great improvements, and made this notably faster. (You can tell because there is a much shorter lag in the camera's image updates.) Here's the revised code that incorporates Brian's suggestions:

import processing.video.*;
import com.google.zxing.*;
import java.awt.image.BufferedImage;

Capture cam; //Set up the camera
com.google.zxing.Reader reader = new com.google.zxing.qrcode.QRCodeReader();

int WIDTH = 640;
int HEIGHT = 480;

PImage cover;  //This will have the cover image
String lastISBNAcquired = "";  //This is the last ISBN we acquired

// Grabs the image file    

void setup() {
  size(640, 480);
    PFont metaBold;
  // The font "Meta-Bold.vlw" must be located in the 
  // current sketch's "data" directory to load successfully
  metaBold = loadFont("SansSerif-48.vlw");
  textFont(metaBold, 36); 

  cam = new Capture(this, WIDTH, HEIGHT);
}
 

void draw() {
  if (cam.available() == true) {
    cam.read();    
    image(cam, 0,0);
    try {
       // Now test to see if it has a QR code embedded in it
       LuminanceSource source = new BufferedImageLuminanceSource((BufferedImage)cam.getImage());
       BinaryBitmap bitmap = new BinaryBitmap(new HybridBinarizer(source));       
       Result result = reader.decode(bitmap); 
       //Once we get the results, we can do some display
       if (result.getText() != null) { 
          println(result.getText());
          ResultPoint[] points = result.getResultPoints();
          //Draw some ellipses on at the control points
          for (int i = 0; i < points.length; i++) {
            fill(#ff8c00);
            ellipse(points[i].getX(), points[i].getY(), 20,20);
            fill(#ff0000);
            text(i, points[i].getX(), points[i].getY());
          }
          //Now fetch the book cover, if it is found
          if (!result.getText().equals(lastISBNAcquired)) {
             String url = "http://covers.oreilly.com/images/" + result.getText() + "/cat.gif";
             try {
                cover = loadImage(url,"gif");
                lastISBNAcquired = result.getText();
             } catch (Exception e) {
               cover = null;
             }
          }
          //Superimpose the cover on the image
          if (cover != null) {
            image(cover, points[1].getX(), points[1].getY());
          }
       }
    } catch (Exception e) {
//         println(e.toString()); 
    }
  }
}

 : May 15 2010 02:57 PM
Oh.. its 'Processing'! :-)
 : Feb 05 2013 03:37 AM
Great demo.

Is it possible to port this to an Android app? If so how? Are there any pitfalls in the code used?

Could be most useful in book shops...