Java / Processing / OpenCV: camera trigger

Posted by Armin on Sunday, January 02, 2011

Processing icon

How the program works

The software utilizes libraries provided by the Processing language (http://www.processing.org/), as well as the open-source computer vision framework OpenCV (http://ubaa.net/shared/processing/opencv/).

camtriggerScShot.jpg

On my laptop, it runs between 15 and 20 frames/s, so it is not useful for triggering a digital SLR camera to capture fast action, for example. The relatively long delays (about 100ms) don't matter when using high-speed cameras, however. These cameras continuously capture video to an internal, circular buffer. When a trigger is received, it can be set to define the last frame of the recording. The saved video history is on the order of seconds (depending on the resolution and recording rate), so that the 100 ms trigger delay can be ignored.

Source Code

Screenshot
Cam trigger java project folder: Screenshot of how the files are organized in the project, and which libraries are used.

The left image is a link to a screenshot that shows how the various source code and libraries are organized in the project folder created in the Eclipse IDE (http://www.eclipse.org/).

The following external libraries are needed

  1. Core Processing library: http://processing.org/learning/eclipse/
  2. OpenCV: http://ubaa.net/shared/processing/opencv/
  3. Minim audio library: http://code.compartmental.net/tools/minim/
  4. Serial library: http://www.processing.org/reference/libraries/serial/

MeasureMotionController.java - creating the window

The Java code in the listing below creates the window for the main applet, which mainly uses code from the Processing.org library (http://processing.org/). To create the window, it uses Java's AWT ("advanced windowing manager") package.

  1. package org.amphioxus.measuremotionv1;
  2.  
  3. import java.applet.Applet;
  4. import java.awt.BorderLayout;
  5. import java.awt.Frame;
  6. import java.awt.event.WindowAdapter;
  7. import java.awt.event.WindowEvent;
  8.  
  9. public class MeasureMotionController extends Frame {
  10.  
  11. 	private static final long serialVersionUID = 1L;
  12. 	private int w = 640, h = 480;
  13.  
  14. 	public MeasureMotionController() {
  15. 		// call to superclass needs to come first in constructor
  16. 		super("MotionMeasure v.1");
  17. 		// won't allow frame to be resized
  18. 		setResizable(false);
  19. 		// set up frame (which will hold applet)
  20. 		setSize(w, h);
  21. 		setLocationRelativeTo(null);  // centers the frame on the screen
  22.  
  23. 		Applet cam1 = new MeasureMotion();
  24. 		add(cam1, BorderLayout.CENTER);
  25.  
  26. 		// allow window and application to be closed
  27. 		addWindowListener(new WindowAdapter() {
  28. 			public void windowClosing(WindowEvent e) {
  29. //				System.out.println("Bye!");
  30. 				System.exit(0);
  31. 			}
  32. 		});
  33. 		cam1.init();
  34. 	}
  35.  
  36. 	public static void main(String[] args) {
  37. 		MeasureMotionController c = new MeasureMotionController();
  38. 		c.setVisible(true);
  39. 	}
  40. }

MeasureMotion.java - the main applet

The applet defined by MeasureMotion.java above provides the main functionality for the software. It uses the OpenCV framework, which needs to be installed first, to grab frames from the computer's default web camera. Framet captured at time t is first converted to grayscale. It is then compared to a thresholded and slightly blurred version of the frame captured one iteration earlier. By taking the absolute difference of each pixel between framet and framet-1, any differences in the two will appear as white pixels. The last step is to put the current image into memory for the comparison in the next iteration. (see http://ubaa.net/shared/processing/opencv/opencv_absdiff.html).

As defined in the keyPressed() function (around line 127), keys are used to adjust the motion threshold (Increase by 0.1: up arrow; Decrease by 0.1: down arrow). The threshold for the binary threshold operation can be changed in steps of 1 with the "," and "." keys, or in steps of 5 with the "m" and "/" keys.

After initial start-up, the alarm is turned off. It can be toggled on and off by pressing the "a" key. Once the alarm is set (as indicated by the text in the application's interface), a trigger action is elicited every time the motion threshold is crossed. The trigger action is carried out within a thread in which a byte is sent to the serial port. An Arduino attached to the computer's USB port can easily be programmed to react to this serial command with an appropriate action. In addition to the serial communication, a sound clip is played. The sound clip is loaded and played back using the Minim audio library (found at http://code.compartmental.net/tools/minim/).

Line 199 of this listing is where you would place a serial command for talking to an Arduino, for example.

  1. package org.amphioxus.measuremotionv1;
  2.  
  3. import hypermedia.video.OpenCV;
  4. import processing.core.PApplet;
  5. import processing.core.PFont;
  6. import processing.serial.Serial;
  7. import ddf.minim.*;
  8.  
  9. @SuppressWarnings("serial")
  10. public class MeasureMotion extends PApplet {
  11.  
  12. 	private int w = 640, h = 480;
  13. 	OpenCV video;
  14. 	private int rectw = 150;
  15. 	private int rectx = w / 2 - rectw / 2;
  16. 	private int recth = 150;
  17. 	private int recty = h / 2 - recth / 2;
  18.  
  19. 	Minim minim;
  20. 	AudioSample alarmSound;
  21. 	boolean alarmFlag = false;
  22. 	int timer = 15;
  23.  
  24. 	TargetRect targetRect = new TargetRect(rectx, recty, rectw, recth, this); // instance
  25. 	private float imagethreshold = 50;
  26. 	private float motionThresh = 4.f; // above brightness/pixel value for which
  27. 	// alarm goes off
  28.  
  29. 	Serial myPort; // create serial port
  30. 	// Fonts for displaying on applet:
  31. 	PFont f1, f2;
  32.  
  33. 	public MeasureMotion() {
  34. 		// TODO Auto-generated constructor stub
  35. 	}
  36.  
  37. 	public void setup() {
  38. 		size(w,h);
  39. 		background(0);
  40. 		//frameRate(30);
  41. 		noStroke();
  42. 		// set up fonts:
  43. 		f1 = createFont("Monospaced", 28, true); // create display fonts
  44. 		f2 = createFont("Monospaced", 16, true);
  45. 		// set up opencv video object:
  46. 		video = new OpenCV(this);
  47. 		video.capture(w, h);
  48.  
  49. 		minim = new Minim(this);
  50. 		alarmSound = minim.loadSample("alarm.aif", 1411);
  51. 		if ( alarmSound == null ) println("Couldn't load alarmSound!");
  52. 		//		  
  53. 		// set up serial communication:
  54. 		println(Serial.list()); // List all the available serial ports
  55. 		myPort = new Serial(this, Serial.list()[0], 115200);
  56. 	}
  57.  
  58. 	public void draw() {
  59. 		background(0);
  60. 		video.read();
  61. 		video.convert(OpenCV.GRAY);
  62. 		video.absDiff();
  63. 		video.threshold(imagethreshold); // get binary image
  64. 		video.blur(OpenCV.BLUR, 3);
  65. 		video.remember();
  66. 		set(0, 0, video.image());
  67. 		targetRect.measureMotion(); // measure brightness under target rectangle
  68. 		targetRect.draw(); // draw the rectangle
  69. 		textFont(f1,28);
  70. 		textAlign(RIGHT);
  71.  
  72. 		// 	ACTION WHEN MOTION > THRESHOLD:
  73. 		if (targetRect.brightnessTotal > motionThresh) {
  74. 			fill(200, 20, 20, 150); // change text color to red
  75. 			if ((alarmFlag == true) && (stillinactive() == false)) {
  76. 				// send serial command out in separate thread
  77. 				Thread serialTx = new SerialTxThread();
  78. 				serialTx.start();
  79. 				alarmSound.trigger(); // sound the alarm bell
  80. 				timer = 15; // reset timer
  81. 			}
  82. 		} else
  83. 			fill(120, 120, 120, 150);
  84. 		String line1 = nfs(targetRect.brightnessTotal, 3, 2);
  85. 		text("Motion: " + line1, width - 50, height - 50);
  86.  
  87. 		fill(120, 120, 120, 150);
  88. 		String line2 = nfs(frameRate, 2, 3);
  89. 		text("FPS: " + line2, width - 50, height - 20);
  90.  
  91. 		textFont(f2);
  92. 		textAlign(LEFT);
  93. 		String line3 = nfs(motionThresh, 1,1);
  94. 		text("motion threshold: " + line3, 20, height-40);
  95.  
  96. 		String line4 = nfs(imagethreshold, 1,1);
  97. 		text("binary threshold: " + line4, 20, height-20);
  98.  
  99. 		if (alarmFlag == true) text("trigger alarm: ON", 20, height-60);
  100. 		else text("trigger alarm: OFF", 20, height-60);
  101.  
  102. 		// make timer count down each frame:
  103. 		if (timer > 0) {
  104. 			timer -= 1;
  105. 		}
  106. 	}
  107.  
  108. 	public void mousePressed() {
  109. 		rectx = mouseX; // remember x and y coords when mouse is pressed
  110. 		recty = mouseY;
  111. 	}
  112.  
  113. 	public void mouseDragged() {
  114. 		int tempw = mouseX - rectx; // distance from x/y of mouse click to
  115. 		// current position
  116. 		int temph = mouseY - recty;
  117. 		targetRect.update(rectx, recty, tempw, temph);
  118. 	}
  119.  
  120. 	public void mouseReleased() {
  121. 		rectw = mouseX - rectx;
  122. 		recth = mouseY - recty;
  123. 		targetRect.update(rectx, recty, rectw, recth);
  124. 	}
  125.  
  126. 	public void keyPressed() {
  127. 		if( key==CODED ){
  128. 			if( keyCode == UP ){ 
  129. 				if (motionThresh < 5.9) {
  130. 					motionThresh += 0.1f;
  131. 				} 
  132. 			}
  133. 			if( keyCode == DOWN ){ 
  134. 				if (motionThresh >= 0.2) {
  135. 					motionThresh -=  0.1f;
  136. 				} 
  137. 			}
  138. 		} // end if key == coded
  139.  
  140. 		if (key == 'm') {
  141. 			if (imagethreshold > 0) {
  142. 				imagethreshold -= 5;
  143. //				System.out.println("image binary threshold: " + imagethreshold);
  144. 			}
  145. 		}
  146. 		if (key == '/') {
  147. 			if (imagethreshold < 250) {
  148. 				imagethreshold += 5;
  149. //				System.out.println("image binary threshold: " + imagethreshold);
  150. 			}
  151. 		}
  152. 		if (key == ',') {
  153. 			if (imagethreshold > 0) {
  154. 				imagethreshold -= 1;
  155. //				System.out.println("image binary threshold: " + imagethreshold);
  156. 			}
  157. 		}
  158. 		if (key == '.') {
  159. 			if (imagethreshold < 250) {
  160. 				imagethreshold += 1;
  161. //				System.out.println("image binary threshold: " + imagethreshold);
  162. 			}
  163. 		}
  164. 		if (key == 'a'){
  165. 			if (alarmFlag == false) {
  166. 				alarmFlag = true;
  167. //				println("Audio alarm ON");
  168. 			}
  169. 			else {
  170. 				alarmFlag = false;
  171. //				println("Audio alarm OFF");
  172. 			}
  173. 		}
  174. 	}
  175. 	// check whether next action can be triggered:
  176. 	public boolean stillinactive() {
  177. 		if (timer <= 0.0) {
  178. 			return false;
  179. 		} 
  180. 		else {
  181. 			return true;
  182. 		}
  183. 	}
  184.  
  185. 	public void stop() {
  186. 		alarmSound.close();
  187. 		minim.stop();
  188. 		video.stop();
  189. 		println("Program stopped");
  190. 	}
  191.  
  192. 	class SerialTxThread extends Thread {		 
  193.  
  194. 		  SerialTxThread() {
  195. 		  }
  196. 	 // This method is called when the thread runs
  197. 	  public void run() {
  198. 	    println("Thread started when alarm was triggered!!! ");
  199. 	   // communicate with the Arduino out of this thread
  200. 	  }
  201. 	}
  202.  
  203. } // end of MeasureMotion applet class

TargetRect.java: target rectangle for defining ROI

This class defines a region of interest (ROI) under which image motion is measured for the threshold calculation. The ROI can be set by clicking and dragging to place the rectangle.

  1. package org.amphioxus.measuremotionv1;
  2.  
  3. import processing.core.PApplet;
  4.  
  5. public class TargetRect {
  6. 	PApplet parent;
  7.  
  8. 	int x, y, w, h;
  9. 	float brightnessTotal;
  10.  
  11. 	TargetRect(int x, int y, int w, int h, PApplet p) {
  12. 		this.x = x;
  13. 		this.y = y;
  14. 		this.w = w;
  15. 		this.h = h;
  16. 		parent = p;
  17. 	}
  18.  
  19. 	public void draw() {
  20. 		parent.stroke(200, 20, 20);
  21. 		parent.fill(255, 255, 255, 25);
  22. 		parent.rect(x, y, w, h);
  23. 	}
  24.  
  25. 	public void update(int x, int y, int w, int h) {
  26. 		this.x = x;
  27. 		this.y = y;
  28. 		this.w = w;
  29. 		this.h = h;
  30. 	}
  31.  
  32. 	public void measureMotion() {
  33. 		brightnessTotal = 0;
  34.  
  35. 		parent.loadPixels(); // operate on the pixels
  36. 		for (int i = x; i < x + w; i++) {
  37. 			for (int k = y; k < y + h; k++) {
  38. 				int loc = i + k * parent.width;
  39. 				brightnessTotal += parent.brightness(parent.pixels[loc]);
  40. 			}
  41. 		}
  42. 		// average pixel brightness in target rectangle:
  43. 		brightnessTotal = brightnessTotal / (w * h);
  44. 	}
  45. }
Tags: 
electronics
Arduino
videography
photography
Java
code
Category: 
Code