Controlling Java with the Leap Motion
Leap Motion Controller
The Leap Motion Controller is a device that uses two cameras to track the hands and fingers. This makes it possible to use gestures for controlling the computer or applications. It is possible to buy or download applications through the Leap Motion app store, but there is also an SDK for different languages available to integrate the controller in your own application.
With this article I aim to give an insight in the usability of the Leap Motion Controller in combination with Java. For this I describe the controller and Java API itself and have written an example application which uses the controller. The application is written in Java and is available on github.
Functionality of the controller
The basic functionality for the controller and API is working without problems. This makes it possible to make the interaction with devices and computers more intuitive. In the next screenshot an example is shown from the supplied Visualizer application with the detected hands and fingers.
Limitations of the controller
The problems arise when the used gestures are too small or there is interference between objects or the hands. Because the controller uses two infrared cameras it is not possible to correctly detect the overlapping hands. The same problems exist for the fingers. If the fingers are not detected correctly the software tries to guess the location of the fingers. This is not always done correctly.
The software also has the possibility of detecting gestures like a swipe or a tap. Because these where limited and not always correctly detected these gestures where not used in my program.
Java integration
The SDK for Java works trough a jar file with JNI connections to OS specific native libraries. These are available trough the SDK so this requires some configuration in the build system like Maven or Gradle for including and releasing the build application. In my application Gradle is used so you can check the configuration for that build system.
API
Controller
The API starts with an Controller object to connect to the leap motion controller. This object is used to configure the settings like gesture detection and retrieve frames with the detection data from the device.
There are two methods for getting the frames from the controller. The first is adding a subclass of the Listener class to the controller which listens to events dispatched from the controller object. One of these events is the onFrame event which is called when a new frame is available.
public void onFrame (Controller controller){
System.out.println("New Frame");
}
There are more events in the Listener like onConnect and onDeviceChange, but they are not important here.
The other way to receive frames is polling of the controller object with the frame method. This retrieves the latest available frame. you can use a parameter to return a number of previous frames. There are a maximum of 60 stored frames. This method is for instance useful for gesture detection.
Frame frame = controller.frame(); //The latest frame Frame previous = controller.frame(1); //The previous frame
I use the polling method in combination with a TimerTask to retrieve the frames in intervals because the listener method gave problems after retrieving a couple a frames. Because the polling method worked I didn’t try to get the event based solution working.
Frame
The Frame represents a snapshot containing the tracked data from the controller. There are a number of methods that are present on most of the classes within the API that do not represent lists of objects.
public boolean isValid();
The controller can return invalid frames when returning a frame from the frame method. This is done so the methods on the frame can be called without a null check first. You should still check the isValid result so there is not much difference because an extra check is still needed. An example for when the controller returns an invalid frame is when you try to retrieve a too old frame from the controller.
public long id();
The id method returns the id of the current detected object so it is possible to see if the detected object like a hand is the same between different frames. For the frames this can be used to make sure the retrieved frame is different from the last retrieved frame.
The Leapcontrol application makes use of these two methods in the run method of the LeapControlFrameRetrieveTask to check if a new valid frame is retrieved before the frame is processed further.
@Override
public void run() {
Frame frame = controller.frame();
if (frame.isValid() && frame.id() != lastProcessedFrameId) {
frameHandlers.processFrame(frame);
lastProcessedFrameId = frame.id();
}
}
Besides these methods the Frame gives access to the detected objects like hands and fingers trough the API. A useful method is the method that returns all detected hands in the frame.
public Handlist hands();
The Hand objects present in this list have further information like the fingers detected for that hand and methods to indicate if the hand is left or right.
public FingerList fingers(); public boolean isLeft(); public boolean isRight();
The following image shows the sequence for retrieving the objects beginning from the Controller class and the methods used.
These objects and methods allows us to create motion controlled applications.
Application
The Leapcontrol application itself is an JavaFX application using AWT to create a tray menu for the main interaction with the application. Check the GitHub location for the complete source.The JavaFX functionality is not yet used because there was no need for extra windows at this time.
The start of the main application consists of four main parts
public class LeapControl extends Application {
/**
* @param args the command line arguments
*/
public static void main(String[] args) {
(1) launch(args);
}
public void start(Stage stage) throws Exception {
Platform.setImplicitExit(false);
(2) FrameHandlers frameHandlers = FrameHandlersFactory.getFrameHandlers();
(3) LeapControlTray leapControlTray = new LeapControlTray(stage, frameHandlers);
SwingUtilities.invokeLater(leapControlTray::start);
(4) Controller controller = new Controller();
controller.setPolicy(Controller.PolicyFlag.POLICY_BACKGROUND_FRAMES);
Arrays.stream(Gesture.Type.values())
.forEach(g -> controller.enableGesture(g));
(5) TimerTask timerTask = new LeapControlFrameRetrieveTask(controller,
frameHandlers);
new Timer(true).schedule(timerTask, new Date(), 100);
}
- Start call for the JavaFX application.
- Initialize the framehandlers with the specific frameactions in a factory and add them to the framehandlers holder.
- Initialize the traymenu with the framehandlers holder and create the menu.
- Initialize and configure the Leap controller.
- Start a timerTask that retreives a frame from the controller and send it to the framehandlers holder.
The FrameHandlersFactory constructs the frameHandlers with the different known frameActions and adds them to the FrameHandlers holder.
/**
* Factory class for initializing the {@link FrameHandlers}
*/
public class FrameHandlersFactory {
private static final FrameHandlers frameHandlers = new FrameHandlers();
private static final Robot robot;
private FrameHandlersFactory() {
}
static {
try {
(1) robot = new Robot();
} catch (AWTException e) {
throw new LeapControlRuntimeException(e);
}
(2) frameHandlers.addHandler(
new FrameHandler(new LoggingFrameAction(), false));
(2) frameHandlers.addHandler(
new FrameHandler(new CloseApplicationFrameAction(robot)));
(2) frameHandlers.addHandler(new FrameHandler(
new MinimizeApplicationFrameAction(robot)));
}
public static FrameHandlers getFrameHandlers() {
return frameHandlers;
}
}
- Initialize a Robot to use in the frameactions for generating the keyevents.
- Initialize the framehandlers with the different frameactions.
The FrameHandlers class is used to hold all the initialized framehandlers.
/**
* Class that handles a list of {@link FrameHandler}s.
*/
public class FrameHandlers {
private static final Logger LOGGER =
LoggerFactory.getLogger(FrameHandlers.class);
private final Map<String, FrameHandler> frameHandlers = new HashMap<>();
public void addHandler(FrameHandler frameHandler) {
if (frameHandlers.containsKey(frameHandler.getActionName())) {
LOGGER.error("Handler with name {} is already added.",
frameHandler.getActionName());
throw new LeapControlRuntimeException("Error adding handler.");
}
frameHandlers.put(frameHandler.getActionName(), frameHandler);
}
/**
* Calls the processing of the frame for each framehandler.
*
* @param frame The frame to process.
*/
(1) public void processFrame(Frame frame) {
frameHandlers.values()
.stream().forEach(f -> processFrameWithErrorHandling(f, frame));
}
(2) private void processFrameWithErrorHandling(FrameHandler frameHandler,
Frame frame) {
try {
frameHandler.processFrame(frame);
} catch (Exception e) {
LOGGER.error("Exception while processing frame", e);
}
}
public Collection<String> getFrameHandlerNames() {
return frameHandlers.values().stream().map(FrameHandler::getActionName)
.sorted().collect(Collectors.toList());
}
public boolean isEnabled(String name) {
return getFrameHandler(name).isEnabled();
}
public void setEnabled(String name, boolean enabled) {
getFrameHandler(name).setEnabled(enabled);
}
private FrameHandler getFrameHandler(String name) {
FrameHandler frameHandler = frameHandlers.get(name);
if (frameHandler == null) {
throw new LeapControlRuntimeException(
String.format("No frame handler with name %s found", name));
}
return frameHandler;
}
}
- Calls the processFrame of each framehandler using a lambda expression. This is the method that is called from the LeapControlFrameRetrieveTask
- Calls the processFrame method of a specific framehandler and handles any exception so it can be used in the lambda expression.
The other methods in the FrameHandlers class are used for access to the framehandlers from the tray menu.
The specific processing of the frame for each framehandler is delegated to an implementation of a FrameAction interface that checks if an action is needed based on the information in the retrieved frame and performs this action.
/**
* Action that processes a frame from the {@link Controller}
*
*/
public interface FrameAction {
/**
* Returns the name for this action. The name should be unique for the list
*of actions
*
* @return The name for the action.
*/
String getName();
void processFrame(Frame frame);
}
An example of an action in the application is the CloseApplicationFrameAction.
*
* {@link FrameAction} that sends a close application call if the conditions
* are met. Conditions: Closed (no more then one extended finger detected)
* right hand detected after open (five extended fingers detected) right hand
* detected.
* Action: Send the ALT + F4 key combination.
*/
public class CloseApplicationFrameAction implements FrameAction {
private int processedHandId;
private final Robot robot;
private int previousExtendedFingersCount;
public CloseApplicationFrameAction(Robot robot) {
this.robot = robot;
}
@Override
public String getName() {
return "Close application";
}
@Override
public void processFrame(Frame frame) {
if (frame.hands().count() == 1) {
Hand hand = frame.hands().get(0);
//only process for valid right hands and a different detected hand
if (hand.isValid() && hand.isRight()
&& hand.id() != processedHandId) {
if (hand.fingers().extended().count() <= 1
&& previousExtendedFingersCount == 5) {
performAction();
processedHandId = hand.id();
previousExtendedFingersCount = 0;
} else {
previousExtendedFingersCount = hand.fingers()
.extended().count();
}
}
}
}
private void performAction() {
robot.keyPress(KeyEvent.VK_ALT);
robot.keyPress(KeyEvent.VK_F4);
robot.keyRelease(KeyEvent.VK_F4);
robot.keyRelease(KeyEvent.VK_ALT);
}
}
This action checks if a right hand is detected where a mostly closed hand is detected when in the previous frame a right hand is detected with all fingers extended. If this is the case then the AWT Robot class is used to send the ALT + F4 key combination and the id of the hand is set to make sure that the same detected hand is not processed multiple times.
The reason that the check is not done on a completely closed hand is so we have an error margin in the detection of the controller in combination with the fingers.
Conclusion
Using the Leap Controller in combination with the SDK it is possible to create motion controlled Java applications. The API has enough methods to get all the needed information to implement needed functionality.
While the use of JNI in combination with platform specific libraries makes cross platform deployments harder Windows, Linux and Mac is supported.
The main problem is that the detection is not perfect and this has to be taken into account when using the controller. Small gestures and gestures where interference from a hand or fingers is possible should not be used.
