{"id":14461,"date":"2015-11-17T13:09:26","date_gmt":"2015-11-17T12:09:26","guid":{"rendered":"https:\/\/blog.trifork.com\/?p=14461"},"modified":"2015-11-17T13:09:26","modified_gmt":"2015-11-17T12:09:26","slug":"controlling-java-with-the-leap-motion","status":"publish","type":"post","link":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/","title":{"rendered":"Controlling Java with the Leap Motion"},"content":{"rendered":"<h2 id=\"leap-motion-controller\">Leap Motion Controller<\/h2>\n<p>The <a href=\"https:\/\/www.leapmotion.com\/\">Leap Motion Controller<\/a> is a device that uses two cameras to track the hands and fingers. This makes it possible to use gestures for controlling the computer or applications. It is possible to buy or download applications through the Leap Motion app store, but there is also an <a href=\"https:\/\/developer.leapmotion.com\/\">SDK<\/a> for different languages available to integrate the controller in your own application.<\/p>\n<p>With this article I aim to give an insight in the usability of the Leap Motion Controller in combination with Java. For this I describe the controller and Java API itself and have written an example application which uses the controller. The application is written in Java and is available on <a href=\"https:\/\/github.com\/plaarakkers\/leapcontrol\">github<\/a>.<\/p>\n<h3 id=\"functionality-of-the-controller\">Functionality of the controller<\/h3>\n<p>The basic functionality for the controller and API is working without problems. This makes it possible to make the interaction with devices and computers more intuitive. In the next screenshot an example is shown from the supplied Visualizer application with the detected hands and fingers.<br \/>\n<!--more--><\/p>\n<p><a href=\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone  wp-image-14464\" src=\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png\" alt=\"visualizer\" width=\"422\" height=\"273\" srcset=\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png 300w, https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-1024x661.png 1024w, https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-768x496.png 768w, https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer.png 1282w\" sizes=\"auto, (max-width: 422px) 100vw, 422px\" \/><\/a><\/p>\n<h3 id=\"limitations-of-the-controller\">Limitations of the controller<\/h3>\n<p>The problems arise when the used gestures are too small or there is interference between objects or the hands. Because the controller uses two infrared cameras it is not possible to correctly detect the overlapping hands. The same problems exist for the fingers. If the fingers are not detected correctly the software tries to guess the location of the fingers. This is not always done correctly.<\/p>\n<p>The software also has the possibility of detecting gestures like a swipe or a tap. Because these where limited and not always correctly detected these gestures where not used in my program.<\/p>\n<h2 id=\"java-integration\">Java integration<\/h2>\n<p>The SDK for Java works trough a jar file with JNI connections to OS specific native libraries. These are available trough the SDK so this requires some configuration in the build system like Maven or Gradle for including and releasing the build application. In my application Gradle is used so you can check the configuration for that build system.<\/p>\n<h2 id=\"api\">API<\/h2>\n<h3 id=\"controller\">Controller<\/h3>\n<p>The API starts with an <a href=\"https:\/\/developer.leapmotion.com\/documentation\/java\/api\/Leap.Controller.html\">Controller<\/a> object to connect to the leap motion controller. This object is used to configure the settings like gesture detection and retrieve frames with the detection data from the device.<\/p>\n<p>There are two methods for getting the frames from the controller. The first is adding a subclass of the <a href=\"https:\/\/developer.leapmotion.com\/documentation\/java\/api\/Leap.Listener.html\">Listener<\/a> class to the controller which listens to events dispatched from the controller object. One of these events is the onFrame event which is called when a new frame is available.<\/p>\n<pre class=\"brush:java\">public void onFrame (Controller controller){\n\u00a0\u00a0\u00a0 System.out.println(\"New Frame\");\n}<\/pre>\n<p>There are more events in the Listener like onConnect and onDeviceChange, but they are not important here.<\/p>\n<p>The other way to receive frames is polling of the controller object with the frame method. This retrieves the latest available frame. you can use a parameter to return a number of previous frames. There are a maximum of 60 stored frames. This method is for instance useful for gesture detection.<\/p>\n<pre class=\"brush:java\">Frame frame = controller.frame(); \/\/The latest frame\nFrame previous = controller.frame(1); \/\/The previous frame<\/pre>\n<p>I use the polling method in combination with a TimerTask to retrieve the frames in intervals because the listener method gave problems after retrieving a couple a frames. Because the polling method worked I didn&#8217;t try to get the event based solution working.<\/p>\n<h3 id=\"frame\">Frame<\/h3>\n<p>The <a href=\"https:\/\/developer.leapmotion.com\/documentation\/java\/api\/Leap.Frame.html\">Frame<\/a> represents a snapshot containing the tracked data from the controller. There are a number of methods that are present on most of the classes within the API that do not represent lists of objects.<\/p>\n<pre>public boolean isValid();<\/pre>\n<p>The controller can return invalid frames when returning a frame from the frame method. This is done so the methods on the frame can be called without a null check first. You should still check the isValid result so there is not much difference because an extra check is still needed. An example for when the controller returns an invalid frame is when you try to retrieve a too old frame from the controller.<\/p>\n<pre>public long id();<\/pre>\n<p>The id method returns the id of the current detected object so it is possible to see if the detected object like a hand is the same between different frames. For the frames this can be used to make sure the retrieved frame is different from the last retrieved frame.<\/p>\n<p>The Leapcontrol application makes use of these two methods in the run method of the LeapControlFrameRetrieveTask to check if a new valid frame is retrieved before the frame is processed further.<\/p>\n<pre class=\"brush:java\">\n@Override\n\u00a0\u00a0\u00a0 public void run() {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Frame frame = controller.frame();\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if (frame.isValid() &amp;&amp; frame.id() != lastProcessedFrameId) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 frameHandlers.processFrame(frame);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 lastProcessedFrameId = frame.id();\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\n\u00a0\u00a0\u00a0 }\n<\/pre>\n<p>Besides these methods the Frame gives access to the detected objects like hands and fingers trough the API. A useful method is the method that returns all detected hands in the frame.<\/p>\n<pre>public Handlist hands();<\/pre>\n<p>The <a href=\"https:\/\/developer.leapmotion.com\/documentation\/java\/api\/Leap.Hand.html\">Hand<\/a> objects present in this list have further information like the fingers detected for that hand and methods to indicate if the hand is left or right.<\/p>\n<pre class=\"brush:java\">public FingerList fingers();\n\npublic boolean isLeft();\n\npublic boolean isRight();<\/pre>\n<p>The following image shows the sequence for retrieving the objects beginning from the Controller class and the methods used.<\/p>\n<p><a href=\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/sequence.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-14463\" src=\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/sequence.png\" alt=\"sequence\" width=\"763\" height=\"63\" srcset=\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2015\/11\/sequence.png 763w, https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2015\/11\/sequence-300x25.png 300w\" sizes=\"auto, (max-width: 763px) 100vw, 763px\" \/><\/a><\/p>\n<p>These objects and methods allows us to create motion controlled applications.<\/p>\n<h2 id=\"application\">Application<\/h2>\n<p>The Leapcontrol application itself is an JavaFX application using AWT to create a tray menu for the main interaction with the application. Check the GitHub location for the complete source.The JavaFX functionality is not yet used because there was no need for extra windows at this time.<\/p>\n<p>The start of the main application consists of four main parts<\/p>\n<pre class=\"brush:java\">public class LeapControl extends Application {\n\n\u00a0 \/**\n\u00a0\u00a0 * @param args the command line arguments\n\u00a0\u00a0 *\/\n\u00a0 public static void main(String[] args) {\n(1)\u00a0\u00a0 launch(args);\n\u00a0 }\n\n\u00a0 public void start(Stage stage) throws Exception {\n\u00a0\u00a0\u00a0\u00a0\u00a0 Platform.setImplicitExit(false);\n\n(2)\u00a0\u00a0 FrameHandlers frameHandlers = FrameHandlersFactory.getFrameHandlers();\n\n(3)\u00a0\u00a0 LeapControlTray leapControlTray = new LeapControlTray(stage, frameHandlers);\n\n\u00a0\u00a0\u00a0\u00a0\u00a0 SwingUtilities.invokeLater(leapControlTray::start);\n\n(4)\u00a0\u00a0 Controller controller = new Controller();\n\u00a0\u00a0\u00a0\u00a0\u00a0 controller.setPolicy(Controller.PolicyFlag.POLICY_BACKGROUND_FRAMES);\n\u00a0\u00a0\u00a0\u00a0\u00a0 Arrays.stream(Gesture.Type.values())\n\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0 .forEach(g -&gt; controller.enableGesture(g));\n\n(5)\u00a0\u00a0 TimerTask timerTask = new LeapControlFrameRetrieveTask(controller,\n \u00a0\u00a0\u00a0\u00a0 frameHandlers);\n \u00a0\u00a0\u00a0\u00a0 new Timer(true).schedule(timerTask, new Date(), 100);\n}<\/pre>\n<ol>\n<li>Start call for the JavaFX application.<\/li>\n<li>Initialize the framehandlers with the specific frameactions in a factory and add them to the framehandlers holder.<\/li>\n<li>Initialize the traymenu with the framehandlers holder and create the menu.<\/li>\n<li>Initialize and configure the Leap controller.<\/li>\n<li>Start a timerTask that retreives a frame from the controller and send it to the framehandlers holder.<\/li>\n<\/ol>\n<p>The FrameHandlersFactory constructs the frameHandlers with the different known frameActions and adds them to the FrameHandlers holder.<\/p>\n<pre class=\"brush:java\">\/**\n\u00a0* Factory class for initializing the {@link FrameHandlers}\n\u00a0*\/\npublic class FrameHandlersFactory {\n\n\u00a0\u00a0\u00a0 private static final FrameHandlers frameHandlers = new FrameHandlers();\n\n\u00a0\u00a0\u00a0 private static final Robot robot;\n\n\u00a0\u00a0\u00a0 private FrameHandlersFactory() {\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 static {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 try {\n(1)\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 robot = new Robot();\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 } catch (AWTException e) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 throw new LeapControlRuntimeException(e);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\n(2)\u00a0\u00a0\u00a0\u00a0 frameHandlers.addHandler(\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 new FrameHandler(new LoggingFrameAction(), false));\n(2)\u00a0\u00a0\u00a0\u00a0 frameHandlers.addHandler(\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 new FrameHandler(new CloseApplicationFrameAction(robot)));\n(2)\u00a0\u00a0\u00a0\u00a0 frameHandlers.addHandler(new FrameHandler(\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 new MinimizeApplicationFrameAction(robot)));\n\u00a0\u00a0\u00a0\u00a0 }\n\n \u00a0\u00a0\u00a0 public static FrameHandlers getFrameHandlers() {\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return frameHandlers;\n     }\n}<\/pre>\n<ol>\n<li>Initialize a <a href=\"http:\/\/docs.oracle.com\/javase\/8\/docs\/api\/java\/awt\/Robot.html\">Robot<\/a> to use in the frameactions for generating the keyevents.<\/li>\n<li>Initialize the framehandlers with the different frameactions.<\/li>\n<\/ol>\n<p>The FrameHandlers class is used to hold all the initialized framehandlers.<\/p>\n<pre class=\"brush:java\">\n\/**\n\u00a0* Class that handles a list of {@link FrameHandler}s.\n\u00a0*\/\npublic class FrameHandlers {\n\n\u00a0\u00a0\u00a0 private static final Logger LOGGER =\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0LoggerFactory.getLogger(FrameHandlers.class);\n\n\u00a0\u00a0\u00a0 private final Map&lt;String, FrameHandler&gt; frameHandlers = new HashMap&lt;&gt;();\n\n\u00a0\u00a0\u00a0 public void addHandler(FrameHandler frameHandler) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if (frameHandlers.containsKey(frameHandler.getActionName())) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 LOGGER.error(\"Handler with name {} is already added.\",\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0frameHandler.getActionName());\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 throw new LeapControlRuntimeException(\"Error adding handler.\");\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 frameHandlers.put(frameHandler.getActionName(), frameHandler);\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 \/**\n\u00a0\u00a0\u00a0\u00a0 * Calls the processing of the frame for each framehandler.\n\u00a0\u00a0\u00a0\u00a0 *\n\u00a0\u00a0\u00a0\u00a0 * @param frame The frame to process.\n\u00a0\u00a0\u00a0\u00a0 *\/\n(1) public void processFrame(Frame frame) {\n\u00a0\u00a0\u00a0\u00a0\u00a0 frameHandlers.values()\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0.stream().forEach(f -&gt; processFrameWithErrorHandling(f, frame));\n\u00a0\u00a0\u00a0 }\n\n(2) private void processFrameWithErrorHandling(FrameHandler frameHandler,\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0Frame frame) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 try {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 frameHandler.processFrame(frame);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 } catch (Exception e) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 LOGGER.error(\"Exception while processing frame\", e);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 public Collection&lt;String&gt; getFrameHandlerNames() {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return frameHandlers.values().stream().map(FrameHandler::getActionName)\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0.sorted().collect(Collectors.toList());\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 public boolean isEnabled(String name) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return getFrameHandler(name).isEnabled();\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 public void setEnabled(String name, boolean enabled) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 getFrameHandler(name).setEnabled(enabled);\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 private FrameHandler getFrameHandler(String name) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 FrameHandler frameHandler = frameHandlers.get(name);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if (frameHandler == null) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 throw new LeapControlRuntimeException(\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0String.format(\"No frame handler with name %s found\", name));\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return frameHandler;\n\u00a0\u00a0\u00a0 }\n}\n<\/pre>\n<ol>\n<li>Calls the processFrame of each framehandler using a lambda expression. This is the method that is called from the LeapControlFrameRetrieveTask<\/li>\n<li>Calls the processFrame method of a specific framehandler and handles any exception so it can be used in the lambda expression.<\/li>\n<\/ol>\n<p>The other methods in the FrameHandlers class are used for access to the framehandlers from the tray menu.<\/p>\n<p>The specific processing of the frame for each framehandler is delegated to an implementation of a FrameAction interface that checks if an action is needed based on the information in the retrieved frame and performs this action.<\/p>\n<pre class=\"brush:java\">\/**\n * Action that processes a frame from the {@link Controller}\n *\n *\/\npublic interface FrameAction {\n\n \/**\n * Returns the name for this action. The name should be unique for the list\n *of actions\n *\n * @return The name for the action.\n *\/\n String getName();\n\n void processFrame(Frame frame);\n}\n<\/pre>\n<p>An example of an action in the application is the CloseApplicationFrameAction.<\/p>\n<pre class=\"brush:java\">\n\u00a0*\n\u00a0* {@link FrameAction} that sends a close application call if the conditions\n\u00a0* are met. Conditions: Closed (no more then one extended finger detected)\n\u00a0* right hand detected after open (five extended fingers detected) right hand\n\u00a0* detected.\n\u00a0* Action: Send the ALT + F4 key combination.\n\u00a0*\/\npublic class CloseApplicationFrameAction implements FrameAction {\n\n\u00a0\u00a0\u00a0 private int processedHandId;\n\n\u00a0\u00a0\u00a0 private final Robot robot;\n\n\u00a0\u00a0\u00a0 private int previousExtendedFingersCount;\n\n\u00a0\u00a0\u00a0 public CloseApplicationFrameAction(Robot robot) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 this.robot = robot;\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 @Override\n\u00a0\u00a0\u00a0 public String getName() {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return \"Close application\";\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 @Override\n\u00a0\u00a0\u00a0 public void processFrame(Frame frame) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if (frame.hands().count() == 1) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Hand hand = frame.hands().get(0);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \/\/only process for valid right hands and a different detected hand\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if (hand.isValid() &amp;&amp; hand.isRight()\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0&amp;&amp; hand.id() != processedHandId) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if (hand.fingers().extended().count() &lt;= 1\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0&amp;&amp; previousExtendedFingersCount == 5) {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 performAction();\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 processedHandId = hand.id();\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 previousExtendedFingersCount = 0;\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 } else {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 previousExtendedFingersCount = hand.fingers()\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0.extended().count();\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\n\u00a0\u00a0\u00a0 }\n\n\u00a0\u00a0\u00a0 private void performAction() {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 robot.keyPress(KeyEvent.VK_ALT);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 robot.keyPress(KeyEvent.VK_F4);\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 robot.keyRelease(KeyEvent.VK_F4);\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 robot.keyRelease(KeyEvent.VK_ALT);\n\u00a0\u00a0\u00a0 }\n\n}\n<\/pre>\n<p>This action checks if a right hand is detected where a mostly closed hand is detected when in the previous frame a right hand is detected with all fingers extended. If this is the case then the AWT Robot class is used to send the ALT + F4 key combination and the id of the hand is set to make sure that the same detected hand is not processed multiple times.<\/p>\n<p>The reason that the check is not done on a completely closed hand is so we have an error margin in the detection of the controller in combination with the fingers.<\/p>\n<h2 id=\"conclusion\">Conclusion<\/h2>\n<p>Using the Leap Controller in combination with the SDK it is possible to create motion controlled Java applications. The API has enough methods to get all the needed information to implement needed functionality.<\/p>\n<p>While the use of JNI in combination with platform specific libraries makes cross platform deployments harder Windows, Linux and Mac is supported.<\/p>\n<p>The main problem is that the detection is not perfect and this has to be taken into account when using the controller. Small gestures and gestures where interference from a hand or fingers is possible should not be used.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Leap Motion Controller The Leap Motion Controller is a device that uses two cameras to track the hands and fingers. This makes it possible to use gestures for controlling the computer or applications. It is possible to buy or download applications through the Leap Motion app store, but there is also an SDK for different [&hellip;]<\/p>\n","protected":false},"author":90,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[31,10],"tags":[],"class_list":["post-14461","post","type-post","status-publish","format-standard","hentry","category-java","category-development"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Controlling Java with the Leap Motion - Trifork Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Controlling Java with the Leap Motion - Trifork Blog\" \/>\n<meta property=\"og:description\" content=\"Leap Motion Controller The Leap Motion Controller is a device that uses two cameras to track the hands and fingers. This makes it possible to use gestures for controlling the computer or applications. It is possible to buy or download applications through the Leap Motion app store, but there is also an SDK for different [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/\" \/>\n<meta property=\"og:site_name\" content=\"Trifork Blog\" \/>\n<meta property=\"article:published_time\" content=\"2015-11-17T12:09:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png\" \/>\n<meta name=\"author\" content=\"Perry Laarakkers\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Perry Laarakkers\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/\",\"url\":\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/\",\"name\":\"Controlling Java with the Leap Motion - Trifork Blog\",\"isPartOf\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png\",\"datePublished\":\"2015-11-17T12:09:26+00:00\",\"author\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/401b3e5d4b42af73eb3877baf6c29ce6\"},\"breadcrumb\":{\"@id\":\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#primaryimage\",\"url\":\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png\",\"contentUrl\":\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/trifork.nl\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Controlling Java with the Leap Motion\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/trifork.nl\/blog\/#website\",\"url\":\"https:\/\/trifork.nl\/blog\/\",\"name\":\"Trifork Blog\",\"description\":\"Keep updated on the technical solutions Trifork is working on!\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/trifork.nl\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/401b3e5d4b42af73eb3877baf6c29ce6\",\"name\":\"Perry Laarakkers\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/25aac8fc9e04dcb126328a75758999b8?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/25aac8fc9e04dcb126328a75758999b8?s=96&d=mm&r=g\",\"caption\":\"Perry Laarakkers\"},\"sameAs\":[\"http:\/\/www.trifork.com\"],\"url\":\"https:\/\/trifork.nl\/blog\/author\/perryl\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Controlling Java with the Leap Motion - Trifork Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/","og_locale":"en_US","og_type":"article","og_title":"Controlling Java with the Leap Motion - Trifork Blog","og_description":"Leap Motion Controller The Leap Motion Controller is a device that uses two cameras to track the hands and fingers. This makes it possible to use gestures for controlling the computer or applications. It is possible to buy or download applications through the Leap Motion app store, but there is also an SDK for different [&hellip;]","og_url":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/","og_site_name":"Trifork Blog","article_published_time":"2015-11-17T12:09:26+00:00","og_image":[{"url":"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png","type":"","width":"","height":""}],"author":"Perry Laarakkers","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Perry Laarakkers","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/","url":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/","name":"Controlling Java with the Leap Motion - Trifork Blog","isPartOf":{"@id":"https:\/\/trifork.nl\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#primaryimage"},"image":{"@id":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#primaryimage"},"thumbnailUrl":"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png","datePublished":"2015-11-17T12:09:26+00:00","author":{"@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/401b3e5d4b42af73eb3877baf6c29ce6"},"breadcrumb":{"@id":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#primaryimage","url":"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png","contentUrl":"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/11\/visualizer-300x194.png"},{"@type":"BreadcrumbList","@id":"https:\/\/trifork.nl\/blog\/controlling-java-with-the-leap-motion\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/trifork.nl\/blog\/"},{"@type":"ListItem","position":2,"name":"Controlling Java with the Leap Motion"}]},{"@type":"WebSite","@id":"https:\/\/trifork.nl\/blog\/#website","url":"https:\/\/trifork.nl\/blog\/","name":"Trifork Blog","description":"Keep updated on the technical solutions Trifork is working on!","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/trifork.nl\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/401b3e5d4b42af73eb3877baf6c29ce6","name":"Perry Laarakkers","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/25aac8fc9e04dcb126328a75758999b8?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/25aac8fc9e04dcb126328a75758999b8?s=96&d=mm&r=g","caption":"Perry Laarakkers"},"sameAs":["http:\/\/www.trifork.com"],"url":"https:\/\/trifork.nl\/blog\/author\/perryl\/"}]}},"_links":{"self":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/14461","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/users\/90"}],"replies":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/comments?post=14461"}],"version-history":[{"count":0,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/14461\/revisions"}],"wp:attachment":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/media?parent=14461"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/categories?post=14461"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/tags?post=14461"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}