A Leap Motion Controller is required to use this data provider.
The Leap Motion controller lets you interact directly with digital content on Windows PCs using your bare hands (Legacy support for Mac OS X is also available ) quick setup download the Leap Motion software plug the device into your USB port and you're ready to go Download free apps for desktop and virtual reality on the Leap Motion gallery If you're a developer use the Leap Motion platform. I'm new to this whole Leap Motion thing and I've loved the product so far, but I'm still trying to get over the initial learning curve. I started with the 4.0.0 version of the software (for Windows), but after I couldn't get that to work with any applications, I looked through this forum and saw that 3.2.1 would be more likely to work so that's what I installed next.
The Leap Motion Data Provider enables articulated hand tracking for VR and could be useful for rapid prototyping in the editor. The data provider can be configured to use the Leap Motion Controller mounted on a headset or placed on a desk face up.
This provider can be used in editor and on device while on the Standalone platform. It can also be used in editor while on the UWP platform but NOT in a UWP build.
Leap Motion Unity Modules Versions Supported |
---|
4.5.0 |
4.5.1 |
Using Leap Motion (by Ultraleap) hand tracking in MRTK
- Importing MRTK and the Leap Motion Unity Modules
- Install the Leap Motion SDK 4.0.0 if it is not already installed
- Import the Microsoft.MixedReality.Toolkit.Foundation package into the Unity project.
- Download and import the latest version of the Leap Motion Unity Modules into the project
- Only import the Core package within the Unity Modules
- Integrate the Leap Motion Unity Modules with MRTK
- After the Unity Modules are in the project, navigate to Mixed Reality Toolkit > Leap Motion > Integrate Leap Motion Unity Modules
Note
Integrating the Unity Modules to MRTK adds 10 assembly definitions to the project and adds references to the Microsoft.MixedReality.Toolkit.Providers.LeapMotion assembly definition. Make sure Visual Studio is closed. - Adding the Leap Motion Data Provider
- Create a new Unity scene
- Add MRTK to the scene by navigating to Mixed Reality Toolkit > Add to Scene and Configure
- Select the MixedRealityToolkit game object in the hierarchy and select Copy and Customize to clone the default mixed reality profile.
- Select the Input Configuration Profile
- Select Clone in the input system profile to enable modification.
- Open the Input Data Providers section, select Add Data Provider at the top, a new data provider will be added at the end of the list. Open the new data provider and set the Type to Microsoft.MixedReality.Toolkit.LeapMotion.Input > LeapMotionDeviceManager
- Select Clone to change the default Leap Motion settings.
- The Leap Motion Data Provider contains the
LeapControllerOrientation
property which is the location of the Leap Motion Controller.LeapControllerOrientation.Headset
indicates the controller is mounted on a headset.LeapControllerOrientation.Desk
indicates the controller is placed flat on desk. The default value is set toLeapControllerOrientation.Headset
. - Each controller orientation contains offset properties:
- The Headset orientation offset properties mirror the offset properties in the LeapXRServiceProvider component. The
LeapVRDeviceOffsetMode
has three options: Default, Manual Head Offset and Transform. If the offset mode is Default, then an offset will not be applied to the Leap Motion Controller. The Manual Head Offset mode allows for the modification of three properties:LeapVRDeviceOffsetY
,LeapVRDeviceOffsetZ
andLeapVRDeviceTiltX
. The axis offset property values are then applied to default controller placement. The Transform offset mode contains theLeapVRDeviceOrigin
Transform property which specifies a new origin for the Leap Motion Controller. - The Desk orientation contains the
LeapControllerOffset
property which defines the anchor position of the desk leap hands. The offset is calculated relative to the main camera position and the default value is (0,-0.2, 0.35) to make sure the hands appear in front and in view of the camera.Note
The offset properties in the profile are applied once when the application starts. To modify the values during runtime, get the Leap Motion Service Provider from the Leap Motion Device Manager:
EnterPinchDistance
andExitPinchDistance
are the distance thresholds for pinch/air tap gesture detection. The pinch gesture is calculated by measuring the distance between the index finger tip and the thumb tip. To raise an on input down event, the defaultEnterPinchDistance
is set to 0.02. To raise an on input up event (exiting the pinch), the default distance between the index finger tip and the thumb tip is 0.05.
LeapControllerOrientation
: Headset (Default)LeapControllerOrientation
: Desk - Testing the Leap Motion Data Provider
- After Leap Motion Data Provider has been added to the input system profile, press play, move your hand in front of the Leap Motion Controller and you should see the joint representation of the hand.
- Building your project
- Navigate to File > Build Settings
- Only Standalone builds are supported if using the Leap Motion Data Provider.
- For instructions on how to use a Windows Mixed Reality headset for Standalone builds, see Build and Deploy.
Getting the hand joints
Getting joints using the Leap Motion Data Provider is identical to hand joint retrieval for an MRTK Articulated Hand. For more information, see Hand Tracking.
With MRTK in a unity scene and the Leap Motion Data Provider added as an Input Data Provider in the Input System profile, create an empty game object and attach the following example script.
This script is a simple example of how to retrieve the pose of the palm joint in a Leap Motion Hand. A sphere follows the left Leap hand while a cube follows the right Leap hand.
Unity editor workflow tip
Using the Leap Motion Data Provider does not require a VR headset. Changes to an MRTK app can be tested in the editor with the Leap hands without a headset.
![Applications Applications](/uploads/1/1/8/2/118216552/916390799.png)
The Leap Motion Hands will show up in the editor, without a VR headset plugged in. If the
LeapControllerOrientation
is set to Headset, then the Leap Motion controller will need to be held up by one hand with the camera facing forward.Note
If the camera is moved using WASD keys in the editor and the
LeapControllerOrientation
is Headset, the hands will not follow the camera. The hands will only follow camera movement if a VR headset is plugged in while the LeapControllerOrientation
is set Headset. The Leap hands will follow the camera movement in the editor if the LeapControllerOrientation
is set to Desk.Removing Leap Motion from the Project
- Navigate to the Mixed Reality Toolkit > Leap Motion > Separate Leap Motion Unity Modules
- Let Unity refresh as references in the Microsoft.MixedReality.Toolkit.Providers.LeapMotion.asmdef file are modified in this step
- Close Unity
- Close Visual Studio, if it's open
- Open File Explorer and navigate to the root of the MRTK Unity project
- Delete the UnityProjectName/Library directory
- Delete the UnityProjectName/Assets/Plugins/LeapMotion directory
- Delete the UnityProjectName/Assets/Plugins/LeapMotion.meta file
- Reopen Unity
In Unity 2018.4, you might notice that errors still remain in the console after deleting the Library and the Leap Motion Core Assets.If errors are logged after reopening, restart Unity again.
Common Errors
Leap Motion has not integrated with MRTK
To test if the Leap Motion Unity Modules have integrated with MRTK:
- Navigate to Mixed Reality Toolkit > Utilities > Leap Motion > Check Integration Status
- This will display a pop up window with a message about whether or not the Leap Motion Unity Modules have integrated with MRTK.
- If the message says that the assets have not been integrated:
- Make sure the Leap Motion Unity Modules are in the project
- Make sure that the version added is supported, see the table at the top of the page for versions supported.
- Try Mixed Reality Toolkit > Utilities > Leap Motion > Integrate Leap Motion Unity Modules
Copying assembly Multiplayer HLAPI failed
On import of the Leap Motion Unity Core Assets this error might be logged:
Solution
- A short term solution is to restart Unity. See Issue 7761 for more information.
Leap Motion Example Scene
The example scene uses the DefaultLeapMotionConfiguration profile and determines if the Unity project has been configured correctly to use the Leap Motion Data Provider.
The example scene is contained in the Microsoft.MixedReality.Toolkit.Examples package in the MRTK/Examples/Demos/HandTracking/ directory.
See also
![Leap Motion Applications Leap Motion Applications](/uploads/1/1/8/2/118216552/611201991.jpg)
Last week, we took an in-depth look at how the Leap Motion Controller works, from sensor data to the application interface. Today, we’re digging into our API to see how developers can access the data and use it in their own applications. We’ll also review some SDK fundamentals and great resources you can use to get started.
Frames
At the most basic level, the Leap Motion API returns the tracking data in the form of frames. Each Frame object contains lists of tracked entities, such as hands, fingers, and tools, as well as objects representing recognized gestures and factors describing the overall motion of hands in the scene. Our software also includes a new Image API that allows you to access the raw sensor data.
Positional tracking
At the top of this post, you can see some of the positional tracking data available through the API, as well as how it’s organized. Every tracked entity in the Leap Motion interaction space falls within a hierarchy that starts with the hand. A Hand object includes:
Leap Motion App Store
- palm position and velocity
- direction and normal vectors
- orthonormal basis
- Fingers
- tip position and velocity
- direction vector
- orthonormal basis
- length and width
- Bones
- joint positions
- orthonormal basis
- length and width
- Arm
- wrist and elbow positions
- direction vector
- orthonormal basis
- length and width
- Pointable Tools
- tip position and velocity
- direction vector
- orthonormal basis
- length and width
Motions and gestures
Motionsare continuous hand movements – estimates of how the position of tracked objects (hands, fingers, and tools) change over time. These consist of translation, rotation, and scale. Comparing any two frames containing the same hand allows you to compute the change in motion through time. By mapping motions and position data to their applications, developers have controlled flying drones, conducted orchestras, and designed art installations.
Gestures are movement patterns that can be used to trigger certain actions. The Leap Motion API includes swipe, circle, and tap gestures, which are emitted as event objects in frames when recognized. You can use gestures to swipe through recipes or twirl your finger to watch videos.
V2 tracking
With V2 tracking, our API is now able to provide a wide range of additional tracking data, including left vs. right hands, tracking confidence values, as well as grab and pinch strength. Finger tracking is now persistent (so that each hand always has five fingers), digit types are identified (thumb, index, middle, ring, and pinky), and individual bones and joints are tracked.
Diagnostic visualizer
Whether you’re testing an app or just want to see the tracking data in action while you wiggle your fingers, the Diagnostic Visualizer is a great resource for any developer. Toggle different settings (press ‘H’ to see all options) to get a feel for the Leap Motion tracking system.
The Leap Motion SDK
Once you download our SDK, you can start building right away with extensive libraries, examples, and documentation. It’s available for Windows, Mac, and Linux in six languages:
Our APIs are also extensible to other languages, such as ActionScript3, Flash/AIR, MatLab, Ruby, and more. For native app development, you’ll find a variety of community-contributed wrappers and libraries on our developer portal, as well as full documentation, tutorials and examples. Be sure to check out the v2 gallery for new examples and interaction experiments.
Unity
Want to integrate Leap Motion with your Unity project? Check out our setup guide for Unity Free or Pro and our v2 Skeletal Assets, featuring:
- quick and easy Leap Motion integration for your existing projects
- an interaction engine that allows you to grab, rotate, and scale objects, as well as pass them from hand to hand
- customizable hand models and a rigged hand
- several demo scenes and interactions
LeapJS
The Leap Motion software also includes support for modern browsers through a WebSocket connection. You can create in-browser Leap Motion experiences in JavaScript and CoffeeScript. LeapJS, our JavaScript client library, is hosted on a dedicated CDN using versioned URLs to make it easier for you to develop web apps and faster for those apps to load.
With LeapJS, you can access a powerful and flexible plugin framework to share common code and reduce boilerplate development. In particular, the rigged hand lets you add an onscreen hand to your web app with just a few lines of code. To get started with LeapJS, check out this beginners guide and try out our web examples.
Development and design tips
When developing for a new user interface like the Leap Motion Controller, user experience design can make or break your project. Here are some key resources to help guide you:
Leap Motion Store
From our experience, we’ve found the best development process is rapid and iterative. Start by observing interaction styles in other motion-controlled applications and examples. Try to hone in on what types of interactions are intuitive and easily learnable, and focus on refining these core interactions before adding too much complexity. An effective strategy is to build rapid prototypes (we use Unity and JavaScript with great success), and test with users early and often. Watch and listen to users and carefully observe how their expectations differ from your intended use.
Finally, if you run into any problems, or want to share your project, we hope you’ll join us on the community forums. Happy hacking!