Kinect Hand Cursor for Unity3D

Part 1 – Intro

One of the curitial parts of a game is your UI and it will impact your gameplay a lot.  Today I’m not going to talk about it’s design, but I’m going to show you how to develop a UI system for Kinect. On desktop you can control your UI with your cursor but if you have ever done a game with Kinect, mouse cursor is not an option. For Unity you can come up with different solutions for this, like raycasting and making bunch of prefabs but it’s hard to implement and takes so much time. Thus, I believe there is an easier solution for this, which is of course Unity’s new UI system.

Here is a quick preview;

Before we begin here is the tools I’m using for this tutorial;

• Unity 4.6.6, you can use any of 4.6+ or 5x versions.
• Kinect v2 and you can download it’s drivers and also Unity plugin from here.
• After downloading and importing Unity Kinect plugin, add KinectView folder into your hierarcy.

Thats all you need for your basic setup. You can check the demo scene inside KinectView folder and see if your setup is correct!

Part 2 – Setup for Input Module

Here is a quick summary of what we are going to do:

• Use BaseInputModule class to make our own input module(You can check out StandaloneInputModule or TouchInputModule classes to get an idea what’s going on under the hood! You can find them under EventSystem gameobject after you create your Canvas for the first time.)
• Get tracking body from Kinect and send this data to our KinectInputModule.(We can use example scripts under /KinectView folder.)
• Get the hand or hands from Kinect body, and process their positions and update input module relative to those positions.
• Process the actions like hovering, clicking and dragging.
• Add clicking options like waiting over a button or tracking open and closed states of your hands.

Let’s begin with a basic setup;

• First we are going to make our KinectInputModule class which is going to be a subclass of Unity’s BaseInputModule.

using UnityEngine;
using System.Collections;
using UnityEngine.EventSystems;
using System.Collections.Generic;
using Windows.Kinect;

[AddComponentMenu("Kinect/Kinect Input Module")]
[RequireComponent(typeof(EventSystem))]
public class KinectInputModule : BaseInputModule
{

}

• There are some overridable methods inside BaseInputModule which are ActivateModule()(Start method), Process()(Update method). You can check out documentation for more information abour them. So we can use Process() method to track hovering, pressing or drag actions.

public override void Process()
{
   ProcessHover();
   ProcessPress();
   ProcessDrag();
   ProcessWaitOver();
}

• Before we start coding, we have to consider we are going to track maybe both hands(later more than one body), so we need a class like a holder.

[System.Serializable]
public class KinectInputData
{

}

• And some enumarations for a clean look.

// Handstate will be our default click gesture
public enum KinectUIClickGesture
{
  HandState, WaitOver
}
public enum KinectUIHandType
{
  Right,Left
}

Part 3 – Coding Input Module

• Let’s start with our Class KinectInputData , this is how it’s going to look like (it’s not the final implementation there is a download link at the end!):

[System.Serializable]
public class KinectInputData { // Which hand we are tracking
 public KinectUIHandType trackingHandType = KinectUIHandType.Right;
 // We can normalize camera z position with this
 public float handScreenPositionMultiplier = 5f;
 // Is hand in pressing condition
 private bool _isPressing;
 // Hovering Gameobject, needed for WaitOver like clicking detection
 private GameObject _hoveringObject;

 // Joint type, we need it for getting body's hand world position
 private JointType handType{}...
 // Hovering Gameobject getter setter, needed for WaitOver like clicking detection
 public GameObject HoveringObject{}...
 public HandState CurrentHandState { get; private set; }
 // Click gesture of button
 public KinectUIClickGesture ClickGesture { get; private set; }
 // Is this hand tracking started
 public bool IsTracking { get; private set; }
 // Is this hand over a UI component
 public bool IsHovering { get; set; }
 // Is hand dragging a component
 public bool IsDraging { get; set; }
 // Is hand pressing a button
 public bool IsPressing{}...
 // Global position of tracked hand
 public Vector3 HandPosition { get; private set; }
 // Temporary hand position of hand, used for draging check
 public Vector3 TempHandPosition { get; private set; }
 // Hover start time, used for waitover type buttons
 public float HoverTime { get; set; }
 // Amout of wait over , between 1 - 0 , when reaches 1 button is clicked
 public float WaitOverAmount { get; set; }

 // Must be called for each hand
 public void UpdateComponent(Body body){}...
 // Get hand state data from kinect body
 private HandState GetStateFromJointType(Body body, JointType type){}...
// Get Vector3 position from Joint position
 private Vector3 GetVector3FromJoint(Windows.Kinect.Joint joint){}... }
}

We are going to use this class for tracking seperate hands and make sure you read all the comments, those will give you a good idea about this implementation! KinectInputData has getters like IsTracking, IsHovering, IsPressing and IsDraging which we can use later to visualize a cursor hand. We have set those values from our KinectInputModule. Let’s have look at that:

public class KinectInputModule : BaseInputModule
{
 // Kinect input data, we can set it's size to two for both hands
 public KinectInputData[] _inputData;
 // Scroll or drag start treshold
 [SerializeField]private float _scrollTreshold = .5f;
 // Drag or scroll speed for scrollView
 [SerializeField]private float _scrollSpeed = 3.5f;
 // Wait over time for wait over buttons
 [SerializeField]private float _waitOverTime = 2f;
 // Pointer event data for hand tracking
 PointerEventData _handPointerData;

 // Call this from your Kinect body view
 public void TrackBody(Body body){}...

 // get a pointer event data for a screen position
 private PointerEventData GetLookPointerEventData(Vector3 componentPosition){}...

 public override void Process()
 {
   ProcessHover();
   ProcessPress();
   ProcessDrag();
   ProcessWaitOver();
 }
 // Process waiting over components, if hovererd buttons click type is waitover, process it!
 private void ProcessWaitOver(){}...
 // Process draging, drag starts with closing your hand and end with opening it, sends scroll event to gameobject over hierarcy
 private void ProcessDrag(){}...
 // Process pressing, event click trigered on button by closing and opening hand,sends submit event to gameobject
 private void ProcessPress(){}...

 // Process hovering over component, sends pointer enter exit event to gameobject
 private void ProcessHover(){}...

 // Gets current kinect hand data if tracking it. Used from components like hand cursor
 public KinectInputData GetHandData(KinectUIHandType handType){}...
}

So in this class we have KinectInputData array, it’s lenght is going to be 2 for our right and left hands. And we are going to update it with TrackBody method.

1. With this part of the code, we are updating our each hand components position, currentstate and tracking option:

// Call this from your Kinect body view
public void TrackBody(Body body)
{
   for (int i = 0; i < _inputData.Length; i++)
   {
     // update kinectInput data
     _inputData[i].UpdateComponent(body);
   }
}

//Here is the UpdateComponent method inside KinectInputData:
// Must be called for each hand
public void UpdateComponent(Body body)
{
  HandPosition = GetVector3FromJoint(body.Joints[handType]);
  CurrentHandState = GetStateFromJointType(body, handType);
  IsTracking = true;
}

2. For handling hover, we are going to use BaseInputModule’s HandlePointerExitAndEnter method. The method needs a pointer event data and a gameobject as parameters. For pointereventdata we are geting it from our hand screen position. And the gameobject is raycast result of our pointereventdata. After that we are updating our inputdata’s Hovering properties:

// Process hovering over component, sends pointer enter exit event to gameobject
private void ProcessHover()
{
   for (int i = 0; i < _inputData.Length; i++)
   {
     PointerEventData pointer = GetLookPointerEventData(_inputData[i].GetHandScreenPosition(    ));
     var obj = _handPointerData.pointerCurrentRaycast.gameObject;
     HandlePointerExitAndEnter(pointer, obj);
     // Hover update
     _inputData[i].IsHovering = obj != null ? true : false;
     _inputData[i].HoveringObject = obj;
   }
}

3. For processing press we are going to use a really basic solution which is just tracking our handstate. If our hand is closed we are setting the data as pressed and when we open our hand we send a submit event to UI component if there is any under our cursor:

 // Process pressing, event click trigered on button by closing and opening hand,sends submit event to gameobject
private void ProcessPress()
{
  for (int i = 0; i < _inputData.Length; i++)
  {
     //Check if we are tracking hand state not wait over
     if (!_inputData[i].IsHovering || _inputData[i].ClickGesture != KinectUIClickGesture.HandState) continue;
     // If hand state is not tracked reset properties
     if (_inputData[i].CurrentHandState == HandState.NotTracked)
     {
       _inputData[i].IsPressing = false;
       _inputData[i].IsDraging = false;
     }
     // When we close hand and we are not pressing set property as pressed
     if (!_inputData[i].IsPressing && _inputData[i].CurrentHandState == HandState.Closed)
     {
       _inputData[i].IsPressing = true;
     }
     // If hand state is opened and is pressed, make click action
     else if (_inputData[i].IsPressing && (_inputData[i].CurrentHandState == HandState.Open)
     {
       PointerEventData lookData = GetLookPointerEventData(_inputData[i].GetHandScreenPosition());
       eventSystem.SetSelectedGameObject(null);
       // If we are not draging, and there is a gameobject under cursor click it
       if (lookData.pointerCurrentRaycast.gameObject != null && !_inputData[i].IsDraging)
       {
          GameObject go = lookData.pointerCurrentRaycast.gameObject;
          ExecuteEvents.ExecuteHierarchy(go, lookData, ExecuteEvents.submitHandler);
       }
        _inputData[i].IsPressing = false;
      }
   }
 }

4. You must have noticed there is IsDraging getter in the code above so we have to check if we are draging our hand. We can check it like this:

• When we press a component(with processpress method), set a temporary screen position.
• While processing drag compare current hand position axis values with temporary one.
• If values are bigger than a treshold set IsDraging as true.

 // Process draging, drag starts with closing your hand and end with opening it, sends scroll event to gameobject over hierarcy
private void ProcessDrag()
{
  for (int i = 0; i < _inputData.Length; i++)
  {
   // if not pressing we can't drag
   if (!_inputData[i].IsPressing)continue;
   // Check if we reach drag treshold for any axis, temporary position set when we press an object
   if (Mathf.Abs(_inputData[i].TempHandPosition.x - _inputData[i].HandPosition.x) > _scrollTreshold || Mathf.Abs(_inputData[i].TempHandPosition.y - _inputData[i].HandPosition.y) > _scrollTreshold)
   {
     _inputData[i].IsDraging = true;
   }
   else
   {
     _inputData[i].IsDraging = false;
   }
   // If dragging use unit's eventhandler to send an event to a scrollview like component
   if (_inputData[i].IsDraging)
   {
     PointerEventData lookData = GetLookPointerEventData(_inputData[i].GetHandScreenPosition());
     eventSystem.SetSelectedGameObject(null);
     GameObject go = lookData.pointerCurrentRaycast.gameObject;
     PointerEventData pEvent = new PointerEventData(eventSystem);
     pEvent.dragging = true;
     pEvent.scrollDelta = (_inputData[i].TempHandPosition - _inputData[i].HandPosition) * _scrollSpeed;
     pEvent.useDragThreshold = true;
     ExecuteEvents.ExecuteHierarchy(go, pEvent, ExecuteEvents.scrollHandler);
   }
 }
}

That’s all we need to do to make a Kinect hand cursor.  You can download the full code from here. Have fun with it!

 

If you like this article please support my work on Patreon, thank you!