Kinect Hand Cursor for Unity3D

Part 1 – Intro

One of the curitial parts of a game is your UI and it will impact your gameplay a lot.  Today I’m not going to talk about it’s design, but I’m going to show you how to develop a UI system for Kinect. On desktop you can control your UI with your cursor but if you have ever done a game with Kinect, mouse cursor is not an option. For Unity you can come up with different solutions for this, like raycasting and making bunch of prefabs but it’s hard to implement and takes so much time. Thus, I believe there is an easier solution for this, which is of course Unity’s new UI system.

Here is a quick preview;

Before we begin here is the tools I’m using for this tutorial;

  • Unity 4.6.6, you can use any of 4.6+ or 5x versions.
  • Kinect v2 and you can download it’s drivers and also Unity plugin from here.
  • After downloading and importing Unity Kinect plugin, add KinectView folder into your hierarcy.

Thats all you need for your basic setup. You can check the demo scene inside KinectView folder and see if your setup is correct!

Part 2 – Setup for Input Module

Here is a quick summary of what we are going to do:

  • Use BaseInputModule class to make our own input module(You can check out StandaloneInputModule or TouchInputModule classes to get an idea what’s going on under the hood! You can find them under EventSystem gameobject after you create your Canvas for the first time.)
  • Get tracking body from Kinect and send this data to our KinectInputModule.(We can use example scripts under /KinectView folder.)
  • Get the hand or hands from Kinect body, and process their positions and update input module relative to those positions.
  • Process the actions like hovering, clicking and dragging.
  • Add clicking options like waiting over a button or tracking open and closed states of your hands.

Let’s begin with a basic setup;

  • First we are going to make our KinectInputModule class which is going to be a subclass of Unity’s BaseInputModule.
using UnityEngine;
using System.Collections;
using UnityEngine.EventSystems;
using System.Collections.Generic;
using Windows.Kinect;

[AddComponentMenu("Kinect/Kinect Input Module")]
[RequireComponent(typeof(EventSystem))]
public class KinectInputModule : BaseInputModule
{

}
public override void Process()
{
   ProcessHover();
   ProcessPress();
   ProcessDrag();
   ProcessWaitOver();
}
  • Before we start coding, we have to consider we are going to track maybe both hands(later more than one body), so we need a class like a holder.
[System.Serializable]
public class KinectInputData
{

}
  • And some enumarations for a clean look.
// Handstate will be our default click gesture
public enum KinectUIClickGesture
{
  HandState, WaitOver
}
public enum KinectUIHandType
{
  Right,Left
}

Part 3 – Coding Input Module

  • Let’s start with our Class KinectInputData , this is how it’s going to look like (it’s not the final implementation there is a download link at the end!):
[System.Serializable]
public class KinectInputData { // Which hand we are tracking
 public KinectUIHandType trackingHandType = KinectUIHandType.Right;
 // We can normalize camera z position with this
 public float handScreenPositionMultiplier = 5f;
 // Is hand in pressing condition
 private bool _isPressing;
 // Hovering Gameobject, needed for WaitOver like clicking detection
 private GameObject _hoveringObject;

 // Joint type, we need it for getting body's hand world position
 private JointType handType{}...
 // Hovering Gameobject getter setter, needed for WaitOver like clicking detection
 public GameObject HoveringObject{}...
 public HandState CurrentHandState { get; private set; }
 // Click gesture of button
 public KinectUIClickGesture ClickGesture { get; private set; }
 // Is this hand tracking started
 public bool IsTracking { get; private set; }
 // Is this hand over a UI component
 public bool IsHovering { get; set; }
 // Is hand dragging a component
 public bool IsDraging { get; set; }
 // Is hand pressing a button
 public bool IsPressing{}...
 // Global position of tracked hand
 public Vector3 HandPosition { get; private set; }
 // Temporary hand position of hand, used for draging check
 public Vector3 TempHandPosition { get; private set; }
 // Hover start time, used for waitover type buttons
 public float HoverTime { get; set; }
 // Amout of wait over , between 1 - 0 , when reaches 1 button is clicked
 public float WaitOverAmount { get; set; }

 // Must be called for each hand
 public void UpdateComponent(Body body){}...
 // Get hand state data from kinect body
 private HandState GetStateFromJointType(Body body, JointType type){}...
// Get Vector3 position from Joint position
 private Vector3 GetVector3FromJoint(Windows.Kinect.Joint joint){}... }
}

We are going to use this class for tracking seperate hands and make sure you read all the comments, those will give you a good idea about this implementation! KinectInputData has getters like IsTracking, IsHovering, IsPressing and IsDraging which we can use later to visualize a cursor hand. We have set those values from our KinectInputModule. Let’s have look at that:

public class KinectInputModule : BaseInputModule
{
 // Kinect input data, we can set it's size to two for both hands
 public KinectInputData[] _inputData;
 // Scroll or drag start treshold
 [SerializeField]private float _scrollTreshold = .5f;
 // Drag or scroll speed for scrollView
 [SerializeField]private float _scrollSpeed = 3.5f;
 // Wait over time for wait over buttons
 [SerializeField]private float _waitOverTime = 2f;
 // Pointer event data for hand tracking
 PointerEventData _handPointerData;

 // Call this from your Kinect body view
 public void TrackBody(Body body){}...

 // get a pointer event data for a screen position
 private PointerEventData GetLookPointerEventData(Vector3 componentPosition){}...

 public override void Process()
 {
   ProcessHover();
   ProcessPress();
   ProcessDrag();
   ProcessWaitOver();
 }
 // Process waiting over components, if hovererd buttons click type is waitover, process it!
 private void ProcessWaitOver(){}...
 // Process draging, drag starts with closing your hand and end with opening it, sends scroll event to gameobject over hierarcy
 private void ProcessDrag(){}...
 // Process pressing, event click trigered on button by closing and opening hand,sends submit event to gameobject
 private void ProcessPress(){}...

 // Process hovering over component, sends pointer enter exit event to gameobject
 private void ProcessHover(){}...

 // Gets current kinect hand data if tracking it. Used from components like hand cursor
 public KinectInputData GetHandData(KinectUIHandType handType){}...
}

So in this class we have KinectInputData array, it’s lenght is going to be 2 for our right and left hands. And we are going to update it with TrackBody method.

1. With this part of the code, we are updating our each hand components position, currentstate and tracking option:

// Call this from your Kinect body view
public void TrackBody(Body body)
{
   for (int i = 0; i < _inputData.Length; i++)
   {
     // update kinectInput data
     _inputData[i].UpdateComponent(body);
   }
}

//Here is the UpdateComponent method inside KinectInputData:
// Must be called for each hand
public void UpdateComponent(Body body)
{
  HandPosition = GetVector3FromJoint(body.Joints[handType]);
  CurrentHandState = GetStateFromJointType(body, handType);
  IsTracking = true;
}

2. For handling hover, we are going to use BaseInputModule’s HandlePointerExitAndEnter method. The method needs a pointer event data and a gameobject as parameters. For pointereventdata we are geting it from our hand screen position. And the gameobject is raycast result of our pointereventdata. After that we are updating our inputdata’s Hovering properties:

// Process hovering over component, sends pointer enter exit event to gameobject
private void ProcessHover()
{
   for (int i = 0; i < _inputData.Length; i++)
   {
     PointerEventData pointer = GetLookPointerEventData(_inputData[i].GetHandScreenPosition(    ));
     var obj = _handPointerData.pointerCurrentRaycast.gameObject;
     HandlePointerExitAndEnter(pointer, obj);
     // Hover update
     _inputData[i].IsHovering = obj != null ? true : false;
     _inputData[i].HoveringObject = obj;
   }
}

3. For processing press we are going to use a really basic solution which is just tracking our handstate. If our hand is closed we are setting the data as pressed and when we open our hand we send a submit event to UI component if there is any under our cursor:

 // Process pressing, event click trigered on button by closing and opening hand,sends submit event to gameobject
private void ProcessPress()
{
  for (int i = 0; i < _inputData.Length; i++)
  {
     //Check if we are tracking hand state not wait over
     if (!_inputData[i].IsHovering || _inputData[i].ClickGesture != KinectUIClickGesture.HandState) continue;
     // If hand state is not tracked reset properties
     if (_inputData[i].CurrentHandState == HandState.NotTracked)
     {
       _inputData[i].IsPressing = false;
       _inputData[i].IsDraging = false;
     }
     // When we close hand and we are not pressing set property as pressed
     if (!_inputData[i].IsPressing && _inputData[i].CurrentHandState == HandState.Closed)
     {
       _inputData[i].IsPressing = true;
     }
     // If hand state is opened and is pressed, make click action
     else if (_inputData[i].IsPressing && (_inputData[i].CurrentHandState == HandState.Open)
     {
       PointerEventData lookData = GetLookPointerEventData(_inputData[i].GetHandScreenPosition());
       eventSystem.SetSelectedGameObject(null);
       // If we are not draging, and there is a gameobject under cursor click it
       if (lookData.pointerCurrentRaycast.gameObject != null && !_inputData[i].IsDraging)
       {
          GameObject go = lookData.pointerCurrentRaycast.gameObject;
          ExecuteEvents.ExecuteHierarchy(go, lookData, ExecuteEvents.submitHandler);
       }
        _inputData[i].IsPressing = false;
      }
   }
 }

4. You must have noticed there is IsDraging getter in the code above so we have to check if we are draging our hand. We can check it like this:

  • When we press a component(with processpress method), set a temporary screen position.
  • While processing drag compare current hand position axis values with temporary one.
  • If values are bigger than a treshold set IsDraging as true.
 // Process draging, drag starts with closing your hand and end with opening it, sends scroll event to gameobject over hierarcy
private void ProcessDrag()
{
  for (int i = 0; i < _inputData.Length; i++)
  {
   // if not pressing we can't drag
   if (!_inputData[i].IsPressing)continue;
   // Check if we reach drag treshold for any axis, temporary position set when we press an object
   if (Mathf.Abs(_inputData[i].TempHandPosition.x - _inputData[i].HandPosition.x) > _scrollTreshold || Mathf.Abs(_inputData[i].TempHandPosition.y - _inputData[i].HandPosition.y) > _scrollTreshold)
   {
     _inputData[i].IsDraging = true;
   }
   else
   {
     _inputData[i].IsDraging = false;
   }
   // If dragging use unit's eventhandler to send an event to a scrollview like component
   if (_inputData[i].IsDraging)
   {
     PointerEventData lookData = GetLookPointerEventData(_inputData[i].GetHandScreenPosition());
     eventSystem.SetSelectedGameObject(null);
     GameObject go = lookData.pointerCurrentRaycast.gameObject;
     PointerEventData pEvent = new PointerEventData(eventSystem);
     pEvent.dragging = true;
     pEvent.scrollDelta = (_inputData[i].TempHandPosition - _inputData[i].HandPosition) * _scrollSpeed;
     pEvent.useDragThreshold = true;
     ExecuteEvents.ExecuteHierarchy(go, pEvent, ExecuteEvents.scrollHandler);
   }
 }
}

That’s all we need to do to make a Kinect hand cursor.  You can download the full code from here. Have fun with it!

Advertisements

35 Comments Add yours

  1. Garrett Davis says:

    Hey man! Great job! This is a major issue for people who want to make kinect games and use unitys built in gui system.

    One quick question. When I open this up in a new unity proj, I get and error on this line

    if (_hoveringObject.GetComponent())

    from KinectUIWaitOverButton.

    It says Type or Namespace can not be found. I am wondering if I need an include or something.

    Thanks in advance!

    1. nevzatarman says:

      Hey thanks for asking, this is embarrassing but I forgot to upload my whole project to git and update the article, I have been dealing with my new game E5:)
      So to get rid of that error you can create an empty script named as “KinectUIWaitOverButton” and fix it for now and wait for me to upload the whole project to github it will take just one or two days. My github is https://github.com/paganini24/ you can follow me there, thanks!

  2. Garrett Davis says:

    Thanks for the reply! I totally understand that, and you are followed!

    The current game I am working on uses the kinect to help victims of stroke therapy do their rehab at home, and I have been pulling my hair out trying to figure out how to make unitys canvas system work with the Kinect. (Its tough for many of the users to use a controller for menus)

    P.S. After looking on the web for this for a while I saw many people are having the same problem for Kinect games. You should put this on the Unity Asset Store!!

    1. nevzatarman says:

      Hi there, I just updated the article, you can check the same link down below to download the sample project from git. It’s on github now so feel free to use, It’s not necessary but if you can add my name to your game’s credits I would be so glad.
      I don’t about Asset Store, if I can prepare a good documentation in the upcoming days I would surely. Thanks for the tip!

  3. Garrett Davis says:

    Ya let me know if you do! I would love to give you good reviews! Ill add you in for sure, and link to it when I get it done!

    I would get a Unity tutorial scene/readme up and submit asap!
    Here is a link to a unity asset store kinect pakage. (They don’t have canvas support though!)
    https://www.assetstore.unity3d.com/en/#!/content/7747

    1. nevzatarman says:

      Ok thanks for that, good luck with your game! I’ll post an update if I it’s published on asset store.

  4. adityosn says:

    Hii,

    Thanks for the tutorial, this is great :D.
    I download your project from GitHub, and I open it using Unity 5.x, then project upgraded. But, Unity found error on ClothTest.cs file. It said that Type or namespace name “SkinnedCloth” could not be found. Is there any file or dll that I missed ?
    Thanks

    1. nevzatarman says:

      Hi, it’s a test script so you can delete it. I was using it with Unity4’s cloth component and with Unity 5 that component has been changed that’s why you are getting the error, you can safely delete it!

  5. Thanks for making the tutorial! I am a new bee learning unity and kinect. May I ask if this compatible with using Kinect V1?

    1. nevzatarman says:

      Never tested with Kinect V1 sorry, please write me an update if you try with that also, thanks.

  6. Same with LindyWong(53074912) ,
    I’ve been looking into various forums and I did not find forums that discuss tutorials, explanations and UI that is easy to understand.
    Has anyone found another clue?

  7. Moi says:

    Im trying to run your code in Unity 5.3, if i play the test scene all the buttons appear but i can click them or make scroll. help me please. can youYou code run in unity 5?

  8. JasoniOosh says:

    Hey there, great tutorial you’ve posted here,

    I’m trying to integrate your input module as part of a small project I’m doing. However, there seem to be an issue where the UIs will break after the first place.

    The problem: The problem is that the first time I press play, the UI works okay, but if i exit play mode and re-enter play mode again, the hand cursor no longer works.

    Do you have any idea what is happening?

    Regards,
    Jason

  9. JasoniOosh says:

    Hey there,

    Great tutorial, but recently I was trying to integrate this into my project and I was in the test scene and I’ve faced some problems.

    I’m using Unity5.3.1f and after the first play, whenever I try to play again, the hand cursor doesn’t work anymore. Not sure if you know why.

    Regards,
    Jason

  10. Medhini Narasimhan says:

    Hi!

    Great tutorial. Im using unity 5.3 and when I play the test scene it does not detect my hand. Could you help me out with this please?

    1. Sophiya Yusuf says:

      yes, i have the same problem.. did u find the solution yet?

  11. Sophiya Yusuf says:

    hey… this is a life saver, and just what i needed for my current project.. The only problem is,that, though kinect is detecting me, once i click play, i cant find the cursor and am not able to move anything on the canvas.. can you please help me out?

  12. Sophiya Yusuf says:

    Can you show me the code for capturing screen shot from it too?

    1. Jeet says:

      Same problem here

  13. jacky angara says:

    i find this really helpful. this is my first time coding in c# and unity. can u make a video tutorial on this? it would be really helpful. anyways, great job! keep making tutorials!

  14. postklee15 says:

    I think that there is problem with Unity 5.3. Handposition variable can’t be updated in KinectInputModule class. Actually, it is updated in UpdateComponent(). but it can’t be referred in GetHandScreenPosition(). Do you have any idea to solve this?

    1. nevzatarman says:

      Hi, I have lost my access to my Kinect recently so I am not able to update the package, if you find a solution feel free to contribute on Github, thanks.

  15. Ahmad Najib says:

    Would it work if I use Kinect V1?

    1. nevzatarman says:

      No it wont work with V1

  16. Alexander Trofimov says:

    Hello, all.

    Firstly, big thanks for the module.

    I also get error with unavailable hand cursor.
    The problem is that possible number of tracking bodies is two.
    If you run “TestUIModule” alone (in your room), you will have only one tracking body. Second tracking body will be with zeroes values.
    These values overwrite values of the tracking body and cursor always stay at (0, 0, 0) position.

    Solution:
    Call
    KinectInputModule.instance.TrackBody(body);
    in BodySourceView.cs only if body is tracked.

    if (body.IsTracked)
    {
    if (!_Bodies.ContainsKey(body.TrackingId))
    {
    _Bodies[body.TrackingId] = CreateBodyObject(body.TrackingId);
    }

    RefreshBodyObject(body, _Bodies[body.TrackingId]);

    // Tracking for UI Component
    KinectInputModule.instance.TrackBody(body);
    }

    1. nevzatarman says:

      Hi, thank you for your reply. Please feel free to contribute to github project also, thanks!

  17. dreadscarlet says:

    Even running the test scene it rarely attaches the cursors to my hands. I don’t change anything in between it doing so or not doing so. It looks like this problem occurred for other people in the comments, but I don’t see any replies to it. Does anyone know what this issue is or how to fix it? It’s awesome when it works, but I can’t get it to function reliably.

    1. nevzatarman says:

      Hi, test scene has been fixed please checkout latest commit from git

  18. Ryan Boyd says:

    Does this work for the Kinect with MS-SDK 1.8? I only have an xbox 360 kinect and the sdk 2.0 only works with the xbox one kinect.

  19. kunal paul says:

    great work man

  20. chap says:

    Are you thinking about making a video?

  21. baher says:

    Hello,
    Please i have a question , in the README it is written (*First import Microsoft’s own Kinect library for Unity3D, you can find it inside the repository.) , i couldn’t find this in the repository , so i can’t use your kinect-inputmodule in my project .
    can you please explain the steps of importing your module in a unity project more clearly, as i need it urgently for my project ?

    Thank you very much .

    1. nevzatarman says:

      Hi,
      If you check out the repository, Kinect library is located under Assets as Kinect.2.0.1410.19000.unitypackage

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s