Hi // Hard Ware III

‘ Hi ‘  is a project by MRAC 01 student  Tomas Quijano , Abanoub Nagy  ,Christopher Booth and
Alberto Martínez   during the Hardware III seminar of the third term. Hardware was taught by IAAC faculty Starsk Lara ,Daniil Koshelyuk

During the Hardware III course, students had to explore ::
Using Unity and use it as a tool to explore ways of human robot interactions



In this project is foucasing on using Unity as a broker or middle man to communicate with human interaction using a leap motion camera. The communication from the robot side is using ROS to control the robot.





As new tools are growing every to help designer to be more close to their design and more interactive. construction site also have its emerging technologies and application in sites.What we saw is lagging  and  opportunity to contribute is the amount of  set up you need and tools to do specific thing on site. This kind of  set ups and installation needed is maybe good for laps but is not convenient to construction site with its harsh environment that is changing every day. So in our project we  wanted  to eliminate or reduce the  the equipment needed  for a human interact with robotic arm. Taking in consideration that the workers would be working with robot, so our system allow easy usage and less installation.






The Project

The project was divided into two teams for the two main tasks :

A) Front end

This team of the project which responsible to designing the user interaction .  Build a virtual box  in a virtual space.  , The pattern of the user’s intersection with the surfaces is logged by the interface and sent to the project back end via an ROS Bridge web-socket.



We used a leap motion camera package and from this package there is a model  for the hands . Another hierarchy object surrounding the right hand object, that was the collusion object.A bounding box created virtually  0.80 m * 0.80 m  and its center was locked in a proportional to the camera distance so can be move the camera around and the hand movement will be always in relation without a fixed places  and without a certain calibration.

After setting up our box and the collusion object.We created a script collusion object that will be activate the box when the the hand (Colluding object) will trigger it.While the box is triggered or  the right hand is inside the box will get the position of the hands and collect this data coordinate .


The coordinate data collected:

  1. Pose

The initial idea was to collect and send the the Pose of the hands and this would be sent Pose target position for  the robot End factor  would be the target position to go to for the robot.Sooner we faced problems from the back-end team with the move it simulation of not receiving or not reading the pose position.


2.  Twist

Reading more in the documentation of ‘move it’  the type of messages that could be read was a Twist message. The different between Pose and Twist. If Pose is the end factor target point that the  robot should goes too . The twist is like a joy stick controller that move the robot with liner and angular direction.


3. Twist Stamp

After getting the  twist position and proven that it is working and read by the ‘Moveit servo’. It was needed to add the update it with other type of messages which is Twist stamp message. The diffrenece between the twist messages and twist stamp, the twist stamp is the same data with added header.


Communication :

To communicate this date from the unity side or from the Human iteration to the robot. We used ROS bridge to send the Twist Stamp message from unity to ROS (Moveit Servo) to control the robot through web-socket protocol.

  1. Units:

In the script there is a function used to convert the unity coordinate to normal world coordinate. Since Unity  Z axes and the Y axes are switched. So we used the function

—- Unity2Ros() — to convert and prepare it before publishing this message.

2. Scalier:

The movement of a human arm is not the same like the robot movement. So a Scalier number added to fine tune the movement of the robot. Scale up or scale down this movement of the robot if someone witth shorter hand or moving to fast or to slow. There  is two Scalier  one for the liner motion and one for the angular motion. The recommended for the angular motion is to have very small Scalier since it was creating a lot of singularity.



B) Back end

The twist transformation of hands done by the user is sent through ROSBridge .






Our project repo

for source files:

  • Repo



Resources + Documentation

The leap motion

  • Package



The unity

  • Octocopler style


  • ROSSharp Docmentation


  • ROS real time