Gym pybullet environment

0. Useful links

   Universal robots repo

   OpenAI Gym repo

   OpenAI Robotic environments

   Pybullet Quickstart guide

   Pybullet Code (including gym examples)

1. Preface

   This is a repository created for funzies to try different RL techniques using an UR5 robot (at this moment).

   So far, basic work has been done in setting up the environment. The test.py file in examples folder runs a basic loop, and gets state, reward and done info. I've written the initial UR5 reach environment in the most simple way possible for now to start any kind of loop. This project is written almost from scratch so there is a lot to add/improve.

2. Installation

   Project is written using python 3!

   These installation instructions are for Linux, but I guess they could be run under Windows or OSX as well, for bullet3 SDK installation refer to the pybullet quickstart guide. I guess that is all that is needed outside the python packages.

   First you need bullet3 SDK for pybullet to work. Other than that, for now the only python packages required are gym, numpy and pybullet (I suggest using virtualenv). After getting those and cloning the folder you need to run python setup.py install in the main folder, and then pip install -e . (if this command doesn't work check if you didn't miss the dot, running this comand can also fix problem).

3. TODO

   Since I just started this project, there is much to be done. For starters, rewards and states need to be defined and reformatted for usage with different RL algorithms.

   The first algorithm that I'll try to implement will be DDPG (I haven't yet decided if I'll use OpenAI's algorithms, or something simpler like this TORCS DDPG example). The robotic environments provide a lot to think about when it comes to the actions, for now I've decided to use velocity control, but position or 'delta' approaches can also be defined.

   Pybullet provides also an easy way to insert multiple robot models into one client, so that can also provide sone incentive to use them for something like genetic algorithms.

   Also, if the reach env will be completed, next environments could be defined (pick and place, or pushing some objects)

   For now, the plan is to:

   1. define rewards, states and targets
   2. add docs to the environment functions (most of the gym examples don't seem beginner friendly, at least to me)
   3. setup and run basic ddpg sim (if computer will allow this) using Tensorflow and maybe Keras

4. Additional info

   UR5 URDF model used in simulation has been generated from xacro files present in UR5 description package (from github). Although it seems okay, if this project would be moved to a real robot I'd check if the information got agrees with real data. When starting the env there is some info about inertia values missing, but I don't want to bother with this now. Also, I've deleted the visual parts of the robot, since the shapes generated by pybullet seem clearer to me. Since the model is thrown without much thought into the environment, below is a simple return of the getJointInfo command from pybullet, to show joint names, indices and other info (very useful!).

getJointInfo legend:

name	type	description
jointIndex	int	the same joint index as the input parameter
jointName	string	the name of the joint, as specified in the URDF (or SDF etc) file
jointType	int	"type of the joint, this also implies the number of position and velocity variables. JOINT_REVOLUTE, JOINT_PRISMATIC, JOINT_SPHERICAL, JOINT_PLANAR, JOINT_FIXED. See the section on Base, Joint and Links for more details."
qIndex	int	the first position index in the positional state variables for this body
uIndex	int	the first velocity index in the velocity state variables for this body
flags	int	reserved
jointDamping	float	the joint damping value, as specified in the URDF file
jointFriction	float	the joint friction value, as specified in the URDF file
jointLowerLimit	float	Positional lower limit for slider and revolute (hinge) joints.
jointUpperLimit	float	Positional upper limit for slider and revolute joints. Values ignored in case upper limit smaller than lower limit..
jointMaxForce	float	Maximum force specified in URDF (possibly other file formats) Note that this value is not automatically used. You can use maxForce in 'setJointMotorControl2'.
jointMaxVelocity	float	Maximum velocity specified in URDF. Note that the maximum velocity is not used in actual motor control commands at the moment.
linkName	string	the name of the link, as specified in the URDF (or SDF etc.) file
jointAxis	vec3	joint axis in local frame (ignored for JOINT_FIXED)
parentFramePos	vec3	joint position in parent frame
parentFrameOrn	vec3	joint orientation in parent frame
parentIndex	int	parent link index, -1 for base

[(0,b'world_joint', 4, -1, -1, 0, 0.0, 0.0, 0.0, -1.0, 0.0, 0.0, b'base_link', (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0, 1.0), -1),
 (1, b'shoulder_pan_joint', 0, 7, 6, 1, 0.0, 0.0, -6.283185307179586, 6.283185307179586, 150.0, 3.15, b'shoulder_link', (0.0, 0.0, 1.0), (0.0, 0.0, 0.08915899693965912), (0.0, 0.0, 0.0, 1.0), 0),
 (2, b'shoulder_lift_joint', 0, 8, 7, 1, 0.0, 0.0, -6.283185307179586, 6.283185307179586, 150.0, 3.15, b'upper_arm_link', (0.0, 1.0, 0.0), (0.0, 0.1358499974012375, 0.0), (0.0, -0.7071067966408575, 0.0, 0.7071067657322372), 1),
 (3, b'elbow_joint', 0, 9, 8, 1, 0.0, 0.0, -3.141592653589793, 3.141592653589793, 150.0, 3.15, b'forearm_link', (0.0, 1.0, 0.0), (0.0, -0.11969999969005585, 0.14500001072883606), (0.0, 0.0, 0.0, 1.0), 2),
 (4, b'wrist_1_joint', 0, 10, 9, 1, 0.0, 0.0, -6.283185307179586, 6.283185307179586, 28.0, 3.2, b'wrist_1_link', (0.0, 1.0, 0.0), (0.0, 0.0, 0.14225000143051147), (0.0, -0.7071067966408575, 0.0, 0.7071067657322372), 3),
 (5, b'wrist_2_joint', 0, 11, 10, 1, 0.0, 0.0, -6.283185307179586, 6.283185307179586, 28.0, 3.2, b'wrist_2_link', (0.0, 0.0, 1.0), (0.0, 0.09300000220537186, 0.0), (0.0, 0.0, 0.0, 1.0), 4),
 (6, b'wrist_3_joint', 0, 12, 11, 1, 0.0, 0.0, -6.283185307179586, 6.283185307179586, 28.0, 3.2, b'wrist_3_link', (0.0, 1.0, 0.0), (0.0, 0.0, 0.09465000033378601), (0.0, 0.0, 0.0, 1.0), 5),
 (7, b'ee_fixed_joint', 4, -1, -1, 0, 0.0, 0.0, 0.0, -1.0, 0.0, 0.0, b'ee_link', (0.0, 0.0, 0.0), (0.0, 0.08229999989271164, 0.0), (0.0, 0.0, -0.7071067966408575, 0.7071067657322372), 6),
 (8, b'wrist_3_link-tool0_fixed_joint', 4, -1, -1, 0, 0.0, 0.0, 0.0, -1.0, 0.0, 0.0, b'tool0', (0.0, 0.0, 0.0), (0.0, 0.08229999989271164, 0.0), (0.7071067966408575, 0.0, 0.0, 0.7071067657322372), 6),
 (9, b'base_link-base_fixed_joint', 4, -1, -1, 0, 0.0, 0.0, 0.0, -1.0, 0.0, 0.0, b'base', (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.999999999999999, -4.371139000186241e-08), 0)]