position control

[SshiJwu]

Hi, I want to execute position control in SAPIEN. But it's not quite what I thought. This is my code:
def step(self, action: np.ndarray):

    self.robot.set_drive_target(action)

    print("action:",action)

    qf = self.robot.compute_passive_force(gravity=True, coriolis_and_centrifugal=True)

    self.robot.set_qf(qf)

    for i in range(self.frame_skip):
        self.scene.step()
        self.render()

    self.current_step += 1

    print("joint_position", self.robot.get_qpos())
    print("\n")

    obs = self.get_observation()
    return obs

The returned results are:

action: [ 1.0080942 0.07700591 -0.26779976 0.45527703 -2.4566367 -0.02794247
-0.41149104 -0.28804654 -0.12245759 0.21643543 0.5214863 0.92012674
0.00890571 0.45191255 0.72066396 0.6122276 -0.14939168 0.5470857
0.614574 0.7317573 0.09097223 -0.34906584 0.5128116 0.5564713
1.4659702 0.65489566 0.70423144 0.0875565 -0.29759088 0.29122272]
[2024-03-29 21:38:32.082] [SAPIEN] [warning] external force is deprecated and ignored in passive force computation.
joint_position [ 5.3307290e+00 -2.7980363e+00 -2.2715263e+00 5.7231402e+00
5.9122033e+00 2.4233885e+00 -5.1686323e-01 -5.9985149e-01
3.4906119e-01 -2.6257661e-01 -9.9840690e-04 2.6208350e-01
2.6045841e-01 1.4904318e+00 1.5557226e+00 1.3111534e+00
-3.4904963e-01 -2.6334012e-01 -2.0598492e-03 7.4446361e-06
5.1040280e-01 -3.3417100e-01 1.5658530e+00 1.5584077e+00
1.3176466e+00 -1.0141965e+00 7.2087932e-01 2.0217310e-01
-6.6974652e-01 -2.5934604e-01]

action: [ 1.0077728 0.21889764 -0.7015863 0.39141732 -1.9953026 1.8720453
-0.38592446 -0.19408308 -0.15859017 0.06714711 0.4636697 0.6357209
0.05944189 0.25535285 0.6238109 0.29917058 -0.19588962 0.45438582
0.5641642 0.51035583 0.13572612 -0.34906584 0.4169842 0.3803882
1.4329351 0.6368792 0.9655694 0.12643999 -0.4998034 0.32632107]
[2024-03-29 21:38:33.266] [SAPIEN] [warning] external force is deprecated and ignored in passive force computation.
joint_position [ 5.2683725e+00 -2.7284687e+00 -2.2209368e+00 5.5126810e+00
5.5400486e+00 3.0390871e+00 -5.2358037e-01 -5.0338590e-01
3.4906495e-01 -2.4317463e-01 1.3675858e-02 2.6462221e-01
2.5787476e-01 1.5027406e+00 1.5534712e+00 1.3072052e+00
-3.4906074e-01 -2.5425181e-01 6.3799964e-03 9.5727848e-04
5.1789987e-01 -3.3145249e-01 1.5706725e+00 1.5550016e+00
1.3148304e+00 -1.0055885e+00 7.2721928e-01 2.0141706e-01
-6.5943998e-01 -2.5790009e-01]

action: [ 1.0283355 0.25253054 -0.6489771 0.42385158 -2.0095935 1.8318564
-0.35773814 -0.14250918 -0.15065043 0.11706779 0.42541528 0.623275
0.03811157 0.3041759 0.635507 0.30414644 -0.16541882 0.44699684
0.5534656 0.5770409 0.10752289 -0.34906584 0.38562608 0.39418247
1.4362597 0.65269834 0.9621447 0.146154 -0.50012773 0.3357537 ]
[2024-03-29 21:38:34.417] [SAPIEN] [warning] external force is deprecated and ignored in passive force computation.
joint_position [ 5.2101369e+00 -2.6591597e+00 -2.1719723e+00 5.3027258e+00
5.1823421e+00 3.6911435e+00 -5.2360207e-01 -5.1060206e-01
3.4906504e-01 -2.5150260e-01 8.2643982e-03 2.6403001e-01
2.5711003e-01 1.4986508e+00 1.5507315e+00 1.3075269e+00
-3.4906173e-01 -2.6233345e-01 -6.7510048e-04 -3.0399292e-07
5.1571220e-01 -3.3028448e-01 1.5631597e+00 1.5541407e+00
1.3159699e+00 -1.0027348e+00 7.2851586e-01 2.0109747e-01
-6.5709531e-01 -2.5787556e-01]

How do I make the robot reach the target position?

[fbxiang]

I believe the issue is that set_qf with passive force needs to be called every step to compensate gravity. In SPAIEN 3 (currently dev releases), we provide a better way to achieve this by disabling gravity of the robot completely.

[SshiJwu]

How do I fix it?

[fbxiang]

You can try set_qf every step.

    self.robot.set_drive_target(action)

    print("action:",action)

    qf = self.robot.compute_passive_force(gravity=True, coriolis_and_centrifugal=True)
    self.robot.set_qf(qf)

    for i in range(self.frame_skip):
        
        # add these lines
        qf = self.robot.compute_passive_force(gravity=True, coriolis_and_centrifugal=True)
        self.robot.set_qf(qf)

        self.scene.step()
        self.render()

    self.current_step += 1

    print("joint_position", self.robot.get_qpos())
    print("\n")

    obs = self.get_observation()
    return obs

[SshiJwu]

Thanks for your reply! I have one more question: How can I get a GT point cloud of objects in the scene?

[fbxiang]

All geometries in SAPIEN are mesh so there is no GT point cloud. I guess you could mean 2 things.

1. Point cloud from GT depth map of cameras. This has been discussed in SAPIEN's documentation rendering section.

2. Sampled point cloud from collision or visual meshes. For this I would suggest converting SAPIEN meshes to trimesh and use trimesh sampling functions. Assuming you are using SAPIEN 2, the example code for this conversion can be found here https://github.com/haosulab/ManiSkill2/blob/main/mani_skill2%2Futils%2Ftrimesh_utils.py

[SshiJwu]

How do I fix it?

Traceback (most recent call last):
File "examples/gen_demonstration_expert.py", line 236, in
main()
File "examples/gen_demonstration_expert.py", line 75, in main
env = create_env(task_name=task_name,
File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/create_env.py", line 37, in create_env
env = LaptopRLEnv(**env_params, friction=5)
File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/rl_env/laptop_env.py", line 20, in init
super().init(use_gui, frame_skip, friction=friction, index=index, **renderer_kwargs)
File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/sim_env/laptop_env.py", line 15, in init
super().init(use_gui=use_gui, frame_skip=frame_skip, **renderer_kwargs)
File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/rl_env/base.py", line 32, in init
super().init(use_gui=use_gui, frame_skip=frame_skip, use_visual_obs=use_visual_obs, **renderer_kwargs)
File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/sim_env/base.py", line 25, in init
engine, renderer = get_engine_and_renderer(use_gui=use_gui, need_offscreen_render=need_offscreen_render,
File "/home/user/Downloads/diffusion_rl/third_party/dexart-release/dexart/env/sim_env/constructor.py", line 36, in get_engine_and_renderer
_renderer = sapien.SapienRenderer(default_mipmap_levels=mipmap_levels, offscreen_only=not use_gui,
RuntimeError: Cannot find any cuda device suitable for rendering.

I run nvidia-smi,nvcc -Vand vulkaninfo,then return no error. I think this is a error about SAPIEN.

[fbxiang]

Can you start a new issue and post the result of vulkaninfo and nvidia-smi?