OpenCV methods return unexpected results when calibrating the simulated RIWMPBOT #994

Kazadhum · 2024-10-12T15:12:46Z

Calibrations using either the upper arm or forearm pattern achieve a better comparison to the GT than using the hand pattern. This is wierd because these methods were clearly designed to use a pattern at the end of the kinematic chain. No joint noise is used here.

Critically, this does not occur when calibrating using real data.

Calibrating: Simulated, no joint noise added, upper arm pattern, Tsai method

export DATASET_PATH=$ATOM_DATASETS/riwmpbot/train \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-ctgt -uic -mn tsai \
-sfr -sfrn /tmp/comparison_to_gt.csv

Results:

+--------------------------------------+----------+------------+
|              Transform               |  Et [m]  | Erot [rad] |
+--------------------------------------+----------+------------+
| tripod_center_support-rgb_world_link | 0.000573 |  0.000972  |
+--------------------------------------+----------+------------+

Calibrating: Simulated, no joint noise added, forearm pattern, Tsai method

export DATASET_PATH=$ATOM_DATASETS/riwmpbot/train \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset.json \
-c rgb_world -p forearm_pattern \
-hl forearm_link -bl base_link \
-ctgt -uic -mn tsai \
-sfr -sfrn /tmp/comparison_to_gt.csv

Results:

+--------------------------------------+---------+------------+
|              Transform               |  Et [m] | Erot [rad] |
+--------------------------------------+---------+------------+
| tripod_center_support-rgb_world_link | 0.00206 |  0.004331  |
+--------------------------------------+---------+------------+

Calibrating: Simulated, no joint noise added, hand pattern, Tsai method

export DATASET_PATH=$ATOM_DATASETS/riwmpbot/train \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset.json \
-c rgb_world -p hand_pattern \
-hl flange -bl base_link \
-ctgt -uic -mn tsai \
-sfr -sfrn /tmp/comparison_to_gt.csv

Results:

+--------------------------------------+----------+------------+
|              Transform               |  Et [m]  | Erot [rad] |
+--------------------------------------+----------+------------+
| tripod_center_support-rgb_world_link | 0.002176 |  0.002587  |
+--------------------------------------+----------+------------+

The text was updated successfully, but these errors were encountered:

miguelriemoliveira · 2024-10-12T16:57:11Z

Hi @Kazadhum ,

I am going crazy thinking about these. I am considering the hyphotesis that its the real results that do not make sense.

Can you tell me how to run these?

I would need the real dataset as well

Kazadhum · 2024-10-12T17:16:22Z

Hi @miguelriemoliveira!

The command I used for the real data:

export DATASET_PATH="$ATOM_DATASETS/riwmpbot_real/diogo_tests" \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_filtered.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset_filtered_train.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-uic -mn tsai \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_tsai_rgb_world.json \
-test_json $DATASET_PATH/dataset_filtered_test.json \
-uic -sfr -sfrn /tmp/single_rgb_evaluation.csv

If you want to test with different patterns and methods, change the -p, -hl and -mn arguments accordingly.

I'll send you the dataset via e-mail right away!

miguelriemoliveira · 2024-10-12T18:03:16Z

Collection 27, dataset filtered

collection 57, dataset filtered

This should disrupt all results, right?

Kazadhum · 2024-10-12T18:17:38Z

I think so! Seems like I didn't catch these when I filtered the dataset...

Kazadhum · 2024-10-12T18:55:31Z

For testing (delete collections 27 and 57 from dataset_filtered.json before doing this)....

export DATASET_PATH="$ATOM_DATASETS/riwmpbot_real/diogo_tests" \
&& \ 
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_filtered.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& mv $DATASET_PATH/atom_calibration.json \
$DATASET_PATH/dataset_patterns_corrected.json \
&& \
rosrun atom_evaluation copy_tfs_from_dataset.py \
-sd $DATASET_PATH/dataset_patterns_corrected.json \
-td $DATASET_PATH/dataset_filtered.json \
-pll flange forearm_link upper_arm_link \
-cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6 \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_patterns_corrected.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset_patterns_corrected_train.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-uic -mn tsai \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_tsai_rgb_world.json \
-test_json $DATASET_PATH/dataset_patterns_corrected_test.json \
-uic -sfr -sfrn /tmp/single_rgb_evaluation.csv

Alternatively, instead of copying the TFs we can perform a sanity check by calibrating using OpenCV with the ATOM-calibrated dataset as input (there is no need to split into train and test here, since the -ctgt from the calibration will give us an indication of how far the OpenCV solution is from the ATOM solution straight away)...

rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_filtered.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/atom_calibration.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-ctgt -uic -mn tsai

miguelriemoliveira · 2024-10-13T08:15:55Z

From this we should remove also collection 34

created dataset_filtered2.json without collections 27, 34 and 57.

Its attached here.

dataset_filtered2.json

miguelriemoliveira · 2024-10-13T08:22:54Z

Hi @Kazadhum ,

not sure I agree with all the commands:

Here my suggestions for change, including using the dataset_filtered2.json.

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot_real/diogo_tests" \
&& \ 
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_filtered2.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& mv $DATASET_PATH/atom_calibration.json \
$DATASET_PATH/dataset_patterns_corrected.json \
&& \
rosrun atom_evaluation copy_tfs_from_dataset.py \
-sd $DATASET_PATH/dataset_patterns_corrected.json \
-td $DATASET_PATH/dataset_filtered_with_atom_patterns.json \
-pll flange forearm_link upper_arm_link \
-cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6 \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_filtered_with_atom_patterns.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset_filtered_with_atom_patterns_train.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-uic -mn tsai \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_tsai_rgb_world.json \
-test_json $DATASET_PATH/dataset_filtered_with_atom_patterns.json \
-uic -sfr -sfrn /tmp/single_rgb_evaluation.csv

Kazadhum · 2024-10-14T08:37:06Z

Hi @miguelriemoliveira! That command doesn't work for me, because the dataset_filtered_with_atom_patterns.json given as a target to copy the tfs onto does not exist. Here's what works for me:

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot_real/diogo_tests" \
&& \ 
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_filtered2.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& mv $DATASET_PATH/atom_calibration.json \
$DATASET_PATH/dataset_patterns_corrected.json \
&& \
rosrun atom_evaluation copy_tfs_from_dataset.py \
-sd $DATASET_PATH/dataset_patterns_corrected.json \
-td $DATASET_PATH/dataset_filtered2.json \
-pll flange forearm_link upper_arm_link \
-cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6 \
&& \
mv $DATASET_PATH/dataset_filtered2_tfs_copied.json \
$DATASET_PATH/dataset_filtered_with_atom_patterns.json \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_filtered_with_atom_patterns.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset_filtered_with_atom_patterns_train.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-uic -mn tsai \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_tsai_rgb_world.json \
-test_json $DATASET_PATH/dataset_filtered_with_atom_patterns_test.json \
-uic -sfr -sfrn /tmp/single_rgb_evaluation.csv

Regardless, I think I'm going to follow your suggestion and use the atom-calibrated dataset as input for the OpenCV calibrations to see how much their solutions deviate from ours.

So, the command I'm running for this:

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot_real/diogo_tests" \
&& \
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_filtered2.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/atom_calibration.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-ctgt -uic -mn tsai

Kazadhum · 2024-10-14T08:56:32Z

I have some results for the real system.

Once again, I'm giving an ATOM-calibrated dataset as input for the OpenCV calibrations. Since these are closed-form solutions and are invariable w.r.t. an initial guess, the choice of calibrated dataset vs. un-calibrated dataset for the input does not impact the final result.

The goal here is to use the -ctgt mechanism to evaluate how different the OpenCV solutions are to the ATOM solutions.

For the Tsai method, here are my results:

Upper Arm Pattern

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot_real/diogo_tests" \
&& \
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_filtered2.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/atom_calibration.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-ctgt -uic -mn tsai

+----------------------+----------+------------+
|      Transform       |  Et [m]  | Erot [rad] |
+----------------------+----------+------------+
| world-rgb_world_link | 0.023379 |  0.037112  |
+----------------------+----------+------------+

Forearm Pattern

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot_real/diogo_tests" \
&& \
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_filtered2.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/atom_calibration.json \
-c rgb_world -p forearm_pattern \
-hl forearm_link -bl base_link \
-ctgt -uic -mn tsai

+----------------------+----------+------------+
|      Transform       |  Et [m]  | Erot [rad] |
+----------------------+----------+------------+
| world-rgb_world_link | 0.008735 |  0.048095  |
+----------------------+----------+------------+

Hand Pattern Calibration

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot_real/diogo_tests" \
&& \
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_filtered2.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/atom_calibration.json \
-c rgb_world -p hand_pattern \
-hl flange -bl base_link \
-ctgt -uic -mn tsai

+----------------------+----------+------------+
|      Transform       |  Et [m]  | Erot [rad] |
+----------------------+----------+------------+
| world-rgb_world_link | 0.004741 |  0.017117  |
+----------------------+----------+------------+

So I'd say this is following the expected behaviour. These methods were intended to be used with a pattern in the manipulator's end-effector.

miguelriemoliveira · 2024-10-14T09:52:58Z

Hi @Kazadhum ,

I think what this proves is that we get a more similar result between Tsai and Atom when we use the hand, in comparison to when we use the upper arm. Do you agree?

miguelriemoliveira · 2024-10-14T09:54:13Z

So I would be expecting to see a larger difference in the reprojection errors evaluation of

atom vs Tsai for the upper arm

in comparison to the

atom vs Tsai for the hand

Can you confirm this is what occurs?

Kazadhum · 2024-10-14T13:17:18Z

Hi @Kazadhum ,

I think what this proves is that we get a more similar result between Tsai and Atom when we use the hand, in comparison to when we use the upper arm. Do you agree?

Hi @miguelriemoliveira! Yes, I agree with that!

When it comes to reprojection errors, I believe we already have this data, but I'll run the calibrations anyways.

So, we have the following after evaluation...

ATOM

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             1.1115             |            1.3416           |              0.5142              |  1.1024  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Tsai

Upper Arm

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |            45.5151             |           72.7517           |             99.5919              | 69.4841  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Hand Pattern

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |            10.5768             |           10.8224           |             14.8335              | 12.1143  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

So it checks out -- the calibration using the hand pattern is much closer in reprojection error to ATOM than the calibration using the upper arm pattern.

Kazadhum · 2024-10-14T20:43:12Z

Repeating this experiment for the simulated system to see if this behaviour is consistent when compared to the real system...

ATOM

export DATASET_PATH=/home/diogo/atom_datasets/riwmpbot/train/ \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_train.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/atom_calibration.json \
-test_json $DATASET_PATH/dataset_test.json \
-uic -sfr -sfrn /tmp/single_rgb_evaluation.csv

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             0.2522             |            0.6279           |              0.3667              |  0.4218  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Tsai

Upper Arm Pattern

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot/train" \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_train.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/atom_calibration.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-ctgt -uic -mn tsai \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_tsai_rgb_world.json \
-test_json $DATASET_PATH/dataset_test.json \
-uic -cpt -sfr -sfrn /tmp/single_rgb_evaluation.csv

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             1.3488             |            0.9109           |              0.5186              |  0.9093  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Hand Pattern

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot/train" \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset_train.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/atom_calibration.json \
-c rgb_world -p hand_pattern \
-hl flange -bl base_link \
-ctgt -uic -mn tsai \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_tsai_rgb_world.json \
-test_json $DATASET_PATH/dataset_test.json \
-uic -cpt -sfr -sfrn /tmp/single_rgb_evaluation.csv

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             5.0728             |            3.3486           |              5.9803              |  4.7830  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

So this is contradictory to the results from the real system. Strange.

miguelriemoliveira · 2024-10-14T21:37:53Z

But you are not copying the atom estimated pattern transforms to the dataset before running the evaluation, right?
Isn't that missing?

Kazadhum · 2024-10-14T22:15:48Z

You're absolutely right!

Here are the new commands I run. First I calibrate with ATOM, then I copy the pattern TFs to the dataset, split it into the train and test dataset, calibrate using Tsai and then evaluate.

Upper Arm Pattern

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot/train" \
&& \ 
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& mv $DATASET_PATH/atom_calibration.json \
$DATASET_PATH/dataset_patterns_corrected.json \
&& \
rosrun atom_evaluation copy_tfs_from_dataset.py \
-sd $DATASET_PATH/dataset_patterns_corrected.json \
-td $DATASET_PATH/dataset.json \
-pll flange forearm_link upper_arm_link \
-cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6 \
&& \
mv $DATASET_PATH/dataset_tfs_copied.json \
$DATASET_PATH/dataset_with_atom_patterns.json \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_with_atom_patterns.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset_with_atom_patterns_train.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-ctgt -uic -mn tsai \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_tsai_rgb_world.json \
-test_json $DATASET_PATH/dataset_with_atom_patterns_test.json \
-uic -cpt -sfr -sfrn /tmp/single_rgb_evaluation.csv

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             1.2352             |            0.8994           |              0.4960              |  0.8623  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Hand Pattern

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot/train" \
&& \ 
rosrun atom_calibration calibrate \
-json $DATASET_PATH/dataset.json \
-uic -v -gtpp \
-jsf "lambda x: False" \
-jpsf "lambda x: False" \
&& mv $DATASET_PATH/atom_calibration.json \
$DATASET_PATH/dataset_patterns_corrected.json \
&& \
rosrun atom_evaluation copy_tfs_from_dataset.py \
-sd $DATASET_PATH/dataset_patterns_corrected.json \
-td $DATASET_PATH/dataset.json \
-pll flange forearm_link upper_arm_link \
-cll charuco_200x200_8x8 charuco_170x100_3x6 charuco_200x120_3x6 \
&& \
mv $DATASET_PATH/dataset_tfs_copied.json \
$DATASET_PATH/dataset_with_atom_patterns.json \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_with_atom_patterns.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand.py \
-json $DATASET_PATH/dataset_with_atom_patterns_train.json \
-c rgb_world -p hand_pattern \
-hl flange -bl base_link \
-ctgt -uic -mn tsai \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_tsai_rgb_world.json \
-test_json $DATASET_PATH/dataset_with_atom_patterns_test.json \
-uic -cpt -sfr -sfrn /tmp/single_rgb_evaluation.csv

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             5.1842             |            3.3819           |              5.9984              |  4.8351  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

The errors are slightly different but not significantly and the same strange observation remains, it seems...

Kazadhum · 2024-10-15T13:49:25Z

Hello @miguelriemoliveira!

I have new results for the simulated system using the Shah and Li methods, implemented in #995, but the same observation can still be made...

Since these methods estimate the pattern's pose, it isn't necessary to copy these TF's from a previous ATOM calibration. The command I used was the following:

clear && export DATASET_PATH="$ATOM_DATASETS/riwmpbot/train" \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand_robot_world.py \
-json $DATASET_PATH/dataset_train.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-ctgt -uic -mn shah \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_shah_rgb_world.json \
-test_json $DATASET_PATH/dataset_test.json \
-uic -cpt -sfr -sfrn /tmp/single_rgb_evaluation.csv

The only thing I changed to get these results was the pattern.

So we have the following...

Upper Arm Pattern

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             0.9200             |            0.8343           |              0.0984              |  0.6061  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Hand Pattern

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             1.3340             |            0.9389           |              0.9203              |  1.0583  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Now testing with the real data...

Kazadhum · 2024-10-15T13:54:42Z

Now with the real dataset...

Upper Arm Pattern

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |            95.0187             |           56.0023           |              2.1066              | 61.3168  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Hand Pattern

Errors per collection
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
| Collection | rgb_world-forearm_pattern [px] | rgb_world-hand_pattern [px] | rgb_world-upper_arm_pattern [px] | all [px] |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+
|  Averages  |             6.3186             |            2.9056           |             14.8358              |  6.9588  |
+------------+--------------------------------+-----------------------------+----------------------------------+----------+

Kazadhum · 2024-10-16T09:18:19Z

Hi @miguelriemoliveira!

The command I used for testing with the real dataset for the hand pattern was:

export DATASET_PATH=$ATOM_DATASETS/riwmpbot_real/diogo_tests/ \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_filtered2.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand_robot_world.py \
-json $DATASET_PATH/dataset_filtered2_train.json \
-c rgb_world -p hand_pattern \
-hl flange -bl base_link \
-ctgt -uic -mn shah \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_shah_rgb_world.json \
-test_json $DATASET_PATH/dataset_filtered2_test.json \
-uic -cpt -sfr -sfrn /tmp/single_rgb_evaluation.csv

and for the Upper Arm Pattern:

export DATASET_PATH=$ATOM_DATASETS/riwmpbot_real/diogo_tests/ \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_filtered2.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand_robot_world.py \
-json $DATASET_PATH/dataset_filtered2_train.json \
-c rgb_world -p upper_arm_pattern \
-hl upper_arm_link -bl base_link \
-ctgt -uic -mn shah \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_shah_rgb_world.json \
-test_json $DATASET_PATH/dataset_filtered2_test.json \
-uic -cpt -sfr -sfrn /tmp/single_rgb_evaluation.csv

For the forearm (added by Miguel)

export DATASET_PATH=$ATOM_DATASETS/riwmpbot_real/diogo_tests/ \
&& \
rosrun atom_calibration split_atom_dataset \
-json $DATASET_PATH/dataset_filtered2.json \
-tp 70 \
-ss 1 \
&& \
rosrun atom_evaluation cv_eye_to_hand_robot_world.py \
-json $DATASET_PATH/dataset_filtered2_train.json \
-c rgb_world -p forearm_pattern \
-hl forearm_link -bl base_link \
-ctgt -uic -mn shah \
&& \
rosrun atom_evaluation single_rgb_evaluation \
-train_json $DATASET_PATH/hand_eye_shah_rgb_world.json \
-test_json $DATASET_PATH/dataset_filtered2_test.json \
-uic -cpt -sfr -sfrn /tmp/single_rgb_evaluation.csv

Note that the pattern transformations from the robot-world-hand-eye calibration are used for evaluation, since I'm using the -cpt flag.

miguelriemoliveira · 2024-10-16T09:37:22Z

Hi @Kazadhum ,

I am a bit lost. The results for me seem very good. They make sense.

This is with the upper arm:

Only the upper arm has good reprojection error of 2.1. All others have large errors which makes perfect sense (since we are calibrating the upper arm).

As for the forearm:

Again, the foream has very low error 6.6 and the upperarm (which is behind in the kinematic chain) also has small error. The hand, which is after in the kinematic chain, has a very large error (42.6).

For the hand results

Hand, very small results (2.9), and the others also small results because they are behind in the kinemaic chain.

These results make perfect sense, and are not like the ones in the results excel.

miguelriemoliveira · 2024-10-16T09:40:28Z

I also revisited the sim results in your comments above and I also think we can explain those well. Let's talk this afternoon.

Kazadhum added the bug Something isn't working label Oct 12, 2024

Kazadhum assigned miguelriemoliveira and Kazadhum Oct 12, 2024

Kazadhum added this to Ongoing work on General Calibration Oct 12, 2024

github-project-automation bot moved this to To do in Ongoing work on General Calibration Oct 12, 2024

miguelriemoliveira mentioned this issue Oct 15, 2024

Implement opencv methods robotworld hand eye camera #995

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenCV methods return unexpected results when calibrating the simulated RIWMPBOT #994

OpenCV methods return unexpected results when calibrating the simulated RIWMPBOT #994

Kazadhum commented Oct 12, 2024

miguelriemoliveira commented Oct 12, 2024

Kazadhum commented Oct 12, 2024

miguelriemoliveira commented Oct 12, 2024

Kazadhum commented Oct 12, 2024

Kazadhum commented Oct 12, 2024

miguelriemoliveira commented Oct 13, 2024 •

edited

Loading

miguelriemoliveira commented Oct 13, 2024 •

edited

Loading

Kazadhum commented Oct 14, 2024 •

edited

Loading

Kazadhum commented Oct 14, 2024

miguelriemoliveira commented Oct 14, 2024

miguelriemoliveira commented Oct 14, 2024

Kazadhum commented Oct 14, 2024

Kazadhum commented Oct 14, 2024 •

edited

Loading

miguelriemoliveira commented Oct 14, 2024

Kazadhum commented Oct 14, 2024 •

edited

Loading

Kazadhum commented Oct 15, 2024

Kazadhum commented Oct 15, 2024 •

edited

Loading

Kazadhum commented Oct 16, 2024 •

edited by miguelriemoliveira

Loading

miguelriemoliveira commented Oct 16, 2024

miguelriemoliveira commented Oct 16, 2024

OpenCV methods return unexpected results when calibrating the simulated RIWMPBOT #994

OpenCV methods return unexpected results when calibrating the simulated RIWMPBOT #994

Comments

Kazadhum commented Oct 12, 2024

Calibrating: Simulated, no joint noise added, upper arm pattern, Tsai method

Calibrating: Simulated, no joint noise added, forearm pattern, Tsai method

Calibrating: Simulated, no joint noise added, hand pattern, Tsai method

miguelriemoliveira commented Oct 12, 2024

Kazadhum commented Oct 12, 2024

miguelriemoliveira commented Oct 12, 2024

Kazadhum commented Oct 12, 2024

Kazadhum commented Oct 12, 2024

miguelriemoliveira commented Oct 13, 2024 • edited Loading

miguelriemoliveira commented Oct 13, 2024 • edited Loading

Kazadhum commented Oct 14, 2024 • edited Loading

Kazadhum commented Oct 14, 2024

Upper Arm Pattern

Forearm Pattern

Hand Pattern Calibration

miguelriemoliveira commented Oct 14, 2024

miguelriemoliveira commented Oct 14, 2024

Kazadhum commented Oct 14, 2024

ATOM

Tsai

Upper Arm

Hand Pattern

Kazadhum commented Oct 14, 2024 • edited Loading

ATOM

Tsai

Upper Arm Pattern

Hand Pattern

miguelriemoliveira commented Oct 14, 2024

Kazadhum commented Oct 14, 2024 • edited Loading

Upper Arm Pattern

Hand Pattern

Kazadhum commented Oct 15, 2024

Upper Arm Pattern

Hand Pattern

Kazadhum commented Oct 15, 2024 • edited Loading

Upper Arm Pattern

Hand Pattern

Kazadhum commented Oct 16, 2024 • edited by miguelriemoliveira Loading

miguelriemoliveira commented Oct 16, 2024

miguelriemoliveira commented Oct 16, 2024

miguelriemoliveira commented Oct 13, 2024 •

edited

Loading

miguelriemoliveira commented Oct 13, 2024 •

edited

Loading

Kazadhum commented Oct 14, 2024 •

edited

Loading

Kazadhum commented Oct 14, 2024 •

edited

Loading

Kazadhum commented Oct 14, 2024 •

edited

Loading

Kazadhum commented Oct 15, 2024 •

edited

Loading

Kazadhum commented Oct 16, 2024 •

edited by miguelriemoliveira

Loading