Zau batch preparations

[brunofavs]

Hey @miguelriemoliveira @Kazadhum,

Since our last meeting I was meaning to do a issue summing up the tasks we planned to do but I forgot. This is a followup to #986 .

• Write a organized script to test manually dataset splitting, calibration and evaluation,
• Annotate the filtered dataset,
• Test each type of evaluation to assess if they are working as intended, (Rgb to Lidar evaluations outdated. #988)
• Run multiple times the whole pipeline in this semi-automatic way to ensure everything is working properly before doing a big batch.

Write a organized script to test manually dataset splitting, calibration and evaluation.

I ditched the one bash script for everything ideia we were working on for something more structured and easier to interpret.
So I tinkered and came up with a makefile I'm pretty happy with :

.PHONY: all split calibrate

.EXPORT_ALL_VARIABLES:

DATASET_PATH   =${ATOM_DATASETS}/zau/dataset_filtered
DATASET        =${DATASET_PATH}/dataset_corrected_with_odometry_and_depth_and_rgb_and_pattern_poses_filtered.json
DATASET_TRAIN  =${DATASET_PATH}/dataset_corrected_with_odometry_and_depth_and_rgb_and_pattern_poses_filtered_train.json
DATASET_TRAINED=${DATASET_PATH}/atom_calibration.json
DATASET_TEST   =${DATASET_PATH}/dataset_corrected_with_odometry_and_depth_and_rgb_and_pattern_poses_filtered_test.json
CSF            =lambda x: True
RNM						 =0.01
TOL						 =1e-3

all: split calibrate evaluate

test:
	@echo $$DATASET_PATH 

split:
	@ echo "..." && \
		echo "..." && \
		echo "\n Splitting Dataset\n" && \
		echo "..." && \
		echo "..." && \
		rosrun atom_calibration split_atom_dataset \
		-json ${DATASET} \
		-ss 1 -tp 70 

calibrate:

	@ echo "..." && \
		echo "..." && \
		echo "\n Calibrating\n" && \
		echo "..." && \
		echo "..." && \
		rosrun atom_calibration calibrate \
		-json ${DATASET_TRAIN} -uic -v  \
		-ssf "lambda x: x in ['rgb_body_left','rgb_body_right','rgbd_hand_color','rgbd_hand_depth','lidar_body']" \
		 -ftol ${TOL} -xtol ${TOL} -gtol ${TOL} -rnm ${RNM} && \
		echo "..." && \
		echo "..." && \
		echo "\n Evaluating RGB to RGB \n" && \
		echo "..." && \
		echo "..."

evaluate: rgb_rgb_evaluations rgb_lidar_evaluations

rgb_rgb_evaluations:
	@ ./rgb_rgb_eval "rgb_body_left" "rgbd_hand_color"
	@ ./rgb_rgb_eval "rgb_body_right" "rgbd_hand_color"

rgb_lidar_evaluations:
	@ ./rgb_lidar_eval "lidar_body" "rgb_body_right"
	@ ./rgb_lidar_eval "lidar_body" "rgb_body_left"
	@ ./rgb_lidar_eval "lidar_body" "rgbd_hand_color"

With this script, one can choose to run the whole pipeline, or just all evaluations, of just a certain type of evaluations. All the env variables declared in the beginning are available to every target in the makefile (including the scripts called within it). Each env variable can also be changed in runtime when calling the makefile like this example:

make RNM=0.0001 calibrate

For the individual evaluations, I made two simple scripts that takes in 2 arguments to make the makefile less clutered.

#! /bin/bash

# Check if the correct number of arguments are passed
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <sensor_source> <sensor_target>"
    exit 1
fi

SENSOR_SOURCE=$1
SENSOR_TARGET=$2

echo "..."
echo "..."
echo "Evaluating $SENSOR_SOURCE to $SENSOR_TARGET"
echo "..."
echo "..."

rosrun atom_evaluation rgb_to_rgb_evaluation \
-train_json $DATASET_TRAINED \
-test_json $DATASET_TEST \
-csf "$CSF" \
-ss "$SENSOR_SOURCE" \
-st "$SENSOR_TARGET" \
-pn "pattern_1" \
-uic

The other script for rgb_to_lidar_eval is a sibling script of this one shown here. I didn't make similar scripts for the calibrate and split because the code would not repeated and it would hurt readability IMO.

This method is also easy to expand, so I think its the best midground between doing everything manual and doing batches.

Annotate the filtered dataset

This is done for rgb_body_left,rgb_body_right,rgbd_hand_color and rgbd_hand_color.

Test each type of evaluation to assess if they are working as intended.

• RGB to RGB works well.
• RGB to Lidar is giving errors. will create a issue and tag it here.

[Kazadhum]

Hi @brunofavs! I see you have the rgb-rgb evaluations working. What results have you gotten for these?

[miguelriemoliveira]

Hi @brunofavs ,

I would not go with a makefile but if it works for you great.

[brunofavs]

Hey @Kazadhum, I'm first testing which evaluation scripts are working before making conclusions. Haven't gotten results to show yet.

[brunofavs]

Rgb to lidar evaluation is working. #988 fixed it.

[miguelriemoliveira]

Nice. Thanks.

[brunofavs]

Hey @miguelriemoliveira @Kazadhum !

I have a lot of results to show. We should meet soon. I have a personal matter tomorrow morning and classes on Tuesday afternoon. Other than that, Im free all week.

I have results from every evaluation for RNM values between 1 and 0.00005.
From what I saw, I found the sweetspot for RNM around 0.125, where for example for the evaluation rgb_body_left to rgbd_hand_color had a RMS error of 8.7px.

I have a lot of raw data. Until the meeting Im working on developing the best method to visualize it.

[Kazadhum]

Hi @brunofavs! Could you post the results from the evaluations you ran for RNM 0.125?

[brunofavs]

rgbd_hand_depth_to_lidar_body_rnm_0.125.csv
rgbd_hand_color_to_rgbd_hand_depth_rnm_0.125.csv
rgb_body_right_to_rgbd_hand_depth_rnm_0.125.csv
rgb_body_left_to_rgbd_hand_depth_rnm_0.125.csv
rgbd_hand_color_to_lidar_body_rnm_0.125.csv
rgb_body_left_to_lidar_body_rnm_0.125.csv
rgb_body_right_to_lidar_body_rnm_0.125.csv
rgb_body_right_to_rgbd_hand_color_rnm_0.125.csv
rgb_body_left_to_rgbd_hand_color_rnm_0.125.csv

[Kazadhum]

Thank you! I can meet anytime to discuss these!

[miguelriemoliveira]

Hi @brunofavs ,

Thanks for the results. We can meet tomorrow morning? At 9h is fine by me ...

[brunofavs]

Hey @miguelriemoliveira , tomorrow at 9am is great.

[Kazadhum]

Tomorrow at 9 AM sounds good!

[brunofavs]

Hey @miguelriemoliveira @Kazadhum :)

Turns out my plotting code was failing because I decided to align the data csv before our meeting to make it more readable and saved it on accident, thus changing the whitespace header. Oh well....

Anyway, this is the heatmap I wanted to show you :

heatmap_rms_rnm_sensors.pdf

The color represents the error, meaning a darker color square has higher error. It's hard to figure out the best RNM from this but we can see the influence it has on each evaluation.

And this is the plot I showed :

mean_rms_varying_rnm.pdf

And this is the pivot table I showed :

RNM    ,rgb_body_left_to_lidar_body ,rgb_body_left_to_rgbd_hand_color ,rgb_body_left_to_rgbd_hand_depth ,rgb_body_right_to_lidar_body ,rgb_body_right_to_rgbd_hand_color ,rgb_body_right_to_rgbd_hand_depth ,rgbd_hand_color_to_lidar_body ,rgbd_hand_color_to_rgbd_hand_depth ,rgbd_hand_depth_to_lidar_body 
5e-06  ,                     9.2114 ,                          9.9563 ,                          7.2326 ,                      9.559  ,                           4.6254 ,                           9.4765 ,                      14.732  ,                            7.0586 ,                       9.396  
5e-05  ,                     8.6228 ,                          9.9653 ,                          7.2453 ,                      9.4257 ,                           4.6234 ,                           9.4969 ,                      14.4075 ,                            7.0704 ,                       9.3271 
0.0005 ,                     8.6303 ,                          9.9555 ,                          7.2449 ,                      9.4138 ,                           4.6255 ,                           9.5235 ,                      14.3974 ,                            7.0742 ,                       9.3376 
0.005  ,                     8.5802 ,                         10.9481 ,                          7.9056 ,                     10.9031 ,                           4.4073 ,                           8.4462 ,                      15.2869 ,                            6.5245 ,                       9.6579 
0.05   ,                     4.3595 ,                         10.7553 ,                          7.5344 ,                      7.6999 ,                           3.6265 ,                           8.2836 ,                      10.3164 ,                            5.4549 ,                       7.6275 
0.125  ,                     4.4011 ,                          8.7354 ,                          6.3801 ,                      8.9587 ,                           6.3399 ,                           7.5553 ,                      11.3071 ,                            4.3812 ,                       7.7707 
0.25   ,                     4.2865 ,                         10.6351 ,                          7.3357 ,                      8.4938 ,                           6.0173 ,                           7.8417 ,                      11.5463 ,                            4.3771 ,                       7.6213 
0.5    ,                     4.4152 ,                         11.9839 ,                          7.8181 ,                      7.4377 ,                           6.005  ,                           8.7278 ,                      11.9266 ,                            4.1051 ,                       7.6529 
1.0    ,                     4.5638 ,                         12.3756 ,                          8.4718 ,                      7.7448 ,                           7.0084 ,                           9.6116 ,                      12.3623 ,                            3.9774 ,                       7.786  

[miguelriemoliveira]

Thanks @brunofavs . Nice plot. I think the conclusions we extracted during the meeting remain the same, so keep up the good work.

[brunofavs]

Closing as we've moved to the next phase already.