Creating an image display rviz plugin with clicking functionality

[miguelriemoliveira]

For the atom dataset reviewer to be fully integrated we need this functionality.

The idea is to develop an rviz plugin in a package called

atom_rviz_plugin which will have this functionality ...

@manuelgitgomes will work with me on this.

@danifpdra FYI ...

[miguelriemoliveira]

Publication is working now

[miguelriemoliveira]

Now for the topic type I suggest to use this:

http://docs.ros.org/en/api/geometry_msgs/html/msg/PointStamped.html

will try to implement this afternoon ...

[miguelriemoliveira]

Some progress. Now the topic name is automatically configured. For example, if the image topic is

camera_1/rgb/image_raw

then the topic which will contain the clicked coordinates will be:

camera_1/rgb/image_raw/click      

Also, it works if we change the image topic on runtime.

[miguelriemoliveira]

Hi @manuelgitgomes ,

Just commited a first functional version. Call me when you want to look into it.

[manuelgitgomes]

Thank you professor @miguelriemoliveira!
I think tomorrow morning I will work on jt!

[miguelriemoliveira]

Made a simple example of a python node that subscribes the click msgs and draws lines on the image...

[miguelriemoliveira]

Hi @manuelgitgomes ,

This last commit creates a complete pipeline of what we want. I will now stop working on this and leave it for you ok?
If you need help just let me know.

[manuelgitgomes]

Ok, sorry for not doing it this morning.
Thank you very much!

[miguelriemoliveira]

The idea was you worked with me. But its not a problem. Now if you need help let me know.

The goal is to have a python node that receives the click messages and allows the user to define the boundary of the pattern.

[miguelriemoliveira]

Perhaps a new issue?

[manuelgitgomes]

I have the issue #374 for that!

[manuelgitgomes]

Added the "black bars on the side" functionality. I believe with this everything on this plugin is done, now work can be done on a dataset_reviewer for depth/rgb images. Results can be seen here.
I believe this issue can be close, with further work in #374.

[miguelriemoliveira]

I @manuelgitgomes .

Nice. Agree, the issue can be closed for now. Perhaps later we could include some key pressing event.

[miguelriemoliveira]

I have the issue #374 for that!

ok, great.

[manuelgitgomes]

Hello @miguelriemoliveira and @danifpdra!
I'm reopening this issue due to a bug on this plugin. When opening an image_click window with an image window open or vice versa, rviz crashes with this error:

[ WARN] [1649321690.502808192]: SEVERE WARNING!!! A namespace collision has occured with plugin factory for class atom_rviz::ImageDisplayWithClick. New factory will OVERWRITE existing one. This situation occurs when libraries containing plugins are directly linked against an executable (the one running right now generating this message). Please separate plugins out into their own library or just don't link against the library and use either class_loader::ClassLoader/MultiLibraryClassLoader to open.

It seems like a namespace colision between our plugin and the image plugin. I will try to fix it.
If any of you have any idea, please share it!

[manuelgitgomes]

Found this website giving some answers.
I think I will leave this issue behind and work fully on issue #374 for now

[miguelriemoliveira]

Hi @manuelgitgomes ,

I had this problem also, but it was a warning only, rviz did not crash.

I will try to solve.

[miguelriemoliveira]

I did several tests but for now could not make it work ... Will continue in the afternoon.

[manuelgitgomes]

Ok, thank you very much.
To verify I also had this writing after the warning. It may help us better understand the problem

[ WARN] [1649336619.799353577]: OGRE EXCEPTION(4:ItemIdentityException): SceneManager instance called 'ImageDisplay0' already exists in SceneManagerEnumerator::createSceneManager at /build/ogre-1.9-kiU5_5/ogre-1.9-1.9.0+dfsg1/OgreMain/src/OgreSceneManagerEnumerator.cpp (line 204) terminate called after throwing an instance of 'Ogre::ItemIdentityException' what(): OGRE EXCEPTION(4:ItemIdentityException): SceneManager instance called 'ImageDisplay0' already exists in SceneManagerEnumerator::createSceneManager at /build/ogre-1.9-kiU5_5/ogre-1.9-1.9.0+dfsg1/OgreMain/src/OgreSceneManagerEnumerator.cpp (line 204)

[miguelriemoliveira]

Thanks @manuelgitgomes ,

I searched around. Actually, these are different errors. The Ogre one only occurs when you have in rviz both a display of type ImageDisplayWithClick and another of ImageDisplay.

I was able to solve this problem, however, the first warning about the namespace collision I still don't know how to solve ...

[manuelgitgomes]

Hello @miguelriemoliveira!
Solving this OGRE problem, rviz does not crash anymore.
There is a bug that might possibly be from the namespace warning.
When loading an ImageDisplayWithClick window when an ImageDisplay window is already open, the first one always displays the same as the second one. When the ImageDisplay window is closed, the ImageDisplayWithClick displays the "No Image" display.
This bug can be seen in this video. At the end, we can see the plugin working when there only exists an ImageDisplayWithClick window.
If no solution can be found, we can replace every ImageDisplay with an ImageDisplayWithClick,

[miguelriemoliveira]

I have tried all afternoon to bring stuff from rviz that is needed. I am now with 10 copied files from rviz in an attempt to compute our image_display_with_click.

I give up for now.

Another even more interesting avenue would be to have this functionality integrated into rviz.

I wrote a message here.

ros-visualization/rviz#916

Let's see the replies.

[miguelriemoliveira]

I was able to integrate this in rviz and just created a pull request. Lets see it is accepted.

ros-visualization/rviz#1737

@manuelgitgomes and @danifpdra , in the meantime you could use my rviz fork which works fine:

https://github.com/miguelriemoliveira/rviz

Just do a

sudo apt-get remove ros-noetic-rviz

and then git clone to your catkin_ws.

[manuelgitgomes]

That's great! Thank you!

[manuelgitgomes]

@miguelriemoliveira they requested some changes on:
ros-visualization/rviz#1737
Should we think about this in the future? To keep this available for everyone

[miguelriemoliveira]

Hi @manuelgitgomes ,

thanks for the reminder. Definitely we want to advance with this.
I plan to look into this sometime on Friday. If you want we can do it together.

[manuelgitgomes]

Sure!