Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

model based tracking for a M12 bolt #1455

Open
TAO-YT opened this issue Aug 16, 2024 · 8 comments
Open

model based tracking for a M12 bolt #1455

TAO-YT opened this issue Aug 16, 2024 · 8 comments

Comments

@TAO-YT
Copy link

TAO-YT commented Aug 16, 2024

I followed the instructions of Tutorial: Markerless generic model-based tracking using a color camera on VISP to track a M12 bolt. But the tracked Z-axis is always unstable when the bolt was still.

12315d489c8be64d4d1a22c93e7fa368.mp4

I have only changed the CAO model and some parameters of XML file. I saw the videos of some tracking results , they are always precise and stable. I want to know how to make it. Please someone help me.

@TAO-YT
Copy link
Author

TAO-YT commented Aug 16, 2024

what I want is to estimae the pose of M12 bolt so that I can control the UR robot to attach it but the pose is not accurate because the Z-axis (blue) is always moving

@s-trinh
Copy link
Contributor

s-trinh commented Aug 16, 2024

Hello,

Could you try without using depth information?
Looks like at this distance, you are outside of the range of the depth sensor.

@TAO-YT
Copy link
Author

TAO-YT commented Aug 17, 2024

The camera I use is realsense D435i, and what I want to do is to estimate the CMO which is the pose of the M12 bolt related to the camera by the model based tracking, then I can control the UR by PBVS. So I think it's nesscary to use depth information.What actually troubles me is the Z-axis is always moving like these videoes especially when the camera is very close.

wyt1_20240817_14565030.mp4
wyt2_20240817_14572966_20240817_14591260.mp4

The Z-axis is always inaccurate and unstable,whether the bolts are placed separately or installed on components. But I have read the instrctions of Tutorial: Markerless generic model-based tracking using a color camera, and seen the videoes of tracking teabox and lego square. I saw the results of the three axis are very precise and stable even they are moving. I don't know what is the key except for changing the .cao file and xml file. I have been stuck for several months but I think that the model based tracking should be suitable for tracking the bolt like this. So I really don't know how to solve that.

@TAO-YT
Copy link
Author

TAO-YT commented Aug 17, 2024

And the distance between the camera and blot is about 20cm-50cm

@s-trinh
Copy link
Contributor

s-trinh commented Aug 17, 2024

Thanks for the videos.

My opinion on this:


image

Here you can see some contours are being falsely tracked. You can try to adjust the visibility parameter for faces.

In any cases, the object to be tracked is very small. Modeling also the support should improve the tracking.

@TAO-YT
Copy link
Author

TAO-YT commented Aug 19, 2024

Thanks for your opinions. Well I know the bolt is too small, but the real thing I want to do is to tighten the bolts by UR16e. So it's necessary to tracking the bolt. But I wonder that maybe the tracking object is too small so that the Z-axis is unstable like that. And I'm sorry that I don't know how to change the HighAccuracyPreset.json file of the Realsense camera in VISP which maybe can improve the depth information, so could you please tell me? This vedio is the process of tracking and visiual servoing the bolt

89f98c9fc878d702b64e2167283ccd43.mp4

I have adjust the paraments of xml file for weeks including the visibility parameter for faces. I think this xml file is the best one by testing.
luoshuan12(back)xml.txt
But the problem is still here, it doesn't help too much. My abilities are really limited and I'm wondering that maybe this MBT can't track small things well? Is there anyways which can track the bolt well? I have seen the last instruction of tracking part in VISP is about tracking with megapose. Do you think this way can track bolt well?

@fspindle
Copy link
Contributor

As mentioned, it seems that your D435i does not provide depth information on your bolt (corresponds to the black areas in the depth image). I can see two possible reasons for this:

  • The camera specification states that the minimum depth distance (Min-Z) at maximum resolution is approximately 28cm. The first reason is therefore that the camera is less than 30 cm from the bolt.
  • If the distance between your camera and the bolt is greater than 30 cm, this may also be due to the material and/or the grey colour of the bolt which completely absorbs infrared, so that the infrared camera is blind and unable to provide depth information on the grey parts of the object.
    If you can't overcome one of these 2 reasons, you can remove the use of depth in the tracker.

I don't know of any 3D tracker capable of providing an accurate pose. As already mentioned, the solution in your case is to track a larger object. I would start with the planar support drawn in red in the following image.

bolt

The cad model is defined in a frame (I arbitrarily choose the one in the bottom right-hand corner, labelled o). If the tracking is successful, you'll get the cMo transformation. Since you're interested in the top bolt (with the frame named b), you ‘just’ need to add the additional oMb constant transformation. The PBVS only needs to consider the product (cMo * oMb).

If the tracking is more stable, you can then enrich the model with the top bolt and why not the bottom ones.

If that doesn't work any better, you can try Megapose, but you'll need a PC with a graphics card.

@TAO-YT
Copy link
Author

TAO-YT commented Aug 22, 2024

Thank you for your reply. Your suggestions are very insightful. Then, I will study issues related to depth information. It seems that my method did not take depth information into consideration, which may be the reason why the z-axis keeps bouncing. As for the method you mentioned of servoing the big part, I have also considered it. But the bolt I want to track is installed on many different devices, which means modeling work needs to be done lots of times, so this idea was rejected. And know I will continue to improve it. If the result is still not good, I will consider using the megapose method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants