feat(multi_object_tracker): multi object input #6820

technolojin · 2024-04-16T00:37:46Z

Description

Enable multi_object_tracker subscribe multiple detected objects topics.

This PR contains the following components/updates.

Channel configuration: Provide all available sources of detections.
Input manager: Manage multiple detected object message channels.
Process timing algorithm: Operate the tracking process in an optimal set of objects to reduce latency.
Existence model: Model the existence probability of each detection source to understand the contributions of detections.
Debugger: Visualize association results and existence probability on each channel.

The following is an example of possible perception pipeline diagram (the PR do not contain changes of the pipeline).

Tests performed

multi_input_mot_test1.mp4

debug message
- topic: /perception/object_recognition/tracking/multi_object_tracker/debug/objects_markers
- type: Marker
- Lines: Association
- Cubic
  - Colored: detection position
  - White: tracked object position
- Detections (In this example) name, short name, color-code
  - Lidar Centterpoint, Lcp, blue
  - Detection by Tracker, dbT, green
  - Radar far, Rf, yellow
Tests on Autoware enviornments
Tests on the simulations provided by the Autoware tutorials were operated and the system worked.
- Rosbag replay simulation sample data https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ad-hoc-simulation/rosbag-replay-simulation/
- AWSIM https://tier4.github.io/AWSIM/GettingStarted/QuickStartDemo/

Notes for reviewers

prior PR on autoware_launch.
This PR will be merged after the launcher is updated.

Interface changes

Additional configuration of input channel selection is added.

Effects on system behavior

Default value is single input.
The overall perception pipeline will work as same as before.

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.
The PR has been properly tested.
The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.
The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp

…ebugger/debug_object.hpp Co-authored-by: Shunsuke Miura <37187849+miursh@users.noreply.github.com> Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp

perception/multi_object_tracker/src/debugger/debugger.cpp

perception/multi_object_tracker/src/processor/input_manager.cpp

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

YoshiRi

LGTM with minor comments

perception/multi_object_tracker/src/processor/input_manager.cpp

perception/multi_object_tracker/src/tracker/model/tracker_base.cpp

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

wep21 · 2024-05-26T13:21:31Z

perception/multi_object_tracker/src/processor/input_manager.cpp

+  const rclcpp::Time & object_latest_time, const rclcpp::Time & object_oldest_time,
+  ObjectsList & objects_list)
+{
+  assert(object_latest_time.nanoseconds() > object_oldest_time.nanoseconds());


@YoshiRi @technolojin @yukkysaito @miursh When launching planning simulator, this assertion always occurs at the start time in my environment. I appreciate it if someone could check if this assertion is valid.

@wep21 Thanks.
@technolojin May I ask for confirmation?

@wep21 @yukkysaito
Thank you for your report. I will investigate it.

@wep21
I made a fix. #7127
However, I couldn't reproduce the situation with the planning simulator.

Can you test this modification and let me know if the issue is resolved?

* refactor: frequently used types, namespace Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * test: multiple inputs Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: check latest measurement time Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: define input manager class Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: interval measures Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: store and sort inputs PoC Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: rename classes Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: object collector Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * impl input manager, no subscribe Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: subscribe and trigger callback Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: subscriber and callbacks are working Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: callback object is fixed, tracker is working Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: get object time argument revise Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: back to periodic publish, analyze input latency and timings Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: enable timing debugger Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: separate object interval function Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: prepare message triggered process Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: trigger tracker by main message arrive Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: clean-up, set namespace Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: object lists with detector index Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: define input channel struct Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: define type for object list Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add channel wise existence probability Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: relocate debugger Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: total existence logic change Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: publishing object debug info, need to fix marker id Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: indexing marker step 1 Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: uuid management Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: association line fix Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: print channel names Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: association lines are color-coded Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: association debug marker bugfix Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * style(pre-commit): autofix Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add option for debug marker Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: skip time statistics update in case of outlier Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: auto-tune latency band Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: pre-defined channels, select on launcher Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: add input channels Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: remove marker idx map Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: to do not miss the latest message of the target stream Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: remove priority, separate timing optimization Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: time interval bug fix Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: refactoring timing state update Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: set parameters optionally Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: revise object time range logic Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: launcher to set input channels Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: exempt spell check 'pointpainting' Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: remove expected interval Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: implement spawn switch Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: remove debug messages Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: update readme Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: change tentative object topic Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * Revert "fix: remove debug messages" This reverts commit 725a49e. Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: reset times when jumps to past Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: check if interval is negative Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: missing config, default value Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: remove debug messages Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: change no-object message level Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * Update perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp Co-authored-by: Shunsuke Miura <37187849+miursh@users.noreply.github.com> Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: Update copyright to uppercase Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: fix readme links to config files Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: move and rename uuid functions Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: fix debug topic to use node name Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * chore: express meaning of threshold Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * feat: revise decay rate, update function Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * fix: define constants with explanation Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> * style(pre-commit): autofix --------- Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Shunsuke Miura <37187849+miursh@users.noreply.github.com>

technolojin self-assigned this Apr 16, 2024

github-actions bot added the component:perception Advanced sensor data processing and environment understanding. (auto-assigned) label Apr 16, 2024

technolojin force-pushed the feat/mot-multi-object-input branch from cad733b to d8d35cd Compare May 8, 2024 04:02

github-actions bot added the component:launch Launch files, scripts and initialization tools. (auto-assigned) label May 8, 2024

technolojin force-pushed the feat/mot-multi-object-input branch from d8d35cd to d0bd1b2 Compare May 9, 2024 08:24

technolojin linked an issue May 9, 2024 that may be closed by this pull request

Replace serialized detected object mergers to reduce complexity #6958

Closed

3 tasks

technolojin force-pushed the feat/mot-multi-object-input branch from 13dfe38 to 1f533f0 Compare May 13, 2024 06:33

github-actions bot added the type:documentation Creating or refining documentation. (auto-assigned) label May 13, 2024

technolojin mentioned this pull request May 14, 2024

feat(multi_object_tracker): add multi object input config file autowarefoundation/autoware_launch#989

Merged

4 tasks

technolojin added 21 commits May 14, 2024 15:03

refactor: frequently used types, namespace

5412a55

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

test: multiple inputs

4176cec

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: check latest measurement time

96be0cc

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: define input manager class

fd086dc

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: interval measures

1462eef

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: store and sort inputs PoC

7757212

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

chore: rename classes

2a443e7

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: object collector

866b656

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

impl input manager, no subscribe

4b71b57

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

fix: subscribe and trigger callback

a9fa209

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

fix: subscriber and callbacks are working

cf2874d

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

fix: callback object is fixed, tracker is working

246cb4a

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

fix: get object time argument revise

9ebaf6d

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: back to periodic publish, analyze input latency and timings

c496a25

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

fix: enable timing debugger

969483c

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

fix: separate object interval function

ceaacc3

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: prepare message triggered process

c556460

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: trigger tracker by main message arrive

4afab3f

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

chore: clean-up, set namespace

23a9df4

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: object lists with detector index

787d48b

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

feat: define input channel struct

b4993c2

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

technolojin marked this pull request as ready for review May 14, 2024 08:29

technolojin requested review from YoshiRi, yukkysaito and miursh as code owners May 14, 2024 08:29

miursh reviewed May 15, 2024

View reviewed changes

perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp Outdated Show resolved Hide resolved

technolojin added the tag:run-build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label May 16, 2024

technolojin and others added 3 commits May 16, 2024 16:54

Update perception/multi_object_tracker/include/multi_object_tracker/d…

f873ab8

…ebugger/debug_object.hpp Co-authored-by: Shunsuke Miura <37187849+miursh@users.noreply.github.com> Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

chore: Update copyright to uppercase

1ba2c79

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

chore: fix readme links to config files

9edd2e0

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

technolojin force-pushed the feat/mot-multi-object-input branch from 827ead6 to 9edd2e0 Compare May 16, 2024 07:55

yukkysaito reviewed May 16, 2024

View reviewed changes

perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp Outdated Show resolved Hide resolved

yukkysaito reviewed May 16, 2024

View reviewed changes

perception/multi_object_tracker/src/debugger/debugger.cpp Outdated Show resolved Hide resolved

yukkysaito reviewed May 16, 2024

View reviewed changes

perception/multi_object_tracker/src/processor/input_manager.cpp Outdated Show resolved Hide resolved

technolojin added 3 commits May 17, 2024 10:21

chore: move and rename uuid functions

cf6cf3c

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

chore: fix debug topic to use node name

d5efe10

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

chore: express meaning of threshold

4445c45

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

technolojin force-pushed the feat/mot-multi-object-input branch from 2ffb7d8 to 4445c45 Compare May 17, 2024 01:22

technolojin requested a review from yukkysaito May 17, 2024 01:25

YoshiRi approved these changes May 23, 2024

View reviewed changes

perception/multi_object_tracker/src/processor/input_manager.cpp Outdated Show resolved Hide resolved

perception/multi_object_tracker/src/tracker/model/tracker_base.cpp Outdated Show resolved Hide resolved

technolojin and others added 3 commits May 23, 2024 11:55

feat: revise decay rate, update function

f218f84

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

fix: define constants with explanation

b623993

Signed-off-by: Taekjin LEE <taekjin.lee@tier4.jp>

style(pre-commit): autofix

13adc73

technolojin merged commit 3aa84b0 into autowarefoundation:main May 23, 2024
22 of 24 checks passed

wep21 reviewed May 26, 2024

View reviewed changes

technolojin mentioned this pull request May 27, 2024

fix(multi_object_tracker): input stream timimg functions #7127

Merged

4 tasks

technolojin mentioned this pull request Jun 12, 2024

feat(tier4_perception_launch): enable multi channel tracker merger #7459

Merged

4 tasks

technolojin deleted the feat/mot-multi-object-input branch July 26, 2024 02:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(multi_object_tracker): multi object input #6820

feat(multi_object_tracker): multi object input #6820

technolojin commented Apr 16, 2024 •

edited

Loading

YoshiRi left a comment

wep21 May 26, 2024 •

edited

Loading

yukkysaito May 26, 2024

technolojin May 27, 2024

technolojin May 27, 2024

feat(multi_object_tracker): multi object input #6820

feat(multi_object_tracker): multi object input #6820

Conversation

technolojin commented Apr 16, 2024 • edited Loading

Description

Related links

Tests performed

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

YoshiRi left a comment

Choose a reason for hiding this comment

wep21 May 26, 2024 • edited Loading

Choose a reason for hiding this comment

yukkysaito May 26, 2024

Choose a reason for hiding this comment

technolojin May 27, 2024

Choose a reason for hiding this comment

technolojin May 27, 2024

Choose a reason for hiding this comment

technolojin commented Apr 16, 2024 •

edited

Loading

wep21 May 26, 2024 •

edited

Loading