feat(inputs.mavlink): Add plugin #16221

chrisdalke · 2024-11-22T12:48:30Z

Summary

Add a Mavlink input plugin.

The mavlink plugin connects to a MavLink-compatible flight controller such as as ArduPilot or PX4. and translates all incoming messages into metrics.

The purpose of this plugin is to allow Telegraf to be used to ingest live flight metrics from unmanned systems (drones, planes, boats, etc.)

Telegraf is already often used on flight computers (eg a Raspberry Pi) to collect system metrics for drones and it would be valuable to extend this to also provide a convenient way to record flight telemetry.

TODO

gomavlib is currently on a fork to support reading serial ports on i386 and darwin. Will need to upstream the fix for this and move back to the main repo after merge, or before merge if the forked gomavlib is not acceptable

Checklist

No AI generated code was used in this PR

Related issues

Resolves feat(inputs): Add Mavlink input plugin #16220

…to feature/mavlink-input-plugin

…nput-plugin

plugins/inputs/mavlink/mavlink.go

plugins/inputs/mavlink/sample.conf

plugins/inputs/mavlink/README.md

plugins/inputs/mavlink/mavlink.go

plugins/inputs/mavlink/mavlink_utils.go

plugins/inputs/mavlink/mavlink.go

plugins/inputs/mavlink/mavlink_utils.go

Hipska

Going in good direction!

plugins/inputs/mavlink/mavlink.go

plugins/inputs/mavlink/mavlink_utils.go

plugins/inputs/mavlink/mavlink.go

plugins/inputs/mavlink/sample.conf

plugins/inputs/mavlink/mavlink_utils.go

srebhan

@chrisdalke thanks for your contribution! Please check my comments in the code. Furthermore, I do have some general remarks

I'm hesitant to merge plugins using forked one-man dependencies as experience shows that this is not maintainable on the long run. In your case I see two possibilities, the first being we hold-up the plugin until the upstream library merged your PR. This will likely take longer if possible at all.
The second option is to base your plugin on the upstream library and limit the availablility of the plugin to the platforms supported (excluding i386 and darwin if I understand correctly). What do you think?
Please keep the comments in the sample.conf file brief and intend the file with two spaces.
Is there some way of mocking a Mavlink controller endpoint? E.g. using a minimal implementation mimicking some messages or a test-container/simulator?

Looking forward to your update!

srebhan · 2024-12-11T14:10:59Z

docs/LICENSE_OF_DEPENDENCIES.md

@@ -96,6 +96,8 @@ following works:
 - github.com/caio/go-tdigest [MIT License](https://github.com/caio/go-tdigest/blob/master/LICENSE)
 - github.com/cenkalti/backoff [MIT License](https://github.com/cenkalti/backoff/blob/master/LICENSE)
 - github.com/cespare/xxhash [MIT License](https://github.com/cespare/xxhash/blob/master/LICENSE.txt)
+- github.com/chrisdalke/go-serial [BSD 3-Clause License](https://github.com/chrisdalke/go-serial/blob/master/LICENSE)


Why not using the upstream bugst/serial instead of your fork?

Returning this to the upstream bugst/serial, there is still one more forked dependency gomavlib which I have a PR to try and fix platform support for.

In your case I see two possibilities, the first being we hold-up the plugin until the upstream library merged your PR. This will likely take longer if possible at all.
The second option is to base your plugin on the upstream library and limit the availablility of the plugin to the platforms supported (excluding i386 and darwin if I understand correctly). What do you think?

Hoping to get the upstream PR to gomavlib merged. The PR just changes the serial library to add wider platform support. I'd like this plugin to work on all platforms that Telegraf supports.

Fork is resolved now, all new libraries are pointing at their main repo.

srebhan · 2024-12-11T14:20:59Z

plugins/inputs/mavlink/README.md

+The `mavlink` plugin connects to a MavLink-compatible flight controller such as
+ [ArduPilot](https://ardupilot.org/) or [PX4](https://px4.io/). and translates
+all incoming messages into metrics.
+
+The purpose of this plugin is to allow Telegraf to be used to ingest live
+ flight metrics from unmanned systems (drones, planes, boats, etc.)
+
+Warning: This input plugin potentially generates a large amount of data! Use
+the configuration to limit the set of messages, or use another telegraf plugin
+to filter the output.
+
+This plugin currently uses the ArduPilot-specific Mavlink dialect. See the
+[Mavlink docs](https://mavlink.io/en/messages/ardupilotmega.html) for more
+info on dialects and the various messages that this plugin can receive.


Please use shortened links and add the metadata to this plugin

Suggested change

The `mavlink` plugin connects to a MavLink-compatible flight controller such as

[ArduPilot](https://ardupilot.org/) or [PX4](https://px4.io/). and translates

all incoming messages into metrics.

The purpose of this plugin is to allow Telegraf to be used to ingest live

flight metrics from unmanned systems (drones, planes, boats, etc.)

Warning: This input plugin potentially generates a large amount of data! Use

the configuration to limit the set of messages, or use another telegraf plugin

to filter the output.

This plugin currently uses the ArduPilot-specific Mavlink dialect. See the

[Mavlink docs](https://mavlink.io/en/messages/ardupilotmega.html) for more

info on dialects and the various messages that this plugin can receive.

This plugin collects metrics from [MavLink][mavlink]-compatible flight controllers

such as [ArduPilot][ardupilor] or [PX4][px4] to live ingest flight metrics from

unmanned systems (drones, planes, boats, etc.)

Currently the ArduPilot-specific Mavlink dialect is used, check the

[Mavlink documentation][mavlink_docs] for more details and the various

messages available.

> [!WARNING]

> This plugin potentially generates a large amount of data! Please use metric

> filters to limit the amount of metrics produced!

⭐ Telegraf v1.34.0

🏷️ iot

💻 all

[mavlink]: https://mavlink.io

[ardupilor]: https://ardupilot.org/

[px4]: https://px4.io/

[mavlink_docs]: https://mavlink.io/en/messages/ardupilotmega.html

srebhan · 2024-12-11T14:21:42Z

plugins/inputs/mavlink/sample.conf

@@ -0,0 +1,39 @@
+# Read metrics from a Mavlink connection to a flight controller.


Suggested change

# Read metrics from a Mavlink connection to a flight controller.

# Read metrics from a Mavlink flight controller.

srebhan · 2024-12-11T14:25:10Z

plugins/inputs/mavlink/sample.conf

+     ## Flight controller URL. Must be a valid Mavlink connection string in one
+     ## of the following formats:
+     ##
+     ## - Serial port:  serial://<device name>:<baud rate> 
+     ##            eg: "serial:///dev/ttyACM0:57600"
+     ## 
+     ## - TCP client:   tcp://<target ip or hostname>:<port>
+     ##            eg: "tcp://192.168.1.12:14550"
+     ## 
+     ## - UDP client:   udp://<target ip or hostname>:<port>
+     ##            eg: "udp://192.168.1.12:14550"
+     ## 
+     ## - UDP server:   udp://:<listen port>
+     ##            eg: "udp://:14540"
+     ## 
+     ## The meaning of each of these modes is documented by
+     ## https://mavsdk.mavlink.io/v1.4/en/cpp/guide/connections.html.


How about shortening this?

Suggested change

## Flight controller URL. Must be a valid Mavlink connection string in one

## of the following formats:

##

## - Serial port: serial://<device name>:<baud rate>

## eg: "serial:///dev/ttyACM0:57600"

##

## - TCP client: tcp://<target ip or hostname>:<port>

## eg: "tcp://192.168.1.12:14550"

##

## - UDP client: udp://<target ip or hostname>:<port>

## eg: "udp://192.168.1.12:14550"

##

## - UDP server: udp://:<listen port>

## eg: "udp://:14540"

##

## The meaning of each of these modes is documented by

## https://mavsdk.mavlink.io/v1.4/en/cpp/guide/connections.html.

## Flight controller URL supporting serial port, UDP and TCP connections.

## The modes are documented at

## https://mavsdk.mavlink.io/v1.4/en/cpp/guide/connections.html.

##

## Examples:

## - Serial port: serial:///dev/ttyACM0:57600

## - TCP client: tcp://192.168.1.12:14550

## - UDP client: udp://192.168.1.12:14550

## - UDP server: udp://:14540

srebhan · 2024-12-11T14:25:18Z

plugins/inputs/mavlink/sample.conf

+     ## 
+     ## The meaning of each of these modes is documented by
+     ## https://mavsdk.mavlink.io/v1.4/en/cpp/guide/connections.html.
+     fcu_url = "udp://:14540"


Suggested change

fcu_url = "udp://:14540"

url = "udp://:14540"

srebhan · 2024-12-11T15:25:30Z

plugins/inputs/mavlink/mavlink.go

+
+			case *gomavlib.EventChannelOpen:
+				s.Log.Debugf("Mavlink channel opened")
+
+			case *gomavlib.EventChannelClose:
+				s.Log.Debugf("Mavlink channel closed")


Why do we need those? Should we do something about it if the channel is opened/closed?

These messages indicate that the Mavlink connection has started/stopped, which is separate from the Mavlink client initializing its background thread.

There's nothing Telegraf can do in this case but it helps the user debug the Mavlink setup which can be difficult. One example of where this could occur is if you had two TCP client connections with the same system_id one would get kicked off. Another example is a flaky USB port.

Without these messages it's hard to tell the difference between (1) Mavlink is not connected vs (2) Robot has a problem and is not broadcasting messages

srebhan · 2024-12-11T15:26:04Z

plugins/inputs/mavlink/mavlink.go

+	return nil
+}
+
+func (s *Mavlink) Gather(_ telegraf.Accumulator) error {


Please remove unused receivers and variables.

Suggested change

func (s *Mavlink) Gather(_ telegraf.Accumulator) error {

func (*Mavlink) Gather(telegraf.Accumulator) error {

srebhan · 2024-12-11T15:27:12Z

plugins/inputs/mavlink/mavlink.go

+			FcuURL:                 "udp://:14540",
+			MessageFilter:          make([]string, 0),
+			SystemID:               254,
+			StreamRequestEnable:    true,
+			StreamRequestFrequency: 4,


This should be moved to Init except for StreamRequestEnable and SystemID (if zero is a valid id).

srebhan · 2024-12-11T15:27:54Z

plugins/inputs/mavlink/mavlink.go

+	"github.com/chrisdalke/gomavlib/v3"
+	"github.com/chrisdalke/gomavlib/v3/pkg/dialects/ardupilotmega"
+	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/internal"
+	"github.com/influxdata/telegraf/internal/choice"
+	"github.com/influxdata/telegraf/plugins/inputs"


Please split the internal and 3rd party packages by an empty line

Suggested change

"github.com/chrisdalke/gomavlib/v3"

"github.com/chrisdalke/gomavlib/v3/pkg/dialects/ardupilotmega"

"github.com/influxdata/telegraf"

"github.com/influxdata/telegraf/internal"

"github.com/influxdata/telegraf/internal/choice"

"github.com/influxdata/telegraf/plugins/inputs"

"github.com/chrisdalke/gomavlib/v3"

"github.com/chrisdalke/gomavlib/v3/pkg/dialects/ardupilotmega"

"github.com/influxdata/telegraf"

"github.com/influxdata/telegraf/internal"

"github.com/influxdata/telegraf/internal/choice"

"github.com/influxdata/telegraf/plugins/inputs"

srebhan · 2024-12-11T15:28:33Z

plugins/inputs/mavlink/mavlink_test.go

+	}
+
+	_, err := ParseMavlinkEndpointConfig(testConfig.FcuURL)
+	require.Equal(t, "could not parse fcu_url ftp://not-a-valid-fcu-url", err.Error())


Suggested change

require.Equal(t, "could not parse fcu_url ftp://not-a-valid-fcu-url", err.Error())

require.ErrorContains(t, err, "could not parse fcu_url")

…mavlink-input-plugin

telegraf-tiger · 2024-12-15T23:08:54Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

chrisdalke · 2024-12-16T00:23:46Z

3. Is there some way of mocking a Mavlink controller endpoint? E.g. using a minimal implementation mimicking some messages or a test-container/simulator?

I can prepare an ArduPilot container which will output data over UDP -- Is there a reference to how you'd like this setup to match other plugins?

chrisdalke and others added 6 commits November 21, 2024 08:31

Mavlink test

290d0bf

Working threaded mavlink plugin

8d95aac

fix config

8105d40

Support all client types

24d47db

wip

89264c4

fix

084395f

telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Nov 22, 2024

chrisdalke and others added 13 commits November 22, 2024 07:54

Fix README

12c598f

Fixing README

b8b5030

Fixes

cb3b1b6

Lint

93899a1

Remove unused imports for now

bc756ea

Fix

1151fda

Add some tests, split things into util file

a85be7a

Working tests

4d8de2c

Lint fixes

18e2180

More fixes

24f8c06

Fix problems

9c07968

Fix dependencies

c9f3c80

Test with cross platform fork of gomavlib

b368ccb

chrisdalke mentioned this pull request Nov 25, 2024

Consider changing serial library to improve multiplatform support bluenviron/gomavlib#130

Closed

chrisdalke and others added 7 commits November 24, 2024 19:36

Fix dependency license file

c656844

Don't export the metric frame data type

ba1f7e5

Merge remote-tracking branch 'origin/feature/mavlink-input-plugin' in…

231b70a

…to feature/mavlink-input-plugin

wip

7579342

Revert dep changes that had nothing to do with this PR

0c6a561

Fix

66adb19

Merge remote-tracking branch 'upstream/master' into feature/mavlink-i…

fbf605c

…nput-plugin

Hipska suggested changes Nov 27, 2024

View reviewed changes

WIP PR review

56e345d

cdalke-havoc added 4 commits November 29, 2024 12:24

Fix README example output, message filtering

2d170ef

Lint fixes

3b504e5

Move client configuration to Init, connection to Start

e8db31b

Lint + test fixes

2c1dfd6

Hipska suggested changes Dec 2, 2024

View reviewed changes

chrisdalke and others added 5 commits December 3, 2024 23:05

Working on PR feedback, simplify parsing logic

7de58ea

Improve test coverage, address linting issues

b95ccda

Update docs

67d40d7

Fix host/port parsing

d009325

Rename source tag back to fcu_url

1fb5aa0

chrisdalke marked this pull request as ready for review December 6, 2024 03:50

srebhan self-assigned this Dec 9, 2024

Hipska approved these changes Dec 9, 2024

View reviewed changes

srebhan reviewed Dec 11, 2024

View reviewed changes

srebhan added new plugin plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins labels Dec 11, 2024

srebhan changed the title ~~feat(inputs): Add Mavlink input plugin~~ feat(inputs.mavlink): Add plugin Dec 11, 2024

Working on PR feedback

1699ec5

chrisdalke mentioned this pull request Dec 15, 2024

feat: Switch serial library to one with larger cross-platform support bluenviron/gomavlib#138

Merged

chrisdalke added 6 commits December 15, 2024 09:09

Lint

64e30ac

Merge commit '516b8cfbd16dc1eade0034fe873bb23a242bee04' into feature/…

9a80fe5

…mavlink-input-plugin

Working on PR feedback

2647227

Move URL parsing out of utils into main file, fix linting issues

0e42711

Lint

70d9732

Swap forked gomavlib for upstream

89a9649

Hipska approved these changes Dec 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(inputs.mavlink): Add plugin #16221

feat(inputs.mavlink): Add plugin #16221

chrisdalke commented Nov 22, 2024 •

edited

Loading

Hipska left a comment

srebhan left a comment

srebhan Dec 11, 2024

chrisdalke Dec 15, 2024

chrisdalke Dec 15, 2024

srebhan Dec 11, 2024

srebhan Dec 11, 2024

srebhan Dec 11, 2024

srebhan Dec 11, 2024

srebhan Dec 11, 2024

chrisdalke Dec 15, 2024

srebhan Dec 11, 2024

srebhan Dec 11, 2024

srebhan Dec 11, 2024

srebhan Dec 11, 2024

telegraf-tiger bot commented Dec 15, 2024

Artifact URLs

chrisdalke commented Dec 16, 2024

-The `mavlink` plugin connects to a MavLink-compatible flight controller such as
- [ArduPilot](https://ardupilot.org/) or [PX4](https://px4.io/). and translates
-all incoming messages into metrics.
-The purpose of this plugin is to allow Telegraf to be used to ingest live
- flight metrics from unmanned systems (drones, planes, boats, etc.)
-Warning: This input plugin potentially generates a large amount of data! Use
-the configuration to limit the set of messages, or use another telegraf plugin
-to filter the output.
-This plugin currently uses the ArduPilot-specific Mavlink dialect. See the
-[Mavlink docs](https://mavlink.io/en/messages/ardupilotmega.html) for more
-info on dialects and the various messages that this plugin can receive.
+This plugin collects metrics from [MavLink][mavlink]-compatible flight controllers
+such as [ArduPilot][ardupilor] or [PX4][px4] to live ingest flight metrics from
+unmanned systems (drones, planes, boats, etc.)
+Currently the ArduPilot-specific Mavlink dialect is used, check the
+[Mavlink documentation][mavlink_docs] for more details and the various
+messages available.
+> [!WARNING]
+> This plugin potentially generates a large amount of data! Please use metric
+> filters to limit the amount of metrics produced!
+⭐ Telegraf v1.34.0
+🏷️ iot
+💻 all
+[mavlink]: https://mavlink.io
+[ardupilor]: https://ardupilot.org/
+[px4]: https://px4.io/
+[mavlink_docs]: https://mavlink.io/en/messages/ardupilotmega.html

		@@ -0,0 +1,39 @@
		# Read metrics from a Mavlink connection to a flight controller.

	# Read metrics from a Mavlink connection to a flight controller.
	# Read metrics from a Mavlink flight controller.

	func (s *Mavlink) Gather(_ telegraf.Accumulator) error {
	func (*Mavlink) Gather(telegraf.Accumulator) error {

	require.Equal(t, "could not parse fcu_url ftp://not-a-valid-fcu-url", err.Error())
	require.ErrorContains(t, err, "could not parse fcu_url")

feat(inputs.mavlink): Add plugin #16221

Are you sure you want to change the base?

feat(inputs.mavlink): Add plugin #16221

Conversation

chrisdalke commented Nov 22, 2024 • edited Loading

Summary

TODO

Checklist

Related issues

Hipska left a comment

Choose a reason for hiding this comment

srebhan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

telegraf-tiger bot commented Dec 15, 2024

Artifact URLs

chrisdalke commented Dec 16, 2024

chrisdalke commented Nov 22, 2024 •

edited

Loading