Crash/lockup reading ttyACM device

[craigerl]

Describe the bug

When PiZero2W is connected to an icom705 (ham radio) via USB, the following command instantly locks up Raspberry Pi OS Bookworm, with all patches applied. Console unresponsive, net-dead. Even a non-root user can crash the system. Activity light blinks once per second. No USB, eventually ext4 reports io errors. I suspect the usb/sd/wifi hardware stops at this point. Scheduler and at least systemd are still running.

sudo /usr/sbin/gpsd -n -b -N -D 2 /dev/ttyACM1

Not observed on a generic ublox gps. Not observed on a Pi5. PiZero2W locks up every time. No oops on console, kern.log, syslog.

A non root user can crash the the system simply by running gpsd ttyACM1.

The radio presents an audio device, a serial port and a gps serial device (ttyACM0, and ttyACM1 respectively).

Kernel bug introduced sometime after 6.1.74(jan24), as regressing to this kernel resolves the problem.
rpi-update d86b5843d68b9972a5430a6d3da1b271cfc83521

Steps to reproduce the behaviour

PiZero2W, icom705 radio, gpsd.

sudo /usr/sbin/gpsd -n -b -N -D 2 /dev/ttyACM1

Device (s)

Raspberry Pi Zero 2 W

System

Raspberry Pi reference 2023-10-10
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, fb56ad562991cf3ae5c96ab50983e1deeaefc7b6, stage2

Aug 30 2024 19:19:24
Copyright (c) 2012 Broadcom
version 2808975b80149bbfe86844655fe45c7de66fc078 (clean) (release) (start_cd)

Linux digipi 6.6.51+rpt-rpi-v7 #1 SMP Raspbian 1:6.6.51-1+rpt3 (2024-10-08) armv7l GNU/Linux

Logs

root@digipi:~# /usr/sbin/gpsd -n -b -N -D 9 /dev/ttyACM1
gpsd:WARN: This system has a 32-bit time_t.  This gpsd will fail at 2038-01-19T03:14:07Z.
gpsd:INFO: launching (Version 3.22)
gpsd:IO: opening IPv4 socket
gpsd:SPIN: passivesock_af() -> 3
gpsd:IO: opening IPv6 socket
gpsd:SPIN: passivesock_af() -> 4
gpsd:INFO: listening on port gpsd
gpsd:PROG: NTP: shmat(0,0,0) succeeded, segment 0
gpsd:PROG: NTP: shmat(1,0,0) succeeded, segment 1
gpsd:PROG: NTP: shmat(2,0,0) succeeded, segment 2
gpsd:PROG: NTP: shmat(3,0,0) succeeded, segment 3
gpsd:PROG: NTP: shmat(4,0,0) succeeded, segment 4
gpsd:PROG: NTP: shmat(5,0,0) succeeded, segment 5
gpsd:PROG: NTP: shmat(6,0,0) succeeded, segment 6
gpsd:PROG: NTP: shmat(7,0,0) succeeded, segment 7
gpsd:PROG: successfully connected to the DBUS system bus
gpsd:PROG: shmget(0x47505344, 24232, 0666) for SHM export succeeded
gpsd:PROG: shmat() for SHM export succeeded, segment 8
gpsd:INFO: stashing device /dev/ttyACM1 at slot 0
gpsd:PROG: no /etc/gpsd/device-hook present, skipped running ACTIVATE hook
gpsd:INFO: SER: opening read-only GPS data source type 3 and at '/dev/ttyACM1'

At this point the sdcard and usb interface is gone... systemd can still write to the console.

Additional context

Rolling back kernel to 6.1.74 (Jan 24) resolves the issue.

• rpi-update d86b5843d68b9972a5430a6d3da1b271cfc83521

Problem observed with:

• Raspberry Pi Zero2w
• Raspberry Pi OS Bookworm, Linux 6.6.51+rpt-rpi-v7
• Icom705 GPS on /dev/ttyACM1

Not a problem if i switch any one of these to

• Pi5
• ublox gps
• kernel 6.1.74

https://gitlab.com/gpsd/gpsd/-/issues/303#note_2174127901

[craigerl]

tailing kernl.log and syslog during lockup provided no information

[pelwell]

Does the Icom705 GPS have it's own power supply? Have you tried putting the GPS behind a powered hub?

[craigerl]

Does the Icom705 GPS have it's own power supply? Have you tried putting the GPS behind a powered hub?

good questions, yes and yes. No change.

[garyemiller]

Hi, I'm Gary Miller from the gpsd project.

I still don't understand:

the sdcard [,,,] is gone

Can it be read after a reboot?

[craigerl]

Hi, I'm Gary Miller from the gpsd project.

I still don't understand:

the sdcard [,,,] is gone

Can it be read after a reboot?

Yes, sorry for being vague. It reboots fine (until the gpsd service starts and the gps is connected).

It's as if the PiZero2W "io chip" stops the same exact moment gpsd starts. no wifi, no usb, no sd card.

systemd starts complaining (countdown) for three services which are timing out.

The kernel ext4 driver reports an io error on an inode

you can "cat /dev/ttyACM0" anb see the output just fine. It's something about how gpsd talks to ttyACM0 that kills sd/usb/wifi.

thanks,
-craig

[craigerl]

Appears to be resolved with Kernel 6.6.58 15869f639ce259cca9d3857eb86316f963d4b1e9

[craigerl]

I take that back, it worked once. I'll play the high/low game and see if I can isolate which kernel commit broke ttyACM on PiZero2w using gpsd.

[craigerl]

Bug was introduced in this commit (kernel went from 6.1.x to 6.6.x)

kernel: Bump to 6.6.16 kernel from next branch
popcornmix committed on Feb 8
e632362b0399b4ce331aacd9386685bc60938ab7

[pelwell]

Oh that's good - only about 70,000 commits between the two....