Here's a small openpilot bug clip:
replicate-prediction-3iy65fzbw3bygdqvuid3duganu.mp4
Capture and develop clips of openpilot from comma.ai's Comma Connect.
The clipper can produce clips of:
- comma.ai openpilot UI (including desired path, lane lines, modes, etc.)
- Origin of codebase, Segment ID, and seconds marker branded into the clip for debugging/reporting. Useful for posting clips in the comma.ai Discord's #driving-feedback and/or #openpilot-experience channel, reddit, Facebook, or anywhere else that takes video. Very useful for making outstanding bug reports as well as feedback on good behavior.
- Forward, Wide, and Driver Camera with no UI
- Concatenate, cut, and convert the raw, low-compatibility, and separated HEVC files to one fairly compatible HEVC MP4 or super-compatible H.264 MP4 for easy sharing.
- 360 Video
- Rendered from Wide and Driver Camera. Uploadable to YouTube, viewable in VLC, loadable in 360 video editing software such as Insta360 Studio, and accepted by any video players or web services that take 360 videos.
- Forward Upon Wide and 360 Forward Upon Wide
- Forward video is overlayed atop the wide video. Not perfect, but usable.
- 360 Forward Upon Wide scales and renders the final result at a higher resolution to assist in reframing the 360 video to a normal video if that's what you want.
All clip options have a configurable target file size option as platforms like Discord limit file upload sizes.
The clipper is deployed on Replicate:
https://replicate.com/nelsonjchen/op-replay-clipper
Replicate is an ultra-low-cost pay-as-you-go compute platform for running software jobs. Replicate is a great way to run this clipper as it's fast, easy to use, and you don't need to install anything on your computer or even deploy anything yourself. Just enter in the required information into the form, and Replicate will generate a clip. Expect to pay about ~$0.01 per clip but not even need to put in any payment details until you've reached a generously large level of usage.
Warning
comma devices should not be used as primary dashcams for numerous reasons!
They are still great as a backup dashcam and for other purposes though.
- Route - A drive recorded by openpilot. Generally from Ignition On to Ignition Off.
- comma.ai device that can upload to comma Connect.
- Free GitHub account to log into Replicate with
- A comma lite or prime subscription.
- Clipping was a comma connect prime-only feature but was removed for refurbishment. This is a free and open source tool to do the same.
We assume you've already paired your device and have access to the device with your comma connect account.
- Visit comma connect and select a route.
- Scrub to the time you want to clip.
- In this example, I've scrubbed to a time where I want to make a small clip of behind this cool car.
- Now I need to select the portion of the route I want to clip. Here's a video of what that UI looks like
-
See how I drag and select a portion.
-
You can see me make a mistake but pressing the left arrow (←) in the top-left corner lets me re-expand and try to trim again.
-
The clipper has a maximum length of 5 minutes. Try to select a portion that's less than that. Try to aim for 20 seconds to a minute though as everybody else has short attention spans.
-
Video:
scrubdown.mp4
-
- Once satisified with the selected portion, prepare the route and files for rendering.
- Make sure all files are uploaded. Select "Upload All" under the "Files" dropdown if you haven't already and make sure it says
uploaded. You may need to wait and your device may need to be on for a while for all files to upload.- The clipper only works with high-resolution files and needs all files that are part of the clip to be uploaded.
- Make sure the route has "Public access" under "More info" turned on. You can set this to off after you're done with clip making.
- Make sure all files are uploaded. Select "Upload All" under the "Files" dropdown if you haven't already and make sure it says
- Copy the URL in the address bar of your browser to your clipboard. This is not the segment ID underneath the More Info button. In the case above, I've copied an old URL of "https://connect.comma.ai/fe18f736cb0d7813/1698203405863/1698203460702" to my clipboard. Note: comma has changed the URL format since this step was originally written. New URLs are like "https://connect.comma.ai/fe18f736cb0d7813/000001bb--4c0c0efba9/21/90".
- When you were adjusting the selected portion of the route in a previous step, it was changing those last two numbers in the browser address bar URL which is the start time and end time respectively.
- "Share This Route" button if it is present will work too. Choose "copy to clipboard" or similar.
- Visit https://replicate.com/nelsonjchen/op-replay-clipper
- Under
route, paste the URL you copied in the previous step. - Tweak any settings you like.
- Press
Run. - Wait for the clip to render. It may take a few minutes.
- Once done, you can download the clip. If you want, turn off "Public access" on the route after you're done.
-
Here's a generated clip with the
widerendering type with no UI:cog-clip.1.mp4
-
If you have issues downloading the clip with the "Download" button in Replicate's UI, click on the vertical ellipsis button or whatever is available in your browser for video in the lower right corner of the video and download via that. This is a strange issue in Replicate's UI that this clipper can't do anything about.
-
Demonstration of speed or longitudinal behavior of openpilot with model-based longitudinal is nearly impossible or hard without this clipper. This video is of a good model based long behavior at highway speeds.
ae_driving_highway.mp4
Cars can have bugs themselves. Here's my 2020 Corolla Hatchback phantomly braking on metal strips in stop and go traffic probably from the radar. Perhaps a future openpilot that doesn't depend on radar might be the one sanity checking the radar instead of the other way around currently. And another example of that in Portland.
phantom-radar-braking.mp4
funky-ramp-redux.mp4
This is a video of a bug report where openpilot's lateral handling lost the lane.
lost-line.mp4
Lane cutting?
cutty-uphills.mp4
Nav-assisted follow the road instead of taking the side road.
2023-06-27--15-58-05--14_nav_stays_left_woooo.mp4
Copying the car in front to get around someone waiting for the left turn
interesting-assisted-right-pass.mp4
- The UI replayed is comma.ai's latest stock UI on their master branch; routes from forks that differ alot from stock may not render correctly. Your experience may and will vary. Please make sure to note these replays are from fork data and may not be representative of the stock behavior. The comma team really does not like it if you ask them to debug fork code as "it just takes too much time to be sidetracked by hidden and unclear changes".
Learn how to bookmark, preserve, and flag interesting points on a drive/route.
Preservation saves the last couple segments from being deleted on your device as well.
With the car on, within a minute after an incident when it is safe to do so:
- Tap the screen to reveal a bookmark flag button in the bottom left if it isn't there already.
- Tap that icon.
- This will result in small slivers of yellow in the timeline you can quickly hone in on.
- You should also set the route to preserve under More Info while you're working on it. Non-comma Prime users need to heed this especially since while files aren't deleted on the device, visiblity in and through comma connect sunsets after 3 days.
- With regards to the clipper usage, during the process in which you are honing in on the start and end boundaries of the clip, your upper bound of the clip will nearly all the time be at that yellow so your first or early drags to hone down should basically top out there and be very generous with the start time before the yellow.
Tip
If you find it a hassle to reach out and touch the device or it is too inconvenient, try installing a custom macropad like the 🦾 comma three Faux-Touch keyboard!
There is a JWT Token input field. This is for users who do not wish to set a route to be "Public access". There is a major catch though. The JWT Token is valid for 90 days and is irrevocable in any way. Password changes from SSO account logins like in Comma Connect will not invalidate the token. Addtionally, it is not granular, meaning it will give access to all routes for the user if leaked.
If you share a JWT Token with anyone, they will be able to access all your routes for 90 days with no possibility of revocation from you. This is why it's not recommended to use this feature unless you know what you're doing compared to the "Public access" method which is much easier to revoke access to.
Tokens can be obtained from visiting https://jwt.comma.ai/ and logging in with the same comma connect account type. Tokens should be about 181 characters or longer.
After you run something, just use your browser to "Duplicate" the tab, change the settings for the next thing, and press Run. Replicate will queue up jobs and if necessary, even scale up to run multiple jobs in parallel. Very cool!
360 videos are cool but sometimes you want a normal video pointing at a specific direction or directions from that data.
reframed-bright.mp4
With 360 videos, it is possible to reframe the 360 video so it is a non-360 video to a normal video pointing at a specific direction.
The best current way to do this is to use a 360 video editor like Insta360 Studio to reframe the video to a normal video. Simply load the 360 video into the editor and reframe the video to the desired direction.
There may be alternative software that'll do it and I will take pull requests to add them to this README, but this is the best way I know how to do it and it is free.
The 360 Forward Upon Wide rendering option scales input videos and renders the final result in a much higher 8K resolution to assist reframing with a high resolution forward video. The normal 360 option just glues the videos together.
If wanting to use 360 Forward Upon Wide, test with the non-360 Forward Upon Wide option and start with a value for forwardUponWideH around "1.9" to dial in that value in first for that overlay before using the 360 Forward Upon Wide option to save money and time.
The real MVP is @deanlee for the replay tool in the openpilot project. The level of effort to develop the replay tool is far beyond this script. The script is just a wrapper around the replay tool to make it easy to use for clipping videos.
https://github.com/commaai/openpilot/blame/master/tools/replay/main.cc
A lot of the FFmpeg commands is based off of @ntegan1's research and documentation including a small disclosure of some but not all details by @incognitojam when @incognitojam was at comma.
https://discord.com/channels/469524606043160576/819046761287909446/1068406169317675078
@morrislee provided original data suitable to try to reverse engineer 360 clips.