Comparing across repeated bursts as an alternative to checksums? #2437
Replies: 1 comment 4 replies
-
|
I think this idea has a lot of merit. Currently, doctrine is that each transmission is simply decoded. Certainly we don't want to lose that ability. But, adding a mode that does inter-transmission processing would be really useful. I will set aside the issue of whether it should be builtin or a external pipeline stage. Combining multiple transmissions into 1 is a form of error correction. As a user I would generally rather see one json line with the best guess at what was sent vs 4 separate ones. I think this has to be a pipeline stage, where the decoder decodes but then some later process merges them. That leads to thinking about the intermediate representation, which could involve a pivot to json as a basic plan and the pipeline stage followed by json->whatever conversion. A second thing is to enable a decoder (for eventual output) only after there has been some threshold number of messages, where the threshold might vary by no-checksum/checksum/crc8/crc16/crc32. It would be further cool to store information about this (including frequency) so that on restarting rtl_433 the previous knowledge would be present, allowing prompt output. That probably needs a "no detects in 30-90 days leads to dropping". All of this is a lot of work. |
Beta Was this translation helpful? Give feedback.
-
|
note that a common error case is not noise but a different protocol that roughly matches what a decoder wants to see, and then we easily get multiple rows with matching data. In other words: a solid number of repeats is no guarantee of validity. |
Beta Was this translation helpful? Give feedback.
-
|
To your point @gdt, some devices do as you describe: sending an entire transmission multiple times, with a significant pause in-between, such that the decoder is called multiple times with what might be an identical pulse stream. I see 3 such repeats on every report from my Acurite-5n1. But working with the Acurite Rain Gauge, I see periodic transmissions, each just a single stream of pulse codes with no long gap between 16 repeats. The whole transmission is passed to the decoder in just one call. So for this kind of repeats, it would not take any infrastructure work for logic in a separate protocol decoder to better verify that it's not reading noise, by validating the repeated pulse codes are (mostly) consistent. |
Beta Was this translation helpful? Give feedback.
-
|
The functionality you describe "a second thing is to enable a decoder (for eventual output) only after there has been some threshold number of messages" is effectively what I do, but post-processing, in NodeRED. I've been a long-time RFLink user, but will be moving most of my devices over to rtl_433, now that it has surpassed, IMHO, the capabilities of RFLink. (I've waited wayyy too long for the next release of RFLink to fix issues I reported YEARS ago. It's not an open-source project :-(. So, in NodeRED I process and store "discovered" devices, and don't send them on to Home Assistant unless I have specifically configured the device to be "observed". The time of the first discovery, the time of the last transmission, the number of msgs seen and the entire last message received are all stored in a database. A partial list of what my combined RFLink 315/433MHz and rtl_433 (just 433MHz for now) have currently: I periodically clean out all of the devices that have only been detected a small number of times (and are likely noise). If a device changes its ID when batteries are replaced, I can add the "map_id," so it's mapped to the prior ID it once had, and Home Assistant won't see it as a newly discovered device. |
Beta Was this translation helpful? Give feedback.
-
|
I'm experimenting with another mechanism that is helping greatly to reduce false positives. I reordered ALL of the protocols, using a configuration file that puts my known-good protocols at the top of the list. I went looking in the code for a way to do this, found, @zuckschwerdt's work that allows for explicit priorities, and then realized that the order the protocols were initially developed implies an implicit priority if they are all auto-added. So, by explicitly adding them in a config file, the user has complete control over the effective priorities. With my known-good protocols at the top of the list, I'm seeing much fewer false positives. We could do something like this, dynamically. @gdt you should think about this, along with your idea of an option to activate a protocol only when it has detected repeated transmissions... Make the protocols that have successfully received more messages, move up higher in the list, edging out any other protocols likely to sometimes cause false positives. |
Beta Was this translation helpful? Give feedback.
-
I recently contributed a change to the Acurite Rain Gauge decoder, helping to make it a lot less prone to false positives. It took little work, just making sure there were 12 or more repeated sends from the device, since the main offender(s) for false positives were much shorter broadcasts.
It left me thinking, though, about another more detailed approach, given the many repeats the Acurite sends: For protocols that are missing checksums, could we add logic to check that there are consistent values in most of the repeats? Is there a concern for the amount of time each protocol decoder takes to (quickly?) decide if it can handle a given transmission?
Beta Was this translation helpful? Give feedback.
All reactions