FDRS and timekeeping

[aviateur17]

Started this discussion to talk about timekeeping and FDRS and implementation and features.

I've implemented NTP via ESP32 WiFi on FDRS gateway. The resolution is in seconds and I hope that is okay.
The gateway will query NTP server upon startup (after Wifi and MQTT) and check to see if the time is valid. If it is valid it will keep the time and update the time and periodically poll the NTP server to correct any inaccuracies. I'm not sure off the top of my head how accurate the time stays on different platforms without querying the NTP server.
I've also implemented gateways sending time via LoRa broadcast to remote nodes. The node will check for accurate time and then keep the time internally.

As we discovered via FDRS ping, the ping responses are quite sub 1 second (5ms ESP-NOW, 100ms LoRa) so I'm not sure that it would be of any use for time correction but I welcome comments on that as the time is only precise down to the 1 second interval and I'm not sure if a more precise time would be required.

Features to do that I can think of would be:

• NTP via Ethernet on gateway (I can try to implement this but I cannot test as I do not have the hardware)
• periodic gateway time send to ESP-NOW nodes (maybe send to subscribed nodes??)
• periodic gateway time send to LoRa nodes via directed unicast (directly to the node). This would be difficult as LoRa node addresses are random. (maybe send to subscribed nodes??)
• Timekeeping in RTC module (I can try to implement this but I cannot test as I do not have the hardware)
• ESP-NOW node time request to gateway and gateway response
• LoRa node time request to gateway and gateway response
• periodic gateway time send via serial to adjacent gateway
• gateway response to serial time requests of adjacent gateway
• inclusion of timestamping in logging records

I will hopefully send the first PR for my code in the next 24 hours so that people can comment and test. I will submit the PR against the release_2023_01 branch or maybe it would be better to start a new branch for this so it can be developed and tested before included into mainstream release.

Sort of off topic but thinking about all of these broadcasts make me wonder how collisions are handed in ESP-NOW and LoRA and if we need to implement some channel detection features or if retries and CRC checking are sufficient.

[timmbogner]

Mulling this over and will present more thoughts at a later date, but some things of note:

1. LoRa pings actually reached >4000ms with SF12, so there's some chance for it to be a bit off when it arrives. I guess this is still kind of negligible though, overall.
2. I just put together a WeMos OLED test because I realized it was untested on 8266, and pings are up to 23ms.
3. Check out this WeMos breadboard I designed 😁.
   

[aviateur17]

1. LoRa pings actually reached >4000ms with SF12, so there's some chance for it to be a bit off when it arrives. I guess this is still kind of negligible though, overall.

Yeah, using time adjustment with LoRa is probably a good idea. Shouldn't be an issue to include that in the LoRa node code. Will make that a higher priority.

[timmbogner]

Something that just flashed into my mind:

Access to time will make up for one of the system's major shortcomings: slow speed. The age of a DataReading when it arrives at the front-end could be seconds to a minute depending on LoRa settings and number of repeaters. As such we'll eventually need to update the DataReading structure to hold timestamps as well. This shouldn't be too painful hopefully, but I definitely hadn't considered it.

[timmbogner]

The major issue is that it will be the first break in sensor compatibility in a very long time, so if possible maybe we can avoid it. Admittedly the system derives most simplicity from having a standardized data type, so keeping a second, "timeless" DR structure probably wouldn't make sense.

[aviateur17]

Perhaps we could do some checking in the gateway if the DataReading packet does or does not have a timestamp and process both situations? I agree that your idea of adding timestamps adds a great deal of improvement as then the reading does not need to be sent right away to the gateway and can even be buffered if the connection to the gateway is lost for a time while keeping the timestamps of the data.

[timmbogner]

The problem there is that so much is currently based off of dividing a packet by sizeof(DataReading) or 7. When a packet arrives, we know that if its size is smaller than 7 it's a SystemPacket, else it's one or more DataReadings and we can divide by 7 to find out how many. If there are different sizes in one packet, then the system goes caput. I think we might need to break compatibility tbh. Maybe leaving an untimed_mode as a compile option.
We can still implement separate timed and untimed DataReadings if it makes anything easier, but I think we'd just have to pad the untimed one, kinda defeating the point.

What's the minimum data size we'd need for a timestamp?

[timmbogner]

then the reading does not need to be sent right away to the gateway and can even be buffered if the connection to the gateway is lost for a time while keeping the timestamps of the data.

Yaaaaaa! I didn't even think of that.

[aviateur17]

The problem there is that so much is currently based off of dividing a packet by sizeof(DataReading) or 7. When a packet arrives, we know that if its size is smaller than 7 it's a SystemPacket, else it's one or more DataReadings and we can divide by 7 to find out how many. If there are different sizes in one packet, then the system goes caput. I think we might need to break compatibility tbh. Maybe leaving an untimed_mode as a compile option. We can still implement separate timed and untimed DataReadings if it makes anything easier, but I think we'd just have to pad the untimed one, kinda defeating the point.

What's the minimum data size we'd need for a timestamp?

Yeah, we'll have to look into this and see if it is an option. I believe the seconds since 1970 (epoch) is 4 bytes. I don't recall off the top of my head what the sizes of the DataReading and SystemPackets are. I'm pretty sure we can interpret between DataReadings with extra 4 bytes with time and those without the 4 bytes with the time in the code. Like you said earlier, I'd hate to break backward compatibility.