Library Updates affecting Features and general usage understanding

[AlHasanSameh]

Thanks a lot for this great library, I make this discussion to understand several key points about its usage and to report several problems I've been facing over the last year and a half.
I might be a bit unprofessional at using github as I am a student and this is my first ever post, but I'll make sure to provide all necessary details to describe the situation
Issue 1 ) Library updates to improve overall features keep ruining previously fixed features :
Hardware : ESP12E
IDE : Arduino IDE
Core : 3.0.2
Description : My apps have been using RTDB streams a lot, the main one is using RTDB for switching outlets on and off, so I use RTDB streams to listen for updates to my loads and respond accordingly, these updates come via a mobile app I've designed to connect to Firebase and control the outlets, so the system consists of a mobile app controlling home appliances via the hardware ESP board which received those commands and turns stuff on or off, I've developed this software using your library and over the course of development updates in the library kept changing software behavior but all issues I faced previously I was able to fix or things were back to normal on the next update, everything was stable and working properly at release : 3.3.5
Code I'll provide below runs in the loop function IF the ESP board has internet connectivity via WiFi, all WiFi and internet connectivity stuff are handled in my app code important thing is Firebase Tasks are run in the loop with internet connectivity, these tasks are :

1. Monitor stream-connections, if a stream disconnects try reconnecting periodically.
2. Handle received RTDB updates and issue the necessary hardware control to connected devices.
3. Handle hardware updates and patch them to the RTDB
4. Update server timestamp periodically on the RTDB so the app knows if a board is connected and can receive updates or not.

Code runs smooth on 3.3.5, as soon as esp connects to the internet, Firebase begins, streams connect, hardware states get patched to RTDB as well as server TimeStamp, TimeStamp updates return Success as well to the code, Commands from the mobile app have 0.5 secs max lag till they get executed on my boards if not instantaneously.
After updating to 4.3.5 :
TimeStamp updates do take effect in the DB, but they always return Failure, and more importantly, this apparently causes my Stream to disconnect and reconnect whenever TimeStamp is updated, this causes app commands to lag and since the stream disconnects, sometimes they might not get executed until another TimeStamp update makes the stream reconnect again ( every 30s )
when I go back to 3.3.5 this goes away, I haven't tried every release in between unfortunately so I can't tell which update caused this.

Issue 2 ) OTA Cloud Storage Updates don't work (apparently with RTDB streams) :
After I recently came back to this project, I wanted to add firmware updates over the DB, so I tried the Cloud Storage example, as I couldn't figure out exactly how I am supposed to upload the firmware to RTDB after reviewing the examples, so I stuck to Cloud Storage, tried the example, requires the BSSLBuffer capacity to be increased, then it works perfect with one catch, the board keeps updating the firmware indefinitely as long as the firmware file is uploaded to the cloud storage, not the biggest deal, just add an RTDB field that issues firmware updates and listen to it, easy.
I incorporated the cloud storage example into my code, created a new FirebaseData object to use it for firmware updates, result : firmware update request fails to get sent, if I increase the Buffer Capacity more, I get crashes, notice the fact that my Streams already needed me to set the BSSLBuffer capacity, which leads me to my last issue.

Issue 3 ) How does BSSL Buffer Size work :
the examples keep instructing us to set its size, and both my use cases require this, but what is a BSSL buffer actually, how does it work, and on what basis should its size be set, this is the most important request for me and I really wish its usage to be explained as it would improve my code immensely as I'll surely be using this more often in the future, will be adding more streams...etc.
Codes will be in the attachments, this is the first time I ever use github but I tried to write posts based on how people recommended good posts to be written, but still I am sure some stuff aren't ideal so I hope you'll excuse me for it.
There's the code :
SF_OnlineControl.zip

[mobizt]

The library works normally for all provided functions and the recent functions are more stable and reliable. It works on Raspberry Pi Pico smoothly that guarantees the reliability as the platform does not use the wdt to reset the device as in ESP device.

There are two things in your code that you need to avoid.

1. I see that you use async function that it's not really async it's just a function that keeps the session open and sending request by skipping server response reading. The nearly first hundred requests may be ok but after that it fails unexpectedly due to server rejection. Then the library will close the session every 3 minutes. You should avoid using it.

2. You should not rely on the response code from stream object as it will return the same http status code since the stream connection was established from beginStream function call or resume after that its status was unchanged because only event data will be pushed from server without any http header. The response code from the FirebaseData object used for async function will always HTTP 200 OK as the true response code is neglected.

You should update your code based on my suggestion.

The OTA updates work normally, your device free flash space should be twice of firmware file size and you should set the partition correctly in IDE.

The BSSL (BearSSL) buffer size function was used to set the tx and rx buffer size of BearSSL engine used in TCP client directly. This is the size of memory that will be one time allocated for SSL engine to use for buffer the cryptographic data which some server may support SSL/TLS fragmentation that we can send/receive SSL data as small chunks from 512 bytes to 16k which is maximum size of SSL/TLS record.

Firebase supports SSL/TLS fragmentation, we can choose any tx buffer size from smallest 512 to 16k without restriction.

But for rx, the buffer size should large enough to cover the payload in case of large size payload otherwise no decrypted data from SSL engine.

[AlHasanSameh]

Thanks a lot for the quick reply and I hope you forgive my late response, internet issues unfortunately, but anyways these are great insights, so now I take away these things :

1. The BSSL Buffer allocates memory permanently for a FBDO, to be used in the TCP client, which means the allocation should be carried out at the Initialization of Firebase and not further.
2. The Asycn function doesn't wait for a server response which means it won't block code execution, which is why I opted to use it in the first place since the TimeStamp update is the most frequent request in my system so I didn't want a failed update to halt the rest of my code, but it's actually way worse in the sense that keeping a session open with the server and spamming requests will eventually lead it to fail, from past working of my system ( for months continuously ) I never faced problems with it, because you mentioned the library closes the session every 3 minutes which is the time it takes for me to issue only 6 requests, so I never faced problems with it but it's not a best practice.
3. The stream HTTP_Status_Code is always OK and therefore can't be used to monitor stream connection, in that case, what would you suggest I should do to monitor my streams, how do I check if a stream is open and listening, and most importantly does the library manage this in the background or not and if so then should I basically do nothing and restart the streams only when internet drops ?

[mobizt]

For the stream, library provides 3 functions to handle stream.

readStream for polling read the response from server.

streamTimeout for checking the time out.

streamAvailable for checking and get the available event data.

You can call httpConnected for checking the TCP session status.

Firebase-ESP-Client/examples/RTDB/DataChangesListener/NoCallback/NoCallback.ino

Lines 155 to 166 in 96f069f

	if (!Firebase.RTDB.readStream(&stream))
	Serial.printf("sream read error, %s\n\n", stream.errorReason().c_str());
	if (stream.streamTimeout())
	{
	Serial.println("stream timed out, resuming...\n");
	if (!stream.httpConnected())
	Serial.printf("error code: %d, reason: %s\n\n", stream.httpCode(), stream.errorReason().c_str());
	}
	if (stream.streamAvailable())

Library will wait for the keep-alive event which used as heartbeat signal, to be received in predefined period (default is 45 seconds) and will issue the time out status when no heartbeat signal received in time.

Firebase-ESP-Client/examples/RTDB/DataChangesListener/NoCallback/NoCallback.ino

Lines 132 to 133 in 96f069f

	//RTDB Stream keep-alive timeout in ms (20 sec - 2 min) when no server's keep-alive event data received.
	config.timeout.rtdbKeepAlive = 45 * 1000;

If the stream timed out and when readStream was executed, library will close the TCP session and resume it by re-sending the HTTP request with server-sent event header to establish connection in stream mode. At this condition, the httpCode function from stream data object will return the current HTTP status code.

We can't detect the server disconnection in the HTTP stream mode then we use the keep-alive event data as a heartbeat signal to check the connection between server and client.

The function httpConnected will return false when the TCP session was closed which you can rely on to check that the stream is running or not.

[mobizt]

In case stream callback function was assigned, the three functions for stream I described above will run in schedule task repeatedly in case ESP8266 or run in infinite loop in RTOS task in ESP32 case.

[mobizt]

Library checks and parses the response returns from server for validation the operation based on the purpose of that function.

[AlHasanSameh]

great insights, thanks a lot for your efforts, all good now

[mobizt]

I recommend Raspberry Pi Pico W as ESP8266 replacement. It uses the BearSSL for SSL engine which 16k memory buffer is small enough for this device when compare to nearly 200k available. You can use many FirebaseData objects you want.

In ESP32 that uses mbedTLS for SSL engine, it takes 50k+ for mbedTLS to work and you cannot use more than three FirebaseData objects that sessions opened simultaneously.

[AlHasanSameh]

Thanks a lot, Pi Pico W is a great recommendation indeed, only downside is it's not available in Electronics distributors in my country apparently ( Egypt ), but it is indeed very suitable if I really want to take my system to the next level with OTA updates and so on as you mentioned, I am surprised that even ESP32 won't be the best, I never used it before but always assumed it was a substantial upgrade to ESP8266 with all the extra IOs and such, but after a bit of research it turns out even most of those pins aren't 100% safe to use for all purposes same as ESP12E.

[AlHasanSameh]

On a side note, after having read my code, if you have the time to answer me further I would like to know your assessment/impression of the quality of my software and the way I designed things, I appreciate your time and effort a lot so you don't have to give a detailed answer but just your impression after having read it, was it easy to understand what each block and each method does ?, was it clear what tasks are done and why they're needed ?, variable names ?, was my documentation any good and if there's room for improvement, would be a great help.

[mobizt]

I can't mention all your code as there are dependencies (class or library) that I don't know.

For the code that related to the library, you don't have to call beginStream again as the library handles the stream for you as long as readStream was called in loop or the callback function was assigned.

The recent version library introduces runStream function that allows user to run the stream task manually in loop and all the data and event will send to the callback functions.

Firebase-ESP-Client/examples/RTDB/DataChangesListener/Callback/Callback.ino

Line 194 in 96f069f

Firebase.RTDB.runStream();

All functions related to database storing/reading will return the Boolean status of operation which you can rely on for success or fail.
This confirms that the data on database was successfully updated or read.

[AlHasanSameh]

that will be very useful as well, thanks a lot