Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No way to respect RetryAfter for retryable errors #603

Open
froodian opened this issue Feb 12, 2024 · 3 comments
Open

No way to respect RetryAfter for retryable errors #603

froodian opened this issue Feb 12, 2024 · 3 comments

Comments

@froodian
Copy link

froodian commented Feb 12, 2024

Describe your environment

  • Operating System version: Ubuntu latest
  • Firebase SDK version: v4.13.0
  • Library version: v4.13.0
  • Firebase Product: messaging

Describe the problem

Some messaging errors are retryable, for instance errors for which messaging.IsQuotaExceeded(err) returns true. The FCM API requests that the Retry-After response header is respected in retrying these. The Firebase SDK reads the RetryAfter header and respects it when retrying internally, however it does not expose this value to callers, leaving them unable to respect this retry-after.

Steps to reproduce:

  • As an integrator of "firebase.google.com/go/v4/messaging", send many messages in a loop until you receive a quota exceeded error
  • Attempt to retry the quota exceeded error at an appropriate time - this information is not exposed by the Firebase SDK despite coming from the Firebase API
@lahirumaramba
Copy link
Member

lahirumaramba commented Mar 20, 2024

Hi @froodian , thanks for filing this feature request. We can't promise a timeline on this, but we will use this issue to track any progress.

Are you looking for a feature to disable auto retry in the SDK or would you prefer to access the error responses with retry-after headers keeping the auto-retry feature enabled?

@lahirumaramba lahirumaramba self-assigned this Mar 20, 2024
@froodian
Copy link
Author

froodian commented Mar 26, 2024

Hi @lahirumaramba - thanks for the response.

In regards to the SDK's internal auto retry, we don't mind that behavior exisitng, but in order to keep requests within our existing timeout expectations, we've actually forked the repo and lowered MaxDelay and ExpBackoffFactor on our fork - so ideally those values would be configurable, or the auto retry would be disableable, so that we don't have to maintain that fork.

Regardless of the outcom there, we would want the ability to expose the Retry-After, so that we can retry any requests that make it through that auto retry at an appropriate time - since the initial filing of this issue, we've discovered this comment from a previous issue requesting this, and it appears we may be able to use this firebase.google.com/go/v4/errorutils package to accomplish that with today's code - we'll attempt that approach imminently.

@muthu3107
Copy link

@lahirumaramba

We've set a 5-second timeout both at the context and HTTP request levels. However, we're observing that requests are still taking longer than 5 seconds, with seemingly random delays. We suspect the server is setting Retry-After headers due to potential spiking issues.

We're already implementing jitter and random backoff within our system to handle retries. It would be extremely helpful if we had an option to disable the Retry-After behavior altogether. This way, if a request times out after the configured 5 seconds, we can handle the failure gracefully with our own retry mechanism.

Please let us know if there's a way to achieve this or if there are alternative solutions to address the timeout issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants