You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When a function yields results from a failure prone process (i.e., a REST API pagination series), it would be ideal to bake the retry into that iteration in a clean fashion. This currently doesn't work with backoff, because the on_exception handler always uses return which breaks the yield chain and once the generator starts getting consumed, the backoff will not trigger.
I've come up with a minimal example to illustrate the point, and I think it could be fixed by checking if the return value is a generator and using yield from instead of returnhere.
It seems like retrying exceptions as the generator is iterating would be the preferred behavior and may be backwards compatible, but I don't think I'll be able to sit down and prove out all of the edge cases and get it tested properly, so I'm putting it here as a request for now 😄
Illustration (if you step into the call to backoffgen, you can see it skip the decorator on the iterations):
@backoff.on_exception(backoff.expo, Exception, on_backoff=lambda details: print("Backing off!"))
def backoffgen():
for _ in range(1,10):
if random.randint(0,10) < 5:
raise Exception("BOOM!!!")
yield 5
time.sleep(1)
import ipdb; ipdb.set_trace()
for a in backoffgen():
print(a)
The text was updated successfully, but these errors were encountered:
When a function yields results from a failure prone process (i.e., a REST API pagination series), it would be ideal to bake the retry into that iteration in a clean fashion. This currently doesn't work with backoff, because the
on_exception
handler always usesreturn
which breaks the yield chain and once the generator starts getting consumed, the backoff will not trigger.I've come up with a minimal example to illustrate the point, and I think it could be fixed by checking if the return value is a generator and using
yield from
instead ofreturn
here.It seems like retrying exceptions as the generator is iterating would be the preferred behavior and may be backwards compatible, but I don't think I'll be able to sit down and prove out all of the edge cases and get it tested properly, so I'm putting it here as a request for now 😄
Illustration (if you step into the call to
backoffgen
, you can see it skip the decorator on the iterations):The text was updated successfully, but these errors were encountered: