LD relay fails to reconnect to Redis after Redis restarts

[carlosjgp]

https://support.launchdarkly.com/hc/en-us/requests/new

Describe the bug
LD relay fails to connect to Redis after Redis has restarted

To reproduce

• Install LD relay Helm chart
• Install Bitnami Redis chart version 16.5.0

architecture: standalone
master:
  persistence:
    enabled: false
    medium: Memory
    sizeLimit: 30Mi

Expected behavior
Fast automatic recovery whether this is fallback to "relay without store" or reconnect to Redis

Logs
If applicable, add any log output related to your problem.

SDK version

• NodeJSClient_6.4.1
• PythonClient_7.5.1

Language version, developer tools
N/A

OS/platform

• LaunchDarkly relay version 6.7.14

Additional context
possibly related to redis/go-redis#1737

Using Sentinel could mitigate this
#114

[cwaldren-ld]

Hi @carlosjgp, really appreciate this report. We'll look into it and get back to you.

Filed internally as 178424.

[keelerm84]

@carlosjgp in your report when you mention "after Redis has restarted", how exactly were you simulating that? Scaling the stateful set back down and up again shows the relay reconnecting appropriately.

[carlosjgp]

@carlosjgp in your report when you mention "after Redis has restarted", how exactly were you simulating that? Scaling the stateful set back down and up again shows the relay reconnecting appropriately.

@keelerm84 you can just terminate the POD

kubectl delete po -n <namespace> <redis master pod>

or as you mention

kubectl -n <namespace> scale statefulset <redis-master-sts> --replicas 0 

and after a while --replicas 1

[keelerm84]

Hmmm, I have tried reproducing this with both options, and each worked as expected with the relay re-establishing connection to the redis instance.

You state that the relay fails to reconnect; looking at yours logs, can you confirm if it successfully establishes connection initially before restarting redis?

How are you installing the redis helm chart? Are you using the default parameters, or are you overriding some of them, like the password? And how are you injecting that information in the relay proxy helm chart?

[keelerm84]

Closing due to inactivity. Please feel free to re-open if you continue experiencing this issue.