Tortoise scales unreasonably when application misbehaves

[randytqwjp]

https://mercari.slack.com/archives/C6HC4JBKM/p1717478614923889

When pods are stuck in crashbackloopoff, tortoise recommendation algorithm does not consider this and causes recommendations to scale HPA max replicas to a unreasonable amount

[sanposhiho]

causes recommendations to scale HPA max replicas to a unreasonable amount

Does it mean the tortoise lowered the target utilization of HPA too much and consequently HPA increased the replica number?

[randytqwjp]

there was a bug in application logic and caused some pods to be stuck in crashbackloop while remaining pods utilization went up. Tortoise then increased maxreplica for this service but since new pods also get stuck in crashbackloop, tortoise kept increasing maxreplica

[sanposhiho]

My suggestion is that we can improve Tortoise to check all Pods' status, and then if the ratio of such crashed Pods is higher than the criterion (50% etc), stop changing the max replica (or maybe stop changing any parameters until the situation is stable).