Frequent (~once per minute) reorgs on collator nodes

[niksaak]

We are experiencing frequent chain reorgs on our devnet. This has been known for a while and has been worked around, but it becomes more important now that we are considering relying more on InBlock statuses.

Please let us know what additional info you'd consider helpful.

Excerpt from logs:

2024-10-21 18:40:18 [Relaychain] ♻️  Reorg on #1156604,0x6d50…4900 to #1156604,0xed2b…9c1b, common ancestor #1156603,0x469c…2ccf
2024-10-21 18:40:24 [Parachain] ♻️  Reorg on #1033525,0x5fbb…f2f0 to #1033526,0x3583…bf0f, common ancestor #1033524,0xe815…3a6c
2024-10-21 18:40:36 [Parachain] ♻️  Reorg on #1033526,0x3583…bf0f to #1033527,0x007a…80ab, common ancestor #1033524,0xe815…3a6c
2024-10-21 18:41:36 [Parachain] ♻️  Reorg on #1033535,0x73f9…38dd to #1033536,0xf7b3…6f05, common ancestor #1033533,0xec49…4f71
2024-10-21 18:41:54 [Parachain] ♻️  Reorg on #1033538,0x302b…5ce6 to #1033539,0x7e9a…56dd, common ancestor #1033537,0x220a…e459
2024-10-21 18:42:36 [Relaychain] ♻️  Reorg on #1156626,0xe24b…5704 to #1156627,0xafbb…49c3, common ancestor #1156625,0xa200…753f
2024-10-21 18:42:36 [Parachain] ♻️  Reorg on #1033544,0xbb76…f3e2 to #1033545,0x5812…5c0b, common ancestor #1033542,0xd9d0…56f8
2024-10-21 18:43:36 [Parachain] ♻️  Reorg on #1033553,0xf70b…d29e to #1033554,0xf58a…da1f, common ancestor #1033551,0xcb30…7504
2024-10-21 18:44:36 [Parachain] ♻️  Reorg on #1033562,0xe855…1445 to #1033563,0x937f…63a7, common ancestor #1033560,0xa5e1…dd25
2024-10-21 18:45:36 [Parachain] ♻️  Reorg on #1033571,0x09fb…b517 to #1033572,0x4851…7012, common ancestor #1033569,0xbee2…fcc0
2024-10-21 18:46:36 [Parachain] ♻️  Reorg on #1033580,0xdd8c…e46b to #1033581,0x1b25…a3e2, common ancestor #1033578,0x9aeb…3e98
2024-10-21 18:47:06 [Relaychain] ♻️  Reorg on #1156672,0xa0b8…7ee8 to #1156672,0xa7fe…a855, common ancestor #1156671,0x2e87…1e89
2024-10-21 18:47:12 [Parachain] ♻️  Reorg on #1033586,0x9ae7…6de0 to #1033587,0xb20f…f3a8, common ancestor #1033585,0x352a…502a
2024-10-21 18:47:36 [Parachain] ♻️  Reorg on #1033589,0xf26f…804e to #1033590,0x498f…10c4, common ancestor #1033587,0xb20f…f3a8
2024-10-21 18:48:24 [Relaychain] ♻️  Reorg on #1156685,0xf5a9…3b3b to #1156685,0x4185…603f, common ancestor #1156684,0x98e8…b6e3
2024-10-21 18:48:36 [Parachain] ♻️  Reorg on #1033598,0x6efa…e4db to #1033599,0x99e5…436e, common ancestor #1033596,0xda3c…671b
2024-10-21 18:49:36 [Parachain] ♻️  Reorg on #1033607,0xf6d1…aaa5 to #1033608,0x3c83…4664, common ancestor #1033605,0x63b2…520a
2024-10-21 18:50:06 [Relaychain] ♻️  Reorg on #1156702,0xb6b6…bc08 to #1156702,0x66e0…9ef5, common ancestor #1156701,0x5c42…73ab
2024-10-21 18:50:12 [Parachain] ♻️  Reorg on #1033613,0xc776…355f to #1033614,0xc1d2…7f41, common ancestor #1033612,0xca05…6aba
2024-10-21 18:50:36 [Parachain] ♻️  Reorg on #1033616,0xe9b1…9e34 to #1033617,0x4254…c367, common ancestor #1033614,0xc1d2…7f41
2024-10-21 18:51:36 [Parachain] ♻️  Reorg on #1033625,0x11bc…ce95 to #1033626,0x2d42…43df, common ancestor #1033623,0x5476…721b
2024-10-21 18:52:36 [Parachain] ♻️  Reorg on #1033634,0xd1d9…e1e0 to #1033635,0x8f43…bad3, common ancestor #1033632,0xfaa4…e935
2024-10-21 18:53:36 [Parachain] ♻️  Reorg on #1033643,0x7c38…fb06 to #1033644,0xae04…dd7e, common ancestor #1033641,0x2c76…0e74
2024-10-21 18:53:48 [Relaychain] ♻️  Reorg on #1156738,0x7f8c…1b3d to #1156739,0x9484…b184, common ancestor #1156737,0xb3d7…effd
2024-10-21 18:53:48 [Parachain] ♻️  Reorg on #1033645,0xf044…6424 to #1033646,0x946d…2325, common ancestor #1033644,0xae04…dd7e
2024-10-21 18:53:54 [Relaychain] ♻️  Reorg on #1156740,0xe353…df91 to #1156740,0x5b76…5b96, common ancestor #1156739,0x9484…b184
2024-10-21 18:54:00 [Parachain] ♻️  Reorg on #1033647,0x146b…f81e to #1033648,0xec13…d243, common ancestor #1033646,0x946d…2325
2024-10-21 18:54:36 [Parachain] ♻️  Reorg on #1033652,0xf36e…4ce0 to #1033653,0x0636…4ad2, common ancestor #1033650,0x4896…da64
2024-10-21 18:55:36 [Parachain] ♻️  Reorg on #1033661,0xa6e2…e91a to #1033662,0x9610…6aed, common ancestor #1033659,0x4409…977b
2024-10-21 18:56:24 [Relaychain] ♻️  Reorg on #1156765,0x7e50…fbc1 to #1156765,0xb85e…48d7, common ancestor #1156764,0x54c3…f736
2024-10-21 18:56:36 [Parachain] ♻️  Reorg on #1033670,0x402f…9dc9 to #1033671,0xd78d…2760, common ancestor #1033668,0xed25…290a
2024-10-21 18:56:48 [Relaychain] ♻️  Reorg on #1156769,0xbe78…da85 to #1156769,0x6c5d…70a5, common ancestor #1156768,0x3dd5…f3e9
2024-10-21 18:56:54 [Parachain] ♻️  Reorg on #1033673,0x6579…ba4e to #1033674,0x7590…58ce, common ancestor #1033672,0x4594…e165
2024-10-21 18:57:36 [Parachain] ♻️  Reorg on #1033679,0x5e5b…b952 to #1033680,0x36bf…adb5, common ancestor #1033677,0xe0e6…ebac
2024-10-21 18:58:24 [Relaychain] ♻️  Reorg on #1156785,0x1c99…940c to #1156785,0x1cce…3c1f, common ancestor #1156784,0xdfa6…8d49
2024-10-21 18:58:36 [Parachain] ♻️  Reorg on #1033688,0x393a…64b8 to #1033689,0x8172…89a6, common ancestor #1033686,0x1da6…b968

cc @BulatSaif, @atrybulkevychglobalgames