Spike in block production times

[niksaak]

Today 2024-10-30 we noticed a spike in block production times that lasted approximately between 12:25 and 12:35 UTC. During that time, processing time reached up to 4 minutes for some transactions. Could you please check on your side to see what happened?

cc @atrybulkevychglobalgames @dschedov @BulatSaif

[BulatSaif]

I checked the infrastructure: no restarts, updates, or maintenance on the collator nodes during that time. Blocks were being built but not finalized.
metrics:

[niksaak]

I see, thank you.

Blocks were being built but not finalized

So that I understand, was this a validator problem or a general network problem (e.g. Polkadot outage)? How likely is this to happen again in the future?

It seems that we can't as of now depend on "Best" blocks (see #245) so finalization times are very important to us.

[BulatSaif]

@KarimJedda Does the data team have any information about Polkadot outages?

Polkadot is a decentralized network, and Parity doesn't host any validators, so I don't have any Polkadot validator metrics or logs in our monitoring system.

[KarimJedda]

The data team only has block data, but no further telemetry regarding the nodes themselves like what you shared above @BulatSaif .

Regardless, I'm not aware of any issues on Polkadot around that time, can you ping Pavla & the team to have a look?