Increase default transientstoreMaxBlockRetention

[yeasy]

Increase the default value of transientstoreMaxBlockRetention

Change-Id: I24eb6a6c4be38fbb964970b518d40c72e02e12ff

Type of change

• Improvement (improvement to code, performance, etc)

Description

transientstoreMaxBlockRetention is used to trigger the purge of private data in the transient store.

The current default value is 1000, which is OK for simple cases with several blocks per second. Howerver, in our test with 10K tps case, it is possible to purge the private data too early before the peer gets a chance to commit the tx into the ledger.

With a larger retention window, the only cost is a larger transient database in the peer node, which is not a big deal.

Hence, this patchset suggest to increase the default value to a larger value of 100000, and it is open to have other reasonable value too.

Additional details

N/A

Related issues

N/A

Release Note

N/A

[denyeart]

How often are you cutting blocks when running at 10k tps?

If you cut a block once a second, the current setting would keep transient data for 1000 seconds (16 minutes), which seems reasonable given that the intention is to only store the transient data for the brief period of time between endorsement and commit.

If you are cutting blocks much more frequently though, I can see where 1000 could become a problem (but at the same time, if you cut blocks too frequently you start to lose some efficiencies). Would 10000 be a sufficient setting for your use case?

[yeasy]

The CI error is related to the gateway testing, should have nothing to do with the PR.

[yeasy]

How often are you cutting blocks when running at 10k tps?

If you cut a block once a second, the current setting would keep transient data for 1000 seconds (16 minutes), which seems reasonable given that the intention is to only store the transient data for the brief period of time between endorsement and commit.

If you are cutting blocks much more frequently though, I can see where 1000 could become a problem (but at the same time, if you cut blocks too frequently you start to lose some efficiencies). Would 10000 be a sufficient setting for your use case?

The block is cut frequently in order to reduce the latency. And in some case, the commit can be seconds later after the endorsement.

[denyeart]

@yeasy Would 10000 be a sufficient setting for your use case?

[yeasy]

@yeasy Would 10000 be a sufficient setting for your use case?

100,000 is more safe, but I'm OK to accept 10,000 or other larger value as the default value.
1000 is too small by default.

[denyeart]

@yeasy How about if you update the PR to 10,000 (or maybe 20,000 if you prefer).

[yeasy]

@yeasy How about if you update the PR to 10,000 (or maybe 20,000 if you prefer).

Sure, just pick 20,000 as the new value.