Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get bodies peer task #7828

Draft
wants to merge 194 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
194 commits
Select commit Hold shift + click to select a range
4b80016
7311: Add PeerTask system for use in future PRs
Matilda-Clerke Sep 17, 2024
a8d5a9f
7311: Clean up some warnings
Matilda-Clerke Sep 17, 2024
4c64dbe
7311: Add feature toggle for enabling use of the peertask system wher…
Matilda-Clerke Sep 17, 2024
7a94fe2
7311: Remove log used for testing, apply spotless
Matilda-Clerke Sep 17, 2024
ace5dd1
7311: Add private constructor to PeerTaskFeatureToggle to prevent ins…
Matilda-Clerke Sep 17, 2024
52d440a
7311: Switch to logging a warning instead of throwing an exception wh…
Matilda-Clerke Sep 17, 2024
f392a0f
7311: Update javadoc to match previous commit
Matilda-Clerke Sep 17, 2024
2fb2690
7311: spotless
Matilda-Clerke Sep 17, 2024
f14aaeb
7311: Fix broken BesuCommandTest
Matilda-Clerke Sep 18, 2024
06553be
Merge branch 'refs/heads/7311-add-cli-feature-toggle-for-peertask-sys…
Matilda-Clerke Sep 18, 2024
1ba5184
7311: add class
Matilda-Clerke Sep 18, 2024
f2500dd
7311: Move PeerTaskFeatureToggle to more appropriate location
Matilda-Clerke Sep 18, 2024
0992c35
Merge branch 'refs/heads/7311-add-cli-feature-toggle-for-peertask-sys…
Matilda-Clerke Sep 18, 2024
33b810b
7311: add X prefix to peertask-system-enabled
Matilda-Clerke Sep 18, 2024
a378d22
Merge branch 'refs/heads/7311-add-cli-feature-toggle-for-peertask-sys…
Matilda-Clerke Sep 18, 2024
15b6bdf
7311: Move --Xpeertask-system-enabled out of BesuCommand and make hidden
Matilda-Clerke Sep 18, 2024
a9f6714
Merge branch 'refs/heads/7311-add-cli-feature-toggle-for-peertask-sys…
Matilda-Clerke Sep 18, 2024
e3fbc6c
7311: spotless
Matilda-Clerke Sep 18, 2024
c03bffb
Merge branch 'refs/heads/7311-add-cli-feature-toggle-for-peertask-sys…
Matilda-Clerke Sep 18, 2024
5e8b750
Merge branch 'main' into 7311-add-cli-feature-toggle-for-peertask-system
macfarla Sep 18, 2024
e4be5c0
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 18, 2024
bc11e0c
Merge branch 'main' into 7311-add-cli-feature-toggle-for-peertask-system
Matilda-Clerke Sep 18, 2024
b2a45c2
7311: Add GetReceiptsFromPeerTask
Matilda-Clerke Sep 19, 2024
513b74f
7311: Move isPeerTaskSystemEnabled to SynchronizerOptions
Matilda-Clerke Sep 19, 2024
6b86919
Merge remote-tracking branch 'origin/7311-add-cli-feature-toggle-for-…
Matilda-Clerke Sep 19, 2024
645e0e3
Merge branch 'refs/heads/7311-add-cli-feature-toggle-for-peertask-sys…
Matilda-Clerke Sep 19, 2024
4522414
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Sep 19, 2024
28fc2cd
Merge remote-tracking branch 'origin/7311-add-GetReceiptsFromPeerTask…
Matilda-Clerke Sep 19, 2024
2364ed5
7311: Fix javadoc issue
Matilda-Clerke Sep 19, 2024
ced19cd
Merge branch 'refs/heads/7311-add-cli-feature-toggle-for-peertask-sys…
Matilda-Clerke Sep 19, 2024
03f6495
7311: Fix javadoc issue
Matilda-Clerke Sep 19, 2024
5859444
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 19, 2024
9a4f3dd
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Sep 19, 2024
97e5918
7311: Move PeerTaskFeatureToggleTestHelper to TestUtil and fix Runner…
Matilda-Clerke Sep 20, 2024
e0f736d
7311: spotless
Matilda-Clerke Sep 20, 2024
df7f62d
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Sep 20, 2024
c335cbe
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 20, 2024
98aefcd
Merge branch 'main' into 7311-add-cli-feature-toggle-for-peertask-system
Matilda-Clerke Sep 20, 2024
6e734f9
7311: Remove PeerTaskFeatureToggle in favor of including isPeerTaskSy…
Matilda-Clerke Sep 20, 2024
42ca85b
Merge branch 'main' into 7311-add-cli-feature-toggle-for-peertask-system
Matilda-Clerke Sep 20, 2024
76724ed
Merge branch 'refs/heads/7311-add-cli-feature-toggle-for-peertask-sys…
Matilda-Clerke Sep 20, 2024
fc9b3f2
7311: Adjust to the removal of PeerTaskFeatureToggle and use Synchron…
Matilda-Clerke Sep 20, 2024
08c66fd
7311: Reduce timeout in PeerTaskRequestSender to 5s
Matilda-Clerke Sep 20, 2024
049cae2
7311: Refactor PeerManager to be an interface
Matilda-Clerke Sep 20, 2024
b6ec075
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Sep 20, 2024
5afba63
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 20, 2024
3c1178c
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Sep 20, 2024
5c3a61a
7311: Fix up compile errors after merge
Matilda-Clerke Sep 20, 2024
8448898
7311: Fix MetricsAcceptanceTest
Matilda-Clerke Sep 20, 2024
ab21100
7311: Fix MetricsAcceptanceTest
Matilda-Clerke Sep 20, 2024
2ac52f0
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Sep 20, 2024
f2ac53e
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 23, 2024
3de578d
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Sep 23, 2024
e901fdf
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 24, 2024
24e73a8
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Sep 24, 2024
f077206
7311: Fix DownloadReceiptsStep when using peer task system
Matilda-Clerke Sep 25, 2024
fa12495
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Sep 25, 2024
6e349e1
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 25, 2024
ad86ae6
7311: Rename PeerManager to PeerSelector
Matilda-Clerke Sep 25, 2024
38f04ab
7311: Reword PeerSelector javadoc to avoid implementation details
Matilda-Clerke Sep 25, 2024
6de3fb3
7311: Use ConcurrentHashMap in DefaultPeerSelector
Matilda-Clerke Sep 25, 2024
da9cd43
7311: Reword trace log in DefaultPeerSelector
Matilda-Clerke Sep 25, 2024
ce7d245
7311: Remove unused imports
Matilda-Clerke Sep 25, 2024
c9eb22e
7311: Use a 1 second delay between retries in PeerTaskExecutor to mat…
Matilda-Clerke Sep 25, 2024
e2fda73
7311: Add testGetPeerButNoPeerMatchesFilter to DefaultPeerSelectorTest
Matilda-Clerke Sep 25, 2024
608fece
7311: Add testGetPeerButNoPeerMatchesFilter to DefaultPeerSelectorTest
Matilda-Clerke Sep 25, 2024
0e76000
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Sep 25, 2024
2d07800
7311: spotless
Matilda-Clerke Sep 25, 2024
b910b4d
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Sep 25, 2024
ad26297
7311: Fix MetricsAcceptanceTest
Matilda-Clerke Sep 20, 2024
96c8030
7311: Fix MetricsAcceptanceTest
Matilda-Clerke Sep 20, 2024
b0f2ed0
7311: Modify PeerTaskExecutor metric to include response time from peer
Matilda-Clerke Sep 26, 2024
598b519
7311: Use SubProtocol instead of subprotocol name string in PeerTask
Matilda-Clerke Sep 26, 2024
bc25b16
7311: rename timing context to ignored to prevent intellij warnings
Matilda-Clerke Sep 26, 2024
e31bb70
7311: Use constants for number of retries
Matilda-Clerke Sep 26, 2024
41923d3
7311: Convert PeerTaskExecutorResult to a record
Matilda-Clerke Sep 26, 2024
720f94e
7311: Rename PeerTaskBehavior to PeerTaskRetryBehavior
Matilda-Clerke Sep 29, 2024
7d845b3
7311: Move peer selection logic to PeerSelector
Matilda-Clerke Sep 30, 2024
50c26f1
7311: spotless
Matilda-Clerke Sep 30, 2024
b7c0c95
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 30, 2024
a81855d
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Sep 30, 2024
64adedc
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Sep 30, 2024
2c1446e
7311: Fix up everything broken after merge
Matilda-Clerke Oct 1, 2024
3c0c47b
7311: Attempt to improve performance of peer task system in pipeline
Matilda-Clerke Oct 2, 2024
d0bd5ed
7311: fix compile check
Matilda-Clerke Oct 2, 2024
1c25ac5
7311: Fix broken workflow
Matilda-Clerke Oct 2, 2024
2e6dfd9
7311: Reduce logging in JsonRpcExecutor to trace level
Matilda-Clerke Oct 2, 2024
aca8058
7311: More changes in DownloadReceiptsStep
Matilda-Clerke Oct 2, 2024
4d59b10
7311: Rework DownloadReceiptsStep
Matilda-Clerke Oct 3, 2024
8718102
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 3, 2024
e63f473
7311: Make changes as discussed in walkthrough meeting
Matilda-Clerke Oct 3, 2024
fae39a8
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 3, 2024
d1847f2
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 4, 2024
07852dc
7311: Update after merge and make discussed changes from walkthrough …
Matilda-Clerke Oct 4, 2024
c477d70
7311: Change to regular HashMap
Matilda-Clerke Oct 4, 2024
6c57a7c
7311: Remove runtime exception again
Matilda-Clerke Oct 4, 2024
6d2cb95
7311: Rename getPeerTaskBehavior to getPeerTaskRetryBehavior
Matilda-Clerke Oct 6, 2024
d84520a
7311: Rename getPeerTaskBehavior to getPeerTaskRetryBehavior
Matilda-Clerke Oct 6, 2024
77ed748
Merge remote-tracking branch 'origin/7311-add-peertask-foundation-cod…
Matilda-Clerke Oct 6, 2024
0896e31
7311: Rework PeerTaskExecutor retry system to be 0-based
Matilda-Clerke Oct 6, 2024
5f924c4
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 6, 2024
b13ac92
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 6, 2024
5006b39
7311: Fix up compile errors after merge
Matilda-Clerke Oct 6, 2024
07f3a7e
7311: Fix broken DownloadReceiptsStepTest test
Matilda-Clerke Oct 6, 2024
493ac91
7311: Move GetReceipts to services worker for parallelism
Matilda-Clerke Oct 7, 2024
1a30174
7311: Refactor peer task system usage in DownloadReceiptsStep to bett…
Matilda-Clerke Oct 7, 2024
84af9f9
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Oct 7, 2024
2865625
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 7, 2024
2f86ed9
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Oct 7, 2024
82cedb0
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 7, 2024
bdd96ba
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 8, 2024
c047f42
7311: Remove unused async methods in PeerTaskExecutor
Matilda-Clerke Oct 8, 2024
5aa6b0b
7311: Return Optional<EthPeer> in PeerSelector.getPeer and utilise ex…
Matilda-Clerke Oct 8, 2024
4fd4724
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 8, 2024
d6120b0
7311: Update after merge
Matilda-Clerke Oct 8, 2024
8becdb3
7311: Redo getPeer again to include hasAvailableRequestCapacity check
Matilda-Clerke Oct 8, 2024
4ad85e8
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 8, 2024
86a1f0b
7311: Add protocol spec supplier to GetReceiptsFromPeerTask
Matilda-Clerke Oct 8, 2024
8186a77
7311: Rework getPeer again to use LEAST_TO_MOST_BUSY comparator
Matilda-Clerke Oct 8, 2024
37b0ec2
7311: Import PeerNotConnected class instead of using fully qualified …
Matilda-Clerke Oct 8, 2024
545fd5c
7311: Change to specifying retry counts in PeerTask instead of behavi…
Matilda-Clerke Oct 9, 2024
7bd048b
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 9, 2024
e9d08f3
7311: clean up after merge
Matilda-Clerke Oct 9, 2024
20478d3
7311: clean up after merge
Matilda-Clerke Oct 9, 2024
4f544f4
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 9, 2024
66a9de2
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 9, 2024
3ddfe71
7311: Fix up javadoc
Matilda-Clerke Oct 10, 2024
1c268b7
7311: Add additional metrics to PeerTaskExecutor
Matilda-Clerke Oct 10, 2024
b06f38b
7311: Add Predicate to PeerTask to check for partial success
Matilda-Clerke Oct 10, 2024
09ee1c8
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 10, 2024
3c12d3d
7311: Fix incorrect name on isPartialSuccessTest
Matilda-Clerke Oct 10, 2024
b1c47ae
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 10, 2024
4664db9
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 10, 2024
3b8b7d5
7311: Implement isPartialSuccess and add unit tests
Matilda-Clerke Oct 10, 2024
d66dd3a
7311: Add partialSuccessCounter and inflightRequestGauge in PeerTaskE…
Matilda-Clerke Oct 11, 2024
fa22e93
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 11, 2024
cff0099
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 11, 2024
a3f5d4a
7311: Also filter by whether a peer is fully validated
Matilda-Clerke Oct 11, 2024
382f7a5
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 11, 2024
714db0a
7311: Remove unneeded throws in RunnerTest
Matilda-Clerke Oct 11, 2024
3a68980
7311: Fix up inflight requests gauge in PeerTaskExecutor
Matilda-Clerke Oct 11, 2024
3ce476d
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 11, 2024
c422bc5
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 11, 2024
74aa7a0
7311: Update plugin api hash
Matilda-Clerke Oct 11, 2024
56c1f9d
7311: Update plugin api hash
Matilda-Clerke Oct 11, 2024
fe50d95
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 11, 2024
e733452
7311: Add javadoc to LabelledGauge.isLabelsObserved
Matilda-Clerke Oct 13, 2024
b3a252b
7311: Update plugin-api hash
Matilda-Clerke Oct 13, 2024
4f9cf52
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 13, 2024
af93824
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 13, 2024
3c96eba
7311: Update changelog
Matilda-Clerke Oct 13, 2024
e664a51
7311: Handle headers with no receipts as a special case in DownloadRe…
Matilda-Clerke Oct 15, 2024
7daf30f
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 15, 2024
c4e685d
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 15, 2024
6800cdd
7311: Complete merge
Matilda-Clerke Oct 15, 2024
44fd3a8
7311: Use taskName instead of className for labelNames
Matilda-Clerke Oct 16, 2024
ac1c4ed
7311: Use snake_case for metric names
Matilda-Clerke Oct 16, 2024
7503535
7311: Use _total metric name suffix
Matilda-Clerke Oct 16, 2024
a59bd30
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 16, 2024
ed25941
7311: rework partial success handling
Matilda-Clerke Oct 17, 2024
4f4b091
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 17, 2024
c396fb5
7311: Update GetReceiptsFromPeerTask with partialSuccess changes
Matilda-Clerke Oct 17, 2024
5a79636
7311: Add default implementation to LabelledGauge.isLabelsObserved
Matilda-Clerke Oct 17, 2024
0668083
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 17, 2024
b075a91
7311: Fix broken unit test
Matilda-Clerke Oct 17, 2024
74995bb
Merge branch 'main' into 7311-add-peertask-foundation-code
Matilda-Clerke Oct 17, 2024
17edf67
Merge branch 'refs/heads/7311-add-peertask-foundation-code' into 7311…
Matilda-Clerke Oct 17, 2024
e2fea48
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Oct 18, 2024
875375e
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Oct 20, 2024
16bcb1a
Merge remote-tracking branch 'matilda/7311-add-GetReceiptsFromPeerTas…
pinges Oct 22, 2024
095e31d
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Oct 23, 2024
c2205c4
7311: Rename parseResponse to processResponse
Matilda-Clerke Oct 23, 2024
dc1f3bd
add possibility to use the new peer task system when downloading the …
pinges Oct 24, 2024
845b564
fix loop
pinges Oct 24, 2024
1b9922f
7311: Wrap peer task system usage in ethScheduler call to match other…
Matilda-Clerke Oct 24, 2024
023445f
small fixes
pinges Oct 25, 2024
50212dd
Merge remote-tracking branch 'matilda/7311-add-GetReceiptsFromPeerTas…
pinges Oct 25, 2024
512ee0b
update API change
pinges Oct 25, 2024
39542bb
spotless
pinges Oct 25, 2024
d827e68
7311: apply spotless
Matilda-Clerke Oct 27, 2024
26476ae
7311: Move check for empty trie hash into GetReceiptsFromPeerTask and…
Matilda-Clerke Oct 27, 2024
a357ecf
7311: spotless
Matilda-Clerke Oct 27, 2024
e7a13a4
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Oct 27, 2024
3d20f56
7311: Fix compile issue after merge
Matilda-Clerke Oct 27, 2024
f1db42e
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Oct 28, 2024
6ee320b
7311: Remove BodyValidator and update code and test to match
Matilda-Clerke Oct 28, 2024
5fcad25
7311: spotless
Matilda-Clerke Oct 28, 2024
1923996
7311: Fix up pre-fill and add test to test failure scenario
Matilda-Clerke Oct 29, 2024
84b422c
7311: Use ProtocolSchedule.anyMatch to find if any ProtocolSpecs are …
Matilda-Clerke Oct 29, 2024
8e6e2b0
7311: Only attempt to remove headers on successful requests
Matilda-Clerke Oct 29, 2024
24d8f99
Merge branch 'main' into 7311-add-GetReceiptsFromPeerTask
Matilda-Clerke Oct 29, 2024
c7f5a74
Merge remote-tracking branch 'stefan/GetBodiesPeerTask' into GetBodie…
Matilda-Clerke Oct 30, 2024
892f96d
7311: Fix broken stuff after merge
Matilda-Clerke Oct 30, 2024
b2e1069
Merge branch 'main' into GetBodiesPeerTask
Matilda-Clerke Nov 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
/*
* Copyright contributors to Hyperledger Besu.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
* SPDX-License-Identifier: Apache-2.0
*/
package org.hyperledger.besu.ethereum.eth.manager.peertask.task;

import org.hyperledger.besu.ethereum.core.Block;
import org.hyperledger.besu.ethereum.core.BlockBody;
import org.hyperledger.besu.ethereum.core.BlockHeader;
import org.hyperledger.besu.ethereum.eth.EthProtocol;
import org.hyperledger.besu.ethereum.eth.manager.EthPeer;
import org.hyperledger.besu.ethereum.eth.manager.peertask.InvalidPeerTaskResponseException;
import org.hyperledger.besu.ethereum.eth.manager.peertask.PeerTask;
import org.hyperledger.besu.ethereum.eth.messages.BlockBodiesMessage;
import org.hyperledger.besu.ethereum.eth.messages.GetBlockBodiesMessage;
import org.hyperledger.besu.ethereum.mainnet.BodyValidation;
import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
import org.hyperledger.besu.ethereum.p2p.rlpx.wire.MessageData;
import org.hyperledger.besu.ethereum.p2p.rlpx.wire.SubProtocol;

import java.util.ArrayList;
import java.util.List;
import java.util.function.Predicate;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class GetBodiesFromPeerTask implements PeerTask<List<Block>> {

private static final Logger LOG = LoggerFactory.getLogger(GetBodiesFromPeerTask.class);

private final List<BlockHeader> blockHeaders;
private final ProtocolSchedule protocolSchedule;

private final long requiredBlockchainHeight;
private final List<Block> blocks = new ArrayList<>();
private final boolean isPoS;

public GetBodiesFromPeerTask(
final List<BlockHeader> blockHeaders, final ProtocolSchedule protocolSchedule) {
if (blockHeaders == null || blockHeaders.isEmpty()) {
throw new IllegalArgumentException("Block headers must not be empty");
}

this.blockHeaders = blockHeaders;
this.protocolSchedule = protocolSchedule;

this.requiredBlockchainHeight =
blockHeaders.stream()
.mapToLong(BlockHeader::getNumber)
.max()
.orElse(BlockHeader.GENESIS_BLOCK_NUMBER);
this.isPoS = protocolSchedule.getByBlockHeader(blockHeaders.getLast()).isPoS();
}

@Override
public SubProtocol getSubProtocol() {
return EthProtocol.get();
}

@Override
public MessageData getRequestMessage() {
return GetBlockBodiesMessage.create(
blockHeaders.stream().map(BlockHeader::getBlockHash).toList());
}

@Override
public List<Block> processResponse(final MessageData messageData)
throws InvalidPeerTaskResponseException {
// Blocks returned by this method are in the same order as the headers, but might not be
// complete
if (messageData == null) {
throw new InvalidPeerTaskResponseException();
}
final BlockBodiesMessage blocksMessage = BlockBodiesMessage.readFrom(messageData);
final List<BlockBody> blockBodies = blocksMessage.bodies(protocolSchedule);
if (blockBodies.isEmpty() || blockBodies.size() > blockHeaders.size()) {
throw new InvalidPeerTaskResponseException();
}

for (int i = 0; i < blockBodies.size(); i++) {
final BlockBody blockBody = blockBodies.get(i);
final BlockHeader blockHeader = blockHeaders.get(i);
if (!blockBodyMatchesBlockHeader(blockBody, blockHeader)) {
LOG.atDebug().setMessage("Received block body does not match block header").log();
throw new InvalidPeerTaskResponseException();
}

blocks.add(new Block(blockHeader, blockBody));
}
return blocks;
}

private boolean blockBodyMatchesBlockHeader(
final BlockBody blockBody, final BlockHeader blockHeader) {
// this method validates that the block body matches the block header by calculating the roots
// of the block body and comparing them to the roots in the block header
if (!BodyValidation.transactionsRoot(blockBody.getTransactions())
.equals(blockHeader.getTransactionsRoot())) {
return false;
}
if (!BodyValidation.ommersHash(blockBody.getOmmers()).equals(blockHeader.getOmmersHash())) {
return false;
}
if (blockBody.getWithdrawals().isPresent()) {
if (blockHeader.getWithdrawalsRoot().isEmpty()) {
return false;
}
if (!BodyValidation.withdrawalsRoot(blockBody.getWithdrawals().get())
.equals(blockHeader.getWithdrawalsRoot().get())) {
return false;
}
} else if (blockHeader.getWithdrawalsRoot().isPresent()) {
return false;
}
return true;
}

@Override
public Predicate<EthPeer> getPeerRequirementFilter() {
return (ethPeer) ->
isPoS || ethPeer.chainState().getEstimatedHeight() >= requiredBlockchainHeight;
}

@Override
public boolean isSuccess(final List<Block> result) {
return !result.isEmpty();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ public DefaultSynchronizer(
syncState,
metricsSystem,
terminationCondition,
peerTaskExecutor,
syncDurationMetrics));

if (SyncMode.FAST.equals(syncConfig.getSyncMode())) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@
import org.hyperledger.besu.ethereum.core.Block;
import org.hyperledger.besu.ethereum.core.BlockHeader;
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.manager.peertask.PeerTaskExecutor;
import org.hyperledger.besu.ethereum.eth.sync.tasks.CompleteBlocksTask;
import org.hyperledger.besu.ethereum.eth.sync.tasks.CompleteBlocksWithPeerTask;
import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
import org.hyperledger.besu.plugin.services.MetricsSystem;

Expand All @@ -31,19 +33,41 @@ public class DownloadBodiesStep
private final ProtocolSchedule protocolSchedule;
private final EthContext ethContext;
private final MetricsSystem metricsSystem;
private final SynchronizerConfiguration synchronizerConfiguration;
private final PeerTaskExecutor peerTaskExecutor;

public DownloadBodiesStep(
final ProtocolSchedule protocolSchedule,
final EthContext ethContext,
final PeerTaskExecutor peerTaskExecutor,
final SynchronizerConfiguration synchronizerConfiguration,
final MetricsSystem metricsSystem) {
this.protocolSchedule = protocolSchedule;
this.ethContext = ethContext;
this.peerTaskExecutor = peerTaskExecutor;
this.synchronizerConfiguration = synchronizerConfiguration;
this.metricsSystem = metricsSystem;
}

@Override
public CompletableFuture<List<Block>> apply(final List<BlockHeader> blockHeaders) {
return CompleteBlocksTask.forHeaders(protocolSchedule, ethContext, blockHeaders, metricsSystem)
.run();
if (synchronizerConfiguration.isPeerTaskSystemEnabled()) {
return ethContext
.getScheduler()
.scheduleServiceTask(() -> getBodiesWithPeerTaskSystem(blockHeaders));
} else {
return CompleteBlocksTask.forHeaders(
protocolSchedule, ethContext, blockHeaders, metricsSystem)
.run();
}
}

private CompletableFuture<List<Block>> getBodiesWithPeerTaskSystem(
final List<BlockHeader> headers) {

final CompleteBlocksWithPeerTask completeBlocksWithPeerTask =
new CompleteBlocksWithPeerTask(protocolSchedule, headers, peerTaskExecutor);
final List<Block> blocks = completeBlocksWithPeerTask.getBlocks();
return CompletableFuture.completedFuture(blocks);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,8 @@ public Pipeline<SyncTargetRange> createDownloadPipelineForSyncTarget(final SyncT
final RangeHeadersValidationStep validateHeadersJoinUpStep =
new RangeHeadersValidationStep(protocolSchedule, protocolContext, detachedValidationPolicy);
final DownloadBodiesStep downloadBodiesStep =
new DownloadBodiesStep(protocolSchedule, ethContext, metricsSystem);
new DownloadBodiesStep(
protocolSchedule, ethContext, peerTaskExecutor, syncConfig, metricsSystem);
final DownloadReceiptsStep downloadReceiptsStep =
new DownloadReceiptsStep(
protocolSchedule, ethContext, peerTaskExecutor, syncConfig, metricsSystem);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

import org.hyperledger.besu.ethereum.ProtocolContext;
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.manager.peertask.PeerTaskExecutor;
import org.hyperledger.besu.ethereum.eth.sync.ChainDownloader;
import org.hyperledger.besu.ethereum.eth.sync.PipelineChainDownloader;
import org.hyperledger.besu.ethereum.eth.sync.SynchronizerConfiguration;
Expand All @@ -35,7 +36,8 @@ public static ChainDownloader create(
final SyncState syncState,
final MetricsSystem metricsSystem,
final SyncTerminationCondition terminationCondition,
final SyncDurationMetrics syncDurationMetrics) {
final SyncDurationMetrics syncDurationMetrics,
final PeerTaskExecutor peerTaskExecutor) {

final FullSyncTargetManager syncTargetManager =
new FullSyncTargetManager(
Expand All @@ -54,6 +56,7 @@ public static ChainDownloader create(
protocolSchedule,
protocolContext,
ethContext,
peerTaskExecutor,
metricsSystem,
terminationCondition),
ethContext.getScheduler(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.manager.EthPeer;
import org.hyperledger.besu.ethereum.eth.manager.EthScheduler;
import org.hyperledger.besu.ethereum.eth.manager.peertask.PeerTaskExecutor;
import org.hyperledger.besu.ethereum.eth.sync.DownloadBodiesStep;
import org.hyperledger.besu.ethereum.eth.sync.DownloadHeadersStep;
import org.hyperledger.besu.ethereum.eth.sync.DownloadPipelineFactory;
Expand Down Expand Up @@ -53,21 +54,25 @@ public class FullSyncDownloadPipelineFactory implements DownloadPipelineFactory
() -> HeaderValidationMode.DETACHED_ONLY;
private final BetterSyncTargetEvaluator betterSyncTargetEvaluator;
private final SyncTerminationCondition fullSyncTerminationCondition;
private final PeerTaskExecutor peerTaskExecutor;

public FullSyncDownloadPipelineFactory(
final SynchronizerConfiguration syncConfig,
final ProtocolSchedule protocolSchedule,
final ProtocolContext protocolContext,
final EthContext ethContext,
final PeerTaskExecutor peerTaskExecutor,
final MetricsSystem metricsSystem,
final SyncTerminationCondition syncTerminationCondition) {
this.syncConfig = syncConfig;
this.protocolSchedule = protocolSchedule;
this.protocolContext = protocolContext;
this.ethContext = ethContext;
this.peerTaskExecutor = peerTaskExecutor;
this.metricsSystem = metricsSystem;
this.fullSyncTerminationCondition = syncTerminationCondition;
betterSyncTargetEvaluator = new BetterSyncTargetEvaluator(syncConfig, ethContext.getEthPeers());
this.betterSyncTargetEvaluator =
new BetterSyncTargetEvaluator(syncConfig, ethContext.getEthPeers());
}

@Override
Expand Down Expand Up @@ -104,7 +109,8 @@ public Pipeline<?> createDownloadPipelineForSyncTarget(final SyncTarget target)
final RangeHeadersValidationStep validateHeadersJoinUpStep =
new RangeHeadersValidationStep(protocolSchedule, protocolContext, detachedValidationPolicy);
final DownloadBodiesStep downloadBodiesStep =
new DownloadBodiesStep(protocolSchedule, ethContext, metricsSystem);
new DownloadBodiesStep(
protocolSchedule, ethContext, peerTaskExecutor, syncConfig, metricsSystem);
final ExtractTxSignaturesStep extractTxSignaturesStep = new ExtractTxSignaturesStep();
final FullImportBlockStep importBlockStep =
new FullImportBlockStep(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

import org.hyperledger.besu.ethereum.ProtocolContext;
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.manager.peertask.PeerTaskExecutor;
import org.hyperledger.besu.ethereum.eth.sync.ChainDownloader;
import org.hyperledger.besu.ethereum.eth.sync.SynchronizerConfiguration;
import org.hyperledger.besu.ethereum.eth.sync.TrailingPeerRequirements;
Expand Down Expand Up @@ -45,6 +46,7 @@ public FullSyncDownloader(
final SyncState syncState,
final MetricsSystem metricsSystem,
final SyncTerminationCondition terminationCondition,
final PeerTaskExecutor peerTaskExecutor,
final SyncDurationMetrics syncDurationMetrics) {
this.syncConfig = syncConfig;
this.protocolContext = protocolContext;
Expand All @@ -59,7 +61,8 @@ public FullSyncDownloader(
syncState,
metricsSystem,
terminationCondition,
syncDurationMetrics);
syncDurationMetrics,
peerTaskExecutor);
}

public CompletableFuture<Void> start() {
Expand Down
Loading
Loading