Co-authored-by: Aurèle Oulès <aurele@oules.com>
Co-authored-by: TheCharlatan <seb.kung@gmail.com>

The following data is added:
- A newly introduced utxo set magic
- A version number
- The network magic
- The block height

The 'connect_nodes' function in the test framework relies
on a stable number of peer connections to verify the new
connection between the nodes is successfully established.
This approach is fragile, as any of the peers involved in
the process can drop, lose, or create a connection at any
step, causing subsequent 'wait_until' checks to stall
indefinitely even when the peers in question are connected
successfully.

This commit improves the situation by using the nodes' subversion
and the connection direction (inbound/outbound) to identify the
exact peer connection and perform the checks exclusively on it.

This carve out is intended to allow a second child under restricted
circumstances, but this topology is not allowed for v3 transactions.

As CPFP carve out does not explicitly require a second child to actually
exist, it has the effect of granting a free +10KvB descendant size limit
when a single child is enough to bust the descendant limit.

Only build ld64 from cctools.
Disable adhoc codesigning, to avoid non-determinism.

Adjust the security check for:
ld64.lld: warning: Option `-allow_stack_execute' is not yet implemented.
ld64.lld: error: -fixup_chains is incompatible with -no_pie
and to account for the embedding of LLVMs version number.

LLD enables ZERO_AR_DATE by default, setting it to zero would enable
non-determinism, setting it to any other value is ignored.

See:
https://github.com/llvm/llvm-project/blob/main/lld/docs/MachO/ld64-vs-lld.rst.

Not supported by llvm-ar / llvm-ranlib.

This information can be used to check bucketing
logic.

Test addresses are being mapped according to the ASMap
file provided properly. Compare the result of the `getrawaddrman`
RPC with the result from the ASMap Health Check.

The https://miniupnp.tuxfamily.org website is unavailable now.

No change in behavior.

For single transaction acceptance, this is a simple refactor:
Workspace::m_all_conflicting
Workspace::m_conflicting_fees
Workspace::m_conflicting_size
Workspace::m_replaced_transactions

are now grouped under a new SubPackageState struct that is
a member of MemPoolAccept.

And local variables m_total_vsize and m_total_modified_fees are now
SubpackageState members so they can be accessed from
PackageMempoolChecks.

We want these to be package-wide variables because
- Transactions could conflict with the same tx (just not the same
prevout), or their conflicts could share descendants.
- We want to compare conflicts with the package fee rather than
individual transaction fee.

We reset these MemPoolAccept-wide fields for each subpackage
evaluation to not cause state leaking, similar to temporary
coins.

The behavior is not new, but this rule exits earlier than before.
Previously, a carve out could have been granted in PreChecks() but then
nullified in PackageMempoolChecks() when CheckPackageLimits() is called
with the default limits.

This adds a markdown hyperlink check task to the lint test_runner. It
relies on having the [`mlc`](https://crates.io/crates/mlc) binary found
on $PATH, but will fail with `success` if the binary is not found.

`mlc` is also added to the ci/04_install.sh script run by the
containerfile.

Note that broken markdown hyperlinks will be detected in untracked
markdown files found in a dirty working directory (including e.g.
.venv).

This prevents SnapshotMetadata from using any globals implicitly.

Adds error messages that were not being handled. Also removes error
messages that no longer exist.

initialize variable

-BEGIN VERIFY SCRIPT-
sed -i 's/nNextSweep/m_next_sweep/g' $(git grep -l 'nNextSweep')
-END VERIFY SCRIPT-

fixing to match style

This supports lcov 2.x in the sense that we are no-longer hardcoding
version specific options, and instead will use the `LCOV_OPTS` variable
(which is the more correct/flexible thing to do in any case). It's also
quite likely that devs are already having to pass extra options to lcov
2.x, given it's more stringent in terms of coverage generation and error
checking. See this thread for an example:
linux-test-project/lcov#238.

Added an example to the developer notes.

Tested on one machine (LCOV 2.0, gcc 13.2) with:
```bash
./autogen.sh
./configure --enable-lcov CXXFLAGS="-fprofile-update=prefer-atomic" LCOV_OPTS="--rc branch_coverage=1 --ignore-errors mismatch"
make
make cov
<snip>
Processing file src/netaddress.cpp
  lines=521 hit=480 functions=72 hit=72 branches=675 hit=499
Overall coverage rate:
  lines......: 81.8% (79362 of 97002 lines)
  functions......: 77.8% (10356 of 13310 functions)
  branches......: 49.6% (130628 of 263196 branches)
```

and another machine (LCOV 2.1, Clang 18.1.3) with:
```bash
./autogen.sh
./configure --enable-lcov CC=clang CXX=clang++ LCOV_OPTS="--rc branch_coverage=1 --ignore-errors mismatch,inconsistent"
make
make cov
<snip>
Processing file src/util/strencodings.cpp
  lines=315 hit=311 functions=38 hit=38 branches=425 hit=357
Overall coverage rate:
  source files: 622
  lines.......: 79.8% (70311 of 88132 lines)
  functions...: 78.1% (13968 of 17881 functions)
  branches....: 44.5% (157551 of 354317 branches)
Message summary:
  101 warning messages:
    count: 1
    inconsistent: 100
  3528 ignore messages:
    inconsistent: 3528
```

Patch Qts internal libpng to resolve the failure.

I would like to have this patched, so we can continue working on the
removal of `FORCE_USE_SYSTEM_CLANG`. Otherwise builds will be broken using
the default clang (18) on the current Ubuntu LTS (24.04).

Adds missing `g++` for macOS. This is needed by Qt:
```bash
Configuring qt...
Creating qmake...
gmake[1]: Entering directory '/bitcoin/depends/work/build/arm64-apple-darwin/qt/5.15.14-4bca24c8f89/qtbase/qmake'
gmake[1]: g++: No such file or directory
gmake[1]: *** [Makefile:250: main.o] Error 127
```

`xz-utils` was also missing (but generally already installed), and is
needed for the `.tar.xz` tarballs.

Remove bsdmainutils, as this is only needed by the main build system
(for tests), and isn't needed to complete a depends build.

Specify json content type in RPC examples

…into bitcoinsync

Note that, as CURRENT_VERSION = 2, the wallet will not make transactions
with nVersion=3 yet.

See: boostorg/boost-tasks#3

Currently, when the FuzzedDataProvider of a FuzzedSock runs out of data,
FuzzedSock::Wait and WaitMany will simulate endless waiting as the
requested events are never simulated as occured.

Fix this by simulating event occurence when ConsumeBool() returns false
(e.g. when the data provider runs out).

Co-authored-by: dergoegge <n.goeggi@gmail.com>

FuzzedSock only supports peeking at one byte at a time, which is not
fuzzer friendly when trying to receive long data.

Fix this by supporting peek data of arbitrary length instead of only one
byte.

For example:
```bash
ADDITIONAL_GUIX_COMMON_FLAGS set in the ENV
ADDITIONAL_GUIX_ENVIRONMENT_FLAGS="--emulate-fhs" ./contrib/guix/guix-build
<snip>
INFO: Building f75199182133 for platform triple x86_64-linux-gnu:
      ...using reference timestamp: 1716905119
      ...running at most 10 jobs
      ...from worktree directory: '/bitcoin'
          ...bind-mounted in container to: '/bitcoin'
      ...in build directory: '/bitcoin/guix-build-f75199182133/distsrc-f75199182133-x86_64-linux-gnu'
          ...bind-mounted in container to: '/distsrc-base/distsrc-f75199182133-x86_64-linux-gnu'
      ...outputting in: '/bitcoin/guix-build-f75199182133/output/x86_64-linux-gnu'
          ...bind-mounted in container to: '/outdir-base/x86_64-linux-gnu'
      ADDITIONAL FLAGS (if set)
          ADDITIONAL_GUIX_COMMON_FLAGS: --no-substitutes
          ADDITIONAL_GUIX_ENVIRONMENT_FLAGS: --emulate-fhs
          ADDITIONAL_GUIX_TIMEMACHINE_FLAGS:
```

This method was introduced as a pre-requirement for the v2 transport
protocol back then (see PR #14047, commit 463921bb), when it was still
BIP151. With the replacement BIP324, this is not needed anymore, and
it's also unlikely that any other proposal would need to negate private
keys at this abstraction level.
(If there is really demand, it's trivial to reintroduce the method.)

Add support for passing `--lint=LINT_TO_RUN` to the lint runner and
add corresponding help message.

…t failure

Reverts part of fa67f096bdea9db59dd20c470c9e32f3dac5be94, now that we
require a minimum of GCC 11.

See also:
bitcoin/bitcoin#28349 (comment).

Here I've reduced the memory reallocations and copying operations in bech32 encode, making it ~15% faster.

make && ./src/bench/bench_bitcoin --filter='Bech32Encode' --min-time=1000

Before:

|             ns/byte |              byte/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|               19.97 |       50,074,562.72 |    0.1% |      1.06 | `Bech32Encode`
After:

|             ns/byte |              byte/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|               17.33 |       57,687,668.20 |    0.1% |      1.10 | `Bech32Encode`

Co-authored-by: josibake <josibake@protonmail.com>

Includes:
LLVM 18.1.x (#30201)
GCC 13.x (#29881)

git-minimal 2.41.0 -> 2.45.1
Kernel Headers 6.1.80 -> 6.1.92
moreutils 0.68 -> 0.69

Commits like
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=7b0f145802f0c2c785014293d748721678fef824,
which should improve the bootstrap situation (#30042).

The helper assumes that the n and k values have to be provided as a
single byte push operation, which is only possible for values up to 16.
Fix that by passing the numbers directly to the CScript list, where it's
automatically converted to minimally-encoded pushes (see class
method `CScript.__coerce_instance`, branch `isinstance(other, int)`).

In case of 17..20, this means that the data-pushes are done with two
bytes using OP_PUSH1 (0x01), e.g. for n=20: 0x01,0x14

This was disabled in #18738 due to the combo of old gcc and qt, neither of
which are relevant to us anymore.

…pty in all test cases

This new test uses the `vExtraTxnForCompact` (`extra_txn`) vector of
optional orphan/conflicted/etc. transactions to provide a transaction
in a compact block that was not otherwise present in our mempool.

This also covers an improbable nullptr deref bug addressed in
bf031a517c79cec5b43420bcd40291ab0e9f68a8 (#29752) where the
`extra_txn` vec/circular-buffer was sometimes null-initialized and
not yet filled when dereferenced in `PartiallyDownloadedBlock::InitData`.

clang-16 and earlier detect "foo -= foo" and "foo /= foo" as self-assignments.

Belt-and suspenders after #30234. Self-assignment should be safe _and_
discouraged.

We used to opt out of this warning because something deep in our
serialization/byteswapping code could self-assign, but that doesn't appear to
be the case anymore.

This is an STL-like container that interface-wise looks like std::deque, but
is backed by a (fixed size, with vector-like capacity/reserve) circular buffer.

GHA Windows images previously had multiple VC Build Tools installed,
which required specifying the `VCPKG_PLATFORM_TOOLSET_VERSION`
explicitly to avoid linker errors. This issue has been resolved as per
actions/runner-images#9701.

This change improves readability. Also the `Tee-Object` cmdlet is used
when appropriate.

Reverts a bug introduced in b47bd959207e82555f07e028cc2246943d32d4c3
"kernel: De-globalize fReindex". The change leads to a GUI user being
prompted to re-index on a chainstate loading failure more than once as
well as the node actually not reindexing if the user chooses to. Fix
this by setting the reindexing option instead of the atomic, which can
be safely re-used to indicate that a reindex should be attempted.

The bug specifically is caused by the chainman, and thus the blockman
and its m_reindexing atomic being destroyed on every iteration of
the for loop.

The reindex option for ChainstateLoadOptions is currently also set in a
confusing way. By using the reindex atomic, it is not obvious in which
scenario it is true or false.

The atomic is controlled by both the user passing the -reindex option,
the user chosing to reindex if something went wrong during chainstate
loading when running the gui, and by reading the reindexing flag from
the block tree database in LoadBlockIndexDB. In practice this read is
done through the chainstate module's CompleteChainstateInitialization's
call to LoadBlockIndex. Since this is only done after the reindex option
is set already, it does not have an effect on it.

Make this clear by using the reindex option from the blockman opts which
is only controlled by the user.

It does not control any actual logic and the log message as well as the
comment are obsolete, since no database initialization takes place there
anymore. Log messages indicating when indexes and chainstate databases
are loaded exist in other places.

Given that the use of a transaction's nVersion is always as an unsigned
int, it doesn't make sense to store it as signed and then cast it to
unsigned.

Drop confusing kernel options:

  BlockManagerOpts::reindex
  ChainstateLoadOptions::reindex
  ChainstateLoadOptions::reindex_chainstate

Replacing them with more straightforward options:

  ChainstateLoadOptions::wipe_block_tree_db
  ChainstateLoadOptions::wipe_chainstate_db

Having two options called "reindex" which did slightly different things
was needlessly confusing (one option wiped the block tree database, and
the other caused block files to be rescanned). Also the previous set of
options did not allow rebuilding the block database without also
rebuilding the chainstate database, when it should be possible to do
those independently.

Before this change continuing a reindex without the -reindex flag set
would leave the block and coins db intact, but discard the data of the
optional indexes. While not a bug per se, wiping the data again is
wasteful, both in terms of having to write it again, and potentially
leading to longer startup times.

When initially running a reindex, both the block index and any further
activated indexes are wiped. On an index's Init(), both the best block
stored by the index and the chain's tip are null. An index's m_synced
member is therefore true. This means that it will process blocks through
validation events while the reindex is running.

Currently, if the reindex is continued without the user re-specifying
the reindex flag, the block index is preserved but further index data is
wiped. This leads to the stored best block being null, but the chain tip
existing. The m_synced member will be set to false. The index will not
process blocks through the validation interface, but instead use the
background sync once the reindex is completed.

If the index is preserved (this change) after a restart its best block
may potentially match the chain tip. The m_synced member will be set to
true and the index can process validation events during the rest of the
reindex.

Co-authored-by: furszy <matiasfurszyfer@protonmail.com>

This is a just a mechanical change, renaming and inverting the meaning
of the indexing variable.

"m_blockfiles_indexed" is a more straightforward name for this variable
because this variable just indicates whether or not
<datadir>/blocks/blk?????.dat files have been indexed in the
<datadir>/blocks/index LevelDB database. The name "m_reindexing" was
more confusing, it could be true even if -reindex was not specified, and
false when it was specified. Also, the previous name unnecessarily
required thinking about the whole reindexing process just to understand
simple checks in validation code about whether blocks were indexed.

The motivation for this change is to follow up on previous commits,
moving away from having multiple variables called "reindex" internally,
and instead naming variables individually after what they do and
represent.