Skip to content

Commit

Permalink
Fix test-network to work with BFT consensus.
Browse files Browse the repository at this point in the history
Added a new option for creating channel:
Running ./network.sh createChannel -bft will initiate a channel running BFT orderers.
Using ./network.sh up -bft will initiate dockers for bft environment.
Added option for 4 orderers.

Add DOC file for adding and removing BFT nodes in a functioning BFT network.

Signed-off-by: Arkadi Piven <arkadi.piven@ibm.com>
Signed-off-by: arkadipiven <arkadi7770@gmail.com>
  • Loading branch information
arkadiPiven committed Aug 20, 2023
1 parent f8f52ca commit 5436366
Show file tree
Hide file tree
Showing 16 changed files with 1,380 additions and 264 deletions.
310 changes: 310 additions & 0 deletions test-network/ADD_ORDERER_TUTORIAL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,310 @@
# Adding Orderer To An Existing Network

## Create an initial cluster
Fabric supports adding new orderer to an existing functioning network.
We will lay out a simple scenario of such functionality using the **test-network** sample.

### Exending the test network to support the fifth orderer
We extend the `docker-compose-bft` and the `crypto-config-orderer.yaml` to support 5 orderers.\
In the `crypto-config-orderer.yaml` we should add:
```yaml
- Hostname: orderer5
SANS:
- localhost
```
In the `docker-compose-bft` we should create a new volume in the volumes section:
```yaml
volumes:
- ...
- orderer5.example.com
```
Now, add the definition of the new orderer:
(Note that you can change the ports according to your needs)
```yaml
orderer5.example.com:
container_name: orderer5.example.com
image: hyperledger/fabric-orderer:latest
labels:
service: hyperledger-fabric
environment:
- FABRIC_LOGGING_SPEC=DEBUG
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_LISTENPORT=7060
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_GENERAL_BOOTSTRAPMETHOD=none
- ORDERER_CHANNELPARTICIPATION_ENABLED=true
- ORDERER_ADMIN_TLS_ENABLED=true
- ORDERER_ADMIN_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_ADMIN_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_ADMIN_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_ADMIN_TLS_CLIENTROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_ADMIN_LISTENADDRESS=0.0.0.0:7061
- ORDERER_OPERATIONS_LISTENADDRESS=orderer5.example.com:9450
- ORDERER_METRICS_PROVIDER=prometheus
working_dir: /root
command: orderer
volumes:
- ../organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/msp:/var/hyperledger/orderer/msp
- ../organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/tls:/var/hyperledger/orderer/tls
- orderer5.example.com:/var/hyperledger/production/orderer
ports:
- 7060:7060
- 7061:7061
- 9450:9450
networks:
- test
```

We also add the following volume to the CLI container definition:
```yaml
volumes:
- ../organizations/ordererOrganizations/example.com/users/Admin@example.com/msp:/var/hyperledger/orderer/msp
- ../organizations/ordererOrganizations/example.com/users/Admin@example.com/tls:/var/hyperledger/orderer/tls
```

### Running the cluster
Use:
```shell
./network.sh createChannel -bft
```
This command will start a network of 4 orderers and 2 peers and 1 CLI, a container of the fifth orderer will be started
as well, but is not a part of the network at this stage.

## Altering the config
The following commands should be executing from the CLI container.

### Getting the last config block
The `peer` command uses environment variables to define the context of the organization in which it will run, we will
change the context to:
```shell
export CORE_PEER_LOCALMSPID="Org1MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=$PEER0_ORG1_CA
export CORE_PEER_MSPCONFIGPATH=${PWD}/organizations/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp
export CORE_PEER_ADDRESS=localhost:7051
```
Now, in order to get the last config block we will make:
```shell
peer channel fetch config config_block.pb -o orderer.example.com:7050 --ordererTLSHostnameOverride orderer.example.com -c mychannel --tls --cafile "$ORDERER_CA"
```

### Convert the block to a JSON
Convert the block to JSON:
```shell
configtxlator proto_decode --input config_block.pb --type common.Block --output config_block.json
```
Extract the config from the JSON block:
```shell
jq .data.data[0].payload.data.config config_block.json > original_config.json
```

### Add the fifth orderer to the config
The output of this stage is an update TX, you can calculate the TX from the CLI container,
or copy the `original_config.json` and make all the changes on your local machine.\
Create a copy of `original_config.json` named `modified_config.json`.
In the new JSON file we need to make 4 changes:

#### 1. Add the orderer to the known endpoints
Go to **channel_group &rarr; groups &rarr; Orderer &rarr; groups &rarr; OrdererOrg &rarr; values &rarr; Endpoints &rarr; value &rarr; addresses**
and add the new orderer endpoint.
```json lines
[
"orderer.example.com:7050",
"orderer.example.com:7052",
"orderer.example.com:7056",
"orderer.example.com:7058",
"orderer.example.com:7060"
]
```

#### 2. Add the orderer to the known identities
Go to **channel_group &rarr; groups &rarr; Orderer &rarr; policies &rarr; BlockValidation &rarr; policy &rarr; value &rarr; identities**
and add the base64 encode of the identity certificate, please correct the path according to your needs.

```json
{
"principal": {
"id_bytes": ".../test-network/organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/msp/signcerts/orderer5.example.com-cert.pem",
"mspid": "OrdererMSP"
},
"principal_classification": "IDENTITY"
}
```

#### 3. Add the orderer to the policy rules
Go to **channel_group &rarr; groups &rarr; Orderer &rarr; policies &rarr; BlockValidation &rarr; policy &rarr; value &rarr; rule**,
change the **n** to be:
```
# Given that the new number of nodes in cluster is num_of_nodes:
f = int((num_of_nodes - 1) / 3)
n = ceil((num_of_nodes + f + 1) / 2)
```

And add a `signed_by` object for the new orderer:

```json
{
"n_out_of": {
"n": 4,
"rules": [
{
"signed_by": 0
},
{
"signed_by": 1
},
{
"signed_by": 2
},
{
"signed_by": 3
},
{
"signed_by": 4
}
]
}
}
```

#### 4. Add the orderer to the concenter mapping
Go to **channel_group &rarr; groups &rarr; Orderer &rarr; values &rarr; Orderers &rarr; value &rarr; consenter_mapping**
and add the base64 encode of the identity, client TLS and server TLS certificates, please correct the paths according
to your needs.
```json
{
"client_tls_cert": ".../test-network/organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/msp/tlscacerts/tlsca.example.com-cert.pem",
"host": "orderer5.example.com",
"id": 5,
"identity": ".../test-network/organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/msp/signcerts/orderer5.example.com-cert.pem",
"msp_id": "OrdererMSP",
"port": 7060,
"server_tls_cert": ".../test-network/organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/msp/tlscacerts/tlsca.example.com-cert.pem"
}
```

We made this process easy and created a Python script which can be found in the `scripts` subfolder that does just that (steps 1-4)!
Example for the script usage:
```shell
python3 add_new_orderer_to_config.py original_config.json modified_config.json
-a orderer5.example.com:7060
-i .../test-network/organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/msp/signcerts/orderer5.example.com-cert.pem
-s .../test-network/organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/msp/tlscacerts/tlsca.example.com-cert.pem
-c .../test-network/organizations/ordererOrganizations/example.com/orderers/orderer5.example.com/msp/tlscacerts/tlsca.example.com-cert.pem
```

Calculate the update using:
```shell
configtxlator proto_encode --input original_config.json --type common.Config --output original_config.pb
configtxlator proto_encode --input modified_config.json --type common.Config --output modified_config.pb
configtxlator compute_update --channel_id mychannel --original original_config.pb --updated modified_config.pb --output config_update.pb
configtxlator proto_decode --input config_update.pb --type common.ConfigUpdate --output config_update.json
echo '{"payload":{"header":{"channel_header":{"channel_id":"mychannel", "type":2}},"data":{"config_update":'$(cat config_update.json)'}}}' | jq . >config_update_in_envelope.json
configtxlator proto_encode --input config_update_in_envelope.json --type common.Envelope --output envelope.pb
```

`envelope.pb` is the config update TX, note that it does not contain any paths,
if it was created on your local machine, please copy it to the CLI container.

## Make the update
From the CLI we need to sign the TX using one of the peers' organizations and the orderes' organization.
Since we are in the context of the peer organization `Org1`, we can simply:
```shell
peer channel signconfigtx -f envelope.pb
```
Now we switch to the orderer organization `Orderer`:
```shell
export CORE_PEER_TLS_ENABLED=true
export CORE_PEER_LOCALMSPID="OrdererMSP"
export CORE_PEER_TLS_ROOTCERT_FILE=/var/hyperledger/orderer/tls/ca.crt
export CORE_PEER_MSPCONFIGPATH=/var/hyperledger/orderer/msp
export CORE_PEER_ADDRESS=localhost:7050
```
And we update the orderer:
```shell
peer channel update -o orderer.example.com:7050 --ordererTLSHostnameOverride orderer.example.com -c mychannel -f envelope.pb --tls --cafile "$ORDERER_CA"
```

The output of this command looks similar to:
```
INFO [channelCmd] InitCmdFactory -> Endorser and orderer connections initialized
INFO [channelCmd] update -> Successfully submitted channel update
```

The new orderer has been added to the cluster, but not to the test channel.

## Use the `osnadmin` CLI to add the new orderer to the test channel

The new orderer needs to run the following command:

```shell
export OSN_TLS_CA_ROOT_CERT=${PWD}/organizations/ordererOrganizations/example.com/tlsca/tlsca.example.com-cert.pem
export ADMIN_TLS_SIGN_CERT=${PWD}/organizations/ordererOrganizations/example.com/orderers/orderer4.example.com/tls/server.crt
export ADMIN_TLS_PRIVATE_KEY=${PWD}/organizations/ordererOrganizations/example.com/orderers/orderer4.example.com/tls/server.key
osnadmin channel join --channelID [CHANNEL_NAME] --config-block [CHANNEL_CONFIG_BLOCK] -o [ORDERER_ADMIN_LISTENADDRESS] --ca-file $OSN_TLS_CA_ROOT_CERT --client-cert $ADMIN_TLS_SIGN_CERT --client-key $ADMIN_TLS_PRIVATE_KEY
```

Replace:
- `CHANNEL_NAME` with the name you want to call this channel.
- `CHANNEL_CONFIG_BLOCK` with the path and file name of the genesis block or the latest config block.
- `ORDERER_ADMIN_LISTENADDRESS` corresponds to the `Orderer.Admin.ListenAddress` defined in the `orderer.yaml` for this orderer.
- `OSN_TLS_CA_ROOT_CERT` with the path and file name of the orderer organization TLS CA root certificate and intermediate certificate if using an intermediate TLS CA.
- `ADMIN_TLS_SIGN_CERT` with the path and file name of the admin client signed certificate from the TLS CA.
- `ADMIN_TLS_PRIVATE_KEY` with the path and file name of the admin client private key from the TLS CA.

For example:
```shell
osnadmin channel join --channelID mychannel --config-block ./channel-artifacts/mychannel.block -o localhost:7061 --ca-file "$OSN_TLS_CA_ROOT_CERT" --client-cert "$ADMIN_TLS_SIGN_CERT" --client-key "$ADMIN_TLS_PRIVATE_KEY"
```

**Note:** Because the connection between the `osnadmin` CLI and the orderer requires mutual TLS, you need to pass the `--client-cert` and `--client-key` parameters on each `osadmin` command. The `--client-cert` parameter points to the admin client certificate and `--client-key` refers to the admin client private key, both issued by the admin client TLS CA.

The output of this command looks similar to:
```
Status: 201
{
"name": "mychannel",
"url": "/participation/v1/channels/mychannel",
"consensusRelation": "follower",
"status": "onboarding",
"height": 0
}
```

You should see something similar to the following in your orderer logs:
```
INFO [orderer.consensus.smartbft.chain] NewChain -> SmartBFT-v3 is now servicing chain mychannel channel=mychannel
INFO [orderer.common.cluster] ConfigureNodeCerts -> Updating nodes identity, channel: mychannel, nodes: [id:1 host:"orderer.example.com" port:7050 msp_id:"OrdererMSP" identity:"..." client_tls_cert:"..." server_tls_cert:"..." ]
```

You can read further about the osnadmin command [here](https://hyperledger-fabric.readthedocs.io/en/latest/create_channel/create_channel_participation.html#step-two-use-the-osnadmin-cli-to-add-the-first-orderer-to-the-channel).

## Use the `osnadmin` CLI to remove the new orderer from the test channel

Use the command:
```shell
osnadmin channel remove -o localhost:7061 --ca-file "$ORDERER_CA" --client-cert "$ORDERER_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER_ADMIN_TLS_PRIVATE_KEY" --channelID mychannel
```

The result should be:
```
Status: 204
```

You should see something similar to the following in your orderer logs:
```
INFO [orderer.consensus.smartbft.chain] Halt -> Shutting down chain channel=mychannel
INFO [orderer.consensus.smartbft.consensus] func1 -> Exiting channel=mychannel
INFO [orderer.consensus.smartbft.consensus] func1 -> Exiting channel=mychannel
INFO [orderer.common.multichannel] removeMember -> Removed channel: mychannel
```
20 changes: 20 additions & 0 deletions test-network/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,26 @@

You can use the `./network.sh` script to stand up a simple Fabric test network. The test network has two peer organizations with one peer each and a single node raft ordering service. You can also use the `./network.sh` script to create channels and deploy chaincode. For more information, see [Using the Fabric test network](https://hyperledger-fabric.readthedocs.io/en/latest/test_network.html). The test network is being introduced in Fabric v2.0 as the long term replacement for the `first-network` sample.

If you are planning to run the test network with consesnsus type BFT then please pass `-bft` flag as input to the `network.sh` script when creating the channel.
That is to create a network use:
```bash
./network.sh up -bft
```

To create a channel use:

```bash
./network.sh createChannel -bft
```

To restart a running network use:

```bash
./network.sh restart -bft
```

Note that running the createChannel command will start the network, if it is not already running.

Before you can deploy the test network, you need to follow the instructions to [Install the Samples, Binaries and Docker Images](https://hyperledger-fabric.readthedocs.io/en/latest/install.html) in the Hyperledger Fabric documentation.

## Using the Peer commands
Expand Down
Loading

0 comments on commit 5436366

Please sign in to comment.