add blockchain explorer, add transfer explanation

Signed-off-by: Arne Rutjes <arne123@gmail.com>

token-sdk/README.md

@@ -6,17 +6,42 @@ Several instances of this service form a Layer 2 network that can transact among
 
 This sample is intended to get familiar with the features of the Token SDK and as a starting point for a proof of concept. The sample contains a basic development setup with:
 
--   An issuer node
--   An auditor node
--   Two owner nodes, with wallets for Alice, Bob, Carlos and Dan
+-   An issuer service
+-   An auditor service
+-   Two owner services, with wallets for Alice and Bob (on Owner 1), and Carlos and Dan (on Owner 2)
 -   A Certificate Authority
 -   Configuration to use a Fabric test network.
 
+From now on we'll call the services for the issuer, auditor and owners 'nodes' (not to be confused with Hyperledger Fabric peer nodes). Each of them runs as a separate application containing a REST API, the Fabric Smart Client and the Token SDK. The nodes talk to each other via a protocol called libp2p to create token transactions, and each of them also has a Hyperledger Fabric user to be able to submit the transaction to the settlement layer. The settlement layer is just any Fabric network that runs the Token Chaincode, which is configured with the identities of the issuer, auditor and CA to be able to validate transactions.
+
 ![components](./components.png)
 
 [plantuml source](https://www.plantuml.com/plantuml/uml/ZPB1IiD048RlUOgXteHM4nIXbD0O4RnO3mKllKmtssR9PYRiRYWYlhlPNPMsIkrjcFs_d-LZvjQXSNsh4ziewj1W2wsYKgErhwfoDQGtrqdIeMXmAs6qv4OIF4ktOzEC02quRk0z0H3STaoI78nMT123iWX9WJ2RmGRNHecnm6awkPtSGPuVmqLVASScCDXN7ffSOLmESRWekauhWKun7RDFrlOoeihQY2g_-vTSx6W8fIkwX6B8I3_SypfKSHgRU4Vd5cMUBz5ejdvwG8fDsQccZptJZy4JBALr1xvhlVdj-qNwluVtRXXJs3Fj5rEDpXVb-PzazaDcPuCBKqdpfPhZlCV6pGayNaXPeoB1bOod94Iqu_oZ-7uBcfPIrCIQjs_UaZ-wyJZtCfAvfAflzIS0)
 
-The source code contains three separate applications: issuer, auditor and owner. Each of the applications runs as a separate service ('node'). The nodes talk to each other (via a protocol called libp2p) to create token transactions, and each of them also has a Hyperledger Fabric user to be able to submit the transaction to the settlement layer. The settlement layer is just any Fabric network that runs the Token Chaincode, which is configured with the identities of the issuer, auditor and CA to be able to validate transactions.
+# Table of Contents
+
+- [Token SDK Sample API](#token-sdk-sample-api)
+- [Table of Contents](#table-of-contents)
+  - [Features](#features)
+  - [Getting started](#getting-started)
+    - [Install dependencies](#install-dependencies)
+    - [Quick start](#quick-start)
+      - [Using the application](#using-the-application)
+      - [Deep dive: what happens when doing a transfer?](#deep-dive-what-happens-when-doing-a-transfer)
+    - [Alternative: manual start](#alternative-manual-start)
+      - [Generate crypto material](#generate-crypto-material)
+      - [Start Fabric and install the chaincode](#start-fabric-and-install-the-chaincode)
+      - [Start the Token network](#start-the-token-network)
+    - [View the blockchain explorer](#view-the-blockchain-explorer)
+  - [Development](#development)
+    - [End to end tests](#end-to-end-tests)
+    - [Code structure](#code-structure)
+    - [Add or change a REST API endpoint](#add-or-change-a-rest-api-endpoint)
+    - [Upgrade the Token SDK and Fabric Smart Client versions](#upgrade-the-token-sdk-and-fabric-smart-client-versions)
+    - [Use another Fabric network](#use-another-fabric-network)
+    - [Add a user / account](#add-a-user--account)
+    - [Run the service directly (instead of with docker-compose)](#run-the-service-directly-instead-of-with-docker-compose)
+
 
 ## Features
 
@@ -41,7 +66,7 @@ Additional features:
 - [X] Use Idemix (privacy preserving) accounts created by a Fabric CA
 - [X] Pre-configured and easy to start for development
 
-### Out of scope for now
+Out of scope for now:
 
 -   HTLC locks (hashed timelock contracts)
 -   Register/enroll new token accounts on a running network
@@ -65,39 +90,34 @@ Prerequisites:
 
 Download the Fabric docker images and binaries. The code only works with Fabric CA 1.5.7+, so even if you cloned the fabric-samples repo before, you may have to re-run it to get the latest versions.
 
-```bash 
+From the fabric-samples directory:
+
+```bash
 curl -sSLO https://raw.githubusercontent.com/hyperledger/fabric/main/scripts/install-fabric.sh && chmod +x install-fabric.sh
 ./install-fabric.sh docker binary
 ```
 
-Make sure that the new binaries are in your path. Change the following line (replace `<your/path/to/> with the actual path`) and add them to your `~/.bashrc` or `~/.zshrc` file (and restart your terminal):
+Make sure that the new binaries are in your path. Change the following line (replace `<your/path/to/>` with the actual path) and add it to your `~/.bashrc` or `~/.zshrc` file. Restart your terminal or `source` the edited file.
 
 ```bash
 export PATH=</your/path/to/>fabric-samples/bin:$PATH
 ```
 
-> Note: if you are *not* running this code from the fabric-samples/token-sdk folder, set the following environment variable:
+Validate that the CA is at 1.5.7 by executing `fabric-ca-client version`.
+
+> Note: you can run this code from anywhere. If you are *not* running it from the fabric-samples/token-sdk folder, also set the following environment variable:
 > ```bash
 > export TEST_NETWORK_HOME=</your/path/to>/fabric-samples/test-network
 > ```
+>
+> See the bottom of this readme for instructions to use another Fabric network than the test network.
 
-Validate that the CA is at 1.5.7 by executing `fabric-ca-client version`.
-
-Install tokengen.
+Install tokengen. Tokengen is a tool to create the configuration file for the token chaincode (once, when deploying the chaincode). It generates the public parameters that the network participants will use to generate their proofs, and it specifies the public identities of the issuer, auditor and CA for signature validation.
 
 ```bash
 go install github.com/hyperledger-labs/fabric-token-sdk/cmd/tokengen@v0.3.0
 ```
 
-Now go to the root folder of your token-sdk sample, for instance in the fabric-samples dir:
-
-```bash
-cd fabric-samples/token-sdk
-```
-
-> You can have the token-sdk folder anywhere. The commands keep working, as long as you have the TEST_NETWORK_HOME variable set.
-> See the bottom of this readme for instructions to use another Fabric network.
-
 ### Quick start
 
 The quickest way to get going is to run:
@@ -116,13 +136,17 @@ When you're done and want to delete everything:
 
 #### Using the application
 
-The services are running on the following ports:
+The services are accessible on the following ports:
 
-- 8080 API documentation
-- 9000 auditor
-- 9100 issuer
-- 9200 owner 1 (alice and bob)
-- 9300 owner 2 (carlos and dan)
+| port | service                  |
+|------|--------------------------|
+| 8080 | API documentation (web)  |
+| 9000 | auditor                  |
+| 9100 | issuer                   |
+| 9200 | owner 1 (alice and bob)  |
+| 9300 | owner 2 (carlos and dan) |
+
+Besides that, the nodes communicate with each other via 9001, 9101, 9201 and 9301 respectively.
 
 Now let's issue and transfer some tokens! View the API documentation and try some actions at [http://localhost:8080](http://localhost:8080). Or, directly from the commandline:
 
@@ -142,10 +166,36 @@ curl -X POST http://localhost:9200/api/v1/owner/accounts/alice/transfer -H 'Cont
 }'
 
 curl -X GET http://localhost:9200/api/v1/owner/accounts/bob/transactions
+curl -X GET http://localhost:9200/api/v1/owner/accounts/alice/transactions
 ```
 
+Notice that the transaction overview uses the UTXO model (like bitcoin). The issuer created a new EURX token of 1000 and assigned its ownership to alice. When alice transfered 100 EURX to bob, she used the token of 1000 as **input** for her transaction. As **output**, she creates two new tokens:
+
+1. one for 100 EURX with bob as the owner
+2. one with _herself_ as the owner for the remaining 990 EURX.
+
+This way, each transaction can have multiple inputs and multiple outputs. Their sum should always be the same, and every new transfer has to be based on previously created outputs.
+
+#### Deep dive: what happens when doing a transfer?
+
+It may look simple from the outside, but there's a lot going on to securely and privately transfer tokens. Let's take an example of alice (on the Owner 1 node) transfering 100 TOK to dan (on the Owner 2 node).
+
+1. **Create Transaction**: Alice requests an anonymous key from dan that will own the tokens. She then creates the transaction, with commitments that can be verified by anyone, but _only_ be opened (read) by dan and the auditor. The commitments contain the value, sender and recipient of each of the in- and output tokens.
+2. **Get Endorsements**: Alice (or more precisely the TransferView in the Owner 1 node) now submits the transaction to the auditor, who validates and stores it. The auditor _may_ enforce any specific business logic that is needed for this token in this ecosystem (for instance a transaction or holding limit).
+
+   Alice then submits the transaction (which is now also signed by the auditor) to the Token Chaincode which is running on the Fabric peers. The chaincode verifies that all the proofs are valid and all the necessary signatures are there. Note that the peer and token chaincode cannot see what is transferred between who thanks to the zero knowledge proofs.
+3. **Commit Transaction**: Alice submits the endorsed Fabric transaction to the ordering service. Alice (Owner 1), dan (Owner 2) and the Auditor nodes have been listening for Fabric events involving this transaction. When receiving the 'commit' event, they change the status of the stored transaction to 'Confirmed'. The transaction is now final; dan owns the 100 TOK.
+
+The names of the Views below correspond to the code in `owner/service/transfer.go`, `owner/service/accept.go` and `auditor/service/audit.go`.
+
+![transfer](transfer.png)
+
+[plantuml source](http://www.plantuml.com/plantuml/uml/TLD1JoCz3BtdLrZb0X98yEaxhTGLRA5xu50EQ0-hNZA92r6dpcpY3Eg_tsHcYDmoUvb9hFUUxHVxFh8Ed0wjOiSjmclG57SOWCj16tQUbCf_7s3nq3g32z0HjEeopHdNQM9OR3ueK-wsjALFWLyEFmOezt2n-l6qNZ_ESVuhd0TZiEELZk-LrRXvraEoZdqOMELO2JhPob18xFW8YxLkWZFmWXZYWEhA2IuU_ry_hfxEOPjWCNmYVRb8i5ekOHLGCqflOBbK6cw-bpQ_0N-wTtTx2w-Rvosn1wi9Cd3gLnLUhnc5CJKcs-Q-o3OkomRyap1o_cSR71A3isFjIiefgQCQL_bcB5kJf-F1fxYbIqzum-w0DodY5UpnAF5lI1WA8sAcCkoAuR-VNy3umy7n0OcZ2iWf47IfQPqf2jSJN5ayAOhxwa_45Ws3eouniDyZHTb0XTQQHv0qFDS-qA_19nx-NV1-5tDszqQQKy0hwLryrm6bmBzCyXsIRF1wIpq6jpkEoldBFk2MNf2iepSfENanuD12mDXvYWZdJkG9-eaCMS27Y4EMF3zJjJfPyTJvvbXwKyydarxEbTlhrjcC6AFLyzEYncpZ8jHyiYQHzNHRnflWEkhzVdeYywuT6M-nZ7ojPCwaAPE5ox6oAnZNJs9dZ5iDBoD1rRgwgwNRr6JOdEISbr-tl0PEPST9a7fvFAOHRLflze8e76g2rzReo43uCG4jVicUhPsOvqimD3r4SaZdoERvr9kpcr2IqrsL1Bfn4gsJbSCqHoWGTOzaqw7z2m00)
+
 ### Alternative: manual start
 
+To get a better view or have more control on the different layers of the network, you can also start the services manually. If you want to do that, first bring down everything with `./scripts/down.sh`.
+
 #### Generate crypto material
 
 In this step, we create all the identities which are used by the Token network. We use a normal Fabric CA for this. Technically, only the Owner identities (the wallets that will hold the tokens) need some form of hierarchy; they use Idemix credentials which must be issued by a single, known issuer (see [Fabric documentation](https://hyperledger-fabric.readthedocs.io/en/latest/idemix.html) for more info about idemix). To keep things simple, we use the same CA for the other identities too. The Token SDK expects the folders for the identities to be in Fabric's 'msp' structure.
@@ -158,6 +208,7 @@ The following crypto will be generated:
  - Owner identities (idemix credentials)
 
 ```bash
+mkdir -p keys/ca
 docker-compose -f compose-ca.yaml up -d
 ./scripts/enroll-users.sh
 ```
@@ -174,7 +225,7 @@ The Issuer and Auditor identities are used by the Token Chaincode to validate to
 tokengen gen dlog --base 300 --exponent 5 --issuers keys/issuer/iss/msp --idemix keys/owner1/wallet/alice --auditors keys/auditor/aud/msp --output tokenchaincode
 ```
 
-> You only have to do this once. But if for any reason you want to re-generate the material: `rm -rf keys; rm tokenchaincode/zkatdlog_pp.json` and execute the steps above again.
+> You only have to do this once. But if for any reason you want to re-generate the material: `rm -rf keys; rm tokenchaincode/zkatdlog_pp.json` and execute the steps above again. If any owner has existing tokens, they will now be invalid because the old proofs can not be verified with the new parameters.
 
 #### Start Fabric and install the chaincode
 
@@ -183,28 +234,49 @@ For simplicity, in this sample all nodes use the credentials of User1 from Org1M
 Start a Fabric sample network and deploy the Token Chaincode as a service:
 
 ```bash
-bash "$TEST_NETWORK_HOME/network.sh" up createChannel
-INIT_REQUIRED="--init-required" "$TEST_NETWORK_HOME/network.sh" deployCCAAS -ccn tokenchaincode -ccp $(pwd)/tokenchaincode -cci "init" -verbose -ccs 1
+../test-network/network.sh up createChannel
+INIT_REQUIRED="--init-required" ../test-network/network.sh deployCCAAS -ccn tokenchaincode -ccp $(pwd)/tokenchaincode -cci "init" -verbose -ccs 1
 
-mkdir -p keys/fabric
-cp -r "$TEST_NETWORK_HOME/organizations" keys/fabric/
+mkdir -p keys/fabric && cp -r ../test-network/organizations keys/fabric/
 ```
 
 > To fully remove the whole network:
 > ```bash
 > docker stop peer0org1_tokenchaincode_ccaas peer0org2_tokenchaincode_ccaas
-> bash "$TEST_NETWORK_HOME/network.sh" down
+> ../test-network/network.sh" down
 > rm -rf keys/fabric
 > ```
 
 #### Start the Token network
 
+> On the bottom of this document you'll find instructions to run the nodes as golang binaries natively instead of with docker compose.
+
 ```bash
+mkdir -p data/auditor data/issuer data/owner1 data/owner2
 docker-compose up -d
 ```
 
 Visit [http://localhost:8080](http://localhost:8080) to view the API documentation and execute some transactions.
 
+### View the blockchain explorer
+
+As a bonus, this sample contains configuration to connect the [blockchain explorer](https://github.com/hyperledger-labs/blockchain-explorer/) with the fabric-samples network. It allows you to inspect the transactions which are committed to the ledger. It shows more of what the Token SDK does under the covers.
+
+Start it as follows:
+
+```bash
+cd explorer
+docker-compose up -d
+```
+
+And visit it in the browser on [localhost:8081](http://localhost:8081).
+
+To tear it down, do this (the -v is important; it removes the volumes that contain the identities and blocks):
+
+```bash
+docker-compose down -v
+```
+
 ## Development
 
 ### End to end tests

token-sdk/explorer/.env

@@ -0,0 +1,4 @@
+PORT=8081
+EXPLORER_CONFIG_FILE_PATH=./config.json
+EXPLORER_PROFILE_DIR_PATH=./connection-profile
+FABRIC_CRYPTO_PATH=../../test-network/organizations

token-sdk/explorer/config.json

@@ -0,0 +1,9 @@
+{
+	"network-configs": {
+		"test-network": {
+			"name": "Test Network",
+			"profile": "./connection-profile/test-network.json"
+		}
+	},
+	"license": "Apache-2.0"
+}

token-sdk/explorer/connection-profile/test-network.json

@@ -0,0 +1,48 @@
+{
+	"name": "test-network",
+	"version": "1.0.0",
+	"client": {
+		"tlsEnable": true,
+		"adminCredential": {
+			"id": "exploreradmin",
+			"password": "exploreradminpw"
+		},
+		"enableAuthentication": true,
+		"organization": "Org1MSP",
+		"connection": {
+			"timeout": {
+				"peer": {
+					"endorser": "300"
+				},
+				"orderer": "300"
+			}
+		}
+	},
+	"channels": {
+		"mychannel": {
+			"peers": {
+				"peer0.org1.example.com": {}
+			}
+		}
+	},
+	"organizations": {
+		"Org1MSP": {
+			"mspid": "Org1MSP",
+			"adminPrivateKey": {
+				"path": "/tmp/crypto/peerOrganizations/org1.example.com/users/User1@org1.example.com/msp/keystore/priv_sk"
+			},
+			"peers": ["peer0.org1.example.com"],
+			"signedCert": {
+				"path": "/tmp/crypto/peerOrganizations/org1.example.com/users/User1@org1.example.com/msp/signcerts/User1@org1.example.com-cert.pem"
+			}
+		}
+	},
+	"peers": {
+		"peer0.org1.example.com": {
+			"tlsCACerts": {
+				"path": "/tmp/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt"
+			},
+			"url": "grpcs://peer0.org1.example.com:7051"
+		}
+	}
+}

token-sdk/explorer/docker-compose.yaml

@@ -0,0 +1,63 @@
+
+# SPDX-License-Identifier: Apache-2.0
+
+# see https://github.com/hyperledger-labs/blockchain-explorer
+
+version: '2.1'
+
+volumes:
+  pgdata:
+  walletstore:
+
+networks:
+  test:
+    name: fabric_test
+    external: true
+
+services:
+
+  explorerdb.mynetwork.com:
+    image: ghcr.io/hyperledger-labs/explorer-db:latest
+    container_name: explorerdb.mynetwork.com
+    hostname: explorerdb.mynetwork.com
+    environment:
+      - DATABASE_DATABASE=fabricexplorer
+      - DATABASE_USERNAME=hppoc
+      - DATABASE_PASSWORD=password
+    healthcheck:
+      test: "pg_isready -h localhost -p 5432 -q -U postgres"
+      interval: 30s
+      timeout: 10s
+      retries: 5
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    networks:
+      - test
+
+  explorer.mynetwork.com:
+    image: ghcr.io/hyperledger-labs/explorer:latest
+    container_name: explorer.mynetwork.com
+    hostname: explorer.mynetwork.com
+    environment:
+      - DATABASE_HOST=explorerdb.mynetwork.com
+      - DATABASE_DATABASE=fabricexplorer
+      - DATABASE_USERNAME=hppoc
+      - DATABASE_PASSWD=password
+      - LOG_LEVEL_APP=info
+      - LOG_LEVEL_DB=info
+      - LOG_LEVEL_CONSOLE=debug
+      - LOG_CONSOLE_STDOUT=true
+      - DISCOVERY_AS_LOCALHOST=false
+      - PORT=${PORT:-8080}
+    volumes:
+      - ${EXPLORER_CONFIG_FILE_PATH}:/opt/explorer/app/platform/fabric/config.json
+      - ${EXPLORER_PROFILE_DIR_PATH}:/opt/explorer/app/platform/fabric/connection-profile
+      - ${FABRIC_CRYPTO_PATH}:/tmp/crypto
+      - walletstore:/opt/explorer/wallet
+    ports:
+      - ${PORT:-8080}:${PORT:-8080}
+    depends_on:
+      explorerdb.mynetwork.com:
+        condition: service_healthy
+    networks:
+      - test

token-sdk/go.work.sum

@@ -1,5 +1,7 @@
+github.com/bits-and-blooms/bitset v1.2.1 h1:M+/hrU9xlMp7t4TyTDQW97d3tRPVuKFC6zBEK16QnXY=
 github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=
 github.com/dgraph-io/badger v1.6.2 h1:mNw0qs90GVgGGWylh0umH5iag1j6n/PeJtNvL6KY/x8=
+github.com/go-kit/log v0.2.1 h1:MRVx0/zhvdseW+Gza6N9rVzU/IVzaeE1SFI4raAhmBU=
 github.com/grpc-ecosystem/grpc-gateway v1.16.0 h1:gmcG1KaJ57LophUzW0Hy8NmPhnMZb4M0+kPpLofRdBo=
 github.com/russross/blackfriday v1.5.2 h1:HyvC0ARfnZBqnXwABFeSZHpKvJHJJfPz81GNueLj0oo=
 github.com/ugorji/go/codec v1.2.9 h1:rmenucSohSTiyL09Y+l2OCk+FrMxGMzho2+tjr5ticU=

token-sdk/owner/service/transfer.go

@@ -66,7 +66,7 @@ type TransferView struct {
 }
 
 func (v *TransferView) Call(context view.Context) (interface{}, error) {
-	// As a first step operation, the sender tries it's own node or contacts the recipient's
+	// As a first step operation, the sender tries its own node or contacts the recipients
 	// FSC node to ask for the identity to use to assign ownership of the freshly created token.
 	var recipient view.Identity
 	var err error