Setting up a Blockchain Business Network With Hyperledger Fabric & Composer Running in Multiple Physical Machine

The Architecture

Hyperledger Fabric has a set of components each with its own roles and functionalities. Each of these will be running on it’s own Docker instances and are configured to work together. These Docker instances even when running on multiple physical machines, can still communicate with each other. That’s essentially the crux of how a blockchain runs on several physical machines.

Terms

Certificate Authority

The certificate authority is responsible for handling all the access control logic, issuing the identities and permission for the users in the Hyperledger blockchain network.

Orderer

In order to keep the entire network in a synchronized state, the orderer is used. Whenever a new transaction is to be committed, the orderer is the one informing all the peers about the transaction. A network can have multiple orderers, also it’s advised in order to maintain less faults.

Peers

Only peers are allowed to commit transactions in the business network. Also each peer has its own copy of the entire world state. It’s connected with CouchDB instances which acts as the database. An organisation can have multiple peers and one or more anchor peer will be used to communicate with other organisations.

In this example, we’ll be having 1 CA, 1 Orderer, and 3 Peers. With 2 Peers running in the first machine, and the other in a second machine.

Prerequisites

Install Docker
Install Composer
Install Fabric Images (1.0.4)
Install Fabric Tools (Use this)

Setting Up

Since all the components are running inside individual Docker instances, we can easily configure them to run separately.

In this example, I’ll be taking the fabric-dev-serverand modify it to run on multiple machines. This is mainly focused towards easier understanding and to use it like a boilerplate.

Here since we’re using only the hlfv1 we’re removing the hlfv11 folder and bringing all the contents of hlfv1 outside.

The Folder Structure

This is how your codebase should look,

Here if you notice, there are three new files docker-compose-peer2.yml, startFabric-Peer2.sh and stopFabric.sh. As the name implies these are related to the third peer which we will be running on a separate machine.

Peer Configurations

The default dev server comes with configuration for running one Peer. So, the certificates are created correspondingly. Since we have three peers in total, we need to generate certificates for all the three.

If you’ve installed the fabric tools with the above mentioned script, you’ll have configtxgen and cryptogen in your system. We’ll be using this to generate the certificates for our peers, and CA.

In cypto-config.yaml change the count under PeerOrgs → Template → Count to number of peers you want. In our case it is 3. Thus the contents of crypto-config.yaml looks like below.

OrdererOrgs:

- Name: Orderer

  Domain: example.com

  Specs:

    - Hostname: orderer

PeerOrgs:

- Name: Org1

  Domain: org1.example.com

  Template:

    Count: 3

  Users:

    Count: 0

And the configtx.yaml looks like,

---

Profiles:



    ComposerOrdererGenesis:

        Orderer:

            <<: *OrdererDefaults

            Organizations:

                - *OrdererOrg

        Consortiums:

            ComposerConsortium:

                Organizations:

                    - *Org1

    ComposerChannel:

        Consortium: ComposerConsortium

        Application:

            <<: *ApplicationDefaults

            Organizations:

                - *Org1



Organizations:



    - &OrdererOrg

        Name: OrdererOrg

        ID: OrdererMSP

        MSPDir: crypto-config/ordererOrganizations/example.com/msp

        AdminPrincipal: Role.MEMBER



    - &Org1

        Name: Org1

        ID: Org1MSP

        MSPDir: crypto-config/peerOrganizations/org1.example.com/msp

        AdminPrincipal: Role.MEMBER

        AnchorPeers:

            - Host: peer0.org1.example.com

              Port: 7051



Orderer: &OrdererDefaults

    OrdererType: solo



    Addresses:

        - orderer.example.com:7050

    BatchTimeout: 2s

    BatchSize:

        MaxMessageCount: 10

        AbsoluteMaxBytes: 98 MB

        PreferredMaxBytes: 512 KB

        

    Kafka:

        Brokers:

            - 127.0.0.1:9092



    Organizations:



Application: &ApplicationDefaults

    Organizations:

Once you made the modifications, run the following inside the composer folder in order to create the certificates for all the peers.

cd "$(dirname "$0")"

cryptogen generate --config=./crypto-config.yaml

export FABRIC_CFG_PATH=$PWD

configtxgen -profile ComposerOrdererGenesis -outputBlock ./composer-genesis.block

configtxgen -profile ComposerChannel -outputCreateChannelTx ./composer-channel.tx -channelID composerchannel

This creates all the certificates and the key under crypto-config folder. We will be using this to configure our Docker image’s environment variables.

Configuring Docker Services

In the docker-compose.yml we have added the following services.

ca.org1.example.com
orderer.example.com
peer0.org1.example.com
couchdb
peer1.org1.example.com
couchdb1

Here the peer0.org1.example.com uses couchdb as the world state database and peer1.org1.example.com uses couchdb1 as world state database.

In order to configure the Certificate Authority, we’ll be using the certificates that have been generated newly. In the command section of ca.org1.example.com make sure to use the proper private key file. It will be located under composer/crypto-config/peerOrganizations/

org1.example.com/ca/ Once you’ve updated it you’re good to go.

version: '2'

services:

ca.org1.example.com:

image: hyperledger/fabric-ca:$ARCH-1.0.4

environment:

  - FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server

  - FABRIC_CA_SERVER_CA_NAME=ca.org1.example.com

#      - FABRIC_CA_SERVER_CA_CERTFILE=/etc/hyperledger/fabric-ca-server-config/org1.example.com-cert.pem

#      - FABRIC_CA_SERVER_CA_KEYFILE=/etc/hyperledger/fabric-ca-server-config/a22daf356b2aab5792ea53e35f66fccef1d7f1aa2b3a2b92dbfbf96a448ea26a_sk



ports:

  - "7054:7054"

command: sh -c 'fabric-ca-server start --ca.certfile /etc/hyperledger/fabric-ca-server-config/ca.org1.example.com-cert.pem --ca.keyfile /etc/hyperledger/fabric-ca-server-config/d4e3285d47260c640643feda80dd0f4ae76378f0047f1cfba3efca6555600b43_sk -b admin:adminpw -d'

volumes:

  - ./crypto-config/peerOrganizations/org1.example.com/ca/:/etc/hyperledger/fabric-ca-server-config

container_name: ca.org1.example.com



orderer.example.com:

container_name: orderer.example.com

image: hyperledger/fabric-orderer:$ARCH-1.0.4

environment:

  - ORDERER_GENERAL_LOGLEVEL=debug

  - ORDERER_GENERAL_LISTENADDRESS=0.0.0.0

  - ORDERER_GENERAL_GENESISMETHOD=file

  - ORDERER_GENERAL_GENESISFILE=/etc/hyperledger/configtx/composer-genesis.block

  - ORDERER_GENERAL_LOCALMSPID=OrdererMSP

  - ORDERER_GENERAL_LOCALMSPDIR=/etc/hyperledger/msp/orderer/msp

working_dir: /opt/gopath/src/github.com/hyperledger/fabric

command: orderer

ports:

  - 7050:7050

volumes:

    - ./:/etc/hyperledger/configtx

    - ./crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/msp:/etc/hyperledger/msp/orderer/msp



peer0.org1.example.com:

container_name: peer0.org1.example.com

image: hyperledger/fabric-peer:$ARCH-1.0.4

environment:

  - CORE_LOGGING_PEER=debug

  - CORE_CHAINCODE_LOGGING_LEVEL=DEBUG

  - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock

  - CORE_PEER_ID=peer0.org1.example.com

  - CORE_PEER_ADDRESS=peer0.org1.example.com:7051

  - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=composer_default

  - CORE_PEER_LOCALMSPID=Org1MSP

  - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/peer/msp

  - CORE_LEDGER_STATE_STATEDATABASE=CouchDB

  - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb:5984

working_dir: /opt/gopath/src/github.com/hyperledger/fabric

command: peer node start --peer-defaultchain=false

ports:

  - 7051:7051

  - 7053:7053

volumes:

    - /var/run/:/host/var/run/

    - ./:/etc/hyperledger/configtx

    - ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/peer/msp

    - ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users

depends_on:

  - orderer.example.com

  - couchdb


couchdb:

container_name: couchdb

image: hyperledger/fabric-couchdb:$ARCH-1.0.4

ports:

  - 5984:5984

environment:

  DB_URL: http://localhost:5984/member_db


peer1.org1.example.com:

container_name: peer1.org1.example.com

image: hyperledger/fabric-peer:$ARCH-1.0.4

environment:

  - CORE_LOGGING_PEER=debug

  - CORE_CHAINCODE_LOGGING_LEVEL=DEBUG

  - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock

  - CORE_PEER_ID=peer1.org1.example.com

  - CORE_PEER_ADDRESS=peer1.org1.example.com:7051

  - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=composer_default

  - CORE_PEER_LOCALMSPID=Org1MSP

  - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/peer/msp

  - CORE_LEDGER_STATE_STATEDATABASE=CouchDB

  - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb1:5984

working_dir: /opt/gopath/src/github.com/hyperledger/fabric

command: peer node start --peer-defaultchain=false

ports:

  - 8051:7051

  - 8053:7053

volumes:

    - /var/run/:/host/var/run/

    - ./:/etc/hyperledger/configtx

    - ./crypto-config/peerOrganizations/org1.example.com/peers/peer1.org1.example.com/msp:/etc/hyperledger/peer/msp

    - ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users

depends_on:

  - orderer.example.com

  - couchdb1



couchdb1:

container_name: couchdb1

image: hyperledger/fabric-couchdb:$ARCH-1.0.4

ports:

  - 6984:5984

environment:

  DB_URL: http://localhost:6984/member_db

Similarly we’ll create a second docker compose file for Peer 2.

version: '2'

services:

  peer2.org1.example.com:

    container_name: peer2.org1.example.com

    image: hyperledger/fabric-peer:$ARCH-1.0.4

    environment:

      - CORE_LOGGING_PEER=debug

      - CORE_CHAINCODE_LOGGING_LEVEL=DEBUG

      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock

      - CORE_PEER_ID=peer2.org1.example.com

      - CORE_PEER_ADDRESS=peer2.org1.example.com:7051

      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=composer_default

      - CORE_PEER_LOCALMSPID=Org1MSP

      - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/peer/msp

      - CORE_LEDGER_STATE_STATEDATABASE=CouchDB

      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb2:5984

    working_dir: /opt/gopath/src/github.com/hyperledger/fabric

    command: peer node start --peer-defaultchain=false

    ports:

      - 9051:7051

      - 9053:7053

    volumes:

        - /var/run/:/host/var/run/

        - ./:/etc/hyperledger/configtx

        - ./crypto-config/peerOrganizations/org1.example.com/peers/peer2.org1.example.com/msp:/etc/hyperledger/peer/msp

        - ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users

    depends_on:

      - couchdb2



  couchdb2:

    container_name: couchdb2

    image: hyperledger/fabric-couchdb:$ARCH-1.0.4

    ports:

      - 7984:5984

    environment:

      DB_URL: http://localhost:7984/member_db

Configuring the scripts

Now we’re ready to run the Fabric network. We’ll be using the fabric-dev-server’s scripts as the base script with a few modifications as mentioned below.

In the startFabric.sh file, by default the configuration for the first peer (peer0.org1.example.com) to join the channel is provided. Now we need to join the second peer (peer1.org1.example.com) to the same channel.

In order to do that we need to fetch the channel block in the second peer and then use that .block file to join the peer. The following code is used to join the peers to the channel in the network.

# Create the channel

docker exec peer0.org1.example.com peer channel create -o orderer.example.com:7050 -c composerchannel -f /etc/hyperledger/configtx/composer-channel.tx


# Join peer0.org1.example.com to the channel.

docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/[email protected]/msp" peer0.org1.example.com peer channel join -b composerchannel.block


# Create the channel

docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/[email protected]/msp" peer1.org1.example.com peer channel fetch config -o orderer.example.com:7050 -c composerchannel

# docker exec peer1.org1.example.com peer channel create -o orderer.example.com:7050 -c composerchannel -f /etc/hyperledger/configtx/composer-channel.tx


# Join peer1.org1.example.com to the channel.

docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/[email protected]/msp" peer1.org1.example.com peer channel join -b composerchannel_config.block

In the second machine, we’ll be using the same concept of fetching the channel block and joining the channel with the block file.

docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/[email protected]/msp" peer2.org1.example.com peer channel fetch config -o orderer.example.com:7050 -c composerchannel

docker exec -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/[email protected]/msp" peer2.org1.example.com peer channel join -b composerchannel_config.block

Peer Admin Card Creation

Now we need to update the script to create the peer admin card for the network. With the default script createPeerAdminCard.sh we’ll add update the connection profile configuration to add the new peers.

cat << EOF > /tmp/.connection.json

{

    "name": "hlfv1",

    "type": "hlfv1",

    "orderers": \[

       { "url" : "grpc://localhost:7050" }

    \],

    "ca": { 

        "url": "http://localhost:7054", 

        "name": "ca.org1.example.com"

    },

    "peers": \[

        {

            "requestURL": "grpc://localhost:7051",

            "eventURL": "grpc://localhost:7053"

        }, {

            "requestURL": "grpc://localhost:8051",

            "eventURL": "grpc://localhost:8053"

        }, {

            "requestURL": "grpc://<Peer2-IP>:9051",

            "eventURL": "grpc://<Peer2-IP>:9053"

        }

    \],

    "channel": "composerchannel",

    "mspID": "Org1MSP",

    "timeout": 300

}

Also you need to update the PRIVATE_KEY path with the corresponding path to the private key of map keystore.

PRIVATE_KEY="${DIR}"/composer/crypto-config/peerOrganizations/org1.example.com/users/[email protected]/msp/keystore/<PRIVATE_KEY_NAME>

This is how your script should look,

#!/bin/bash



# Exit on first error

set -e

# Grab the current directory

DIR="$( cd "$( dirname "${BASH_SOURCE\[0\]}" )" && pwd )"



echo

# check that the composer command exists at a version >v0.14

if hash composer 2>/dev/null; then

    composer --version | awk -F. '{if ($2<15) exit 1}'

    if \[ $? -eq 1 \]; then

        echo 'Sorry, Use createConnectionProfile for versions before v0.15.0' 

        exit 1

    else

        echo Using composer-cli at $(composer --version)

    fi

else

    echo 'Need to have composer-cli installed at v0.15 or greater'

    exit 1

fi

# need to get the certificate 



cat << EOF > /tmp/.connection.json

{

    "name": "hlfv1",

    "type": "hlfv1",

    "orderers": \[

       { "url" : "grpc://localhost:7050" }

    \],

    "ca": { 

        "url": "http://localhost:7054", 

        "name": "ca.org1.example.com"

    },

    "peers": \[

        {

            "requestURL": "grpc://localhost:7051",

            "eventURL": "grpc://localhost:7053"

        }, {

            "requestURL": "grpc://localhost:8051",

            "eventURL": "grpc://localhost:8053"

        }, {

            "requestURL": "grpc://192.168.31.12:9051",

            "eventURL": "grpc://192.168.31.12:9053"

        }

    \],

    "channel": "composerchannel",

    "mspID": "Org1MSP",

    "timeout": 300

}

EOF



PRIVATE_KEY="${DIR}"/composer/crypto-config/peerOrganizations/org1.example.com/users/[email protected]/msp/keystore/7fe58742a0b6d1102c74293808f1736dea010d3451f9e1a804c0b86ecf90baa0_sk

CERT="${DIR}"/composer/crypto-config/peerOrganizations/org1.example.com/users/[email protected]/msp/signcerts/[email protected]



if composer card list -n PeerAdmin@hlfv1 > /dev/null; then

    composer card delete -n PeerAdmin@hlfv1

fi

composer card create -p /tmp/.connection.json -u PeerAdmin -c "${CERT}" -k "${PRIVATE_KEY}" -r PeerAdmin -r ChannelAdmin --file /tmp/[email protected]

composer card import --file /tmp/[email protected] 



rm -rf /tmp/.connection.json



echo "Hyperledger Composer PeerAdmin card has been imported"

composer card list

That’s it, everything is set and ready to go. Now lets run the following scripts to start the Hyperledger Fabric network on the first machine.

./teardownFabric.sh && ./startFabric.sh && ./createPeerAdminCard.sh

Run the following command on the second machine,

./startFabric-Peer2.sh

If everything is working properly, you’ll notice clean logs and zero errors. Try running docker ps to see all of the services that are running on each of the machines.

In the first machine there are 6 services running.

And in the second machine there are two services running.

When you notice the logs, the peers syncs whenever you create asset of submit transactions. The peer0 will be getting the first transaction submit request and then the orderer distributes to other peers maintaining all the peers in sync.

Checkout this repo for the entire source code of the server : https://github.com/varun-raj/fabric-dev-servers-multipeer

Awesome illustrations by @vectorpocket

Up next

What I will be doing in 2018