error sending: chaincode stream terminated

[spsingh559]

I am doing performance testing for node js-based Fabric Gateway and HLF Network which consist of 2 org and one anchor peer each, sending 2500 tx with the speed of 20-30 tx per second.

We are facing the issue of the chain code container getting terminated when the load is very high.

The current VM infra where HLF Peers and Fabric Gateway adapter is hosted

16 GB RAM, 4 Core CPU

Log of smart contract container getting exit

`goroutine 180834 [running]:
runtime.throw({0xa28fb2?, 0xc0004c5568?})
/usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc0004c5510 sp=0xc0004c54e0 pc=0x435231
runtime.mapassign_faststr(0x974740, 0xc000214030, {0xc0004fe0a8, 0x3})
/usr/local/go/src/runtime/map_faststr.go:295 +0x38b fp=0xc0004c5578 sp=0xc0004c5510 pc=0x4144ab
reflect.mapassign_faststr(0x974740, 0xc0004b6270?, {0xc0004fe0a8?, 0xa2c260?}, 0x19?)
/usr/local/go/src/runtime/map.go:1357 +0x28 fp=0xc0004c55b0 sp=0xc0004c5578 pc=0x45e688
reflect.Value.SetMapIndex({0x974740?, 0xeb34a0?, 0xffe?}, {0x9504c0, 0xc000344340, 0x98}, {0x9446c0, 0xc0004b6270, 0x197})
/usr/local/go/src/reflect/value.go:2232 +0x225 fp=0xc0004c5638 sp=0xc0004c55b0 pc=0x494445
encoding/json.(*decodeState).object(0xc000508000, {0x93b800?, 0xeb34a0?, 0x0?})
/usr/local/go/src/encoding/json/decode.go:816 +0x13d7 fp=0xc0004c5888 sp=0xc0004c5638 pc=0x4ee457
encoding/json.(*decodeState).value(0xc000508000, {0x93b800?, 0xeb34a0?, 0x90?})
/usr/local/go/src/encoding/json/decode.go:373 +0x45 fp=0xc0004c58f8 sp=0xc0004c5888 pc=0x4ec0c5
encoding/json.(*decodeState).unmarshal(0xc000508000, {0x93b800?, 0xeb34a0?})
/usr/local/go/src/encoding/json/decode.go:180 +0x1de fp=0xc0004c5970 sp=0xc0004c58f8 pc=0x4eb9de
encoding/json.Unmarshal({0xc0002d7000, 0xe7c, 0x1000}, {0x93b800, 0xeb34a0})
/usr/local/go/src/encoding/json/decode.go:107 +0x125 fp=0xc0004c59a8 sp=0xc0004c5970 pc=0x4eb4e5
main.(*Customs).Invoke(0xc0004ac758?, {0xb52448?, 0xc0002ac630})
/chaincode/input/src/main.go:90 +0x59d fp=0xc0004c5e38 sp=0xc0004c59a8 pc=0x8a2add
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000302d00)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2 fp=0xc0004c5f40 sp=0xc0004c5e38 pc=0x857722
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction-fm(0x84300c?)
:1 +0x2c fp=0xc0004c5f60 sp=0xc0004c5f40 pc=0x8629ac
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000302d00, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32 fp=0xc0004c5fb0 sp=0xc0004c5f60 pc=0x856f72
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady.func2()
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x32 fp=0xc0004c5fe0 sp=0xc0004c5fb0 pc=0x85d812
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004c5fe8 sp=0xc0004c5fe0 pc=0x4641c1
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 1 [select]:
github.com/hyperledger/fabric-chaincode-go/shim.chatWithPeer({0xc000028017, 0x51}, {0x7f6c391ee0a8?, 0xc00005c170}, {0xb4b4e8?, 0xee6c40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/shim.go:125 +0x40d
github.com/hyperledger/fabric-chaincode-go/shim.chaincodeAsClientChat({0xc000028017, 0x51}, {0x7f6c391ee078?, 0xc00005c170}, {0xb4b4e8, 0xee6c40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/shim.go:89 +0xd2
github.com/hyperledger/fabric-chaincode-go/shim.Start({0xb4b4e8, 0xee6c40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/shim.go:75 +0x116
main.main()
/chaincode/input/src/main.go:22 +0x69

goroutine 19 [select, 4123 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0000ecf80)
/chaincode/input/src/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x73
created by google.golang.org/grpc.newCCBalancerWrapper
/chaincode/input/src/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:76 +0x22a

goroutine 6 [select, 4123 minutes]:
google.golang.org/grpc.newClientStreamWithParams.func4()
/chaincode/input/src/vendor/google.golang.org/grpc/stream.go:366 +0x92
created by google.golang.org/grpc.newClientStreamWithParams
/chaincode/input/src/vendor/google.golang.org/grpc/stream.go:365 +0xe2f

goroutine 180795 [runnable]:
internal/poll.runtime_Semacquire(0xc0004a1150?)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc0000b2060, 0xd6?)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).writeLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:239
internal/poll.(*FD).Write(0xc0000b2060, {0xc0000245f0, 0x50, 0x50})
/usr/local/go/src/internal/poll/fd_unix.go:370 +0x72
os.(*File).write(...)
/usr/local/go/src/os/file_posix.go:48
os.(*File).Write(0xc0000b0008, {0xc0000245f0?, 0x50, 0x2000?})
/usr/local/go/src/os/file.go:176 +0x65
fmt.Fprintln({0xb49aa0, 0xc0000b0008}, {0xc00015aeb0, 0x2, 0x2})
/usr/local/go/src/fmt/print.go:265 +0x75
fmt.Println(...)
/usr/local/go/src/fmt/print.go:274
main.getOnboardingWithOrgType(0x9446c0?, {0xc0002c7f10, 0xa})
/chaincode/input/src/common.go:764 +0x1cc
main.loadPublicKey(0xb49aa0?, {0xc0002c7f10?, 0xc000579808?}, {0x2?, 0x2?})
/chaincode/input/src/signature_verification.go:88 +0x36
main.verifySignature(0xc00015bd88, {0xc0002c7f10, 0xa}, {0xc0002c7833, 0x6dc}, {0xc0002c7632, 0x200}, {0xa1cbc3, 0x5}, {0xc0000270e0, ...}, ...)
/chaincode/input/src/signature_verification.go:136 +0x212
main.(*Customs).Invoke(0xc00018bf58?, {0xb52448?, 0xc000306580})
/chaincode/input/src/main.go:117 +0xcf6
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000367600)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000367600, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 34 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000494000)
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1671 +0x165
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:384 +0x1b5c

goroutine 35 [runnable]:
syscall.Syscall(0x0, 0x3, 0xc000136000, 0x81bf)
/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5
syscall.read(0xc000210b80?, {0xc000136000?, 0xc000136000?, 0xc0000eb558?})
/usr/local/go/src/syscall/zsyscall_linux_amd64.go:696 +0x4d
syscall.Read(...)
/usr/local/go/src/syscall/syscall_unix.go:188
internal/poll.ignoringEINTRIO(...)
/usr/local/go/src/internal/poll/fd_unix.go:794
internal/poll.(*FD).Read(0xc000210b80?, {0xc000136000?, 0x81bf?, 0x81bf?})
/usr/local/go/src/internal/poll/fd_unix.go:163 +0x285
net.(*netFD).Read(0xc000210b80, {0xc000136000?, 0xc0000b5280?, 0xc000136005?})
/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0000b1338, {0xc000136000?, 0x81bf?, 0xc0000b5280?})
/usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0004b6198, {0xc000136000?, 0x0?, 0xc0004aa1a0?})
/usr/local/go/src/crypto/tls/conn.go:785 +0x3d
bytes.(*Buffer).ReadFrom(0xc00030d778, {0xb487a0, 0xc0004b6198})
/usr/local/go/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc00030d500, {0xb49900?, 0xc0000b1338}, 0x81bf?)
/usr/local/go/src/crypto/tls/conn.go:807 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc00030d500, 0x0)
/usr/local/go/src/crypto/tls/conn.go:614 +0x116
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc00030d500, {0xc00045c000, 0x8000, 0x0?})
/usr/local/go/src/crypto/tls/conn.go:1285 +0x16f
bufio.(*Reader).Read(0xc0004531a0, {0xc000488040, 0x9, 0x18?})
/usr/local/go/src/bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0xb48640, 0xc0004531a0}, {0xc000488040, 0x9, 0x9}, 0x9)
/usr/local/go/src/io/io.go:331 +0x9a
io.ReadFull(...)
/usr/local/go/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc000488040?, 0x9?, 0xe10b6005265a?}, {0xb48640?, 0xc0004531a0?})
/chaincode/input/src/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000488000)
/chaincode/input/src/vendor/golang.org/x/net/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000494000, 0x0?)
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1597 +0x1b7
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:394 +0x1bdb

goroutine 5 [runnable]:
runtime.Gosched(...)
/usr/local/go/src/runtime/proc.go:317
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0000701e0)
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:570 +0x1f6
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:448 +0x6a
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:446 +0x2233

goroutine 180746 [chan receive]:
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).sendReceive(0xc00034f668?, 0x634d30?, 0xc00026da20?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:142 +0xe5
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).callPeerWithChaincodeMsg(0xc00007a1e0, 0x3174793863446457?, {0xc000497b20, 0xe}, {0xc000030880, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:230 +0x1eb
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleGetState(0x4d308e?, {0xa21a37, 0xa}, {0xc0000c6380, 0x1e}, {0xc000497b20, 0xe}, {0xc000030880, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:239 +0x206
github.com/hyperledger/fabric-chaincode-go/shim.(*ChaincodeStub).GetPrivateData(0xb49aa0?, {0xa21a37?, 0xc00034fad0?}, {0xc0000c6380?, 0x2?})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/stub.go:231 +0x6f
main.fetchData({{0xb52448, 0xc0004f8580}, {0xc0002047c0, 0x2, 0x2}, {0xc0004364a0, 0xb}, {0x0, 0x0}, 0x18a3b945aff}, ...)
/chaincode/input/src/common.go:112 +0x21e
main.(*Order).getEndToEndOrderDetail(0x9caf80?, 0xc0003539a8, {0xc000351a58, 0x1, 0x0?}, {0xa21a37, 0xa})
/chaincode/input/src/order_procedures.go:3332 +0x1e8
main.(*Order).submitOrder(0xc000587600?, 0xc0003539a8, {0xc00021af00?, 0x18, 0x18}, {0xc000587650?, 0x4d308e?})
/chaincode/input/src/order_procedures.go:2573 +0x19b1
main.(*Customs).submitOrderController(0xc76c51a3c24b8b70?, {{0xb52448, 0xc0004f8580}, {0xc0002047c0, 0x2, 0x2}, {0xc0004364a0, 0xb}, {0x0, 0x0}, ...}, ...)
/chaincode/input/src/main.go:607 +0x9e
main.(*Customs).Invoke(0xc00018b758?, {0xb52448?, 0xc0004f8580})
/chaincode/input/src/main.go:135 +0x103d
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000501900)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000501900, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180719 [runnable]:
internal/poll.runtime_Semacquire(0x21?)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc0000b2060, 0x0?)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).writeLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:239
internal/poll.(*FD).Write(0xc0000b2060, {0xc0004f4000, 0x17, 0x4000})
/usr/local/go/src/internal/poll/fd_unix.go:370 +0x72
os.(*File).write(...)
/usr/local/go/src/os/file_posix.go:48
os.(*File).Write(0xc0000b0008, {0xc0004f4000?, 0x17, 0x0?})
/usr/local/go/src/os/file.go:176 +0x65
fmt.Fprintln({0xb49aa0, 0xc0000b0008}, {0xc0004ddc88, 0x2, 0x2})
/usr/local/go/src/fmt/print.go:265 +0x75
fmt.Println(...)
/usr/local/go/src/fmt/print.go:274
main.(*Order).submitOrder(0xc000583600?, 0xc0004df9a8, {0xc00021a000?, 0x18, 0x18}, {0xc000583650?, 0x4d308e?})
/chaincode/input/src/order_procedures.go:2591 +0x1c93
main.(*Customs).submitOrderController(0xc76c51a3c24b8b70?, {{0xb52448, 0xc0004f8160}, {0xc0002c0800, 0x2, 0x2}, {0xc00025c420, 0xb}, {0x0, 0x0}, ...}, ...)
/chaincode/input/src/main.go:607 +0x9e
main.(*Customs).Invoke(0xc00018f758?, {0xb52448?, 0xc0004f8160})
/chaincode/input/src/main.go:135 +0x103d
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000302380)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0x0?, 0xc000302380, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180727 [chan receive]:
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).sendReceive(0xc0001abb10?, 0x634d30?, 0xc00026da20?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:142 +0xe5
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).callPeerWithChaincodeMsg(0xc00007a1e0, 0x0?, {0xc0001321a0, 0xe}, {0xc0003d2180, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:230 +0x1eb
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleGetState(0x4d308e?, {0xa22d7e, 0xc}, {0xc000030c00, 0x32}, {0xc0001321a0, 0xe}, {0xc0003d2180, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:239 +0x206
github.com/hyperledger/fabric-chaincode-go/shim.(*ChaincodeStub).GetPrivateData(0xb49aa0?, {0xa22d7e?, 0xc0001abf78?}, {0xc000030c00?, 0x2?})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/stub.go:231 +0x6f
main.fetchData({{0xb52448, 0xc0005186e0}, {0xc000302d80, 0x4, 0x4}, {0xc000032d80, 0xb}, {0x0, 0x0}, 0x18a3b957c14}, ...)
/chaincode/input/src/common.go:112 +0x21e
main.(*Order).stampDataToInvoice(_, _, {{0xa22d7e, 0xc}, {0xc000030c00, 0x32}, {0xa20718, 0x9}, {0xc00043303c, 0xa}, ...}, ...)
/chaincode/input/src/order_procedures.go:3268 +0x3bd
main.(*Order).submitOrder(0xc000587600?, 0xc0001af9a8, {0xc000142000?, 0x18, 0x18}, {0xc000587650?, 0x4d308e?})
/chaincode/input/src/order_procedures.go:2720 +0x334d
main.(*Customs).submitOrderController(0xc76c51a3c24b8b70?, {{0xb52448, 0xc0005186e0}, {0xc000302d80, 0x4, 0x4}, {0xc000032d80, 0xb}, {0x0, 0x0}, ...}, ...)
/chaincode/input/src/main.go:607 +0x9e
main.(*Customs).Invoke(0xc000157758?, {0xb52448?, 0xc0005186e0})
/chaincode/input/src/main.go:135 +0x103d
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000302980)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0xc00004ac00?, 0xc00008f680?, 0xc000302980, 0xc0005040b4?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180710 [runnable]:
internal/poll.runtime_Semacquire(0x0?)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc0000b2060, 0x0?)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).writeLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:239
internal/poll.(*FD).Write(0xc0000b2060, {0xc000424150, 0x17, 0x18})
/usr/local/go/src/internal/poll/fd_unix.go:370 +0x72
os.(*File).write(...)
/usr/local/go/src/os/file_posix.go:48
os.(*File).Write(0xc0000b0008, {0xc000424150?, 0x17, 0x0?})
/usr/local/go/src/os/file.go:176 +0x65
fmt.Fprintln({0xb49aa0, 0xc0000b0008}, {0xc0002d1c88, 0x2, 0x2})
/usr/local/go/src/fmt/print.go:265 +0x75
fmt.Println(...)
/usr/local/go/src/fmt/print.go:274
main.(*Order).submitOrder(0xc000583600?, 0xc0002d39a8, {0xc00031b980?, 0x18, 0x18}, {0xc000583650?, 0x4d308e?})
/chaincode/input/src/order_procedures.go:2591 +0x1c93
main.(*Customs).submitOrderController(0xc76c51a3c24b8b70?, {{0xb52448, 0xc0003066e0}, {0xc0002c0300, 0x2, 0x2}, {0xc0001ffeb0, 0xb}, {0x0, 0x0}, ...}, ...)
/chaincode/input/src/main.go:607 +0x9e
main.(*Customs).Invoke(0xc0004ae758?, {0xb52448?, 0xc0003066e0})
/chaincode/input/src/main.go:135 +0x103d
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000394680)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000394680, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180797 [chan receive]:
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).sendReceive(0x44646e756f626e69?, 0x69746172616c6365?, 0x2c5d5b3a22736e6f?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:142 +0xe5
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleInvokeChaincode(0xc00007a1e0, {0xc0004a61e0, 0x24}, {0xc00017c390, 0x2, 0x2}, {0xc00025d0c0, 0xe}, {0xc0003d2740, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:599 +0x3ab
github.com/hyperledger/fabric-chaincode-go/shim.(*ChaincodeStub).InvokeChaincode(0xc000518840?, {0xa28fc7?, 0x40e5c5?}, {0xc00017c390?, 0x2?, 0x2?}, {0xa23ed7?, 0x4141414141?})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/stub.go:156 +0xfe
main.fetchDataFromOtherContract(0xc0004cbd88, {0xc0004caed0, 0x2, 0x62?}, {0xa28fc7, 0x15}, {0xa23ed7, 0xe})
/chaincode/input/src/common.go:401 +0xf3
main.getOnboardingWithOrgType(0x9446c0?, {0xc0004f1f10, 0xa})
/chaincode/input/src/common.go:763 +0x138
main.loadPublicKey(0xb49aa0?, {0xc0004f1f10?, 0xc0004cb808?}, {0x2?, 0x2?})
/chaincode/input/src/signature_verification.go:88 +0x36
main.verifySignature(0xc000157d88, {0xc0004f1f10, 0xa}, {0xc0004f1833, 0x6dc}, {0xc0004f1632, 0x200}, {0xa1cbc3, 0x5}, {0xc0003cab10, ...}, ...)
/chaincode/input/src/signature_verification.go:136 +0x212
main.(*Customs).Invoke(0xc00018f758?, {0xb52448?, 0xc000518840})
/chaincode/input/src/main.go:117 +0xcf6
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000207900)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0x0?, 0xc000207900, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180808 [runnable]:
internal/poll.runtime_Semacquire(0xc0004a1220?)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc0000b2060, 0xd6?)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).writeLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:239
internal/poll.(*FD).Write(0xc0000b2060, {0xc0000246e0, 0x50, 0x50})
/usr/local/go/src/internal/poll/fd_unix.go:370 +0x72
os.(*File).write(...)
/usr/local/go/src/os/file_posix.go:48
os.(*File).Write(0xc0000b0008, {0xc0000246e0?, 0x50, 0x2000?})
/usr/local/go/src/os/file.go:176 +0x65
fmt.Fprintln({0xb49aa0, 0xc0000b0008}, {0xc000586eb0, 0x2, 0x2})
/usr/local/go/src/fmt/print.go:265 +0x75
fmt.Println(...)
/usr/local/go/src/fmt/print.go:274
main.getOnboardingWithOrgType(0x9446c0?, {0xc0002d6f10, 0xa})
/chaincode/input/src/common.go:764 +0x1cc
main.loadPublicKey(0xb49aa0?, {0xc0002d6f10?, 0xc0004c9808?}, {0x2?, 0x2?})
/chaincode/input/src/signature_verification.go:88 +0x36
main.verifySignature(0xc000587d88, {0xc0002d6f10, 0xa}, {0xc0002d6833, 0x6dc}, {0xc0002d6632, 0x200}, {0xa1cbc3, 0x5}, {0xc0003ca828, ...}, ...)
/chaincode/input/src/signature_verification.go:136 +0x212
main.(*Customs).Invoke(0xc00018b758?, {0xb52448?, 0xc000518790})
/chaincode/input/src/main.go:117 +0xcf6
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000394780)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000394780, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180670 [runnable]:
internal/poll.runtime_Semacquire(0x0?)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc0000b2060, 0x10?)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).writeLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:239
internal/poll.(*FD).Write(0xc0000b2060, {0xc000354380, 0x6e, 0x70})
/usr/local/go/src/internal/poll/fd_unix.go:370 +0x72
os.(*File).write(...)
/usr/local/go/src/os/file_posix.go:48
os.(*File).Write(0xc0000b0008, {0xc000354380?, 0x6e, 0xc0003e9d68?})
/usr/local/go/src/os/file.go:176 +0x65
fmt.Fprintln({0xb49aa0, 0xc0000b0008}, {0xc0003e9dc0, 0x1, 0x1})
/usr/local/go/src/fmt/print.go:265 +0x75
fmt.Println(...)
/usr/local/go/src/fmt/print.go:274
main.insertData(0xc0003ed9a8, {0xc00002a2d0, 0x25}, {0xa25542, 0x10}, {0xc00001c340?, 0xc0003e9f50?, 0xc00042e380?})
/chaincode/input/src/common.go:77 +0x18d
main.genericInsertData(0xa04fe0?, {0xc00002a2d0, 0x25}, {0xa25542, 0x10}, {0xa04fe0?, 0xc00042e380?})
/chaincode/input/src/common.go:363 +0x78
main.persistLogs(, {{0xa22d7e, 0xc}, {0xc0005b2300, 0x32}, {0xa20718, 0x9}, {0xc00052303c, 0xa}, {0xc000523000, ...}, ...}, ...)
/chaincode/input/src/order_procedures.go:2949 +0x297
main.(*Order).stampDataToInvoice(, _, {{0xa22d7e, 0xc}, {0xc0005b2300, 0x32}, {0xa20718, 0x9}, {0xc00052303c, 0xa}, ...}, ...)
/chaincode/input/src/order_procedures.go:3261 +0x326
main.(*Order).submitOrder(0xc000167600?, 0xc0003ed9a8, {0xc00050cf00?, 0x18, 0x18}, {0xc000167650?, 0x4d308e?})
/chaincode/input/src/order_procedures.go:2720 +0x334d
main.(*Customs).submitOrderController(0xc76c51a3c24b8b70?, {{0xb52448, 0xc000306420}, {0xc000302e80, 0x4, 0x4}, {0xc0003a67e0, 0xb}, {0x0, 0x0}, ...}, ...)
/chaincode/input/src/main.go:607 +0x9e
main.(*Customs).Invoke(0xc0004ac758?, {0xb52448?, 0xc000306420})
/chaincode/input/src/main.go:135 +0x103d
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000394000)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000394000, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180805 [runnable]:
reflect.Value.Elem({0x9313c0?, 0xc0002baa78?, 0x16?})
/usr/local/go/src/reflect/value.go:1166 +0x1ba
encoding/json.indirect({0x948440?, 0xc0002baa78?, 0x13?}, 0x0)
/usr/local/go/src/encoding/json/decode.go:470 +0x239
encoding/json.(*decodeState).literalStore(0xc000232ab0, {0xc0002f8e74, 0x5, 0x18c}, {0x948440?, 0xc0002baa78?, 0x9a0720?}, 0x0)
/usr/local/go/src/encoding/json/decode.go:869 +0x1a8
encoding/json.(*decodeState).value(0xc000232ab0, {0x948440?, 0xc0002baa78?, 0x13?})
/usr/local/go/src/encoding/json/decode.go:387 +0x126
encoding/json.(*decodeState).object(0xc000232ab0, {0x9a0720?, 0xc0002baa68?, 0xb?})
/usr/local/go/src/encoding/json/decode.go:774 +0xca5
encoding/json.(*decodeState).value(0xc000232ab0, {0x9a0720?, 0xc0002baa68?, 0x4?})
/usr/local/go/src/encoding/json/decode.go:373 +0x45
encoding/json.(*decodeState).array(0xc000232ab0, {0x9446c0?, 0xc0002fcac8?, 0xb?})
/usr/local/go/src/encoding/json/decode.go:561 +0x626
encoding/json.(*decodeState).value(0xc000232ab0, {0x9446c0?, 0xc0002fcac8?, 0x9446c0?})
/usr/local/go/src/encoding/json/decode.go:363 +0x7e
encoding/json.(*decodeState).object(0xc000232ab0, {0x93b800?, 0xeb34a0?, 0x0?})
/usr/local/go/src/encoding/json/decode.go:774 +0xca5
encoding/json.(*decodeState).value(0xc000232ab0, {0x93b800?, 0xeb34a0?, 0x90?})
/usr/local/go/src/encoding/json/decode.go:373 +0x45
encoding/json.(*decodeState).unmarshal(0xc000232ab0, {0x93b800?, 0xeb34a0?})
/usr/local/go/src/encoding/json/decode.go:180 +0x1de
encoding/json.Unmarshal({0xc0002f8000, 0xe7c, 0x1000}, {0x93b800, 0xeb34a0})
/usr/local/go/src/encoding/json/decode.go:107 +0x125
main.(*Customs).Invoke(0xc00018bf58?, {0xb52448?, 0xc0003064d0})
/chaincode/input/src/main.go:90 +0x59d
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000394400)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000394400, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180803 [runnable]:
internal/poll.runtime_Semacquire(0xc0004a0ee0?)
/usr/local/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc0000b2060, 0xd6?)
/usr/local/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).writeLock(...)
/usr/local/go/src/internal/poll/fd_mutex.go:239
internal/poll.(*FD).Write(0xc0000b2060, {0xc000168000, 0x50, 0x4000})
/usr/local/go/src/internal/poll/fd_unix.go:370 +0x72
os.(*File).write(...)
/usr/local/go/src/os/file_posix.go:48
os.(*File).Write(0xc0000b0008, {0xc000168000?, 0x50, 0x2000?})
/usr/local/go/src/os/file.go:176 +0x65
fmt.Fprintln({0xb49aa0, 0xc0000b0008}, {0xc000582eb0, 0x2, 0x2})
/usr/local/go/src/fmt/print.go:265 +0x75
fmt.Println(...)
/usr/local/go/src/fmt/print.go:274
main.getOnboardingWithOrgType(0x9446c0?, {0xc0002c6f10, 0xa})
/chaincode/input/src/common.go:764 +0x1cc
main.loadPublicKey(0xb49aa0?, {0xc0002c6f10?, 0xc000575808?}, {0x2?, 0x2?})
/chaincode/input/src/signature_verification.go:88 +0x36
main.verifySignature(0xc000583d88, {0xc0002c6f10, 0xa}, {0xc0002c6833, 0x6dc}, {0xc0002c6632, 0x200}, {0xa1cbc3, 0x5}, {0xc0000d6150, ...}, ...)
/chaincode/input/src/signature_verification.go:136 +0x212
main.(*Customs).Invoke(0xc00018b758?, {0xb52448?, 0xc0004f8000})
/chaincode/input/src/main.go:117 +0xcf6
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000367180)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000367180, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180749 [chan receive]:
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).sendReceive(0x0?, 0x634d30?, 0xc00026da20?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:142 +0xe5
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).callPeerWithChaincodeMsg(0xc00007a1e0, 0x646a364379753174?, {0xc000428b20, 0xe}, {0xc000030a40, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:230 +0x1eb
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleGetState(0x4d308e?, {0xa21a37, 0xa}, {0xc0000c6340, 0x1e}, {0xc000428b20, 0xe}, {0xc000030a40, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:239 +0x206
github.com/hyperledger/fabric-chaincode-go/shim.(*ChaincodeStub).GetPrivateData(0xb49aa0?, {0xa21a37?, 0xc00033bad0?}, {0xc0000c6340?, 0x2?})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/stub.go:231 +0x6f
main.fetchData({{0xb52448, 0xc0004f8630}, {0xc000204600, 0x2, 0x2}, {0xc0004281c0, 0xb}, {0x0, 0x0}, 0x18a3b9489c9}, ...)
/chaincode/input/src/common.go:112 +0x21e
main.(*Order).getEndToEndOrderDetail(0x9caf80?, 0xc00033f9a8, {0xc00033da58, 0x1, 0x0?}, {0xa21a37, 0xa})
/chaincode/input/src/order_procedures.go:3332 +0x1e8
main.(*Order).submitOrder(0xc000583600?, 0xc00033f9a8, {0xc00021a000?, 0x18, 0x18}, {0xc000583650?, 0x4d308e?})
/chaincode/input/src/order_procedures.go:2573 +0x19b1
main.(*Customs).submitOrderController(0xc76c51a3c24b8b70?, {{0xb52448, 0xc0004f8630}, {0xc000204600, 0x2, 0x2}, {0xc0004281c0, 0xb}, {0x0, 0x0}, ...}, ...)
/chaincode/input/src/main.go:607 +0x9e
main.(*Customs).Invoke(0xc00018b758?, {0xb52448?, 0xc0004f8630})
/chaincode/input/src/main.go:135 +0x103d
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000501b00)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000501b00, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180851 [chan receive]:
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).sendReceive(0xc0005794d8?, 0x634d30?, 0xc00026da20?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:142 +0xe5
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).callPeerWithChaincodeMsg(0xc00007a1e0, 0x9134b8?, {0xc00014e020, 0xe}, {0xc0000de180, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:230 +0x1eb
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleGetState(0x17?, {0x0, 0x0}, {0xa214d3, 0xa}, {0xc00014e020, 0xe}, {0xc0000de180, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:239 +0x206
github.com/hyperledger/fabric-chaincode-go/shim.(*ChaincodeStub).GetState(0xc0003d9ad0?, {0xa214d3?, 0xc000579948?})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/stub.go:165 +0x3e
main.getOrgTypeByMSP({0xb52448?, 0xc0002ad6b0?}, {0xc000424120, 0x12})
/chaincode/input/src/common.go:159 +0x5e
main.(*Customs).Invoke(0xc000048f58?, {0xb52448?, 0xc0002ad6b0})
/chaincode/input/src/main.go:40 +0x12b
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000302000)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0xc000206080?, 0xc00008f680?, 0xc000302000, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180825 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00007a140, {0xc00000e148, 0x5, 0x5})
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/transport.go:195 +0xa5
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00007a140, {0xc00000e148?, 0xc000026030?, 0xc000057a70?})
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/transport.go:175 +0x147
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00007c2d0, {0xc00000e148?, 0xc000057ae8?, 0x79e427?})
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/transport.go:495 +0x32
io.ReadAtLeast({0xb49300, 0xc00007c2d0}, {0xc00000e148, 0x5, 0x5}, 0x5)
/usr/local/go/src/io/io.go:331 +0x9a
io.ReadFull(...)
/usr/local/go/src/io/io.go:350
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000504240, {0xc00000e148, 0x5, 0x5})
/chaincode/input/src/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0xa5
google.golang.org/grpc.(*parser).recvMsg(0xc00000e138, 0x6400000)
/chaincode/input/src/vendor/google.golang.org/grpc/rpc_util.go:561 +0x47
google.golang.org/grpc.recvAndDecompress(0x9f8e80?, 0xc000504240, {0x0, 0x0}, 0x6400000, 0x0, {0x0, 0x0})
/chaincode/input/src/vendor/google.golang.org/grpc/rpc_util.go:692 +0x66
google.golang.org/grpc.recv(0x81f8e4?, {0x7f6c391ee008, 0xee6c40}, 0xc00069f980?, {0x0?, 0x0?}, {0x9f8e80, 0xc000394480}, 0xc0000d2910?, 0x0, ...)
/chaincode/input/src/vendor/google.golang.org/grpc/rpc_util.go:758 +0x6e
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0005020d0, {0x9f8e80?, 0xc000394480}, 0x4?)
/chaincode/input/src/vendor/google.golang.org/grpc/stream.go:1060 +0x2be
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0004aef30?)
/chaincode/input/src/vendor/google.golang.org/grpc/stream.go:900 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc000504000, 0xc0004aef38, 0xc0004aef08)
/chaincode/input/src/vendor/google.golang.org/grpc/stream.go:735 +0x3be
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000504000, {0x9f8e80?, 0xc000394480?})
/chaincode/input/src/vendor/google.golang.org/grpc/stream.go:899 +0x12e
github.com/hyperledger/fabric-protos-go/peer.(*chaincodeSupportRegisterClient).Recv(0xc00005c170)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-protos-go/peer/chaincode_shim.pb.go:1197 +0x4c
github.com/hyperledger/fabric-chaincode-go/shim.chatWithPeer.func1()
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/shim.go:119 +0x2f
created by github.com/hyperledger/fabric-chaincode-go/shim.chatWithPeer
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/shim.go:143 +0x4a7

goroutine 180751 [chan receive]:
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).sendReceive(0xc000317668?, 0x634d30?, 0xc00026da20?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:142 +0xe5
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).callPeerWithChaincodeMsg(0xc00007a1e0, 0x3174793863446457?, {0xc000428b30, 0xe}, {0xc000030a80, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:230 +0x1eb
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleGetState(0x4d308e?, {0xa21a37, 0xa}, {0xc0000c6300, 0x1e}, {0xc000428b30, 0xe}, {0xc000030a80, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:239 +0x206
github.com/hyperledger/fabric-chaincode-go/shim.(*ChaincodeStub).GetPrivateData(0xb49aa0?, {0xa21a37?, 0xc000317ad0?}, {0xc0000c6300?, 0x2?})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/stub.go:231 +0x6f
main.fetchData({{0xb52448, 0xc0004f86e0}, {0xc0002045c0, 0x2, 0x2}, {0xc0002b58e0, 0xb}, {0x0, 0x0}, 0x18a3b948994}, ...)
/chaincode/input/src/common.go:112 +0x21e
main.(*Order).getEndToEndOrderDetail(0x9caf80?, 0xc00031b9a8, {0xc000319a58, 0x1, 0x0?}, {0xa21a37, 0xa})
/chaincode/input/src/order_procedures.go:3332 +0x1e8
main.(*Order).submitOrder(0xc00015b600?, 0xc00031b9a8, {0xc00021a180?, 0x18, 0x18}, {0xc00015b650?, 0x4d308e?})
/chaincode/input/src/order_procedures.go:2573 +0x19b1
main.(*Customs).submitOrderController(0xc76c51a3c24b8b70?, {{0xb52448, 0xc0004f86e0}, {0xc0002045c0, 0x2, 0x2}, {0xc0002b58e0, 0xb}, {0x0, 0x0}, ...}, ...)
/chaincode/input/src/main.go:607 +0x9e
main.(*Customs).Invoke(0xc00018f758?, {0xb52448?, 0xc0004f86e0})
/chaincode/input/src/main.go:135 +0x103d
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000501d00)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0x0?, 0xc000501d00, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e

goroutine 180817 [chan receive]:
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).sendReceive(0xc0005775b0?, 0x634d30?, 0xc00026da20?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:142 +0xe5
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).callPeerWithChaincodeMsg(0xc00007a1e0, 0xc0000b2060?, {0xc00014e000, 0xe}, {0xc0005b2500, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:230 +0x1eb
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleGetState(0xc000142000?, {0x0, 0x0}, {0xa231be, 0xd}, {0xc00014e000, 0xe}, {0xc0005b2500, 0x40})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:239 +0x206
github.com/hyperledger/fabric-chaincode-go/shim.(*ChaincodeStub).GetState(0xb52448?, {0xa231be?, 0xc000577cc8?})
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/stub.go:165 +0x3e
main.(*Customs).Invoke(0x831d40?, {0xb52448?, 0xc0002ad600})
/chaincode/input/src/main.go:75 +0x44e
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleTransaction(0xc00007a1e0, 0xc000395480)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:209 +0x1a2
github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleStubInteraction(0x0?, 0xc00008f680?, 0xc000395480, 0x0?)
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:159 +0x32
created by github.com/hyperledger/fabric-chaincode-go/shim.(*Handler).handleReady
/chaincode/input/src/vendor/github.com/hyperledger/fabric-chaincode-go/shim/handler.go:643 +0x11e`

[spsingh559]

@denyeart @bestbeforetoday can you please let me know if this is a definite bug in the go-shim package and what is the tentative timeline for the fix?

[bestbeforetoday]

I'll leave it to Dave and the other maintainers of this repo to give an authoritative answer.

[denyeart]

It looks like fabric-chaincode-go called Invoke on your custom chaincode successfully, and then your chaincode caused a panic. Here's the call into your chaincode and the line that failed:

main.(*Customs).Invoke(0xc0004ac758?, {0xb52448?, 0xc0002ac630})
/chaincode/input/src/main.go:90 +0x59d fp=0xc0004c5e38 sp=0xc0004c59a8 pc=0x8a2add

You'll need to troubleshoot what your chaincode is doing at line 90.