Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic on leveldb range query fabric peer #5048

Open
sam-at-luther opened this issue Nov 1, 2024 · 0 comments
Open

Panic on leveldb range query fabric peer #5048

sam-at-luther opened this issue Nov 1, 2024 · 0 comments
Labels

Comments

@sam-at-luther
Copy link

sam-at-luther commented Nov 1, 2024

Description

Version: 2.5.5, with external chaincode, levelDB, using custom Go chaincode.

We're seeing this fairly frequently in a busy network.

It seems to only happen when a transaction has exceeded the execution timeout duration, but is in the midst of a range query on the fabric peer.

From what I can tell, the corresponding lines in levelDB is:

func (i *indexedIterator) dataErr() bool {
	if err := i.data.Error(); err != nil {
		if i.errf != nil {
			i.errf(err)
		}
		if i.strict || !errors.IsCorrupted(err) {
			i.err = err
			return true
		}
	}
	return false
}

where

if err := i.data.Error(); err != nil {

is triggering the panic.

I suspect the peer has a read lock on level db, but in the middle of ranging, the peer hits an execution timeout, and the interruption logic it’s not correctly waiting for the iterator to close before releasing the lock. In other words, Close() is called on the iterator before it’s finished the range query.

Possibly relates to: #5001

Stack traces:

2024-10-02 10:05:27.738 UTC c30993 ERRO [endorser] simulateProposal -> failed to invoke chaincode com_luthersystems_chaincode_substrate01, error: timeout expired while executing transaction
github.com/hyperledger/fabric/core/chaincode.(*Handler).Execute
	/core/chaincode/handler.go:1234
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).execute
	/core/chaincode/chaincode_support.go:278
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Invoke
	/core/chaincode/chaincode_support.go:208
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
	/core/chaincode/chaincode_support.go:161
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
	/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
	/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
	/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
	/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
	/core/endorser/endorser.go:350
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
	/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
	/core/handlers/auth/filter/filter.go:32
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler.func1
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:107
github.com/hyperledger/fabric/internal/peer/node.serve.unaryGrpcLimiter.func13
	/internal/peer/node/grpc_limiters.go:56
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
	/common/grpclogging/server.go:92
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/peer/node.serve.UnaryServerInterceptor.func11
	/common/grpcmetrics/interceptor.go:31
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:109
google.golang.org/grpc.(*Server).processUnaryRPC
	/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650
error sending
failed to execute transaction c078f5529e0ec5a22649101fc02db922c36e5c6b20d71e109a3aa639afaf30de
github.com/hyperledger/fabric/core/chaincode.processChaincodeExecutionResult
	/core/chaincode/chaincode_support.go:167
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
	/core/chaincode/chaincode_support.go:162
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
	/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
	/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
	/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
	/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
	/core/endorser/endorser.go:350
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
	/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
	/core/handlers/auth/filter/filter.go:32
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler.func1
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:107
github.com/hyperledger/fabric/internal/peer/node.serve.unaryGrpcLimiter.func13
	/internal/peer/node/grpc_limiters.go:56
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
	/common/grpclogging/server.go:92
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/peer/node.serve.UnaryServerInterceptor.func11
	/common/grpcmetrics/interceptor.go:31
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:109
google.golang.org/grpc.(*Server).processUnaryRPC
	/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650 channel=luther txID=c078f552
2024-10-02 10:05:27.738 UTC c30994 WARN [endorser] ProcessProposal -> Failed to invoke chaincode channel=luther chaincode=com_luthersystems_chaincode_substrate01 error="error in simulation: failed to execute transaction c078f5529e0ec5a22649101fc02db922c36e5c6b20d71e109a3aa639afaf30de: error sending: timeout expired while executing transaction"
2024-10-02 10:05:27.738 UTC c30995 INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.request_deadline=2024-10-02T10:05:29.979Z grpc.peer_address=10.0.6.175:1306 grpc.peer_subject="CN=oracle@private.acreplatforms.net,ST=London,C=UK" grpc.code=OK grpc.call_duration=8.75856771s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x12b5c9f]

goroutine 1046998611 [running]:
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).dataErr(0xc00da8f380)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:59 +0x1f
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).Next(0xc00da8f380)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:150 +0xa5
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).First(0xc00da8f380)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:89 +0xe5
github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).First(0xc00a6b7a00)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:72 +0xb6
github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).Next(0xc00c86dbc0?)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:157 +0x265
github.com/syndtr/goleveldb/leveldb.(*dbIter).Next(0xc0056ac480)
	/vendor/github.com/syndtr/goleveldb/leveldb/db_iter.go:249 +0x7e
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/stateleveldb.(*kvScanner).Next(0xc00c86db60)
	/core/ledger/kvledger/txmgmt/statedb/stateleveldb/stateleveldb.go:313 +0x42
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*resultsItr).Next(0xc0031456d0)
	/core/ledger/kvledger/txmgmt/txmgr/query_executor.go:439 +0x28
github.com/hyperledger/fabric/core/chaincode.(*QueryResponseGenerator).BuildQueryResponse(0xc00561e068, 0xc005b01380?, {0x1d24820, 0xc0031456d0}, {0xc005b01380, 0x24}, 0x0, 0x186a0)
	/core/chaincode/query_response_generator.go:31 +0xc4
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleGetStateByRange(0xc005630360, 0xc005726d80, 0xc008cee320)
	/core/chaincode/handler.go:766 +0x40d
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleTransaction(0xc005630360, 0xc005726d80, 0xc0061e0950)
	/core/chaincode/handler.go:253 +0x373
created by github.com/hyperledger/fabric/core/chaincode.(*Handler).handleMessageReadyState in goroutine 187
	/core/chaincode/handler.go:195 +0x3e9

2024-10-08 13:41:47.064 UTC 2a47c7 ERRO [endorser] simulateProposal -> failed to invoke chaincode com_luthersystems_chaincode_substrate01, error: timeout expired while executing transaction
github.com/hyperledger/fabric/core/chaincode.(*Handler).Execute
	/core/chaincode/handler.go:1234
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).execute
	/core/chaincode/chaincode_support.go:278
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Invoke
	/core/chaincode/chaincode_support.go:208
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
	/core/chaincode/chaincode_support.go:161
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
	/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
	/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
	/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
	/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
	/core/endorser/endorser.go:350
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
	/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
	/core/handlers/auth/filter/filter.go:32
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler.func1
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:107
github.com/hyperledger/fabric/internal/peer/node.serve.unaryGrpcLimiter.func13
	/internal/peer/node/grpc_limiters.go:56
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
	/common/grpclogging/server.go:92
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/peer/node.serve.UnaryServerInterceptor.func11
	/common/grpcmetrics/interceptor.go:31
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:109
google.golang.org/grpc.(*Server).processUnaryRPC
	/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650
error sending
failed to execute transaction f8df1d185389530537d58407709edfa8342dd8e8feed19cdea23f8e4df2256a7
github.com/hyperledger/fabric/core/chaincode.processChaincodeExecutionResult
	/core/chaincode/chaincode_support.go:167
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
	/core/chaincode/chaincode_support.go:162
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
	/core/endorser/support.go:126
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
	/core/endorser/endorser.go:120
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
	/core/endorser/endorser.go:187
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError
	/core/endorser/endorser.go:409
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
	/core/endorser/endorser.go:350
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
	/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
	/core/handlers/auth/filter/filter.go:32
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler.func1
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:107
github.com/hyperledger/fabric/internal/peer/node.serve.unaryGrpcLimiter.func13
	/internal/peer/node/grpc_limiters.go:56
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
	/common/grpclogging/server.go:92
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/peer/node.serve.UnaryServerInterceptor.func11
	/common/grpcmetrics/interceptor.go:31
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9.1.1
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/hyperledger/fabric/internal/pkg/comm.NewGRPCServerFromListener.ChainUnaryServer.func9
	/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/hyperledger/fabric-protos-go/peer._Endorser_ProcessProposal_Handler
	/vendor/github.com/hyperledger/fabric-protos-go/peer/peer.pb.go:109
google.golang.org/grpc.(*Server).processUnaryRPC
	/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650 channel=luther txID=f8df1d18
2024-10-08 13:41:47.064 UTC 2a47c8 WARN [endorser] ProcessProposal -> Failed to invoke chaincode channel=luther chaincode=com_luthersystems_chaincode_substrate01 error="error in simulation: failed to execute transaction f8df1d185389530537d58407709edfa8342dd8e8feed19cdea23f8e4df2256a7: error sending: timeout expired while executing transaction"
2024-10-08 13:41:47.064 UTC 2a47c9 INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.request_deadline=2024-10-08T13:41:49.637Z grpc.peer_address=10.0.5.138:39365 grpc.peer_subject="CN=oracle@private.acreplatforms.net,ST=London,C=UK" grpc.code=OK grpc.call_duration=8.426256026s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x12b5c9f]

goroutine 256351988 [running]:
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).dataErr(0xc00c18e8a0)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:59 +0x1f
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).Next(0xc00c18e8a0)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:150 +0xa5
github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).First(0xc00c18e8a0)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:89 +0xe5
github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).First(0xc003783480)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:72 +0xb6
github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).Next(0xc00a0b23c0?)
	/vendor/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:157 +0x265
github.com/syndtr/goleveldb/leveldb.(*dbIter).Next(0xc007a7b170)
	/vendor/github.com/syndtr/goleveldb/leveldb/db_iter.go:249 +0x7e
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/stateleveldb.(*kvScanner).Next(0xc00a0b2360)
	/core/ledger/kvledger/txmgmt/statedb/stateleveldb/stateleveldb.go:313 +0x42
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*resultsItr).Next(0xc00f5012c0)
	/core/ledger/kvledger/txmgmt/txmgr/query_executor.go:439 +0x28
github.com/hyperledger/fabric/core/chaincode.(*QueryResponseGenerator).BuildQueryResponse(0xc0116f0778, 0xc0079cb4a0?, {0x1d24820, 0xc00f5012c0}, {0xc0079cb4a0, 0x24}, 0x0, 0x186a0)
	/core/chaincode/query_response_generator.go:31 +0xc4
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleGetStateByRange(0xc000248a20, 0xc003783080, 0xc003eb3b80)
	/core/chaincode/handler.go:766 +0x40d
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleTransaction(0xc000248a20, 0xc003783080, 0xc007aadf50)
	/core/chaincode/handler.go:253 +0x373
created by github.com/hyperledger/fabric/core/chaincode.(*Handler).handleMessageReadyState in goroutine 71935
	/core/chaincode/handler.go:195 +0x3e9

Steps to reproduce

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant