velero backup failed: "getting backup volume info: request failed"

[OPSTime]

What steps did you take and what happened:
$ velero schedule get

NAME                               STATUS    CREATED                         SCHEDULE    BACKUP TTL   LAST BACKUP   SELECTOR   PAUSED
velero-cce-udp-prod-daily-backup   Enabled   2024-11-14 19:53:41 +0800 CST   0 6 * * *   2160h0m0s    n/a           <none>     false

$ velero backup create --from-schedule velero-cce-udp-prod-daily-backup
$ velero backup get

NAME                                              STATUS   ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
velero-cce-udp-prod-daily-backup-20241114112709   Failed   0        0          2024-11-14 19:27:09 +0800 CST   89d       cce-udp-prod       <none>

$ velero backup describe velero-cce-udp-prod-daily-backup-20241114112709

Name:         velero-cce-udp-prod-daily-backup-20241114112709
Namespace:    velero
Labels:       app.kubernetes.io/instance=velero-cce-udp-prod
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=velero
              helm.sh/chart=velero-8.0.0
              myenv=cce-udp-prod
              velero.io/schedule-name=velero-cce-udp-prod-daily-backup
              velero.io/storage-location=cce-udp-prod
Annotations:  meta.helm.sh/release-name=velero-cce-udp-prod
              meta.helm.sh/release-namespace=velero
              myenv=cce-udp-prod
              velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.28.7-r0-28.0.39.8
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=28+

Phase:  Failed (run `velero backup logs velero-cce-udp-prod-daily-backup-20241114112709` for more information)


Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  cce-udp-prod

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  2160h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-11-14 19:27:09 +0800 CST
Completed:  <n/a>

Expiration:  2025-02-12 19:27:09 +0800 CST

Total items to be backed up:  27361
Items backed up:              27361

Backup Volumes:
  <error getting backup volume info: request failed: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>000001932AA4A99E44D3F8DCF50773F8</RequestId><HostId>RDRKuvcarRwkrDulPanUlpz9ZQ/KDMfm0c6Wqp2pPPmO0fFkESUs9zQOoUk6AxjK</HostId></Error>>

HooksAttempted:  0
HooksFailed:     0

Environment:

• Velero version (use velero version): 1.15.0

• Velero features (use velero client config get features):

• Kubernetes version (use kubectl version):

• Server Version: version.Info{Major:"1", Minor:"28+", GitVersion:"v1.28.7-r0-28.0.39.8", GitCommit:"b78e81a22974f7b57baa90d97758476072dc4d78", GitTreeState:"clean", BuildDate:"2024-06-18T15:27:10Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}
  WARNING: version difference between client (1.23) and server (1.28) exceeds the supported minor version skew of +/-1

• Kubernetes installer & version: version 1.28

Uploading bundle-2024-11-14-20-14-17.tar.gz…

[blackpiglet]

Could you re-upload the debug bundle?
I couldn't open the download link. When I click it, this issue's page is reloaded.
error getting backup volume info: request failed
I think this error is not the reason why the backup failed. It means the velero backup describe cannot read the needed VolumeInfo to display the volume information of this backup, and the missing VolumeInfo should be caused by the backup didn't reach to upload the VolumeInfo stage.

[OPSTime]

bundle-2024-11-14-20-14-17.tar.gz

[blackpiglet]

time="2024-11-14T19:55:55+08:00" level=error msg="Error uploading log file" backup=velero-cce-udp-prod-daily-backup-20241114115526 bucket=devops-obs error="rpc error: code = Unknown desc = error putting object k8s_config_backup/cce-udp-prod/backups/velero-cce-udp-prod-daily-backup-20241114115526/velero-cce-udp-prod-daily-backup-20241114115526-logs.gz: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 000001932A87AF1D45CD338B0639FF24, HostID: MOCKQjIxQTYxQzZDMDAwMDAxMzQwMzFCRTgwMDUyOTNBQUFBQUFBQWJiYmJiYmJi, api error InvalidArgument: The x-amz-content-sha256 you specified did not match what we received." error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:280" error.function="main.(*ObjectStore).PutObject" logSource="pkg/persistence/object_store.go:256" prefix=k8s_config_backup/cce-udp-prod
time="2024-11-14T19:55:55+08:00" level=info msg="Initial backup processing complete, moving to Finalizing" backup=velero/velero-cce-udp-prod-daily-backup-20241114115526 logSource="pkg/controller/backup_controller.go:745"
time="2024-11-14T19:55:55+08:00" level=info msg="plugin process exited" backup=velero/velero-cce-udp-prod-daily-backup-20241114115526 cmd=/plugins/velero-plugin-for-aws id=79 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-11-14T19:55:55+08:00" level=info msg="plugin process exited" backup=velero/velero-cce-udp-prod-daily-backup-20241114115526 cmd=/velero id=71 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/velero
time="2024-11-14T19:55:55+08:00" level=error msg="backup failed" backuprequest=velero/velero-cce-udp-prod-daily-backup-20241114115526 controller=backup error="rpc error: code = Unknown desc = error putting object k8s_config_backup/cce-udp-prod/backups/velero-cce-udp-prod-daily-backup-20241114115526/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 000001932A87AF4445CE3180AA43CFD3, HostID: MOCKQjIxQTYxQzZDMDAwMDAxMzQwMzFCRTgwMDUyOTNBQUFBQUFBQWJiYmJiYmJi, api error InvalidArgument: The x-amz-content-sha256 you specified did not match what we received." logSource="pkg/controller/backup_controller.go:291"

Two things found from the bundle:

• There are two Velero server pods found in the velero namespace: velero and velero-cce-udp-prod. This should not happen.
• The object uploading failed with The x-amz-content-sha256 you specified did not match what we received. Please check whether this solution works for you. Backup failed with error XAmzContentSHA256Mismatch #7696 (comment)

[OPSTime]

There is only one Velero pod in the velero namespace: velero-cce-udp-prod.
After replacing velero-plugin-for-aws with version 1.5.1, the backup was successful.
However, "Backup Volumes" still shows "Access Denied."

[blackpiglet]

Do you mean the velero backup describe CLI cannot display the volume information with Access Denied failure?

Could you give the detail error information?

[OPSTime]

Backup Volumes:
<error getting backup volume info: request failed: AccessDeniedAccess Denied000001932AA4A99E44D3F8DCF50773F8RDRKuvcarRwkrDulPanUlpz9ZQ/KDMfm0c6Wqp2pPPmO0fFkESUs9zQOoUk6AxjK>