[Flake] Scylla Manager integration should register cluster, sync backup tasks and support manual restore procedure [It] using workaround for consistent_cluster_management for ScyllaDB Enterprise [RequiresObjectStorage]

[rzetelskik]

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2165/pull-scylla-operator-master-e2e-gke-parallel/1854175863751315456#1:test-build-log.txt%3A2094

Snippet of what failed.

  [FAILED] %!(EXTRA string=Error: create restore target, units and views: init units: table vbt4csgx.test, which is a part of restored backup, is missing in the restored cluster. Please either exclude it from the restore (--keyspace '*,!vbt4csgx.test'), or first restore its schema (--restore-schema)
  Trace ID: 0wCzIdjDRT2RzElHZxF07A (grep in scylla-manager logs)
  )
  Unexpected error:
      <exec.CodeExitError>: 
      command terminated with exit code 1
      {
          Err: <*errors.errorString | 0xc000acc1b0>{
              s: "command terminated with exit code 1",
          },
          Code: 1,
      }
  occurred
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllamanager_object_storage.go:436 @ 11/06/24 15:34:37.735

[rzetelskik]

For the record it also flaked with the default ScyllaDB version #2061 (comment), so it's not unique to the workaround procedure.

[Michal-Leszczynski]

So basically SM queries GET /column_family/name Scylla endpoint (on a random node) in order to get all table names, but the restored table is not there. If it is a single node cluster, then I don't know why would this API lag behind schema changes. If that's the case, then it could be a bug in Scylla.