Test failure: TestStorageTieringContinueInxMigration.test_put_object_limit_lt

[alanking]

I have observed two modes of failure for test_plugin_unified_storage_tiering.TestStorageTieringContinueInxMigration.test_put_object_limit_lt as of commit 511cba8 (NOTE: This commit is not necessarily the commit which introduced the problem, just the earliest one with which I tested).

1. The test intermittently fails in about the same amount of time as a passing test (~135 seconds).
2. The test intermittently hangs forever after migrating a few objects out of the 255 objects it was supposed to migrate.

We need to investigate whether this is due to the test or if this is a real issue. We at least need to ensure that the test will fail eventually by removing the infinite loop found here:

irods_capability_storage_tiering/packaging/test_plugin_unified_storage_tiering.py

Lines 162 to 171 in 511cba8

	def wait_for_empty_queue(function):
	done = False
	while done == False:
	out, err, rc = lib.execute_command_permissive(['iqstat'])
	if -1 != out.find('No delayed rules pending'):
	function()
	done = True
	else:
	print(out)
	sleep(1)

[alanking]

For more context, I see hundreds of messages like this in the log:

{
  "log_category": "database",
  "log_level": "error",
  "log_message": "Could not find a delay rule with id [10356].",
  "request_api_name": "",
  "request_api_number": 20013,
  "request_api_version": "d",
  "request_client_user": "rods",
  "request_host": "172.25.0.3",
  "request_proxy_user": "rods",
  "request_release_version": "rods4.3.1",
  "server_host": "2d81d8c90eae",
  "server_pid": 2505,
  "server_timestamp": "2023-12-19T22:54:19.628Z",
  "server_type": "agent",
  "server_zone": "tempZone"
}
{
  "log_category": "api",
  "log_level": "error",
  "log_message": "Could not get delay rule information [rule id=[10356]]",
  "request_api_name": "",
  "request_api_number": 20013,
  "request_api_version": "d",
  "request_client_user": "rods",
  "request_host": "172.25.0.3",
  "request_proxy_user": "rods",
  "request_release_version": "rods4.3.1",
  "server_host": "2d81d8c90eae",
  "server_pid": 2505,
  "server_timestamp": "2023-12-19T22:54:19.628Z",
  "server_type": "agent",
  "server_zone": "tempZone"
}