Updating a preserved replica does not tier out changes

[alanking]

The following issue was seen with iRODS 4.3.2 and storage tiering 4.3.2.0.

I have a very basic 3-tier group:
tier0_A:unixfilesystem
tier1_A:unixfilesystem
tier2:unixfilesystem

irods::storage_tiering::preserve_replicas is set to true on all 3 tiers. Data is annotated and tiered out as expected.

  rods              0 tier0_A           35 2024-05-15.14:02 & test.txt
        generic    /tmp/irods/tier0_A/home/rods/ingest-test/test.txt
  rods              1 tier1_A           35 2024-05-15.14:03 & test.txt
        generic    /tmp/irods/tier1_A/home/rods/ingest-test/test.txt
  rods              2 tier2           35 2024-05-15.14:05 & test.txt
        generic    /tmp/irods/tier2/home/rods/ingest-test/test.txt

AVUs defined for dataObj /tempZone/home/rods/ingest-test/test.txt:
attribute: irods::access_time
value: 1715781924
units: 
----
attribute: irods::storage_tiering::group
value: tier_group_A
units: 2

The problem is when one of the preserved replicas is updated:

  rods              0 tier0_A           48 2024-05-15.14:07 & test.txt
        generic    /tmp/irods/tier0_A/home/rods/ingest-test/test.txt
  rods              1 tier1_A           35 2024-05-15.14:03 X test.txt
        generic    /tmp/irods/tier1_A/home/rods/ingest-test/test.txt
  rods              2 tier2           35 2024-05-15.14:05 X test.txt
        generic    /tmp/irods/tier2/home/rods/ingest-test/test.txt

The access time is not updated:

AVUs defined for dataObj /tempZone/home/rods/ingest-test/test.txt:
attribute: irods::access_time
value: 1715781924
units: 
----
attribute: irods::storage_tiering::group
value: tier_group_A
units: 2

In order to force a tier-out, you have to trim the other two replicas. Only then does the violating query pick up and start tiering out the changes.

This is similar to #235, but not exactly the same.

[alanking]

I've marked this as a bug because it is not behaving as expected on first glance, but it's possible that this is actually expected. We will need to discuss this more.

[alanking]

One way that this can be done is by updating the "tracked" replica when any of the non-tracked, preserved replicas are modified. This would require a query (or inspection of in-memory information) to determine the tracked replica and make an update based on whether the modified replica is the tracked replica. In this way, the replica in the higher tier will be marked stale and then the new tracked replica would tier out appropriately.