util: set group:user on extracted tar members

[efahl]

Set the gid:uid on the tar file members as we extract them from the build archive, so they are owned by a real user in both the container and on the host.

This came about from two things: part of documenting setup of a local server, and attempting to write host-based tools to manage build storage.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.59%. Comparing base (5e65dec) to head (a9a26bd).
Report is 120 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1025      +/-   ##
==========================================
+ Coverage   80.75%   90.59%   +9.84%     
==========================================
  Files          15       14       -1     
  Lines         977     1085     +108     
==========================================
+ Hits          789      983     +194     
+ Misses        188      102      -86     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[aparcar]

Did you check that this is not much slower?

[efahl]

I didn't pay close attention, but the resulting builds were just as fast. My ath79 live test using owut was the usual 30 seconds with and without. Shouldn't cost much as it's just setting some bits in the archive member headers.

Live testing strategy:

1. Run all the usual pytests, when they're good:
2. Start up my local server using the new code.
3. Over on one of my routers, run owut download --device <some test target>
4. Check the total time, usually 30s on ath79 with no packages, 180s on x86 with a bunch of packages.
5. Back on the server, look at the asu_worker_1 logs and public/store dir to make sure everything looks good.

I usually do a few owut runs with and without defaults, add or remove packages, that sort of thing.

[aparcar]

Great thanks for checking

[efahl]

Ok, got solid numbers... I added a little timer, so we can measure wall time for a chunk of code, looks like this:

        with timethis("untar"):
            with TarFile(host_tar.name) as tar_file:
                nmembers = 0
                for member in tar_file:
                    nmembers += 1
                    # Fix the owner of the copied files, change to "us".
                    member.uid = getuid()
                    member.gid = getgid()
                    member.mode = 0o755 if member.isdir() else 0o644
                tar_file.extractall(copy[1])
        logging.info(f"Extracted {nmembers} members to {copy[1]}")

So the timings below are for the whole open/set-uids/extract sequence. Did three builds, extracted this from the logs.

00:43:27 Pulling ghcr.io/openwrt/imagebuilder:ath79-generic-master... done
00:43:54 timethis:untar took 0.009s
00:43:54 Extracted 7 members to /app/public/store

00:44:00 Pulling ghcr.io/openwrt/imagebuilder:x86-64-v23.05.5... done
00:44:40 timethis:untar took 0.065s
00:44:40 Extracted 12 members to /app/public/store

00:44:43 Pulling ghcr.io/openwrt/imagebuilder:x86-64-master... done
00:47:30 timethis:untar took 0.247s
00:47:30 Extracted 13 members to /app/public/store

Then looked at the extracted file sizes. It appears that untar time is purely a function of the file sizes, and overall is of no consequence.

$ du -sh public/store/*
7.7M    public/store/3095a0088b42b236a63c89d28a6811f0    <-- ath79 snapshot
53M     public/store/7911d20550a1eb3b1d6943f689b45b56    <-- x86 23.05
198M    public/store/4d4c1cb3b9b28482258e8c4c2784ec45    <-- x86 snapshot

[aparcar]

Thanks for checking. Back in the day I terribly slowed down ASU by relying on pure python tar extraction