Best approach to read zip of gzip compressed files?

[marbemac]

The real world context is I am attempting to read the response from the Amplitude export API (https://www.docs.developers.amplitude.com/analytics/apis/export-api/#response) in a performant way (memory constrained edge environment).

The amplitude API streams a zip of json files (that are themselves gzip compressed). I am attempting to read the response stream, and process the data from one json file in the response zip at a time without writing anything to disk (still streaming - the compressed json files are jsonl which makes it easy to process one json object at a time if I can get that far).

Here's a simplified example zip that reflects the behavior of the amplitude API (zip of two gzip compressed json files).

export.zip

I was hoping to do something like this, but can't quite wrap my head around how to marry what I know about web streams with the zip.js API.

Any pointers would be much appreciated - thank you so much for creating this library!

// @ts-expect-error ignore
import fs from 'node:fs';
// @ts-expect-error ignore
import { Readable } from 'node:stream';

import * as zip from '@zip.js/zip.js';

zip.configure({
  useWebWorkers: false,
});

export const debug = async () => {
  const nodeReadable = fs.createReadStream('./export.zip');
  const fsStream = Readable.toWeb(nodeReadable);
  const zipReader = new zip.ZipReader(fsStream);

  const entries = zipReader.getEntriesGenerator();

  for await (const entry of entries) {
    console.log('process entry', entry.filename);

    const { writable, readable } = new TransformStream();
    readable
      .pipeThrough(new DecompressionStream('gzip'))
      .pipeThrough(new TextDecoderStream('utf-8'))
      .pipeThrough(
        new TransformStream({
          transform(chunk, controller) {
            console.log('text chunk', chunk);
            // console.log(inflate.append(chunk));
            controller.enqueue(chunk);
          },
        }),
      );
    // .pipeThrough(splitStream('\n'))
    // .pipeThrough(parseJSON());

    await entry.getData!({ writable });
  }

  await zipReader.close();
};

[gildas-lormeau]

Are you sure entries are compressed with gzip? It's not mentioned in the documentation (https://www.docs.developers.amplitude.com/analytics/apis/export-api/#considerations).

[marbemac]

I know it's not mentioned but I am 100% sure :).

[marbemac]

However even if they were not, I'm still having trouble figuring out how to combine the stream utils I have with zip.js (for example I have some TransformStream classes to process a stream of jsonl lines, then pipe that through a json parser, etc). I was hoping I could get a stream from entry.getData, and then pipe that through a DecompressionStream and the other stream utils I have to ultimately parse the jsonl + json objects and then finally pipe those objects elsewhere (such as a DB that supports streaming inserts). But that doesn't seem to be how zip.js works, which is totally cool - just trying to wrap my head around the "zip.js way" :).

[gildas-lormeau]

You can get the readable from getData() once processed, it's the ReadableStream returned by pipeThrough() (which is ignored in your code).

[marbemac]

... omg. Guess it's just one of those days. All set, thank you for the point in the right / obvious direction!

[gildas-lormeau]

You're welcome :)