Requesting feedback on a new feature for Zarr (javascript)

[udras]

Hi everyone,

I recently joined a research group at a Swiss hospital that maintains and continuously develops a data warehouse. Recently, we have been exploring the use of zarr-python for efficiently accessing time-series data on a server backend and displaying it on a client's web browser. Our experience with Zarr has been very positive and we are cofident we will eventually use this in production.

As we further test the viability of using Zarr for our purposes, we have come across a use case that we believe would greatly benefit from a specific feature. We are interested in a scenario where Zarr data stored on the local disk of a client can be accessed and displayed in the browser without the need to communicate with the server. In other words, we are looking for a purely clientside solution. It is my understanding that this would require a special kind of store - similar to the Zarrita File System Store, but that gains access to the file system using the File System Access API [1] [2].

I would like to gauge the general interest of the community in such a feature. We believe that it would unlock a powerful and much requested capability for our users. We would greatly appreciate any insights or feedback you may have, and will influence our management's decision to continue developing the feature internally or open-source.

[joshmoore]

Hi @udras. Great to hear about your in Zarr, and I look forward to hearing more about your use case. If I understand correctly, what you are referring to is already possible. All of the files you see on https://idr.github.io/ome-ngff-samples, for example, are being accessed via HTTP directly by from the browser.

For example, for https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846152.zarr/&viewState={%22zoom%22:-3.5663700395903306,%22target%22:[9560,6675]}

that looks like this:

where each of the individual chunks is a URL of the form:
https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846152.zarr/3/1/45/1/0

[d-v-b]

@joshmoore those examples require accessing data via http, but I think @udras wants to use the browser to access user's local data, i.e. no http required. I think it's a great idea!

[joshmoore]

Ah, sorry for the confusion. I think I understand. Similar goal then as what @will-moore did in ome/ome-zarr-py#285 but without the need for having Python in the middle 😄

(NB: This is one of the reasons I don't like GH discussions)

[will-moore]

Sounds great! So, you're proposing to add a new type of File System store to zarrita.js?

[udras]

Yes, precisely! I'm not sure but Zarrita seems to me to be the best candidate. I also found:

• https://github.com/gzuidhof/zarr.js
• https://github.com/freeman-lab/zarr-js

What do you guys think?

[joshmoore]

@manzt will be able to tell us best, but my understanding is that the general plan is to base the V3 version of the other libraries on zarrita as was done with zarr-python.

[manzt]

Thanks for pinging me @joshmoore. Yes, I've been considering implementing a web version of the FileSystemStore store based on the Web File System API for Zarrita. However, setting up the store initially requires users granting permissions to the file system, and it's a bit unclear to me how much of that "flow" should be a library like Zarrita's responsibility.

One of the main design considerations in the zarrita rewrite was to make it very simple to write custom stores. You just need to implement Readable to get something working.

Here is an example of a WebFileSystemStore based on the WebFileSystemAPI that implements Readable:

WebFileSystemStore implementation

/**
 * Get a file handle to a file in a directory.
 *
 * @param {FileSystemDirectoryHandle} root - A root directory from the Web File System API.
 * @param {string} path - A key to a file in the root directory.
 * @returns {Promise<FileSystemFileHandle>} - A file handle to the file.
 */
async function resolveFileHandleForPath(root, path) {
  const dirs = path.split("/");
  const fname = dirs.pop();
  for (const dir of dirs) {
    root = await root.getDirectoryHandle(dir);
  }
  return root.getFileHandle(fname);
}

/**
 * A store for zarrita based on the Web File System API.
 *
 * Note: usage requires prompting the user to select a directory to grant
 * access to the Web File System API.
 *
 * @see {@link https://developer.mozilla.org/en-US/docs/Web/API/File_System_API}
 *
 * @example
 * ```js
 * import * as zarr from "zarrita";
 *
 * let directoryHandle = await globalThis.showDirectoryPicker();
 * let store = new WebFileSystemStore(directoryHandle);
 * let root = await zarr.root(store);
 * let arr = await zarr.open(root.resolve("/foo"), { kind: "array" });
 * ```
 */
class WebFileSystemStore {
  /** @param {FileSystemDirectoryHandle} root - A web file system directory handle */
  constructor(root) {
    /** @type {FileSystemDirectoryHandle} */
    this.root = root;
  }

  /**
   * @param {`/${string}`} key
   * @returns {Promise<Uint8Array | undefined>}
   */
  async get(key) {
    // TODO: better error handling. We want to catch when file is missing and just return undefined. Other errors should bubble.
    let fh = await resolveFileHandleForPath(
      this.root,
      key.slice(1),
    ).catch(() => null);
    if (!fh) return undefined;
    let file = await fh.getFile();
    let buffer = await file.arrayBuffer();
    return new Uint8Array(buffer);
  }
}

// Usage
import * as zarr from "https://esm.sh/zarrita@next";

// Users must trigger event to be able to call `showDirectoryPicker`
document.querySelector("button").addEventListener("click", async () => {
  let dh = await globalThis.showDirectoryPicker();
  let store = zarr.root(new WebFileSystemStore(dh));
  let node = await zarr.open(store.resolve("/blosc"));
  console.log(node);
});

@udras, I'd be very open to graduating this store to something more official in @zarrita/storage package if that is something you'd be interested in taking on.

[keller-mark]

I would like to gauge the general interest of the community in such a feature.

We have discussed using this functionality for Vitessce as well so would be interested! vitessce/vitessce#1278 (comment)

[udras]

Thank you for the feedback, everyone! I will discuss this with our group leader - it certainly seems to be a worthwhile endeavor.