Can kuzu work with pyodide?

[Analect]

I was wondering if anyone on here has tried to get kuzu working with pyodide (for running python using wasm).

I tried most of the wheel files on https://pypi.org/project/kuzu/#files ... but it seems none are compatible.

import micropip as mp
await mp.install("https://files.pythonhosted.org/packages/a3/86/9c0b35f015ab81c3366dbf4528bfc0453e06f28ae69637b93e10ec6e44ae/kuzu-0.5.0-cp311-cp311-win_amd64.whl")

import kuzu

Any pointers appreciated. Thanks

[prrao87]

Hi @Analect, thanks for trying out Kùzu. WASM support isn't finished yet - we currently don't have all the wheels in place for Kùzu to run in the browser. But it's definitely on the roadmap. This discussion might be relevant (though not directly related to Pyodide). @mewim will get back to you with more technical information specific to Pyodide and how it could be supported.

[Analect]

Thanks for the info @prrao87, @mewim .
I was able to find this on Github at https://github.com/DylanShang/kuzu-lab.
There's a running version here: https://kuzu-lab.netlify.app/lab/index.html
I needed to manually add the csv files ... but it appears to work.

I'm not sure what magic is being used here, but maybe @DylanShang can elaborate! Thanks.

[prrao87]

Ah yes, this isn't something officially built by the Kùzu team, but we did know of this based on @DylanShang 's comments on our Discord. He would be the best person to answer your question.

[mewim]

@Analect,
The current version of kuzu should be buildable with the Emscripten compiler without any change. However, as we pointed out in #3327, Kùzu’s current buffer manager uses memory mapping and requires support of MADV_DONTNEED, which is not available for wasm right now. We worked with @DylanShang a bit when he worked on his WASM version, but I do not think his current implmentation resolves this problem yet. In production use, the lack of eviction can lead to crash eventually, and I would suggest against attempting to use Kùzu with WASM until we officially support it.

[DylanShang]

Thanks @prrao87 @mewim

Hi @Analect, I am so glad that you are interested in kuzu-wasm. I have some knowledge of Pyodide, and I understand that installing wheels via micropip currently only supports pure Python or WebAssembly versions of wheels.

Kuzu-wasm is part of my research project, and my goal is to explore the potential of kuzu in a WebAssembly environment. As @mewim mentioned, kuzu's use of memory mapping leads to MADV_DONTNEED not being supported by Emscripten, so the process of bringing kuzu to wasm hasn't been smooth. Currently, kuzu-wasm works fine as long as it's not handling large-scale operations. The current solutions are:

1. The official team plans to develop an alternative version of the buffer manager that doesn't rely on memory mapping later this year.
2. Modifying the source code in getFrame and releaseFrame so the frame is released properly.

I'm thrilled that you've taken an interest in my project, Feel free to reach out if you need any more details or assistance！

[Analect]

Great. Thank you all. @DylanShang, I'll maybe ping you on your own repo. @prrao87 , should I leave this open for now? Otherwise happy to close.

[mewim]

@Analect I think we will close this for now. If you are interested, please subscribe to #3327. We will update it when we resolve the buffer manager issue and launch wasm version of kuzu officially.