MetalSplatter

Render 3D Gaussian Splats using Metal on Apple platforms (iOS/iPhone/iPad, macOS, and visionOS/Vision Pro)

This is a Swift/Metal library for rendering scenes captured via the techniques described in 3D Gaussian Splatting for Real-Time Radiance Field Rendering. It will let you load up a PLY and visualize it on iOS, macOS, and visionOS (using amplification for rendering in stereo on Vision Pro). Modules include

• MetalSplatter, the core library to render a frame
• PLYIO, for reading and writing binary or ASCII PLY files; this is standalone, feel free to use it if you just have a hankering to load up some PLY files for some reason.
• SplatIO, a thin layer on top of PLYIO to interpret these PLY files (as well as .splat files) as sets of splats
• SampleApp, a mini app to demonstrate how to use the above (based on Apple template code) -- don't expect much, it's intentionally minimal, just an illustration
• SampleBoxRenderer, a drop-in replacement for MetalSplatter for debugging integration, which just renders the cube from Apple Metal template

Getting Started

You're right, the documentation is entirely missing; it's a major TODO list item. In the meantime, feel free to try out the sample app -- just, like I said, don't expect much.

1. Get yourself a gaussian splat PLY (or .splat) file (see Resources below)
2. Clone the repo and open SampleApp/MetalSplatter_SampleApp.xcodeproj
3. If you want to run on iOS/visionOS, select your target and set your development team and bundle ID in Signing & Capabilities. On macOS, just have at it.
4. Make sure your scheme is set to Release mode. Loading large files in Debug is more than an order of magnitude slower.
5. Run
6. Note: framerate will be better if you run without the debugger attached (hit Stop in Xcode, and go run from the app from the Home screen)

Showcase: apps and projects using MetalSplatter

• The MetalSplatter viewer is a simple, official Vision Pro app based on this library. This is different from the minimal included sample app (for instance, it has camera controls and a splat gallery). Confusingly, both the (open source) library and (non-open-source) app are called MetalSplatter, and both are by scier.

• OverSoul for Vision Pro: "Capture, share, and interact with spatial photos, 3D models and immersive spaces in a vibrant social ecosystem designed for the next generation of spatial computing"

• Know of another project using MetalSplatter? Let us know!

Resources

• Looking for .splat files? Here are a few suggestions:
  • Capture your own by using a camera or drone, then use Nerfstudio to train the splat
  • Capture one with Luma AI's iPhone app, and export it in "splat" format
  • Use the scene data from the original paper
• RadianceFields.com is a great source to track news and articles about 3DGS, NeRFs, and related technology and tools (for instance news about MetalSplatter), and the community surrounding it
• MrNeRF's Awesome 3D Gaussian Splatting Resources is exactly what it says on the label - in particular, an exhaustive and frequently-updated list of 3DGS-related research
• There are a variety of other renderer implementations available with source, such as
  • Kevin Kwok's WebGL implementation and online demo
  • Mark Kellogg's three.js implementation and online demo
  • Aras Pranckevičius's Unity implementation and series of very insightful blog posts: 1, 2, 3
  • The original paper's reference implementation