Improve examples/ subfolder with cross-compilation and more docs (#25)

- Adds examples for cross-compilation of EIFs in `./examples`, and gives more detail on what commands to run in order to build EIFs. - Fixes a bug where the examples lockfile would fail to find the parent flake if the flake is not already in the Nix store. - Updates architecture diagram to reflect changes from #24 - Removes quick start example as it is redundant with `./examples/` and means maintaining an extra flake in markdown
monzo · Sep 26, 2024 · 7d75557 · 7d75557
1 parent e0ef65c
commit 7d75557
Show file tree

Hide file tree

Showing 6 changed files with 125 additions and 60 deletions.
diff --git a/README.md b/README.md
@@ -8,66 +8,38 @@ You can think of it as an alternative to `nitro-cli build-enclave` for building
 - give users complete control over their enclave images, providing additional options like BYOK (Bring Your Own Kernel)
 - easily build EIFs on systems other than Amazon Linux, including M1+ Macs (e.g, it's possible to build an x86_64 Linux EIF on an ARM Mac)
 
-We recommend [this excellent blog post](https://blog.trailofbits.com/2024/02/16/a-few-notes-on-aws-nitro-enclaves-images-and-attestation) to learn more about the EIF Nitro image format in general.
+
+> We wrote [a blog post](https://monzo.com/blog/securing-our-software-supply-chain-better-with-reproducible-builds-for)
+> about our motivation for building this tooling at Monzo. We recommend you read it if you use AWS Nitro Enclaves
+> and you are wondering why you might want to use it.
+> We also recommend [this other excellent blog post](https://blog.trailofbits.com/2024/02/16/a-few-notes-on-aws-nitro-enclaves-images-and-attestation) to learn more about the EIF Nitro image format in general.
 
 
 The tradeoffs between using this repo and AWS' `nitro-cli` are:
-| Feature | `nitro-cli build-enclave` | monzo/aws-nitro-util |
-|---------|-----------|----------------------|
-| EIF userspace input | Docker container | plain files, including nix packages and unpacked OCI images
-| EIF bootstrap input | pre-compiled kernel binary provided by AWS | use pre-compiled kernel by AWS or bring your own kernel (see [example](./examples/README.md))
-| dependencies | Docker, linuxkit fork, [aws/aws-nitro-enclaves-image-format](https://github.com/aws/aws-nitro-enclaves-image-format/) | Nix, [aws/aws-nitro-enclaves-image-format](https://github.com/aws/aws-nitro-enclaves-image-format/)
-| Source-reproducible | no, uses pre-compiled blobs provided by AWS | yes, can be built entirely from source
-| Bit-by-bit reproducible EIFs | no, EIFs are timestamped | yes, building the same EIF will result in the same SHA256
-| cross-architecture EIFs | yes, if you provide a container for the right architecture | yes, if you provide binaries for the right architecture
-| OS* | [Amazon Linux](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave-cli-install.html) unless you [compile `nitro-cli` from source](https://github.com/aws/aws-nitro-enclaves-cli/tree/main/docs) for other Linux. No MacOS. | any Linux or MacOS with a Nix installation
 
+| Feature                      | `nitro-cli build-enclave`                                                                                                                                                                                                             | monzo/aws-nitro-util                                                                                |
+|------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
+| EIF userspace input          | Docker container                                                                                                                                                                                                                      | plain files, including nix packages and unpacked OCI images                                         |
+| EIF bootstrap input          | pre-compiled kernel binary provided by AWS                                                                                                                                                                                            | use pre-compiled kernel by AWS or bring your own kernel (see [example](./examples/README.md))       |
+| dependencies                 | Docker, linuxkit fork, [aws/aws-nitro-enclaves-image-format](https://github.com/aws/aws-nitro-enclaves-image-format/)                                                                                                                 | Nix, [aws/aws-nitro-enclaves-image-format](https://github.com/aws/aws-nitro-enclaves-image-format/) |
+| Source-reproducible          | no, uses pre-compiled blobs provided by AWS                                                                                                                                                                                           | yes, can be built entirely from source                                                              |
+| Bit-by-bit reproducible EIFs | no, EIFs are timestamped                                                                                                                                                                                                              | yes, building the same EIF will result in the same SHA256                                           |
+| cross-architecture EIFs      | yes, if you provide a container for the right architecture                                                                                                                                                                            | yes, if you provide binaries for the right architecture                                             |
+| OS*                          | [Amazon Linux](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave-cli-install.html) unless you [compile `nitro-cli` from source](https://github.com/aws/aws-nitro-enclaves-cli/tree/main/docs) for other Linux. No MacOS. | any Linux or MacOS with a Nix installation                                                          |
 
 (*): OS for building EIFs. Note that 
 - to make EIFs on a Mac, you have to be able to cross-compile the userspace binaries from Darwin to Linux
 - even if you make an EIF on a Mac, it can still only run on Linux.
 
 ## Examples
 
-Flake quick start, to build an enclave with nixpkgs' `hello` :
-```nix
-{
-  inputs = {
-    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
-    nitro-util.url = "github:/monzo/aws-nitro-util";
-    nitro-util.inputs.nixpkgs.follows = "nixpkgs";
-  };
-  outputs = { self, nixpkgs, flake-utils, nitro-util, ... }:
-    let system = "x86_64-linux";
-        nitro = nitro-util.lib.${system};
-        eifArch = "x86_64";
-	    pkgs = nixpkgs.legacyPackages."${system}";
-    in {
-        packages.${system}.eif-hello-world = nitro.buildEif {
-            name = "eif-hello-world";
-
-            # use AWS' nitro-cli binary blobs
-            inherit (nitro.blobs.${eifArch}) kernel kernelConfig nsmKo;
-			
-            arch = eifArch;
-
-            entrypoint = "/bin/hello";
-            env = "";
-            copyToRoot = pkgs.buildEnv {
-                name = "image-root";
-                paths = [ pkgs.hello ];
-                pathsToLink = [ "/bin" ];
-            };
-        };
-    };
-}
-```
+You can find examples in [`examples/`](./examples/README.md).
 
-See more examples in [`examples/`](./examples/).
+Note that you need to install [Nix](https://nixos.org/) and [enable flakes](https://nixos.wiki/wiki/Flakes) to use this repo.
 
 ## Design
 
-monzo/aws-nitro-util is made up of a small CLI that wraps  [aws/aws-nitro-enclaves-image-format](https://github.com/aws/aws-nitro-enclaves-image-format/) (which allows building an EIF from a specific file structure) and of Nix utilities to reproducibly build the CLI and its inputs.
+monzo/aws-nitro-util compiles a CLI from [aws/aws-nitro-enclaves-image-format](https://github.com/aws/aws-nitro-enclaves-image-format/) (which allows building an EIF from a specific file structure) and of Nix utilities to reproducibly build AWS' tooling, the EIF, and its dependencies.
 
 A typical EIF build would look like the following:
 
@@ -96,7 +68,7 @@ graph TD
 		yourRepo("your source code \n or OCI image")
 	end
 	initBin("init \n compiled init.c \n (or bring your own)")
-	eifCli("📦 eif-cli \n in this repo")
+	eifCli("📦 eif_build CLI \n")
 	nsm("nsm.ko \n compiled Nitro \n kernel module \n (or bring your own)")
 
     subgraph PCR1
@@ -125,7 +97,7 @@ graph TD
 	nsm-->doSysInit
 	doSysInit ==> sysInit
 
-	eifFormatRepo ---> |pulled as \n build-time Rust \n cargo dependency|eifCli
+	eifFormatRepo ---> |compile \n from source|eifCli
 	eifCli -->doEif
 	kernel -->doEif
 	sysInit ==>doEif

diff --git a/examples/README.md b/examples/README.md
@@ -1,10 +1,53 @@
 # Usage examples
 
-Examples are structured as a single flake containing packages of potential EIFs.
+You need to install [Nix](https://nixos.org/) and [enable flakes](https://nixos.wiki/wiki/Flakes) to use this repo.
+Examples are structured as an additional Nix flake containing [derivations](https://zero-to-nix.com/concepts/derivations) (ie, build recipes, like Dockerfiles) for potential EIFs.
 
 To see the overall plumbing to use the aws-nitro-util flake, see [flake.nix](./flake.nix).
 
 To see examples for specific EIFs, see the individual package definitions:
 
 - Booting an enclave with a shell script only: [`withShellScript.nix`](./withShellScript.nix)
-- Booting an enclave with your own, compiled-from-source kernel: [`bringYourOwnKernel.nix`](./bringYourOwnKernel.nix)
+- Booting an enclave with your own, compiled-from-source kernel: [`bringYourOwnKernel.nix`](./bringYourOwnKernel.nix)
+
+## Building the examples
+
+### To show what examples can be built
+
+```bash
+nix flake show
+```
+
+### To build `shellScriptEif` for your current architecture:
+```bash
+nix build '.#shellScriptEif'
+```
+Note this will produce an `aarch64-linux` EIF if you are running it in an ARM Mac.
+
+
+### To build for a different architecture via a remote builder
+Nix allows compiling 'natively' for other architectures by building in a different machine.
+
+To do this you need to set up a [linux remote builder](https://nix.dev/manual/nix/2.18/advanced-topics/distributed-builds) first.
+This can be any machine you can SSH into, including a VM.
+
+Then, for example, to compile EIFs natively for `x86_64-linux` on an ARM Mac:
+```bash
+nix build '.#packages.x86_64-linux-crossCompiledEif'
+```
+
+Using remote builders makes builds simpler (because it is a linux x86 machine compiling linux x86 binaries) but requires setting
+up that additional machine and telling your local Nix installation about it.
+
+### To build for a different architecture via cross-compilation
+
+If you do not have remote builders, you can cross-compile. Keep in mind this requires all dependencies
+of your EIF to be cross-compiled too (which is tricky for bash scripts).
+
+
+To cross-compile an EIF from your local system
+to `x86_64-linux`:
+
+```bash
+nix build '.#x86_64-linux-crossCompiledEif'
+```
diff --git a/examples/flake.lock b/examples/flake.lock
diff --git a/examples/flake.nix b/examples/flake.nix
@@ -2,7 +2,7 @@
   inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
 
-    nitro-util.url = "path:../";
+    nitro-util.url = "github:monzo/aws-nitro-util";
     nitro-util.inputs.nixpkgs.follows = "nixpkgs";
 
     flake-utils.url = "github:numtide/flake-utils";
@@ -14,7 +14,7 @@
     in
     {
       packages = {
-
+        # the EIFs below will be for your machine's architecture
         shellScriptEif = pkgs.callPackage ./withShellScript.nix {
           inherit nitro;
         };
@@ -28,6 +28,25 @@
           inherit nitro;
         };
 
+        # the EIFs below will be for the architecture in the package name,
+        # even if you build from a different machine
+        x86_64-linux-crossCompiledEif =
+          let
+            crossArch = "x86_64";
+            crossPkgs = import nixpkgs { inherit system; crossSystem = "${crossArch}-linux"; };
+          in
+          crossPkgs.callPackage ./withCrossCompilation.nix {
+            inherit crossArch nitro;
+          };
+
+        aarch64-linux-crossCompiledEif =
+          let
+            crossArch = "aarch64";
+            crossPkgs = import nixpkgs { inherit system; crossSystem = "${crossArch}-linux"; };
+          in
+          crossPkgs.callPackage ./withCrossCompilation.nix {
+            inherit crossArch nitro;
+          };
       };
     }));
 }
diff --git a/examples/withCrossCompilation.nix b/examples/withCrossCompilation.nix
@@ -0,0 +1,28 @@
+{ buildEnv
+, hello
+, nitro # when you call this function pass `nitro-util.lib.${system}` here
+, crossArch
+}:
+nitro.buildEif {
+  arch = crossArch;
+  kernel = nitro.blobs.${crossArch}.kernel;
+  kernelConfig = nitro.blobs.${crossArch}.kernelConfig;
+
+  name = "eif-hello-world";
+
+  nsmKo = nitro.blobs.${crossArch}.nsmKo;
+
+  copyToRoot = buildEnv {
+    name = "image-root";
+    # the image passed here must be a Nix derivation that can be cross-compiled
+    # we did not use a shell script here because that is hard for GNU coreutils
+    paths = [ hello ];
+    pathsToLink = [ "/bin" ];
+  };
+
+  entrypoint = ''
+    /bin/hello
+  '';
+
+  env = "";
+}
diff --git a/examples/withShellScript.nix b/examples/withShellScript.nix
@@ -6,8 +6,8 @@
 }:
 let
   myScript = writeShellScriptBin "hello" ''
-    # note busybox can be used for building EIFs but only on Linux
-    # so remove this line if you are building an EIF on MacOS
+    # this will fail when compiling on MacOS
+    # see cross-compilation examples for alternatives
     export PATH="$PATH:${busybox}/bin"
 
     while true;
@@ -25,7 +25,7 @@ nitro.buildEif {
 
   name = "eif-hello-world";
 
-  nsmKo = nitro.blobs.aarch64.nsmKo;
+  nsmKo = nitro.blobs.${arch}.nsmKo;
 
   copyToRoot = buildEnv {
     name = "image-root";