Execution VFS
One struggle with hermetic build systems such as Nix or Bazel often is the different file system layout they require. They break certain assumptions that build systems make, e.g.:
- input directories are read-only,
- for performance reasons, a lot of symlinks are used to avoid file/directory duplication,
- relative files between source files and generated files break, because those file are stored in different root directories,
- hard-coded paths don’t work anymore,
- “global” dependencies are not at their typical locations, …
If we faked a typical build environment where it is required, we could reuse more existing tooling
unchanged. Executing commands in a chroot or even stronger sandboxed environment with
and underlying VFS that provided input files at expected paths and tracked all writes, would be
useful.
Getting changes back to the source
Where desired, getting the changes on the source files back to the source would be nice. E.g. for code formatting tools. If those run in parallel with the build, changing the source files adhoc is not desirable — at least not without further coordination. But e.g. allowing the user to copy all “proposed” changes to the source files easily, would make fast, incremental code formatting/fixes facilitated by the build system easy.
Potential layout
Similar to chroot envs:
mount -o hidepid=2 -t proc none proc/(https://www.kernel.org/doc/Documentation/filesystems/proc.txt)mount --rbind /sys sys/mount --rbind /dev dev/
Files:
- Base FS from “sysroot” shared between all or most sandboxes
/home/user- fake home directory/home/user/workspace- the mono repo/tmp- tmp space
Capabilities
- Only see files that are part of the sysroot or dependencies. (hermeticity) a. Copy/Hardlink/Symlink, or b. Filter
- Fast to set up.
- Easily extract output files.
- Easily discard temp files.
- Easily detect all written files.
- Easily detect all read files (for dependency pruning).
Implementation Ideas
- Using FUSE, e.g. with fuser
- Overlay filesystem basic
- mergerfs FUSE unionfs with many features
- Merges different branches, e.g. root paths together
- has a follow symlink feature
- wrapfs simple pass through as basis
- cow-shell
- bindfs
- bind mounts might get slow if used excessively?
- fuse passthrough
- crosvm
- filter files
- vagga sets up dev envs
Nix Sandbox?
These are the input files:
/** * All input paths (that is, the union of FS closures of the * immediate input paths). */ StorePathSet inputPaths;Here is how the inputPaths are provisioned on Linux in chroot mode (which includes sandbox mode?):
Which bind-mounts all dependencies:
Nix dependencies are much more coarse-granular, typically, as in a NixOS package. It might be that this is to slow for us.
(The Darwin implementation takes a filtering approach.)
Distri
Chroot beginning:
https://github.com/distr1/distri/blob/34e3a587e20e481c87d621082e6f7eadafafa347/cmd/distri/run.go#L64
Mounting of FUSE: