After my last blog post about OCaml and web development, I got a lot of good feedback from the OCaml community, offering solutions to the pain points I faced during my project setup. Some talked about opam tricks, while others recommended exploring Nix Overlays.

Opam Switches

In case you’re not familiar, you can have independent environments with opam. Those environments are called switches, and whatever you depend on or install will only be accessible within that switch.

If I were to continue using opam, I’d probably have an environment per project. To build a new one, I’d do

$ opam switch create my-new-project 5.2.0
# 5.2.0 being the most recent OCaml compiler release

and install my dependencies and tools there. If I wanted to change back to my old project, I could do

$ opam switch set my-old-project

This isn’t perfect though: While it is possible to have per-terminal opam switch setups by calling eval $(opam env --switch=my-new-project), they are by default global. And setting this up with my editor is probably possible, but I’m afraid working on multiple projects at the same time will confuse Merlin/the LSP server.

I already knew about switches, but it didn’t help me with my most pressing problem: ensuring that my build is reproducible. To ensure that the exact same dependencies are used, I’m probably better off using Esy.

So, Esy Then?

In theory, the natural choice for development would be Esy. Esy has isolated environments per project automatically, lockfiles, and installs not-yet-installed dependencies automatically. My only concern would, once again, be how I’m getting Merlin/the LSP server to work with Esy itself.

The difference from opam, however, is that Esy forces you to handle this, which means there should be documentation on how to get your editor up and running with it. Which in turn means I probably won’t have to investigate as much. Though to be fair to opam, the solution for getting Esy up and running is probably similar to how you’d solve it there.

… but even with a lockfile, this will not ensure my build is reproducible. In fact, it won’t even guarantee that it will be buildable! In the blog post mentioned above, I encountered two issues related to glibc: One in the OCaml compiler itself, and one in the dependency pg_query. I’d have to pin myself out of the last one to get it to work with my newer glibc.

While unlikely, it’s not impossible to imagine that this could happen in the future, and that there’s no immediate fix available. In that case, my build will be broken until either I or someone else fix the dependency, or until I roll down my glibc version.

Nixing Broken Builds

A suggestion from someone who read my blog post was to use Nix Overlays to avoid the glibc mismatch I encountered, pointing me towards nix-ocaml/nix-overlays. Nix Overlays seem to use Nix Flakes under the covers, as the recommended approach is to write a flake.nix file.

One of the things I’ve never really gotten around to is to try out Nix. Nix sounds great to me: It claims to ensure builds are reproducible, thus giving us a guarantee that our stuff will build in the future even if we’re on different machines¹.

Reflecting on it, it’s actually quite weird that I haven’t learnt it in anger: I’m usually one of the few running Linux in a team, which usually leads me to debug and document stupid differences between Linux and macOS. Clearly, Nix could help me out here, right?

Perhaps to some extent, but there’s a nonzero startup cost. For one, I’d have to get the others to install, use and learn Nix. As we’ll see, that’s not trivial. More importantly, through my annoyance with big Docker container images, I’ve learnt to avoid dynamic libraries and rather make standalone executables. And for CI/CD, I’ve usually managed to set up a reproducible image that builds stuff – sort of like a poor man’s Nix I suppose. When that’s up and running, usually there’s not really that big of a deal to have a Nix setup: You can copy the build image’s steps and get your system up and running based on that template. Because there’s basically no dynamic library you have to mess with, there’s very little that ends up going wrong in practice for me (YMMV).

But this comes with a “hidden” cost: If I’m using Go, I use Go’s DNS/TCP/IP implementation, force myself to use e.g. the Go port of SQLite or just use a key-value store implemented in Go rather than the battle-tested LevelDB/RocksDB or similar. When running on the Java runtime, I avoid JNI when possible for the same reason.

This probably results in me either having worse performance, fewer debugging tools and fewer options. It’s not all negative though, reproducibility and portability is much easier if you manage to avoid OS-installed packages. But if I can get the same through Nix without limiting myself to the language I use, then that would be very nice.

Learning Nix

I’ve sort of learnt Nix conceptually through osmosis from all kinds of places, and since it is on my agenda to properly learn it, I thought I may as well try it now. It’s a little hard to know where to start, so I went to Nix pills to learn the stuff bottom-up.

Nix pills seems like a great introduction to all the nitty-gritty details, tips and tricks and all the other things you probably need to know if you want to port existing packages over to Nix. I, on the other hand, wanted to make my OCaml builds reproducible.

That made me a bit antsy after a while: While it’s certainly nice to know how to compress binaries, and exclude GCC from the derivation, that’s not what I’m after. By the time I reached Nix pill 12, I was tired and skimmed through the rest. I felt I had gathered enough information from the earlier pills to navigate Flakes, as I would likely end up copying someone else’s setup and hack on it ‘till it worked.

That being said, my initial impression of Nix wasn’t as reproducible as I thought it would be. Consider the example of a working derivation in pill 7. Here, the author finds the location of Bash 4.2 through some REPL calls:

nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> "${bash}"
"/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45"

When I try to reproduce this behaviour, I get Bash 5.2:

nix-repl> :l <nixpkgs>
Added 21422 variables.
nix-repl> "${bash}"
"/nix/store/h3bhzvz9ipglcybbcvkxvm4vg9lwvqg4-bash-5.2p26"

And from what the pill says, this is REPL sugar for the expression import <nixpkgs> { } in Nix.

I’m sure this makes sense if I dig a bit deeper, but it’s a little weird to me that this is what I’m initially greeted with. Nothing is telling me that <nixpkgs> is a moving target! How can my derivation be reproducible if Nix could be pulling the rug under me by bumping my dependencies?

Anyway, the point was not to learn Nix properly, but to get enough understanding that I could move on to learn about Nix Flakes.

Flakes and Unstable Ground

After looking around for information about Nix Flakes, it seems like the NixOS & Flakes Book is the best way to learn about it.

This sounds great, but there are two reasons why I stopped at this point:

• An entire book? I just wanted reliable builds for my toy OCaml project :(
• Flakes is an experimental feature, and according to the docs:

  Experimental features are considered unstable, which means that they can be changed or removed at any time.

Using Flakes for reproducibility kinda defeats the purpose then: Unstable means it’s by definition not necessarily reproducible in future versions of Nix. Yeah, yeah, I know I could “just” use the current Nix version to the heat death of the universe, but that’s not exactly a viable long-term strategy.

One Thing At a Time

Mind you, this isn’t about Nix being hard – though I certainly find the documentation spread all over the place, and it seems like there is quite a lot of it. This is more about me wanting to focus and learn one thing at a time, just as I skipped ReasonML/OCaml on the frontend to begin with as well. Changing too many variables at the same time is a recipe for disaster.

Nix is still on the agenda for me though. It sounds like NixOS is a great tool for pet servers, so being able to leverage it for small projects where I don’t need a Kubernetes cluster to manage the cattle seems wonderful.

But recall my initial pains from the first post: I’d like to have a reproducible OCaml build system. While having the right glibc available will ensure that my build will always compile, that’s only a tiny part of the problem. My main problem is that I have to ensure the right git revisions are used for my dependencies, and that isn’t something Nix nor Nix Flakes directly help me with. As such, for development, Nix would only help me with a tiny issue I’ve now managed to resolve. And if I were to use Nix, only to find out I need to use some other library that depends on a later glibc version, I’d probably be in a lot of pain again.

So What’s the Solution Then?

The silver bullet for me seems to be a combination of Esy and Nix Overlays. I think I can manage fine with just Esy for now, so that’s what I’m aiming for.

I’d also like to note that the build situation is much better now than when I wrote the first blog post. I already mentioned that the Esy example was fixed, and Dream is now in sync with its documentation. Unless you’re going to use ppx_rapper, all should work out of the box without any pinning pain!