[RFC003] Nix-nickel (Draft, WIP)

[yannham]

Rendered

[MagicRB]

Oh my! It's here

[aspiwack]

If there are written document that describe the shortcomings of stdenv and packages-as-functions. It would be best to link to them here.

[blaggacao]

https://github.com/nixpkgs-architecture/issues/issues/1

[aspiwack]

I don't think that having unix_package be a function would make much of a difference here (except for potentially higher computational cost, that's true, but it's fairly isolated): the function that we don't want is a package being a function of its dependencies. Which forces the fixpoint computation before knowing anything of a package.

The problem of the fixpoint computation is that the cost of computationally heavy packages tends to propagate to others. I would characterise the problem that we are trying to solve as: we want to be able to know a package's metadata without forcing its dependencies.

[yannham]

True, and flakes are one example I think, where output is still a function but that's fine because the other metadata are one level up and therefore accessible without having to evaluate the output function. And back to our example, evaluating a merge and a simple function (which is morally a parametrized record) shouldn't make a big difference in practice.

One reason I could see to do that would be for consistency, as using both functions and merging may end up being confusing to the user, in particular which one is to be used in which case.

[thufschmitt]

One difference between having a function and a record merge here is that the function only goes on way (the function can transform the package definition), while with a record merge, the package definition could itself depend on stuff defined by unix_package.

For example (assuming that we can access to not-yet-defined record fields), one could imagine that unix_package has a shell field, and that the package definition uses that to define a custom build phase:

hello = builders.unix_package & {
  buildPhase = "%{shell} ./build.sh", # Where `shell` will be provided by `unix_package`

  ... # Additional stuff
}

That pattern currently exists in nixpkgs, where stdenv both provides the mkDerivation function to generate the arguments for builtins.derivation, but also exposes a bunch of stuff (stdenv.shell, stdenv.cc, etc..)

(Not saying that we should use record merging to provide that, just pointing out that the two are different in terms of expressiveness)

[thufschmitt]

One difference between having a function and a record merge here is that the function only goes on way (the function can transform the package definition), while with a record merge, the package definition could itself depend on stuff defined by unix_package.

For example (assuming that we can access to not-yet-defined record fields), one could imagine that unix_package has a shell field, and that the package definition uses that to define a custom build phase:

hello = builders.unix_package & {
  buildPhase = "%{shell} ./build.sh", # Where `shell` will be provided by `unix_package`

  ... # Additional stuff
}

That pattern currently exists in nixpkgs, where stdenv both provides the mkDerivation function to generate the arguments for builtins.derivation, but also exposes a bunch of stuff (stdenv.shell, stdenv.cc, etc..)

(Not saying that we should use record merging to provide that, just pointing out that the two are different in terms of expressiveness)

[thufschmitt]

I agree with the confusing-ness of bazel targets, but they have a very strong point for them in that because since they are at a meta-language level, they are only resolved at the very last moment, meaning that

1. The actual path of the dependency really stays fully opaque, which
   1. Is more robust. For example, there was a couple of occurrences of code manipulating the output path of a dependency in nixpkgs, and that broke with CA derivations so I had to change it − like calibre: Fix build with CA derivations NixOS/nixpkgs#124227
   2. (potentially) makes the final UX nicer because you don’t have ugly store paths leaking everywhere whenever you want to do something
2. Evaluating a target doesn’t require evaluating its dependencies (meaning that the whole discussion about packages-as-data vs packages-as-functions doesn’t even have to exist)

We anyways couldn’t do something like what bazel does (because in the Nix model you do need to resolve the paths of your dependencies to generate the build script, which isn’t the case in bazel), but maybe an in-between solution could be interesting

[thufschmitt]

Unless (until?) we can get to a point where the Nix evaluator is just Nickel with a Nix frontend (in which case that problem won’t exist anymore because Nix could just call Nickel code transparently), I think that could be a limiting factor for the adoption of Nickel − at least for open-source projects, but these are the one with the most visibility. So we’ll want to have a decent story around this too

[yannham]

As mentioned during the meeting, it seems going from a data representation to a function representation is easier, so we can hope for a simple generic wrappers around a Nickel package. But that needs to be investigated further before proceeding.

[steshaw]

Are we back to considering a functional representation for Nickel pkgs?

[yannham]

I think what is being discussed here is having a wrapper of some sort to use a Nickel package from Nix code, but just as a legacy compatibility mechanism. Using Nickel packages from another Nickel package would use the data representation and leverage all the benefits. However, if for some reason you need to use a package defined in Nickel from a Nixpkgs-like Nix codebase, you could, thanks to (very schematically) to_nixpkg : Package -> NixPkgLegacyFunction.

[thufschmitt]

I don’t think that’s a bad thing in the long term tbh. Having an interface that would be too transparent could cause the path of least resistance to lead to a weird spaghetti of Nix and Nickel code intertangled, which would probably be a very bad thing.

At least, having some annoying constraints at the boundaries (even if just because both worlds have different idioms) mean that people will care about these boundaries, and things will stay clearly separated

[yannham]

That's a good point. It is also a sign that indeed the PARM and Nickel additions do add value: if we could convert systematically easily between them, we could have adopted the PARM inside Nix itself, incrementally and in a backward-compatible manner without a new language a long time ago.

[milahu]

this could allow inversion of control, where a child package can modify a parent package

# file hello.ncl
{
  # dependencies
  gtk | NickelPackage = {
    version = "2.3.4", # bottom-up pinning. nixpkgs: gtk_2_3_4
    enable_somefeature = true, # require feature from parent
    # nixpkgs: hello = callpackage ./hello.nix { gtk = gtk.override { enable_somefeature = true; }; };
  },

problem: pollution → move to scope

# file hello.ncl
{
  pkgs = {
    gtk | NickelPackage,
  },
  ... pkgs.gtk ...

or, anonymous scope

# file hello.ncl
{
  {
    gtk,
  } | NickelPackages,
  ... gtk ...

[yannham]

Good point. Although this could also be solved by merge priorities, the Nickelpkg layer erasing the local definitions. But it does sound fragile. Another issue is that having everything living at the same level forces all the versions of gtk to be the same in the final fixpoint, preventing local overrides (we want this package to have python=python27 but that package to have python=python3 for example).

For those reasons, I think your first suggestion sounds like the best way forward. Also, it could avoid having to repeat dependencies as say inputs and build_inputs: we could kill two bird with one stone and declare both inputs for Nix and the "merging interface" for Nickel as in

{
  inputs = {
    gtk,
  },

 build_inputs = {
    foocompiler,
  }

  # ... inputs.gtk ...
}

[milahu]

bikeshed:
inputs sounds too general, could be packages and options (enable_somefeature)
packages is too long
pkgs is short & sweet (but also used in nixpkgs)

or we identify "package scopes" by type

{
  p | Pkgs = {
    gtk,
  },
  rundeps | Pkgs = [
    apkg, bpkg, cpkg, dpkg,
  ],
  drv = {
    build = m%"
      cp %{p.gtk}/lib/libgtk.so $out/lib/
      echo PATH=%{lib.makePath rundeps} >$out/bin/wrapper.sh
    "%m,
  },
}

short names like p would compensate for nickel's missing with feature

[steshaw]

If it's a lot of work to reimplement builtins, I was wondering whether it may be possible to leverage the builtins within Nix itself (i.e. the executable / C++ source code) by using Nickel as a library in Nix. I'm not familiar with the Nix implementation, so it may be cumbersome, as often extension functions are tied closely to the interpreter. However, with some refactoring, a shared interface may be extracted for use in Nix proper and Nix-Nickel.

Using Nickel as a library seems like a good path for Nix in the future, whatever happens. Once that's in place, it makes sense to slowly port Nix from C++ to Rust. I believe there is already some interest in that direction.

[yannham]

Yes, that is a possible path forward, but it is in opposition with the constraint _Do not require unreasonable changes to Nix itself_. We could debate if those changes are unreasonable, but I think that for a first approach, the less changes to Nix, the better. Not only from a disturbance/community point of view, but also I fear that even once accepted, the Nix side of the changes would be very long to land given the current development process, and that would hold us back.

[steshaw]

What are g-exprs? Sounds like something Scheme-like (ala s-expression / fexprs). Perhaps a reference and an example of how it is less ergonomic would be helpful.

[yannham]

Those are the Guix equivalent of string contexts, somehow: manual page. At this point this paragraph is just draft notes, but it'll deserve a proper link indeed once finalized. At a quick glance, it's less ergonomic because:

• Use a special syntax to start a g-exp #~ (gexp)
• Use a special syntax to interpolate other g-exp #$ (ungexp)

As opposed to Nix where string contexts just work without having the user to even think about it or convert between strings and derivations. On the other hand, the explicitness of g-exp may actually be a good point: arguably, it makes things less magic. But it would still be one additional thing that Nix users don't need to do today. Although, I think what was written at the time is not true: something akin to g-expr in Nickel using standard syntax for strings, like interpolation %{}, would still need a bit of magic and an extension mechanism, probably.

[steshaw]

I understand that this is a draft. That's no problem!

Thanks for the reference. I have not delved much into Guix. Nix/NixOS is niche enough for me 🙂.

As I mentioned in IRL, my thinking is that the ideal solution is to have overloaded and customisable interpolated strings. Some languages do this such as Scala and Haskell (via quasiquotes and TH). In Scala it requires a prefix. Haskell doesn't have builtin interpolation, so quasiquotation kind of is the way 🤷‍♂️. I experimented a little bit with the the support in Idris the other night. It seems doable. Though I seem to have hit a limitation, misunderstanding, or bug in Idris.

[steshaw]

In Idris, for instance, you can overload the standard string interpolation syntax. It should be possible to make something like this work:

postInstall : PkgExp
postInstall = """
    substituteInPlace \{out}/bin/my-script \\
      --replace /usr/bin/python \{python}/bin/python \\
      --replace /usr/bin/perl "\{perl}/bin/perl -I \{libfoo}"
  """

I'm not fond of that backslash as it evokes the idea of a "lambda". This is simply how string interpolation is defined in Idris (strange as $ was discarded because of it's special meaning).

In this short experiment, pkg3a should be equivalent to the desugared pkg3 but I ran into a snag.