Packaging opam for FreeBSD shows strange error

[hannesm]

Dear all,

I want to update the FreeBSD port of devel/ocaml-opam from 2.0.8 to 2.1.1. I use the full tarball -- i.e. where all dependencies are embedded as tarballs (since the build phase does not have access to the network). While doing that, I noticed that now a full OCaml compiler is bootstrapped. This is done via shell/bootstrap-ocaml.sh, which right at the top has needlessly an exit 1 if neither "curl" nor "wget" is found. As the full tarball contains all tarballs, this shell script will never execute "curl"/"wget", and the shell script should not exit with 1.

For the FreeBSD package, I added a local patch https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259667 (https://bugs.freebsd.org/bugzilla/attachment.cgi?id=229304&action=diff) -- but it would be nice to fix the bootstrap story of the full tarball before a next release. Since I'm not sure how the bootstrap story is put together, I'm not sure what an appropriate fix is (in FreeBSD, we as well pass --disable-checks to configure since otherwise (at least in earlier releases) opam refused to build without curl/wget).

Thanks for reading and have a nice weekend,

Hannes

[hannesm]

Also I'm curious since the full tarball an build process compiles an OCaml compiler, but configure checks for a OCaml compiler as well, if that could be simplified -- and if the shipped OCaml compiler could be used for all OCaml code in the full tarball (to drop the dependency on an external OCaml compiler).

[dra27]

The requirement on curl/wget should be fixed, yes - I'll have a look at that. When you bootstrap a compiler, configure will (or definitely should) begin:

Bootstrap compiler found -- activating
checking for ocamlc... ocamlc
OCaml version is 4.12.0
OCaml library path is /home/dra/opam/bootstrap/ocaml/lib/ocaml

i.e. if you bootstrapped the compiler as part of opam's build, it will completely the presence or not of any preinstalled ocaml compiler.

[hannesm]

I'm not sure I understand what you mean. An experiment I conduct without any ocamlc in my PATH:

opam-full-2.1.1$ ./configure 
checking for ocamlc... no
checking for ocaml... no
checking for ocamldep... no
checking for ocamlmktop... no
checking for ocamlmklib... no
checking for ocamldoc... no
checking for ocamlbuild... no
configure: error: You must install the OCaml compiler, at least version 4.02.3 (or run: make compiler)

Now, the packaged compiler fro FreeBSD is 4.05.0, if this is installed I get:

opam-full-2.1.1$ ./configure
checking for ocamlc... ocamlc
OCaml version is 4.05.0
OCaml library path is /usr/local/lib/ocaml
checking for ocamlopt... ocamlopt
checking for ocamlc.opt... ocamlc.opt
checking for ocamlopt.opt... ocamlopt.opt
checking for ocaml... ocaml
checking for ocamldep... ocamldep
checking for ocamldep.opt... ocamldep.opt
checking for ocamlmktop... ocamlmktop
checking for ocamlmklib... ocamlmklib
checking for ocamldoc... ocamldoc
checking for ocamldoc.opt... ocamldoc.opt
checking for ocamlbuild... no
checking OCaml Sys.os_type... Unix

Maybe I misunderstand how building opam-full should be done -- should there be a make bootstrap ; ./configure ; make done? If this is the case, it is pretty unconvential - would it be possible to achieve this with the common ./configure ; make?

[hannesm]

I also tried to run ./configure without --disable-checks that then (if neither wget nor curl is found) calls our to ${MAKE:-make} -q -C src_ext has-archives, resulting in:

gmake: *** No rule to make target 'archives/findlib-1.9.1.tar.gz', needed by 'has-archives'.  Stop.

That makes me wonder why findlib/ocamlbuild/topkg are in Makefile.sources as it seems they are not used for compilation?

[hannesm]

After reading some more of configure.ac and the recent development on the master branch, I'd hope that before a 2.2 release is done, this could be solved. A hacky solution would be a custom configure script that eventually executes make compiler (if no ocamlc is found or it is < 4.08.0) and then runs the real configure.

[dra27]

I'm not sure there's anything which needs to be solved, but it does sound as though there is a bug in the full tarball in that it forces requirement of a tool which isn't needed (the curl/wget part)! The README specifies that OCaml is a prerequisite and has a link to further in down it for if you don't to https://github.com/ocaml/opam#compiling-without-ocaml which explains make cold and the use of CONFIGURE_ARGS if needed. The process is (or certainly ought to be!) entirely standard if you have prerequisites 🙂

[hannesm]

Thanks @dra27, indeed I missed the make cold target and its documentation.

[dra27]

The findlib, ocamlbuild, etc. tarballs are there for the lib-pkg mode of bootstrapping (this is where we install the dependencies as normal into the bootstrapped compiler). This is a mode which was very useful for me early on in the Windows work (in 2015!), but since the switch to jbuilder/Dune, I think it’s served its purpose and I may well strip it out before 2.2

[hannesm]

Indeed, to wrap up:

• shell/bootstrap-ocaml.sh should not exit 1 if curl and wget are not found
• the src_ext has-archives target is broken (used by configure if curl and wget are not available and enable_checks != no) - my workaround is to pass --disable-checks to configure
• I updated the FreeBSD port PR to do ${MAKE_CMD} compiler before confgure, then configure is invoked, and then make is invoked with the target cold-lib-ext cold-all, afterwards make install

Sorry for the partial noise, and please enjoy your weekend. Special thanks to @dra27 for pointing me to the documentation.

[hannesm]

In FreeBSD, we now have opam 2.1.2 in the ports with two patches (see https://github.com/freebsd/freebsd-ports/tree/65d44673f0de934bf18081e60ca92742cb404324/devel/ocaml-opam/files)

• as mentioned above, bootstrap-ocaml.sh does not exit 1 is curl and wget are not found
• to support x86 (32 bit) on FreeBSD, a patch to the OCaml compiler is needed, thus the patch of bootstrap-ocaml now includes a way to patch the OCaml distribution
• the actual patch for OCaml's configure (see relocation issues with clang on 32bit FreeBSD ocaml#10835)

The sad news is that with opam 2.0.x (where we were able to use OCaml 4.05.0 from ports (also patched https://github.com/freebsd/freebsd-ports/tree/65d44673f0de934bf18081e60ca92742cb404324/lang/ocaml/files), this worked nicely on arm32 bit systems -- now it fails and my attempts to get this working lead nowhere (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261180).

[hannesm]

FWIW, it would be greatly appreciated if these packaging issues could be addressed before the next release (even if it is 2.1.3) -- mainly the bootstrap-ocaml.sh shouldn't exit with 1 if curl / wget are not found (and there's nothing to fetch anyways).

[kit-ty-kate]

#5223 should fix half of this issue. And if I understand correctly, once 4.14.1 is out and opam bootstraps with it, it should fix the remaining issue.

EDIT: Actually it's in 4.14.0