Re-integrate jpm into janet #1444
Comments
|
I think having something like jpm be bundled with janet has merits. I'm not so sure about trying to bring jpm itself back in though. Possibly a new thing might be better? Haven't thought too deeply about it though. |
|
Once mentioned idea of moving jpm into spork and making spork part of the janet build and release sounds the best to me. |
I agree with that. jpm has usablility problems that make that not ideal. So current jpm issues as I see it:
I guess I don't think the entirety of This kind of flow is pretty common in repl-first languages, and avoids some of the annoying and complicated setup of jpm, where it needs to be installed system wide in a compatible way to Janet. Configuration would be done solely by where I also don't think "installation" into a tree should be that complicated by default - a git project that was installable this way could just be a project that had a Lastly, the make-style build system built into jpm can be nice but I think is something that could just be left in spork if people needed it - most projects I've seen really don't need it. Curious on peoples thought |
|
The hardest part is of course an opinionated C build system (which is actually what |
|
I like the ideas of having a few simple primitives for module management etc. in janet itself and keeping more complicated parts in spork to import as needed. |
...
Interested in this kind of decoupling along with having some key feature(s) make it into janet. Having build system bits be in spork or elsewhere might be nice for extensibility and experimentation by 3rd parties. |
|
An observation about the specific example repl session above... IIUC, it demos the idea of being able to install spork with just an installation of janet. ATM, I think a C compiler is required in the case of spork because it has native modules. Possibly the janet-built-in stuff can use a C compiler if one is detected and bail otherwise? |
So spork already actually contains some of the needed functionality for building C modules (see spork/cc) - my idea is that we would eventually have sport be able to self bootstrap with it's own utilities. EDIT: as far as installing jpm without a C compiler, also a possibility but a number of modules depend on other native modules like |
|
So I was able to a working version of the An example of what the spork hooks look like can be found here: https://github.com/janet-lang/spork/tree/bundle-hooks/hooks Usage would be something like the following: Curious on feedback here - I know this does not include all of the features that This could also be put into spork rather than Janet as there really isn't anything here that needs integration with |
|
I just tried it, and it seems to be working well for me. |
|
bakpakin wrote:
This sounds good to me. tionis wrote:
You mean this sort of thing? On a side note, |
|
Did some more work on the branch, and added All that said, I can see functionality here growing and growing and growing, and at some point a line needs to be drawn. Not only should the functionality be simple but genuinely useful, the implementation needs to be fairly simple. |
Tried out briefly -- seemed to work here.
I agree about these endpoint characteristics -- wonder how to get there though (^^; |
|
Tried it briefly on Windows. Encountered at least one issue with |
|
I haven't tested any of this on windows but I wouldn't be surprised if there are path issues. |
|
@bakpakin Is the idea that module authors will include hooks files in the module to run on build and install? Or is this just an aspect of the proof of concept that will have improved UX later on? I will say that I like the way modules all currently have a |
|
This is all proof of concept, having each "hook" be a function inside a single "project" file in some ways makes more sense. I will likely change over to this way, and was playing with this locally. However, one of the issues we ran into with JPM was how to evaluate such files before dependencies were installed - separate files avoids that problem. I'm hesitant to commit even to this whole idea since JPM does already work, but I do want to explore it. |
The idea was that authors would include their own hooks, but UX would certainly be improved. However, not sure the same level of "declarativeness" as is in JPM is likely. |
|
@bakpakin I realise it's annoying when people compare Janet to Clojure and so at the risk of doing that in a way that turns you off completely, you could do something similar to what Clojure did when they introduced Unlike Clojure, Janet's data-only file ( Something like: {
:name "Foo"
:description "The best module"
:author "Janet Doe"
:license "MIT"
:url "https://git.example.org/foo"
:repo "git+https://git.example.org/foo"
:deps [“https://git.example.org/bar”]
:src [“foo/a.janet”
“foo/b.janet”]
:build-file "./build.janet" # potentially this is only required if you want to use a different file from the default
:install-file "./install.janet" # same as above
}JPM could then be updated to look for this file if there is no |
|
I like the idea of a If we went that route, we might want to settle on some clear conventions (maybe enforced by the Janet binary to one extent or other?) about how to include "third party" project metadata consistently. |
Not at all, that is a good comparison and a valid point. The data only file would allow more universal processing of bundles, and is relatively low effort to implement. I don't expect we would do much processing of this data in the bundle/ module besides move it around uncorrupted - not even sure we want to have the capacity to fetch remote dependencies over a network. Currently there really isn't even the concept of dependencies - the user would be responsible for providing them. There are a lot of ways to package and distribute software versions and dependencies that are not quite orthogonal, (npm/cargo/go style language package managers, system package managers, version control (git submodules, monorepos), containers, VMs, tarballs, One thing though that might be feasible and useful though is a way to check all installed bundles and make sure that they are working with all other installed bundles, basically a "check-installation" hook where authors of packages could enforce any constraints they wanted on the environment, and raise a (nice) error if there is a problem. For example, if a software author chooses to use a dependency that releases using semver, incremental version numbers, date ranges, etc., then they could do something like: # check if sporky installed with compatible version
(check-semver-for-installed-bundle "sporky" ">1.2.3,<1.2.8")
# check if sparky is installed with the correct git hash
(require-exact-git-hash "sparky" "abc1234")
# check if spot is installed and built within the specified dates
(require-build-date-range "spot" "2023-05-06" "2024-01-28")
# or use feature checking
(require-import-and-binding "spork/zig" zig-compile-and-link-v2)
# or only warn on mismatched bundle
(warn-exact-git-hash "spark" "abc1234" "Only git hash abc1234 has been tested and verified as dependency for this version")
# And finally run a simple smoke test
(import spot)
(spot/smoke-test)inside a hook. However, trying to automatically "resolve" versioning conflicts and search for the best version of packages for compatibility is something that should be up to users. Or at least, outside the scope of |
Even if you're using vendoring as a way to distribute code together with its dependencies, you still need some way to get the dependencies into the project tree in the first place and if you want to make that something you can do from the REPL, downloading remote dependencies is still an important piece of that, isn't it? |
|
Well in the case of vendoring I would think that the dependencies would be managed using git-subtree, git-subrepo, git submodules or maybe just by extracting an archive and commiting it. |
|
Vendoring increases repository size. Even though rust statically links a binary, it doesn't force repositories to carry dependencies. Haskell allows a program to depend on functional packages installed onto the system or ~/.cabal. I prefer functional packages that you can see in haskell, guix, and nix. I think haskell dependency model is a role model for janet. |
|
Git submodules increase size of archives. |
Yes, but doing so is actually fairly simple. E.g. with git: (defn bundle/install-git
"Install from git"
[repo-url name]
(def tempdir (string (dyn *syspath*) "/" (os/cryptorand 16) ".git"))
(os/mkdir tempdir)
(defer (os/execute ["rm" "-rf" "--" tempdir] :px)
(os/execute ["git" "clone" "--depth=1" repo-url tempdir] :px)
(bundle/install tempdir name))
name)You could do similar with mercurial, curl + tar, svn, cvs, perforce, etc. However, the only lightweight way to embed into janet is to shell out, so I'm hesitant to include it in Janet itself. |
So Haskell + Cabal seems to use something called PVP, which seems like a slightly different flavor of semantic versioning. As for vendoring, it will not be the only way to do things, but I would encourage it and it needs to be something that could be officially well supported.
The idea being that dependency resolution and fetching sources/binaries over a network sits at a layer above this. |
|
I'm not particularly fond of PvP. I prefer semantic versioning for its simplicity. |
|
@bakpakin wrote:
I agree regarding those should be at a separate layer. I tend to think of package installation as a flow of steps:
Fetch should likely be something along the lines of a dispatcher that sees an URI and finds a URI handler that exists for it and calls it to execute a fetch. Build should probably have some nice "shortcuts" available (maybe in spork or something) but is entirely package defined. Extract probably needs a "what's that file end with" dispatcher. I think having Dependency and conflict resolution is a difficult thing and it'd be nice if there was something like a Having Janet able to manage installing packages out of the box sounds like a pretty good idea, but once someone needs more than that, perhaps a library (spork, install-utils, whatever) to provide for managing such at a higher level, handling all the fetching, dependencies, conflicts, and so forth would be the best of both, especially if that library is self contained with no other dependencies other than Janet so it'd be as simple as fetching it manually and doing The one issue I see is, if Janet has a On another note... something like (when (not (bundle/installed? "package"))
(bundle/install package))Having all this easily available from within Janet, REPL or Script or Project/Package, would likely feel more fluid and more easily approachable. |
|
I think the main idea would be to have the base module handling (steps 3 & 4) in janet itself and keep the other parts in spork. |
|
I'm not too keen on all of spork being preinstalled with janet. It now has a fair number of pieces by various authors at various levels of "completeness" / "stability". I wonder if there is a way to get "just the necessary bits"...or perhaps there could be some other arrangement. Edit 2: Perhaps a jpm-like tool could depend on code in spork but not cause spork to be installed as a library (so like what is possible now with So the tool could still end up as part of an installation of janet but it could remain unaffected for its basic functionality by what happens to live in janet's syspath. I hope I'm making sense here (^^; Perhaps I didn't quite understand what was meant by "janet releases" though... Edit 1: It occurs to me that there may be projects that depend on specific versions of spork. It seems possible that some of these might break if a newer version of spork gets installed when an upgrade to janet occurs. Not sure if this is a valid concern, but FWIW. |
|
Hm, these are some valid concerns, I'm not sure what the best solution would be regarding those. |
|
I'm unsure if spork should be an automatically included "batteries" library that essentially achieves a status as a de-facto part of the Janet standard library. It's starting to sound like package management shouldn't be part of spork, if one's goal is to in essence say "use latest version of package management and/or Janet". Which almost sounds like keeping a jpm of sorts somewhere though it might be different than the jpm of today, being that it relies on things built in to the Janet standard library. @sogaiu wrote:
This would be nice but then package management wouldn't be very REPL-ish aside from low level concerns of uninstall/remove/reinstall. If it makes sense for package management to be done via REPL and for the higher level package management tool depend on spork then spork might need a split between "stable" and "evolving", that or the REPL higher level package management library should intern spork basically (privatize all the symbols or namespace them behind a prefix like Having wrote all this it almost seems like we need many small libraries instead of a few conglomerates but I don't really like that idea either (that's Javascript-esque... there's a library for that for everything). So now we have a better and more difficult idea that might satisfy all the things in the next sections. Versioned imports. If we have a directory structure something like: Then we could let spork be installed by default, the higher level package management tool could pin to a version, a user could request the latest, and unversioned imports from within Janet default to the latest. This means one could do something like With that idea, spork could be included as well as a higher level package management library for the REPL, without clashing with what users need. Whether or not it's a good idea is another question, and even if it's a good idea, it's questionable if it's worth the extra complexity outside of the fact that this saga we're currently participating in will not be the only case where such a need is likely to come up wherein one needs 2 versions of the same library to service tools and service user code. |
|
I don't really plan on spork being released with Janet. If that was the goal, it would just be merged into Janet. However, spork contains a lot of the core functionality of JPM, just better. On windows, it might make sense to have a spork setup option in the installer but ultimately it should be pretty easy to setup manually. Currently we have a Makefile target to try and setup JPM, so something like that might make it in. However, the key idea is that you could have different versions of spork per project because it's just another pacakage in the tree |
|
Is this going to create breaking changes? |
No, this should be purely additive. JPM will still work for the foreseeable future, and won't require any changes to existing projects. Existing packages that use JPM will still work as well, and I see no reason why a project could not support both besides a bit of extra work. There are various levels of integration that we could go for here (should jpm installed packages be visible to As per usual though, to avoid breaking changes, stick with release versions of Janet rather than latest master. |
So I'm open to ideas, but at first glance versioned imports seems like something I'm very much not going to implement. Rather, it would be a versioned install and the import would be something like (bundle/install "/downloads/spork" "spork-1.2.3") # install script dynamically chooses what root directory to install to based on bundle name
(import spork-1.2.3 :as spork)
(require (string "spork-" (calculate-needed-version)))This looks awful to me in most cases, but its implementable via the usual mechanisms. |
|
Perhaps, can project.janet specify something like this? This is the haskell way of specifying dependencies. In ~/.cabal, various versions of spork are available. |
IIUC, at least this could help mitigate the problem of installation of one project clobbering the dependencies of another. As an example, with how things work with jpm currently, suppose:
Assume further that X < Y [1] and Y has changes that are not compatible with some functionality in X. If someone has project A installed and then installs project B, project A may break because version X of spork has been replaced with version Y of spork. With the proposed "versioned install" idea, if various projects make use of the mechanism, it seems like the aforementioned problem could be avoided (or at least lessened) -- at least by projects that make use of the mechanism. [1] Strictly speaking, I don't think one needs any kind of version "number" ordering to express the problem mentioned above. IIUC, all that is necessary is that there are two different versions with some incompatibilities (e.g. imagine the case of two different commits from a particular repository where there are no version "numbers" associated with either commit). As a side note, vendoring can also protect against the issue of disappearing dependencies (e.g. this recent PR). It looks like other communities have "immutable" stores that can address this. May be that's not so important at this stage. |
|
@bakpakin wrote:
It does look awful, but versioned install solves the issue I had in mind. Someone could potentially "monkey-patch" You are pretty spot on that built-in low-level @sogaiu wrote:
If various projects make use of it soon it'll be the canonical thing, and somewhere in there someone's going to make it look much nicer probably. @sogaiu wrote:
The immutable store thing sounds awesome, until going about the implementation thereof. Vendoring is a lot easier at than making an immutable store that supports $favorite-vcs, $archive, $something-phenomenally-ridiculous-but -somehow-necessary-to-support. |
I messed up a bit with terminology above -- probably I should have written something like (mostly) "append-only". A specific example of this might be maven central. Git repositories on GitHub with the default branch being protected are somewhat close but IIUC these can just "go away" if a user removes the repository (or like in the recent xz incident, MS decides to just make something unavailable). May be there was no confusion, but FWIW.
Yeah, probably it is not a simple matter, but I suppose it depends on the specifics a bit regarding how complicated things can get. Over time, depending on popularity, "append only" things seem like they could get costly though because the stored content just keeps growing...and growing (^^; Edit: Sorry if this is getting a bit off-topic. May be it wouldn't be bad to make a list somewhere of related "concerns" (e.g. installed dependency clobbering, dependency disappearance, etc.) though. Possibly it would be helpful for present and future discussions? |
|
Been doing some thinking about dependencies and bundle/install and bundle/uninstall, and I do think we should allow users to prevent breaking dependencies. Essentially, this is something that bundle/uninstall and bundle/install would check about what's currently installed before doing anything.
This does some what require that we distinguish between "incidentally" installed bundles and "directly" installed bundles. (defn bundle/install
...
(def deps (read-deps-data-jdn))
(def missing (seq [d :in deps :unless (bundle/installed? d)] d))
(when (next missing) (errorf "missing required dependencies: %s" (string/join missing ", ")))
...)
(defn bundle/uninstall
...
(def deps (read-installed-deps-data-jdn))
(def dependents (seq [d :in deps :if (bundle/installed-directly? d)] d))
(when (next dependents) (errorf "cannot uninstall due to dependent bundles: %s" (string/join dependents ", ")))
# uninstall my files, then uninstall dependent files
(each d deps
(if (= 1 (bundle-dep-ref-count d))
(bundle/uninstall d)))
This works without needing to know anything about how to fetch dependencies or versions - the constraint is a function of what's in the syspath so we could definitely put it into the EDIT: The concept of directly installed bundles vs. incidentally installed bundles is basically "autoremove" in apt-get, or "orphaned packages" in pacman. |
|
I've been trying to figure out how this:
would be done. Is this a matter of checking existing manifest files? |
I was thinking so, just check that existing bundles are installed first. It could also include a manual "deps" hook where you could also search for system dependencies or even do feature testing a la autoconf. |
|
Thanks for the clarification.
That sounds quite interesting. |
Many janet installs out there are broken or only partially correct, simplifying the setup might help improve this.
To quote @bakpakin :
Any approach bringing a tighter integration with the runtime and package management should reduce possible misconfigurations and make it simpler for people to get started with janet.
There are still a few open questions like:
The text was updated successfully, but these errors were encountered: