zig package manager

[photex]

Howdy!

With the introduction of the new package manager (build.zig.zon) I was interested in experimenting with a way to add support to zig-gamedev so that individual libraries can be referred to by other projects, rather than vendoring zig-gamedev as a submodule.

But before I looked too closely I wanted to make sure that this isn't in conflict with your goals for the project. Maybe we can discuss the possibility here and I can put some concrete PRs together from it.

It might seem too early to do this, but my rough thoughts on the topic are:

• It would be ideal to setup Github actions which create 'releases' for each library.
  • Releases are just tarballs with all relevant content resolved (no need for git or lfs downstream)
  • This could be based on tags, but a rolling 'HEAD' release whenever there is a commit is possible (I think)
• Downstream projects could select the release they're interested in and include them via their build.zig.zon.
  • The notes for the release could provide snippets which show the correct entries to add.
• Each library would have their own build.zig.zon generated to refer to their inter-dependencies correctly
  • Details of this particular point are uncertain to me still. For example, can the contents of these files be totally generated?

Alternatively, it might instead be worth starting out with a single broad stroke, and have zig-gamedev itself be a single package that we depend on instead.

Some questions I have, but probably can't answer until I started:

• Does "packaging" any part of zig-gamedev interact badly with the existing approach you've taken with your zig build setup?
• Is it important to keep package versions incremented (does zig build make cache decisions based on package versions or just hashes)?
  • Looking at other examples (https://github.com/andrewrk/ffmpeg/blob/main/build.zig.zon) I only see archive urls and hashes.

[michal-z]

Hi!

Sorry for late reply. Your plan sounds great and doing this work would be much appreciated. I think we could start with one library and see how it goes.

We don't strictly need version numbers - hashes would be fine.

[photex]

Awesome! Is zmath a good choice to start with?

[michal-z]

Awesome! Is zmath a good choice to start with?

Yes!

[Game4all]

Hey,

Not to hurry anyone or things up but has there been any progress on packaging the project libs ?

In any case thanks for the awesome and constant work on these libs 🚀

Cheers

[photex]

Hi, sorry, there hasn't been much progress beyond simple tests. I just moved recently which tossed all my spare time into a tizzy. I'm hoping to get back to this soon though.

[Game4all]

Thanks for the heads up & no worries :) I'll look into that in the following days and will come back here if I ever come up with something usable with Github Actions

---

Another but way more involved direction for packaging individual libs would be to move them under their own repository mirrored from the contents of this monorepo, allowing them to be very easily consumed (just grab the tar for a specific commit / tag). This would however require a lot of plumbing in the build files. Here are my 2 cents.

[photex]

That solves one of the bigger question marks. My packaging script has to calculate hashes for archives and to build the archive the generated build.zig.zon is a part of that.

So the script has to make a little tree to execute the plan correctly.

That’s where I left off before my move.

[Game4all]

re: came up with a workflow which packs and uploads libs as release assets upon pushing a new git tag (or a release) (branch: https://github.com/Game4all/zig-gamedev/tree/packaging-script). This is currently at the same point as your experiments iirc from what I glanced over. Next step would be to modify build files to expose public modules for consumption as a 'package'.

[hazeycode]

All libs now have a build.zig.zon file. Each lib should now at least be somewhat consumable by a local path. However note that some libs are not self-contained, some have a dependency on system-sdk (maybe we should make this a package also and do the necessary build.zig fixes for libs that consume it), and some depend on other libs.

Re publishing packages via GitHub actions. I think this a good way to go. But the details require some thought. Like, where should they be hosted? We want to keep the tarballs out of the git tree. Git LFS is an option but it would increase download size of repo checkouts with no benefit to users of the git repo. Also it could use up the storage budget quickly. We should estimate storage requirements and evaluate different options and costs involved.

[Game4all]

All libs now have a build.zig.zon file. Each lib should now at least be somewhat consumable by a local path. However note that some libs are not self-contained, some have a dependency on system-sdk (maybe we should make this a package also and do the necessary build.zig fixes for libs that consume it), and some depend on other libs.

Since a number of libraries depend on this, making this a dependable package sounds good.

Re publishing packages via GitHub actions. I think this a good way to go. But the details require some thought. Like, where should they be hosted? We want to keep the tarballs out of the git tree. Git LFS is an option but it would increase download size of repo checkouts with no benefit to users of the git repo. Also it could use up the storage budget quickly. We should estimate storage requirements and evaluate different options and costs involved.

Package tarballs could be stored as github release assets, as there are no particular restrictions on artifacts apart from a max file size (2GB) as per github support. Another option to consider, which incurs a non-negligible cost for maintenance, would be to move each package to its own git repo and use tags for release (now that zig can fetch git repos, including commit and tags), but yeah, maintaining 16+ git repos without automation sounds like hell. I'm bringing this option up here as there has been a mention of moving the repo under an organization recently.

[hazeycode]

system-sdk is now also a package: 9c9e475

[hazeycode]

Quick update on this

• Recent build system overhaul resolved some issues when consuming libs using package manager
• Hosting is ready to go. Test: https://zig-gamedev-pkgs.ams3.cdn.digitaloceanspaces.com/zmath-0.9.6.tar.gz

So here's what's left to do:

• A fix or workaround for this issue that is preventing published packages from being consumed
• Automate fixing package interdependencies to published urls instead of local paths
• Publish all the packages!

[zxubian]

A fix or workaround for ziglang/zig#17779 that is preventing published packages from being consumed

Looks like this was resolved by ziglang/zig#19111

[hazeycode]

Looks like this was resolved by ziglang/zig#19111

Can confirm the issue is now resolved.

[terraquad]

Idea on how you could implement this:

You just convert the entire monorepo into a zig package and then you expose some options for choosing the packages you need.
Mach core also does this.
Example:

build.zig.zon:

.{
    .name: "example_game",
    .version: "0.1.0",
    .minimum_zig_version: "0.12.0",
    .dependencies: .{
        .zig_gamedev = .{
            .url = "https://api.github.com/repos/zig-gamedev/zig-gamedev/tarball/<commit-hash>",
            .hash = "<some-hash>",
        },
    },
    "paths": .{
        "",
    },
}

build.zig:

const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "example_game",
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });
    b.installArtifact(exe);

    const zgame = b.dependency("zig_gamedev", .{
        // Package dependencies are automatically set to true as well
        // type of dependencies is `?anytype` so the user can pass build options to the underlying dependency
        .zglfw = .{},
        .zgpu = .{},
    });
    exe.root_module.addImport("zglfw", zgame.zglfw.module("root"));
    exe.linkLibrary(zgame.zglfw.artifact("glfw"));
    @import("zig_gamedev").zgpu.addLibraryPathsTo(exe);
    exe.root_module.addImport("zgpu", zgame.zgpu.module("root"));
    exe.linkLibrary(zgame.zgpu.artifact("zdawn"));
}