Merge pull request #90 from gadget-inc/cached-prep

Small changes to prep for caching daemon

Contributing.md

@@ -0,0 +1,190 @@
+## Contributing to Dateilager
+
+TL;DR:
+
+```bash
+# allow nix to do its thing
+direnv allow
+
+# in one tab, run the background services
+dev
+
+# in another, run the migrations, then start developing!
+make migrate
+make server
+```
+
+### Requirements
+
+Dateilager has a fully managed development environment ready for use via `nix`. To use this environment, ensure you have nix and `direnv` installed, and then run `direnv allow` to have the environment build itself.
+
+#### Manual environment
+
+If you don't want to use nix, you can install the components manually:
+
+- Go 1.22
+- Postgresql
+- Node v20
+- mkcert (https://github.com/FiloSottile/mkcert)
+- Protobuf Compiler (https://grpc.io/docs/protoc-installation/)
+
+Ensure that you have a working [Go development environment](https://golang.org/doc/install) and that you are running at least Go 1.22. You will also require `npm`.
+
+### Run background services
+
+DL requires a running postgres instance in the background for development. The `dev` script will setup and run this database.
+
+Run
+
+```bash
+dev
+```
+
+The default Postgres host is `127.0.0.1` you can override it by exporting `DB_HOST`.
+
+```bash
+export DB_HOST=10.0.0.1
+```
+
+### Setup VSCode (Optional)
+
+We recommend using VSCode for development, and there's an example settings file at `.vscode/settings.example.json` to get started with:
+
+```
+cp .vscode/settings.example.json .vscode/settings.json
+```
+
+### Build
+
+This will build the server and client executables along with the `*.proto` files.
+
+```bash
+make build
+```
+
+### Run migrations
+
+Assuming Postgres is running, you can setup the database using the migrate tool:
+
+```
+make migrate
+```
+
+## API Testing
+
+Ensure there is a Postgres database named `dl_tests`. These tests will write to a real database instance
+but all writes will be rolled back as every test runs within its own transaction.
+
+```bash
+make test
+```
+
+## Local
+
+Set up a local development environment by resetting the local database, and building an example dataset
+within the `input/` directory.
+
+```bash
+make setup-local
+```
+
+Then launch the server process, by default it will run on port 5051.
+
+```
+make server
+```
+
+Our input directory contains 3 folders, all meant as different versions of the same project. There are also
+diff files listing which files have changed between them.
+
+We can now load these into the database.
+
+```bash
+make client-update
+```
+
+And then use the client app to read all files within the latest version.
+
+```bash
+make client-get
+```
+
+You can also filter the results with a prefix search.
+
+```bash
+make client-get prefix=n1
+```
+
+Or filter for a specific version.
+
+```bash
+make client-get to_version=1
+```
+
+If you want to rebuild an entire project's directory locally, use the `rebuild` command.
+
+```bash
+make client-rebuild to_version=3 dir=./rebuild
+ls -lah ./rebuild
+```
+
+## Javascript Client
+
+Ensure a server is running with `make server`.
+
+Import the `DateiLagerGrpcClient` from the module in `js/` and use it to query objects:
+
+```javascript
+const client = new DateiLagerGrpcClient("localhost", 5051);
+
+// Get a single object
+const object = await client.getObject(1n, "a");
+console.log("[getObject] path: " + object.path);
+console.log("[getObject] content:\n" + object.content);
+
+// List all objects
+for await (const object of client.listObjects(1n, "")) {
+  console.log("[listObjects] path: " + object.path);
+  console.log("[listObjects] content:\n" + object.content);
+}
+```
+
+Update objects and await the successful commit of a new version:
+
+```javascript
+const stream = client.updateObjects(1n);
+
+stream.send({
+  path: "a",
+  mode: 0o755,
+  content: "foo bar",
+});
+
+const version = await stream.complete();
+console.log("[updateObject] version: " + version);
+```
+
+## Release
+
+When you're ready to release a new version, perform the following steps. Ensure you are on doing these steps on the `main` branch
+
+1. Update the version in `default.nix`
+2. Update the version in `js/package.json`
+3. Update the version in `js/package-lock.json` (run `cd js && npm install`)
+4. Commit the changes (e.g. `git commit -am "Bump version to 0.0.x"`)
+5. Push the changes upstream to main (e.g. `git push origin HEAD`)
+
+New versions are released and hosted on Github. (https://github.com/gadget-inc/dateilager/releases)
+
+Create a new tag and push it to GitHub, GoReleaser will handle building it.
+
+```bash
+git tag v0.0.x
+git push origin v0.0.x
+```
+
+We also need to build the server docker image and push it to Gadget's container registry.
+
+```bash
+make upload-container-image version=0.0.x
+```

Readme.md

@@ -14,160 +14,6 @@ Dateilager is used in production to power https://gadget.dev to sync the filesys
 
 You can read more about DateiLager's design in the `docs/` directory. A good place to start is the [core concepts file](./docs/core.md).
 
-## Setup
+## Contributing
 
-### Requirements
-
-- Go 1.19
-- Postgresql
-- Node v16
-- mkcert (https://github.com/FiloSottile/mkcert)
-- Protobuf Compiler (https://grpc.io/docs/protoc-installation/)
-
-Create a Postgres database named `dl`. The default Postgres host is `127.0.0.1` you can override it by
-exporting `DB_HOST`.
-
-```bash
-$ export DB_HOST=10.0.0.1
-```
-
-### Install Toolchains
-
-Ensure that you have a working [Go development environment](https://golang.org/doc/install) and that you are running at least Go 1.19.
-
-You will also require `npm`.
-
-### Setup VSCode (Optional)
-
-We recommend using VSCode for development, and there's an example settings file at `.vscode/settings.example.json` to get started with:
-
-```
-cp .vscode/settings.example.json .vscode/settings.json
-```
-
-### Build
-
-This will build the server and client executables along with the `*.proto` files.
-
-```bash
-$ make build
-```
-
-## API Testing
-
-Ensure there is a Postgres database named `dl_tests`. These tests will write to a real database instance
-but all writes will be rolled back as every test runs within its own transaction.
-
-```bash
-$ make test
-```
-
-## Local
-
-Set up a local development environment by resetting the local database, and building an example dataset
-within the `input/` directory.
-
-```bash
-$ make setup-local
-```
-
-Then launch the server process, by default it will run on port 5051.
-
-```
-$ make server
-```
-
-Our input directory contains 3 folders, all meant as different versions of the same project. There are also
-diff files listing which files have changed between them.
-
-We can now load these into the database.
-
-```bash
-$ make client-update
-```
-
-And then use the client app to read all files within the latest version.
-
-```bash
-$ make client-get
-```
-
-You can also filter the results with a prefix search.
-
-```bash
-$ make client-get prefix=n1
-```
-
-Or filter for a specific version.
-
-```bash
-$ make client-get to_version=1
-```
-
-If you want to rebuild an entire project's directory locally, use the `rebuild` command.
-
-```bash
-$ make client-rebuild to_version=3 dir=./rebuild
-$ ls -lah ./rebuild
-```
-
-## Javascript Client
-
-Ensure a server is running with `make server`.
-
-Import the `DateiLagerGrpcClient` from the module in `js/` and use it to query objects:
-
-```javascript
-const client = new DateiLagerGrpcClient("localhost", 5051);
-
-// Get a single object
-const object = await client.getObject(1n, "a");
-console.log("[getObject] path: " + object.path);
-console.log("[getObject] content:\n" + object.content);
-
-// List all objects
-for await (const object of client.listObjects(1n, "")) {
-  console.log("[listObjects] path: " + object.path);
-  console.log("[listObjects] content:\n" + object.content);
-}
-```
-
-Update objects and await the successful commit of a new version:
-
-```javascript
-const stream = client.updateObjects(1n);
-
-stream.send({
-  path: "a",
-  mode: 0o755,
-  content: "foo bar",
-});
-
-const version = await stream.complete();
-console.log("[updateObject] version: " + version);
-```
-
-## Release
-
-When you're ready to release a new version, perform the following steps. Ensure you are on doing these steps on the `main` branch
-
-1. Update the version in `default.nix`
-2. Update the version in `js/package.json`
-3. Update the version in `js/package-lock.json` (run `cd js && npm install`)
-4. Commit the changes (e.g. `git commit -am "Bump version to 0.0.x"`)
-5. Push the changes upstream to main (e.g. `git push origin HEAD`)
-
-New versions are released and hosted on Github. (https://github.com/gadget-inc/dateilager/releases)
-
-Create a new tag and push it to GitHub, GoReleaser will handle building it.
-
-```bash
-$ git tag v0.0.x
-$ git push origin v0.0.x
-```
-
-We also need to build the server docker image and push it to Gadget's container registry.
-
-```bash
-$ make upload-container-image version=0.0.x
-```
+See the development documentation in [Contributing.md](./Contributing.md)

internal/files/writer.go

@@ -90,7 +90,7 @@ func writeObject(rootDir string, cacheObjectsDir string, reader *db.TarReader, h
 			return err
 		}
 		hashHex := hex.EncodeToString(content)
-		return hardlinkDir(filepath.Join(cacheObjectsDir, hashHex, header.Name), path)
+		return HardlinkDir(filepath.Join(cacheObjectsDir, hashHex, header.Name), path)
 
 	case tar.TypeReg:
 		dir := filepath.Dir(path)
@@ -193,7 +193,7 @@ func makeSymlink(oldname, newname string) error {
 	return nil
 }
 
-func hardlinkDir(olddir, newdir string) error {
+func HardlinkDir(olddir, newdir string) error {
 	if fileExists(newdir) {
 		err := os.RemoveAll(newdir)
 		if err != nil {
@@ -203,7 +203,7 @@ func hardlinkDir(olddir, newdir string) error {
 
 	return filepath.Walk(olddir, func(oldpath string, info fs.FileInfo, err error) error {
 		if err != nil {
-			return err
+			return fmt.Errorf("failed to walk dir: %v, %w", info, err)
 		}
 
 		newpath := filepath.Join(newdir, strings.TrimPrefix(oldpath, olddir))

pkg/cli/cache.go

@@ -18,7 +18,7 @@ func NewCmdGetCache() *cobra.Command {
 			ctx := cmd.Context()
 			c := client.FromContext(ctx)
 
-			version, err := c.GetCache(ctx, path)
+			version, _, err := c.GetCache(ctx, path)
 			if err != nil {
 				return err
 			}