Go Open Registry

Crates.io cargo registry Golang implementation using amazing Gin web framework https://github.com/gin-gonic/gin

Based on https://doc.rust-lang.org/cargo/reference/registries.html

Limitations

• Yank feature not implemented yet (https://doc.rust-lang.org/cargo/reference/registries.html#yank)

Configure with environment variables

A shortlist, description bellow

• GIN_MODE
• PORT
• CARGO_API_URL
• GIT_REPO_URL
• GIT_REPO_PATH
• GIT_REPO_USERNAME
• GIT_REPO_PASSWORD
• GIT_REPO_EMAIL
• MONGODB_URI
• MONGO_CONNECTION_TIMEOUT
• STORAGE_TYPE
• LOCAL_STORAGE_PATH
• ARTIFACTORY_URL
• ARTIFACTORY_LOGIN
• ARTIFACTORY_PASSWORD
• ARTIFACTORY_REPO_NAME
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION
• AWS_S3_BUCKET_NAME
• AWS_S3_ENDPOINT

Generic

STORAGE_TYPE

• Description

  Storage type is used to choose backend storage to upload/download binary crates files
  It can be one of choice:
  .. local - local filesystem folder
  .. s3 - AWS S3 bucket, or S3 API compatible system (Minio, Ceph)
  .. artifactory - JFrog Artifactory system

• Example

  STORAGE_TYPE=s3

• Default

  STORAGE_TYPE=local

GIN

GIN_MODE

• Description

  Gin Application working mode
  https://github.com/gin-gonic/gin/blob/master/mode.go#L52
  There is some log verbosity applied based on the mode.
  Debug is more verbose.

• Example

  GIN_MODE=release

• Default

  GIN_MODE=debug

PORT

• Description

  HTTP port to application listen on
  

• Example

  PORT=8000

• Default

  PORT=8000

Cargo config

Cargo config file stored in a git registry generated automatically from environment variables on start If you change variables, the file will be updated and pushed to the repository

https://doc.rust-lang.org/cargo/reference/registries.html#index-format

dl url made from api with concatenation /api/v1/crates in internal/config/config.go:60

CARGO_API_URL

• Description

  This is the base URL for the web API

• Example

  CARGO_API_URL=http://my-registry-api:3000

• Default

  CARGO_API_URL=http://localhost:8000

Git

GIT_REPO_URL

• Description

  Git repository to store index files.
  Cargo use it to search particular package and version.
  Cargo configuration example

• Example

  GIT_REPO_URL=https://github.com/boskiv/open-registry-index

• Default

  Empty

GIT_REPO_PATH

• Description

  Temporary directory to clone repo.
  Application use it to commit and push cargo package information.
  

• Example

  GIT_REPO_PATH=/data/gitRepo

• Default

  GIT_REPO_PATH=tmpGit

GIT_REPO_USERNAME

• Description

  Login to work with git repo.
  

• Example

  GIT_REPO_USERNAME=boskiv

• Default

  Empty

GIT_REPO_PASSWORD

• Description

  Password to work with git repo.
  

• Example

  GIT_REPO_PASSWORD=123123123

• Default

  Empty

GIT_REPO_EMAIL

• Description

  Email used in commit signature.
  

• Example

  GIT_REPO_EMAIL=crates@company.org

• Default

  Empty

Mongo

Mongo database used to store package name and version info.

Multiple field index used to control version uniqueness.

https://docs.mongodb.com/manual/core/index-unique

Keys: bson.M{
  "name": 1,
  "version": 1,
},
Options: options.Index().SetUnique(true)

MONGODB_URI

• Description

  URI formats for defining connections between applications and MongoDB instances in the official MongoDB drivers.
  https://docs.mongodb.com/manual/reference/connection-string

• Example

  MONGODB_URI=mongodb://mongo:27017

• Default

  MONGODB_URI=mongodb://127.0.0.1:27017

MONGO_CONNECTION_TIMEOUT

• Description

  Timeout to check mongo availability
  Application wil exit with code 1, if timeout fires.

• Example

  MONGO_CONNECTION_TIMEOUT=15

• Default

  MONGO_CONNECTION_TIMEOUT=5

Local storage

Be sure that use set STORAGE_TYPE=local to use or remove that env, so this will apply by default.

LOCAL_STORAGE_PATH

• Description

  Filesystem path to store uploaded crates and get it for download.
  If the path does not exist, the application will try to create it.
  You can use docker mounted volumes, shared volumes and network filesystem volumes, to sure about data persistence.
  However, it's not production recommended storage because of maintenance, backup, and support issues.
  So use it for testing only
  

• Example

  LOCAL_STORAGE_PATH=/data/crates

• Default

  LOCAL_STORAGE_PATH=upload

Artifactory storage

Be sure that use set STORAGE_TYPE=artifactory to use it.

ARTIFACTORY_URL

• Description

  Path to artifactory API.
  Should include schema (http/https) and full path to API.
  You can find this in the description at Set Me Up dialog

• Example

  ARTIFACTORY_URL=http://localhost:8081/artifactory

• Default

  Empty

ARTIFACTORY_LOGIN

• Description

  Login to artifactory with write access for binary repository.
  

• Example

  ARTIFACTORY_LOGIN=bot

• Default

  Empty

ARTIFACTORY_PASSWORD

• Description

  Password to access artifactory with login provided in ARTIFACTORY_LOGIN.
  

• Example

  ARTIFACTORY_PASSWORD=password

• Default

  Empty

ARTIFACTORY_REPO_NAME

• Description

  Generic binary repository name created in artifactory.
  https://www.jfrog.com/confluence/display/RTF/Configuring+Repositories

• Example

  ARTIFACTORY_REPO_NAME=crates

• Default

  Empty

S3 storage

Most of variables are common used.

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

AWS_ACCESS_KEY_ID

• Description

  Specifies an AWS access key associated with an IAM user or role.
  

• Example

  AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE

• Default

  Empty

AWS_SECRET_ACCESS_KEY

• Description

  Specifies the secret key associated with the access key. This is essentially the "password" for the access key.

• Example

  AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

• Default

  Empty

AWS_DEFAULT_REGION

• Description

  Specifies the AWS Region to send the request to.

• Example

  AWS_DEFAULT_REGION=us-west-2

• Default

  Empty

AWS_S3_BUCKET_NAME

• Description

  Bucket name to store crate files. It will be created automatically if not exist.

• Example

  AWS_S3_BUCKET_NAME=crates

• Default

  Empty

AWS_S3_USE_SSL

• Description

  Should application use SSL protocol when connect to S3 API

• Example

  AWS_S3_USE_SSL=false

• Default

  AWS_S3_USE_SSL=true

AWS_S3_ENDPOINT

• Description

  S3 API Endpoint to upload file.
  Application use minio sdk to work with S3 like APIs you should put here url without schema f.e. play.minio.io, but you should to put schema in AWS_S3_USE_SSL as boolean param

• Example

  AWS_S3_ENDPOINT=localhost:9000

• Default

  AWS_S3_ENDPOINT=s3

Run

Prerequisite

Make a accessible git repository with config.json

https://doc.rust-lang.org/cargo/reference/registries.html#index-format

{
    "dl": "http://localhost:8000/api/v1/crates",
    "api": "http://localhost:8000"
}

The keys are:
dl: This is the URL for downloading crates listed in the index.
api: This is the base URL for the web API.

Developer

Run this configuration just to have third-party services

• mongo
• minio
• artifactory

Steps

• docker-compose run -f compose/developer.yaml
• go build cmd/go-open-registry
• ./go-open-registry

Test with local storage

• docker-compose up -d -f compose/local.yaml
• open http://localhost:3000 for Gogs and create an account crates and repository crates-index. Dont forget to check button Initialize repo with readme file. Empty repo will raise an error when registry starts.
• run docker-compose up -d -f compose/artifactory.yaml second time to initialize registry
• config.json file will be generated automatically.
• use http://localhost:8000 to Configure cargo

Test with artifactory

• docker-compose up -d -f compose/artifactory.yaml
• open http://localhost:8081 for Artifactory, create a login bot and password password.
• create a repository named crates, and give user bot a write access to this repo.
• open http://localhost:3000 for Gogs and create an account crates and repository crates-index. Dont forget to check button Initialize repo with readme file. Empty repo will raise an error when registry starts.
• run docker-compose up -d -f compose/artifactory.yaml second time to initialize registry
• config.json file will be generated automatically.
• use http://localhost:8000 to Configure cargo

Test with s3 storage

• docker-compose up -d -f compose/minio.yaml
• open http://localhost:9000 for Minio and login with minio as user and minio123 as password.
• bucket crates will be created automatically
• open http://localhost:3000 for Gogs and create an account crates and repository crates-index. Dont forget to check button Initialize repo with readme file. Empty repo will raise an error when registry starts.
• run docker-compose up -d -f compose/artifactory.yaml second time to initialize registry
• config.json file will be generated automatically.
• use http://localhost:8000 to Configure cargo

Cargo configuration example

Publishing package

• setup registry .cargo/config file with

[registries.open-registry]
index = "https://github.com/boskiv/open-registry-index.git"
token = ""

• setup Cargo.toml file of your package with publish settings

[package]
publish = ["open-registry"]

• publish your package

cargo publish --registry open-registry

Getting package

• setup registry .cargo/config file with

[registries.open-registry]
index = "https://github.com/boskiv/open-registry-index.git"
token = ""

• Setup dependency in Cargo.toml file

[dependencies]
bo-helper = { version = "0.1", registry = "open-registry" }

• Run build

cargo update
cargo build