crates.io API returns 403

[suzuki-shunsuke]

aqua info

$ aqua info
{
  "version": "2.24.0",
  "commit_hash": "4df269e479ac3364a3430ce77d52572cf01648e5",
  "os": "darwin",
  "arch": "arm64",
  "pwd": "/Users/(USER)/repos/src/github.com/suzuki-shunsuke/tfaction-example",
  "root_dir": "/Users/(USER)/.local/share/aquaproj-aqua",
  "env": {
    "AQUA_GLOBAL_CONFIG": "/Users/(USER)/repos/src/github.com/suzuki-shunsuke/dotfiles/aqua.yaml:/Users/(USER)/repos/src/github.com/aquaproj/aqua-registry/aqua-all.yaml",
    "AQUA_PROGRESS_BAR": "true"
  },
  "config_files": [
    {
      "path": "/Users/(USER)/repos/src/github.com/suzuki-shunsuke/tfaction-example/aqua/aqua.yaml"
    }
  ]
}

Overview

aqua g, aqua gr, and aqua up commands don't work well for cargo package because crates.io API returns 3xx.

https://aquaproj.github.io/docs/reference/registry-config/cargo-package/

How to reproduce

aqua gr crates.io/gitu
aqua gr crates.io/skim

Debug output

No response

Expected behaviour

These commands succeed.

Actual behaviour

$ aqua gr crates.io/gitu
WARN[0000] get a crate metadata by crates.io API         aqua_version=2.24.0 env=darwin/arm64 error="HTTP status code is greater equal than 300" program=aqua
WARN[0000] get a latest version by crates.io API         aqua_version=2.24.0 env=darwin/arm64 error="HTTP status code is greater equal than 300" program=aqua
packages:
  - name: crates.io/gitu
    type: cargo
    crate: gitu

$ aqua gr crates.io/skim
WARN[0000] get a crate metadata by crates.io API         aqua_version=2.24.0 env=darwin/arm64 error="HTTP status code is greater equal than 300" program=aqua
WARN[0000] get a latest version by crates.io API         aqua_version=2.24.0 env=darwin/arm64 error="HTTP status code is greater equal than 300" program=aqua
packages:
  - name: crates.io/skim
    type: cargo
    crate: skim

Note

aquaproj/aqua-registry#20854 (comment)

aqua/pkg/cargo/versions.go

Line 41 in 90b5cc2

versions, _, err := listInstallableVersions(ctx, c.client, fmt.Sprintf("https://crates.io/api/v1/crates/%s/versions", crate))

aqua/pkg/cargo/versions.go

Line 54 in 90b5cc2

payload, _, err := getCrate(ctx, c.client, "https://crates.io/api/v1/crates/"+crate)

aqua/pkg/cargo/versions.go

Lines 88 to 90 in 90b5cc2

	if resp.StatusCode >= 300 { //nolint:gomnd
	return nil, resp.StatusCode, errors.ErrHTTPStatusCodeIsGreaterEqualThan300
	}

[suzuki-shunsuke]

🤔

$ curl -o /dev/null -w '%{http_code}\n' -s https://crates.io/api/v1/crates/skim/versions
200

$ curl -o /dev/null -w '%{http_code}\n' -s https://crates.io/api/v1/crates/skim
200

[suzuki-shunsuke]

I could reproduce the issue.

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
)

func main() {
	if err := core(); err != nil {
		log.Fatal(err)
	}
}

func core() error {
	req, err := http.NewRequest("GET", "https://crates.io/api/v1/crates/skim/versions", nil)
	if err != nil {
		return err
	}
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return err
	}
	defer resp.Body.Close()
	fmt.Println(resp.StatusCode)
	if _, err := io.Copy(os.Stdout, resp.Body); err != nil {
		return err
	}
	return nil
}

$ go run .
403
{"errors":[{"detail":"We are unable to process your request at this time. This usually means that you are in violation of our crawler policy (https://crates.io/policies#crawlers). Please open an issue at https://github.com/rust-lang/crates.io or email help@crates.io and provide the request id 768e05d9-4654-4cd8-96ce-951261259c20"}]}

We are unable to process your request at this time.
This usually means that you are in violation of our crawler policy (https://crates.io/policies#crawlers)

[suzuki-shunsuke]

req.Header.Add("User-Agent", "aqua")

I could resolve the issue by setting a User-Agent header.