oclif promote shows NoSuchKey: The specified key does not exist.

[tlkiong]

I am not sure if this is a bug. However, based on the docs here, I should be able to push to S3 by using the command oclif promote.

However, after some fiddling around and trying to understand what is going wrong due to lack of documentation, I encounter this issue:

    NoSuchKey: The specified key does not exist.
    Code: NoSuchKey

[RodEsp]

Hey @tlkiong, if you're using Oclif v2 you'll need to use oclif upload first to upload your CLI to S3 and then you can use oclif promote to promote the CLI to a specific release channel.

We are currently working on updating and improving the docs. Sorry for the hassle.
Let me know if that helps!

[creatzor]

I get this error because it attempts to copy <app-name>-v<version>-<sha>-<target>-buildmanifest however, the upload command never uploaded that file.

[mattbishop]

Same failure for me at the buildmanifest file. I would expect oclif pack tarballs to generate that file, but it does not.

I am following the directions here: https://oclif.io/docs/releasing

[mattbishop]

It looks like oclif expects the *-buildmanifest file to be in the S3 bucket already?

oclif/test/integration/publish.test.ts

Lines 52 to 56 in db03bd0

	const list = await aws.s3.listObjects({Bucket: bucket, Prefix: `${basePrefix}/${path}`})
	const manifestFile = list.Contents?.map(listObject => listObject.Key).find(f => f!.includes(target) && f!.endsWith('-buildmanifest'))
	if (!manifestFile) {
	throw new Error(`could not find a buildmanifest file for target ${target}`)
	}

If so, how do I create this file? What are its contents? Is there an oclif command to generate and push? Feels like there is a missing step in the instructions for tarball releases.

[mattbishop]

Just realized this was a test file, but still the question holds--how does one create a {arch}-buildmanifest file?

[DaanSchoukens]

Wondering the same thing.
I'm trying to upload the tarballs to a raw nexus repository, but I don't find documentation on the directory structure and necessary files (apart from the tarballs) to enable autoupdating.

[knvpk]

Facing same problem. is this addressed ?

[RodEsp]

Hey @mattbishop, @DaanSchoukens, @creatzor. Looking into this right now.

I think we're missing a check in the upload and promote commands that exists for pack.
Looks like if there's no s3.host in the oclif config the -buildmanifest files won't be built. See here.

Do you have oclif.update.s3.host set in your package.json as described in the Autoupdater docs?

[mattbishop]

Thank you @RodEsp I hadn't set that value and am using Backblaze. That property is required for the upload command to work. The host property actually needs to be a URL, like https://s3.other-service.com instead of just the host name.

Also, an undocumented requirement for non-AWS services is that the environment variable AWS_S3_ENDPOINT must be set to the host name, not a URL.

Thanks again for your help.

[RodEsp]

Thanks for the thorough follow up @mattbishop! I'll make sure we get that added to the docs soon.

[KASOGIT]

Hi @RodEsp , sorry to bother you so long after this issue have been written but i still have the problem trying to promote a macos build.

My config:

  "oclif": {
    "bin": "pos-cli",
    "dirname": "pos-cli",
    "commands": "./dist/commands",
    "macos": {
      "identifier": "com.hero.pos-cli",
      "host": "https://xxxx.s3.eu-west-3.amazonaws.com"
    },
    "update": {
      "s3": {
        "bucket": "prod-hero-post-transfer-cli-releases-2da7095475674",
        "host": "https://xxxx.s3.eu-west-3.amazonaws.com"
      }
    },
    "plugins": [
      "@oclif/plugin-help",
      "@oclif/plugin-plugins",
      "@oclif/plugin-update"
    ],
    "topicSeparator": " ",
    "topics": {
      "deploy": {
        "description": "deploy the new version of the app"
      },
      "package": {
        "description": "package the app"
      }
    }
  },

I'm running command in this order:

./node_modules/.bin/oclif pack macos
./node_modules/.bin/oclif upload macos
./node_modules/.bin/oclif promote --macos --sha=$(git rev-parse --short HEAD) --version=$(node -p "require('./package.json').version")

Getting:

Promoting buildmanifests & unversioned tarballs to stable
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-linux-x64-buildmanifest to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-linux-x64-buildmanifest
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-linux-x64.tar.gz to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-linux-x64.tar.gz
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4[16](https://github.com/Hero-payments/mono/actions/runs/4562908601/jobs/8050724915#step:9:17)5d/pos-cli-v0.0.0-eb4165d-linux-arm-buildmanifest to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-linux-arm-buildmanifest
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-linux-arm.tar.gz to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-linux-arm.tar.gz
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-win32-x64-buildmanifest to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-win32-x64-buildmanifest
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-win32-x64.tar.gz to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-win32-x64.tar.gz
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-win32-x86-buildmanifest to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-win32-x86-buildmanifest
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-win32-x86.tar.gz to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-win32-x86.tar.gz
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-darwin-x64-buildmanifest to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-darwin-x64-buildmanifest
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-darwin-x64.tar.gz to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-darwin-x64.tar.gz
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-darwin-arm64-buildmanifest to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-darwin-arm64-buildmanifest
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-darwin-arm64.tar.gz to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-darwin-arm64.tar.gz
Promoting macos pkgs to stable
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-x64.pkg to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-x64.pkg
oclif: s3:copyObject from s3://prod-hero-post-transfer-cli-releases-2da7095475674/versions/0.0.0/eb4165d/pos-cli-v0.0.0-eb4165d-arm64.pkg to s3://prod-hero-post-transfer-cli-releases-2da7095475674/channels/stable/pos-cli-arm64.pkg
    NoSuchKey: The specified key does not exist.
    Code: NoSuchKey

Any idea of what happening here and why no manifest are being generated during packing step ?

[RodEsp]

Sorry @KASOGIT but I wouldn't know off the top of my head. Did you try what was mentioned in the comments above already about setting the oclif properties and environment variables?

[KASOGIT]

Yeap i tried everything listed above but i couldn't succeed to generate buildmanifest files. Do you have an working exemple for s3 that generate those files with the command oclif pack macos ?

I always end up with *.pkg but no buildmanifest

[RodEsp]

I don't have a working example unfortunately. You might ask @mdonnalley, he might be able to provide you with one.

[KASOGIT]

Thanks @RodEsp ! It would be awesome if you can provide me one @mdonnalley or if you already know what i have to do in order to generate those buildmanifest files

[KASOGIT]

Hi again @mdonnalley, did u have the opportunity to take a look at my problem ?

[ryandagg]

We have a similar issue, and were able to track down which file is actually missing. This is not straightforward because all the copies appear to happen in parallel and the logging only tells you when the process starts.
This file does not exist in S3:
"oclif: s3:copyObject from s3://***/versions/8.0.4/96472c6/apt/heroku_8.0.4.96472c6-1_i386.deb to ..."

oclif upload:deb does upload files for armel such as "cli/dist/deb/heroku_8.0.3.ddf4a4b-1_armel.deb", but nothing for i386

I'm not sure how oclif pack deb decides targets, but it's not in line with oclif upload deb. I'm not sure if this is a bug or a configuration issue.

[ryandagg]

I spent a good chunk of time looking into this yesterday, and I can find no way to pack/create files ending in _i386.
It looks like this line should be deleted, or caught to prevent the job from failing.
https://github.com/oclif/oclif/blob/main/src/commands/promote.ts#L155

[cristiand391]

Hey all, @ryandagg and I were looking at this failure in the CLI promote pipeline, here's what we found:

1. oclif pack:deb generates deb pkgs for the targets defined in buildConfig.targets:
   https://github.com/oclif/oclif/blob/main/src/commands/pack/deb.ts#L94

these targets can be set by users in the CLI pjson (pjson.oclif.update.node.targets), and will default to these if the previous object is empty:
https://github.com/oclif/oclif/blob/main/src/tarballs/config.ts#L13-L20

So, pack:deb will not generate a deb pkg for linux-x86. But even if you set that target manually in your pjson, oclif will fail to build the pkg because node dropped 32bits build for linux a long time ago:
nodejs/build#885

2. oclif upload:deb tries to upload the i386 pkg:
   

   oclif/src/commands/upload/deb.ts

   Line 54 in f483d1c

   uploadDeb('i386'),

but skips it here because the file doesn't exists:

oclif/src/commands/upload/deb.ts

Line 47 in f483d1c

if (fs.existsSync(dist(deb))) await Promise.all([upload(deb), uploadWorkaround(deb)])

3. oclif promote:deb picks targets the same way pack:deb does BUT you can also specify them via the --targets flag:
   

   oclif/src/commands/promote.ts

   Line 31 in f483d1c

   const buildConfig = await Tarballs.buildConfig(flags.root, {targets: flags?.targets?.split(',')})

   
   

   oclif/src/commands/promote.ts

   Line 20 in f483d1c

   targets: Flags.string({char: 't', description: 'comma-separated targets to promote (e.g.: linux-arm,win32-x64)'}),

the odd thing is that when promoting the deb pkgs it just hardcodes the i386 target (which again, hasn't worked for a very long time now)

oclif/src/commands/promote.ts

Lines 155 to 156 in f483d1c

	templateShortKey('deb', {bin: config.bin, versionShaRevision: debVersion(buildConfig), arch: 'amd64' as any}),
	templateShortKey('deb', {bin: config.bin, versionShaRevision: debVersion(buildConfig), arch: 'i386' as any}),

Another issue with this is that promote:deb can never promote the arm deb pkg that pack:deb builds.

[tuler]

2. oclif upload:deb tries to upload the i386 pkg:
   

   oclif/src/commands/upload/deb.ts

   Line 54 in f483d1c

   uploadDeb('i386'),

My issue is that oclif upload:deb does not upload the armel pkg produced by oclif pack:deb.
Any reason why?

[cristiand391]

Sorry @tuler, can you confirm oclif builds a deb installer for arm? Ask because our CLI (Salesforce CLI) don't publish deb installers so we don't know the current state of it.

If so, seems it should be simple to fix that in upload:deb

[tuler]

I can confirm the arm package is produced by oclif pack:deb, it just is not published. I do think it's an easy fix, probably just add it here.

[omidasadpour]

I can confirm the arm package is produced by oclif pack:deb, it just is not published. I do think it's an easy fix, probably just add it here.

Any updates ?

[git2gus]

This issue has been linked to a new work item: W-13796754

[cristiand391]

@tuler @oap75 this was fixed a few weeks ago, make sure to update to latest oclif to get the fix.
#1158