Add push and pull support using cnab-to-oci #681

radu-matei · 2019-03-27T10:55:41Z

This PR is the very first attempt to make things build. It is a sanity check and a reference for the all dependency versions we need in order to have Duffle and cnab-to-oci compile, and it uses my forks for cnab-to-oci, as well as a CNAB Go package extracted from Duffle in order to avoid circular dependencies.

It also implements naive push and pull commands, to make sure everything works together -- see below.

Obviously, this should not be merged in its current state.

radu:deislabs$ duffle push bundle.json -t docker.io/radumatei/duffle-push-test3
Starting to copy image cnab/helloworld:0.1.1...
Completed image cnab/helloworld:0.1.1 copy
WARN[0007] reference for unknown type: application/vnd.cnab.config.v1+json 
Pushed successfully, with digest "sha256:0ad49bc2506ed60a4f8de90703b84fe3c356b1ca0e526fe7200c3d95c4457826"

radu:deislabs$ duffle pull docker.io/radumatei/duffle-push-test3@sha256:0ad49bc2506ed60a4f8de90703b84fe3c356b1ca0e526fe7200c3d95c4457826 -o test3.json
radu:deislabs$ cat test3.json 
{
	"name": "duffle-puhs-test",
	"version": "0.1.1",
	"description": "please, no more go deps shenanigans",
	"keywords": [
		"helloworld",
		"cnab",
		"tutorial"
	],
	"maintainers": [
		{
			"name": "Jane Doe",
			"email": "jane.doe@example.com",
			"url": "https://example.com"
		}
	],
	"invocationImages": [
		{
			"imageType": "docker",
			"image": "radumatei/duffle-push-test3@sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6",
			"size": 942,
			"mediaType": "application/vnd.docker.distribution.manifest.v2+json"
		}
	],
	"images": null,
	"parameters": null,
	"credentials": null
}

cc @michelleN, @silvin-lubecki

glyn · 2019-03-27T11:37:40Z

This PR does not appear to satisfy the requirements of #668:

It implements a many-one mapping of repositories; issue 668 requires a one-one mapping.
It moves a bundle and its images around as an atomic unit; issue 668 relocates just the images of a bundle and does not require either the original or relocated bundle to be stored in a repository.

I've added an agenda item to today's CNAB community call to discuss these points.

radu-matei · 2019-03-27T13:47:26Z

@glyn -- I do have one question about this discussion -- and keep in mind that I haven't followed the registry and image relocation discussions closely, so I might be asking something obvious here.

Have we decided that the requirements from #668 about image relocation (which as I recall, is a continuation of the rewrite command) are the also the requirements for the simple bundle push / pull to OCI registries?

At least at first, they look like two separate use cases, both with valid scenarios.

glyn · 2019-03-27T14:06:50Z

@glyn -- I do have one question about this discussion -- and keep in mind that I haven't followed the registry and image relocation discussions closely, so I might be asking something obvious here.

Have we decided that the requirements from #668 about image relocation (which as I recall, is a continuation of the rewrite command) are the also the requirements for the simple bundle push / pull to OCI registries?

At least at first, they look like two separate use cases, both with valid scenarios.

No, we certainly haven't decided that, but I wanted to point out the potential conflict and have the discussion so we are all "on the same page".

radu-matei · 2019-03-27T18:08:39Z

Thanks for pointing this out, @glyn!

To sum up the discussion from the CNAB meeting earlier today, we decided that both approaches are valid, and we want to eventually support relocating the images for duffle push, perhaps under a flag.

As we are working on the two implementations and we have a clear view of how they work, we will decide if / how we want to integrate relocating images either directly in Duffle, or as part of the cnab-to-oci library used in Duffle for registry support.

Gopkg.toml

silvin-lubecki

It looks good, excepting the new dependency on Kubernetes 😕

silvin-lubecki · 2019-04-09T15:07:31Z

@radu-matei : the PR moving to github.com/deislabs/cnab-go/bundle has already been merged on cnab-to-oci, but the move in duffle is on this PR, still in draft, and we have some vendoring issues. Could you split this PR, and extract the renaming to another PR we can merge quicker?

silvin-lubecki · 2019-04-09T15:30:49Z

Done with #704

- Use a custom duffle branch using github.com/deislabs/cnab-go/bundle (waiting for cnabio/duffle#681 to be merged)

- Use a custom duffle branch using github.com/deislabs/cnab-go/bundle (waiting for cnabio/duffle#681 to be merged) Signed-off-by: Jean-Christophe Sirot <jean-christophe.sirot@docker.com>

radu-matei · 2019-04-11T14:11:30Z

@silvin-lubecki - absolutely, we shouldn't wait on that PR.
Thanks!

- Use a custom duffle branch using github.com/deislabs/cnab-go/bundle (waiting for cnabio/duffle#681 to be merged) Signed-off-by: Jean-Christophe Sirot <jean-christophe.sirot@docker.com>

radu-matei · 2019-04-21T20:37:25Z

This PR introduces an initial implementation for working with OCI registries, provided by github.com/docker/cnab-to-oci.

duffle push - if you logged in to an OCI registry using the Docker CLI, this command will reuse the same credentials, so you should be able to push.

$ duffle push test.json --target cnabregistry.azurecr.io/duffle-push-target:latest --bundle-is-file
Starting to copy image ubuntu:latest...
Completed image ubuntu:latest copy
WARN[0002] reference for unknown type: application/vnd.cnab.config.v1+json
Pushed successfully, with digest "sha256:8a6482e6e5778354cbd95cac49ce3cb5fd78ded0df744c889ca3b29cb131f310"

As the current implementation uses cnab-to-oci, you can read this WIP document describing the rationale behind cnab-to-oci.

Note that you can push a bundle from the bundle store, or a bundle from a file, by passing the --bundle-is-file flag.

duffle pull - again, this will reuse the Docker CLI credentials

$ rm -rf ~/.duffle/
$ duffle pull cnabregistry.azurecr.io/duffle-push-target:latest
$ duffle bundle list
NAME                    VERSION DIGEST
helloworld-test-push    0.1.2   8d100741027eac95a658d354d73bf090f080a318

Note that you can either pull a tag, for example cnabregistry.azurecr.io/duffle-push-target:latest, or directly pull the SHA256 - for example cnabregistry.azurecr.io/duffle-push-target@sha256:8a6482e6e5778354cbd95cac49ce3cb5fd78ded0df744c889ca3b29cb131f310

This has been tested using Azure Container Registries and Docker Hub.

Feedback needed:

how the repository is passed for push(the --target flag) - this PR does not follow the same tag -> push strategy the Docker CLI follows - meaning the name of the bundle (as defined by the spec) does not dictate the repository that will be used to push to (as in - you need to first tag an image and include the repository name with the Docker CLI before you can push). In its current form, the bundle will be pushed to whatever --target points to, provided you have push access.
what should Duffle do if the images referenced in the bundle aren't already pushed and available to copy? (@glyn might have some insight here, as he just implemented the duffle relocate command`)

Thanks!

glyn · 2019-04-23T07:58:28Z

cmd/duffle/pull.go

+	var pull pullCmd
+	cmd := &cobra.Command{
+		Use:   "pull <ref> [options]",
+		Short: "Pulls an image reference",


Does this command really pull an image reference?

Should this be... pulls a remote bundle from an OCI registry?

Yeah, thanks for catching this. Updating.

glyn · 2019-04-23T08:04:39Z

what should Duffle do if the images referenced in the bundle aren't already pushed and available to copy? (@glyn might have some insight here, as he just implemented the duffle relocate command`)

I'm not sure what the intended spec of duffle push is or should be. This needs to be made clear, preferably in the help text. Specifically:

Does this command operate on just the bundle itself or on referenced images too?
If it operates on the reference images, what assumptions does it make about their location and what does it do to them?

(duffle relocate only deals with images, but it will read them from their current repositories and write them to their relocated repositories.)

glyn

The help text needs beefing up so that the spec of duffle push and pull is clear.

glyn · 2019-04-23T13:25:46Z

Obviously, this should not be merged in its current state.

Nit: have you heard of draft pull requests? :-)

michelleN · 2019-04-23T13:41:48Z

@glyn Radu had in fact made this a draft pull request and changed it as ready for review in the past few days.

glyn · 2019-04-23T14:25:09Z

@glyn Radu had in fact made this a draft pull request and changed it as ready for review in the past few days.

Ah, I missed that. Thanks.

michelleN

If I'm understanding this correctly, --target is something required. If so, I think it should be a required argument and not a flag unless we plan on using some sort of defaulting mechanism later on. This is a non-blocking comment.

radu-matei · 2019-04-24T19:24:53Z

@michelleN - yes, the target flag is required. Adding this as the second argument would look like:

duffle push <bundle> <repository>:<optional-tag>

That indeed seems a better, updating.

Thanks for the feedback.

michelleN

Lmk when you're ready for another review @radu-matei. Heads up, I also saw the following error with $ duffle bundle list --> Error: cannot load bundle: invalid character '-' in numeric literal

Don't think that's an issue in master but I'll double check. Cheers.

radu-matei · 2019-04-30T05:36:38Z

@michelleN - updated the UX with your suggestions, and improved the help messages for both commands.

That error message seems to be tied to some signed bundles (from before pulling out the signing part), and also rebased this, meaning there are no more references to those in this branch, so everything should be fixed now.

glyn

Thanks for clarifying the spec of push. I would like a little more clarity on the handling of images by push and maybe pull.

glyn · 2019-04-30T08:06:37Z

cmd/duffle/push.go

+	const pushDesc = `
+Pushes a CNAB bundle to an OCI registry by pushing all container
+images referenced in the bundle to the target repository (all images are
+pushed to the same repository, and are referenceable through their digest).


Where are images "pushed" from? Are they copied direct from their remote repositories or do they have to be present in a docker daemon?

Yes, they are directly copied from the remote (and actually, as #734 tracks, if the images are only present in the local daemon, this will fail)..

glyn · 2019-04-30T08:07:14Z

cmd/duffle/pull.go

+Pulls a CNAB bundle from an OCI repository.
+The only argument for this command is the repository where
+the bundle can be found, and by default, this command pulls the
+bundle and stores it in the local bundle store.


Does this command do anything with images referenced by the bundle?

No, by default it does not pull the referenced images.

I’m revisiting this PR after the whole OCI discussion - do you think it pulling the bundle should pull the referenced images?
Most likely they will not be used on the same machine that pulls the bundle, so not sure about this - would an optional flag for pulling the images be helpful?

I think it makes sense for a local development use case to pre-populate the local docker image store (just pull the bundle and everything in it and it "works"), so I agree with the optional flag.

By the way I don't think it should be a cnab-to-oci feature, as it has no knowledge of the docker engine, only duffle has. I guess it will then parse the bundle.json and call docker pull on every docker image using the docker client ?

Yeah, I agree - I wasn't thinking this should be in cnab-to-oci.

silvin-lubecki · 2019-05-10T14:46:59Z

cmd/duffle/pull.go

+}
+
+func newPullCmd(w io.Writer) *cobra.Command {
+	const usage = `Pulls a bundle from an OCI repository`


nit: use const block

Signed-off-by: Radu M <root@radu.sh>

radu-matei · 2019-08-07T20:20:58Z

Rebased off #827 - haven't addressed all comments yet.
Using WIP cnabio/cnab-to-oci#61

Signed-off-by: Radu M <root@radu.sh>

ghost assigned radu-matei Mar 27, 2019

ghost added the review label Mar 27, 2019

radu-matei mentioned this pull request Mar 27, 2019

Extract bundle structures in a different project #680

Closed

radu-matei mentioned this pull request Apr 3, 2019

Replace deislabs/duffle dependency with deislabs/cnab-go cnabio/cnab-to-oci#32

Merged

radu-matei force-pushed the registry-forks branch 3 times, most recently from 2611762 to 16b907e Compare April 4, 2019 14:54

silvin-lubecki reviewed Apr 5, 2019

View reviewed changes

Gopkg.toml Outdated Show resolved Hide resolved

silvin-lubecki reviewed Apr 5, 2019

View reviewed changes

carolynvs mentioned this pull request Apr 8, 2019

porter bundle publish command getporter/porter#254

Closed

jcsirot pushed a commit to jcsirot/docker-app that referenced this pull request Apr 10, 2019

- Update k8s dependencies to 1.14.0

f1e12ee

- Use a custom duffle branch using github.com/deislabs/cnab-go/bundle (waiting for cnabio/duffle#681 to be merged)

radu-matei mentioned this pull request Apr 11, 2019

Move to deislabs/cnab-go #704

Merged

silvin-lubecki mentioned this pull request Apr 16, 2019

Image relocation #671

Merged

glyn mentioned this pull request Apr 16, 2019

Support image relocation for docker and oci image types #668

Closed

radu-matei force-pushed the registry-forks branch from 16b907e to 39efc73 Compare April 21, 2019 20:09

radu-matei marked this pull request as ready for review April 21, 2019 20:37

radu-matei changed the title ~~[WIP] Add push and pull support using cnab-to-oci~~ Add push and pull support using cnab-to-oci Apr 21, 2019

radu-matei mentioned this pull request Apr 22, 2019

Purpose of this project radu-matei/coras#3

Open

glyn reviewed Apr 23, 2019

View reviewed changes

glyn suggested changes Apr 23, 2019

View reviewed changes

michelleN added this to In Progress in duffle 0.1.0-ralpha.6+flirtini via automation Apr 23, 2019

michelleN added this to the duffle 0.1.0-ralpha.6+flirtini milestone Apr 23, 2019

michelleN self-requested a review April 24, 2019 16:09

michelleN reviewed Apr 24, 2019

View reviewed changes

michelleN reviewed Apr 25, 2019

View reviewed changes

radu-matei force-pushed the registry-forks branch from 39efc73 to cd9084f Compare April 30, 2019 05:33

radu-matei requested review from bacongobbler, carolynvs, jeremyrickard, technosophos and vdice as code owners April 30, 2019 05:33

radu-matei force-pushed the registry-forks branch from cd9084f to ba25c02 Compare April 30, 2019 05:38

radu-matei mentioned this pull request Apr 30, 2019

duffle push should also push all referenced images #734

Open

glyn suggested changes Apr 30, 2019

View reviewed changes

silvin-lubecki reviewed May 10, 2019

View reviewed changes

Radu M added 3 commits August 6, 2019 18:52

Temporarily bump cnab-go to latest commit from master

45b8e91

Signed-off-by: Radu M <root@radu.sh>

Sync Duffle with cnab-go and spec changes

367ce6b

Signed-off-by: Radu M <root@radu.sh>

Add initial push-pull implementation

61bb2d7

Signed-off-by: Radu M <root@radu.sh>

radu-matei force-pushed the registry-forks branch from ba25c02 to 61bb2d7 Compare August 7, 2019 20:19

Update lock file

363b643

Signed-off-by: Radu M <root@radu.sh>

jeremyrickard mentioned this pull request Aug 13, 2019

duffle push should add invocation image digest to bundle.json if it doesn't exist #690

Open

carolynvs removed their request for review October 14, 2019 18:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add push and pull support using cnab-to-oci #681

Add push and pull support using cnab-to-oci #681

radu-matei commented Mar 27, 2019

glyn commented Mar 27, 2019

radu-matei commented Mar 27, 2019

glyn commented Mar 27, 2019

radu-matei commented Mar 27, 2019

silvin-lubecki left a comment

silvin-lubecki commented Apr 9, 2019

silvin-lubecki commented Apr 9, 2019

radu-matei commented Apr 11, 2019

radu-matei commented Apr 21, 2019 •

edited

glyn Apr 23, 2019

michelleN Apr 24, 2019

radu-matei Apr 24, 2019

glyn commented Apr 23, 2019

glyn left a comment

glyn commented Apr 23, 2019

michelleN commented Apr 23, 2019

glyn commented Apr 23, 2019

michelleN left a comment

radu-matei commented Apr 24, 2019

michelleN left a comment

radu-matei commented Apr 30, 2019

glyn left a comment

glyn Apr 30, 2019

radu-matei Apr 30, 2019

glyn Apr 30, 2019

radu-matei Apr 30, 2019

radu-matei Jun 3, 2019

silvin-lubecki Jun 3, 2019

radu-matei Jun 3, 2019

silvin-lubecki May 10, 2019

radu-matei commented Aug 7, 2019 •

edited

Add push and pull support using cnab-to-oci #681

Are you sure you want to change the base?

Add push and pull support using cnab-to-oci #681

Conversation

radu-matei commented Mar 27, 2019

glyn commented Mar 27, 2019

radu-matei commented Mar 27, 2019

glyn commented Mar 27, 2019

radu-matei commented Mar 27, 2019

silvin-lubecki left a comment

Choose a reason for hiding this comment

silvin-lubecki commented Apr 9, 2019

silvin-lubecki commented Apr 9, 2019

radu-matei commented Apr 11, 2019

radu-matei commented Apr 21, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glyn commented Apr 23, 2019

glyn left a comment

Choose a reason for hiding this comment

glyn commented Apr 23, 2019

michelleN commented Apr 23, 2019

glyn commented Apr 23, 2019

michelleN left a comment

Choose a reason for hiding this comment

radu-matei commented Apr 24, 2019

michelleN left a comment

Choose a reason for hiding this comment

radu-matei commented Apr 30, 2019

glyn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radu-matei commented Aug 7, 2019 • edited

radu-matei commented Apr 21, 2019 •

edited

radu-matei commented Aug 7, 2019 •

edited