containers.conf: implement modules #1599

vrothberg · 2023-08-07T13:30:38Z

Add a new concept to containers.conf called "modules". A "module" is
a containers.conf file located at a specific directory. More than one
modules can be loaded in the specified order, following existing
override semantics.

There are three directories to load modules from:

$CONFIG_HOME/containers/containers.conf.modules
/etc/containers/containers.conf.modules
/usr/share/containers/containers.conf.modules

With CONFIG_HOME pointing to $HOME/.config or, if set, $XDG_CONFIG_HOME.
Absolute paths will be loaded as is, relative paths will be resolved
relative to the three directories above allowing for admin configs
(/etc/) to override system configs (/usr/share/) and user configs
($CONFIG_HOME) to override admin configs.

Also move some functions from config.go for locality.

Signed-off-by: Valentin Rothberg vrothberg@redhat.com

Still a draft but there are fairly comprehensive tests.

@rhatdan @Luap99 @giuseppe @containers/podman-maintainers PTAL. I've received contradicting feedback with respect to the "modules" name. So far, it's my favorite but I am not married to it.

Please also take a look at the initial refactoring. I found it important to first clean up the API a bit to make sure it remains maintainable.

vrothberg · 2023-08-08T08:14:46Z

Now with docs.

Luap99

Look like these are either not used or you are not testing what you think you are testing because the default_network configs are incorrect.

pkg/config/testdata/modules/etc/containers/containers.conf.modules/sub/etc-only.conf

pkg/config/testdata/modules/etc/containers/containers.conf.modules/third.conf

pkg/config/testdata/modules/home/.config/containers/containers.conf.modules/third.conf

pkg/config/testdata/modules/usr/share/containers/containers.conf.modules/third.conf

vrothberg · 2023-08-08T09:17:38Z

Thanks! They're currently not tested but I'll update and add a check for third.conf.

pkg/config/new.go

Luap99 · 2023-08-08T09:49:48Z

Sorry for not looking at the design doc earlier, I will add some comments there as I think there will be some big problem with the podman implementation part.

vrothberg · 2023-08-08T09:50:48Z

Sorry for not looking at the design doc earlier, I will add some comments there as I think there will be some big problem with the podman implementation part.

I am cool to discuss here. I guess you'll also worry about how to handle the defaults on the CLI wrt to container creation etc?

Luap99 · 2023-08-08T10:04:45Z

Yes the current assumption is configs are loaded before cli arguments. This is a hard requirement as the config currently is used to set defaults for said arguments. Loading the configs again when --module is used will not work as the defaults for cli args are already locked at that point.

One example where defaults are locked is podman run --detach-keys, you cannot change the default once the flag is defined. That must happen before argument parsing so loading a module later will not change detach keys.
However callers that fill in defaults at a later point, i.e. somewhere in pkg/specgen will work fine with modules. For example the netns option.

The other problem with that is that there currently is no consistent way of where we fill in defaults from the config, it is all over the place. In some cases modules will work correctly in others they will not. This will be impossible to document and test and thus cause a lot of pain for us and users.

Luap99 · 2023-08-08T10:12:23Z

Another problem is do you want to support modules on the remote client? Because right now most defaults are only set on the server so I am not sure how this should behave at all there.

vrothberg · 2023-08-08T11:01:35Z

Yes the current assumption is configs are loaded before cli arguments. This is a hard requirement as the config currently is used to set defaults for said arguments. Loading the configs again when --module is used will not work as the defaults for cli args are already locked at that point.

Yes, I was thinking about this as well and do not have a good answer yet

Another problem is do you want to support modules on the remote client? Because right now most defaults are only set on the server so I am not sure how this should behave at all there.

The way I look at module is that it's just yet another way for loading containers.conf files. There are numerous issue in Podman wrt. to whether and where it respects the containers.conf settings, so I don't aim at fixing those for now.

The other problem with that is that there currently is no consistent way of where we fill in defaults from the config, it is all over the place. In some cases modules will work correctly in others they will not. This will be impossible to document and test and thus cause a lot of pain for us and users

My goal for now is to be able for cmd/podman to be able to process modules and pass that down. Given the chaos of how containers.conf is being used in Podman, I am extremely hesitant to shave the Yak. But for sure, I don't want to worsen the status quo, so modules must work cmd/podman.

vrothberg · 2023-08-08T11:08:44Z

I think the easy cheat would be passing modules via env variables but the UX is ugly. I want to get to a point where --module tab completion works.

vrothberg · 2023-08-08T11:17:17Z

If we can come up with a means to parse --module in https://github.com/containers/podman/blob/main/cmd/podman/registry/config.go#L48, we should be good.

Luap99 · 2023-08-08T11:34:18Z

I think the easy cheat would be passing modules via env variables but the UX is ugly. I want to get to a point where --module tab completion works.

Agreed, that would make it very hard to use and thus likely defeat the point of this feature.

Another problem is do you want to support modules on the remote client? Because right now most defaults are only set on the server so I am not sure how this should behave at all there.

The way I look at module is that it's just yet another way for loading containers.conf files. There are numerous issue in Podman wrt. to whether and where it respects the containers.conf settings, so I don't aim at fixing those for now.

But I think this assumption is not what a user would expect. If I use --module and the we ignore most option on remote users will file bug reports and get upset when we tell them yes this is expected you must set defaults on the server side. I think this would totally defeat the use case here.

Overall I am worried that such a design causes a lot of pain for us in the future. So far my understanding is that there is exactly one user/customer wanting this so my immediate response would be why can they not keep doing what they are doing now? Is a wrapper around podman really that bad?! There is a point where we just need to say no.

Then it looks like this here is trying to much more than the use case is, I was only part of one meeting so I could be missing context. My understanding is that they only care about container create options? If that is the case I think maybe going with a design that only focuses on that might be better than trying to generalize this to far. I like using the same config format but I don't think it is required to treat it like the real containers.conf

vrothberg · 2023-08-08T11:48:03Z

But I think this assumption is not what a user would expect. If I use --module and the we ignore most option on remote users will file bug reports and get upset when we tell them yes this is expected you must set defaults on the server side. I think this would totally defeat the use case here.

Modules don't change the status quo. This issue already applies to Podman now. I don't think we should block modules on that - it's a connected but orthogonal problem.

Overall I am worried that such a design causes a lot of pain for us in the future. So far my understanding is that there is exactly one user/customer wanting this so my immediate response would be why can they not keep doing what they are doing now? Is a wrapper around podman really that bad?! There is a point where we just need to say no.

I think you're conflating the pre-existing chaotic usage of containers.conf in Podman with this PR. I am convinced all issues you mention already apply to Podman today. Some options are on the client-side, some on the server side, some may be totally ignored. All that is already the case. I am not advertising against fixing these issues but I won't sign up for fixing them right now.

What I think is important is that modules don't worsen the status quo. That's why I think we need to find a way to parse the --module CLI flag here: https://github.com/containers/podman/blob/main/cmd/podman/registry/config.go#L48

That's where the "default config" is parsed (and set) first. All other callers of Default() will just use that.

Then it looks like this here is trying to much more than the use case is, I was only part of one meeting so I could be missing context. My understanding is that they only care about container create options? If that is the case I think maybe going with a design that only focuses on that might be better than trying to generalize this to far. I like using the same config format but I don't think it is required to treat it like the real containers.conf

Let's focus on my above counter point on not changing the status quo before throwing everything over board. We've desired such an on-demand loading mechanism for a long time and I think this way of loading absolute paths and resolving to a "modules" directory is a sound design. I desire it for testing, for instance, podman run --module crun.conf --module cni.conf.

vrothberg · 2023-08-08T11:50:57Z

I think it's worth mentioning that I brought the issue of "maybe some options won't work" up with the stakeholder. So seeing it from another angle, it's an occasion to make sure more options work as intended.

Luap99 · 2023-08-08T12:04:20Z

I think modules worsen the situation because users are not switching from an existing containers.conf to modules but from an existing podman run -v ..., depending on what options they intend to replace with a module it may not work.

Having modules behave differently for remote vs local case is bad design for me and will cause a lot of trouble and we now have a lot of remote users on macos/windows. The current implementation would likely make modules unusable on remote for most things so I think this must be avoided.

I think we totally agreed on containers.conf defaults are set on the server side but for modules this design makes no sense. IMO modules would either work on client side only or we send the module names to server and then the server looks up the files there in it own modules directory.

vrothberg · 2023-08-08T12:07:02Z

I think modules worsen the situation because users are not switching from an existing containers.conf to modules but from an existing podman run -v ..., depending on what options they intend to replace with a module it may not work.

Can you elaborate on where you see the difference between CONTAINERS_CONF=foo.conf podman ... and podman --module foo.conf ...? I don't see a difference.

Same when editing $HOME/.config/containers/containers.conf. I don't see how this would change the status quo.

Having modules behave differently for remote vs local case is bad design for me and will cause a lot of trouble and we now have a lot of remote users on macos/windows. The current implementation would likely make modules unusable on remote for most things so I think this must be avoided.

Same as above.

I think we totally agreed on containers.conf defaults are set on the server side but for modules this design makes no sense. IMO modules would either work on client side only or we send the module names to server and then the server looks up the files there in it own modules directory.

Same as above.

Luap99 · 2023-08-08T12:23:03Z

Because my understanding of this feature is to combine multiple podman options in a single file. Modules will be used to replace podman run --volume /mypath:/ctr --dns 1.1.1.1 --env TEST=123 $IMAGE with podman --module mymod.conf run $IMAGE.
If we treat this as containers.conf equivalent it will not work like I as an user expect it to work. The most common example as mentioned in the design docs is the replacement of options. If two modules declare volumes = [] the later will replace the first one which seems bad, this must be add instead.

vrothberg · 2023-08-08T12:27:12Z

Because my understanding of this feature is to combine multiple podman options in a single file. Modules will be used to replace podman run --volume /mypath:/ctr --dns 1.1.1.1 --env TEST=123 $IMAGE with podman --module mymod.conf run $IMAGE. If we treat this as containers.conf equivalent it will not work like I as an user expect it to work. The most common example as mentioned in the design docs is the replacement of options. If two modules declare volumes = [] the later will replace the first one which seems bad, this must be add instead.

That seems like a very new point in our discussion but a good one and (briefly) mentioned in design doc:

NOTE: We need a mechanism for appending certain lists together when specified in different Modules, or from containers.conf. For example a user may specify a list of Volumes they want mounted into a container in three different files, or environment variables. Currently containers.conf replaces volumes or env variables with those specified in the final containers.conf file

Finding a mechanism for containers.conf files to append/add instead of replace is future work. For now, we focus on the loading mechanism.

Luap99 · 2023-08-08T12:39:14Z

The other thing is I have no problem with containers.conf reading the defaults on the server and I think this is the correct thing to do. We can always tell people to edit config files on the server.
However if the remote client now has an options called --module I would assume it works and not just inserts default into my local containers.conf which then are not used by the server. There is literally no way to make such a module work in the remote context as you cannot send this via the API. And you cannnot tell people to store these modules on the server unless the module names are send via API. SO IMO this design cannot support modules in the containers.conf style on the remote client as these only effect options used by the remote client such as the [machine] section.

This is really my main point I tied to bring across here. A server side containers.conf can be edited and used. A server side module simply is impossible to use with the current design.

vrothberg · 2023-08-08T14:07:47Z

The other thing is I have no problem with containers.conf reading the defaults on the server and I think this is the correct thing to do. We can always tell people to edit config files on the server. However if the remote client now has an options called --module I would assume it works and not just inserts default into my local containers.conf which then are not used by the server.

I agree that it would be bad if there are expectations that $option works in the remote case but doesn't. What I don't agree on that --module would cause this expectation. This (hypothetical) problem is already there as remote clients already parse the various containers.conf files on the system (and the environment).

I am sure there are problems today and that those need to be fixed eventually and limitations need to be documented. But again, I don't think this PR is the source of these problems.

This is really my main point I tied to bring across here. A server side containers.conf can be edited and used. A server side module simply is impossible to use with the current design.

It's as much (or less) use as any other containers.conf file today.

Luap99 · 2023-08-08T14:38:25Z

Problem today: Some containers.conf option are read on the client other on the server. It is not documented what is used where and in it is a total mess in the podman code. However a user can simply add settings on either the server or the client to containers.conf so with a bit of testing you can get it to work just fine.

Module problem: All options that are read on the server side can never be used for podman-remote --module. They will not work and there is no workaround as you cannot set the module on the server side to get what you need. I mean sure you could start podman system service --module ... but that totally defeats the purpose to selective set options for a single command. You would need to restart the service with a new module for each of your podman-remote command in order to make use of it with is of course nonsensical.

So I argue this is a new problem that you are going to introduce! There is nothing I can tell a remote user to make such a module work.

To me the use case described was lets make podman commands shorter and reusable by such small modules but if they do not function with the remote client I think this is a big no go to me. Maybe the customer does not care but feature wise I argue we should not add new things that do not work via remote unless there is a proper reason for it.

rhatdan · 2023-08-08T14:47:07Z

Why not make --module not allowed on --remote? We already do this with a lot of global settings.

Luap99 · 2023-08-09T13:39:06Z

I would like to see a WIP podman PR just to get a better feeling on how it will be implemented and that it does not add regressions.

vrothberg · 2023-08-09T13:46:05Z

I would like to see a WIP podman PR just to get a better feeling on how it will be implemented and that it does not add regressions.

Totally agree. I don't want to merge it here without a Podman PR, so I'll keep it as a Draft.

vrothberg · 2023-08-09T15:55:20Z

Here's the Podman PR: containers/podman#19567

Still raw but the new tests pass locally

Support a new concept in containers.conf called "modules". A "module" is a containers.conf file located at a specific directory. More than one module can be loaded in the specified order, following existing override semantics. There are three directories to load modules from: - $CONFIG_HOME/containers/containers.conf.modules - /etc/containers/containers.conf.modules - /usr/share/containers/containers.conf.modules With CONFIG_HOME pointing to $HOME/.config or, if set, $XDG_CONFIG_HOME. Absolute paths will be loaded as is, relative paths will be resolved relative to the three directories above allowing for admin configs (/etc/) to override system configs (/usr/share/) and user configs ($CONFIG_HOME) to override admin configs. Pulls in containers/common/pull/1599. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

n1hility · 2023-08-12T19:12:04Z

pkg/config/new.go

+// All callers are expected to use the returned Config read only.  Changing
+// data may impact other call sites.
+func Default() (*Config, error) {
+	if cachedConfig != nil || cachedConfigError != nil {


Double checked locking (on its own) is not safe with Golang's memory model. While you may observe the write to cachedConfig, there is no happens-before guarantee on the preceding writes for the config structure as part of initialization (the pointer write may be reordered (speculative execution, compiler reordering etc) or coherence not yet achieved leading to a program to observe an uninitialized / incomplete config). See https://go.dev/ref/mem. Also https://gee.cs.oswego.edu/dl/jmm/cookbook.html is pretty dated but still nice overview/example of how compilers map a model to hardware.

I'd recommend using either sync.Once or an atomic Load Store for the pointer in combo with the DCL as it is (what Once does under the hood) . The go memory model does have a happens-before established at the location of an atomic store, but without the lock queuing or CAS overhead.

Thanks a lot, @n1hility ! Will fix accordingly

According to https://go.dev/ref/mem it's sufficient to remove this check. Lock() + check + Unlock() makes sure the variables are synchronized.

@vrothberg +1, only reason i suggested sync.Once was if you were trying to avoid the lock penalty. putting the Lock/Unlock in Do would give you that same DCL lock for just init behavior. I agree the opto isn't critical in this case.

Thanks a lot for checking! I did consider a sync.Once but with this PR New() may override the "default" config which would have broken the contract Once would guarantee otherwise.

Because that's what it actually does. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

It has no external user and should not be exported to avoid any API misuse; built-in defaults are an implementation detail. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

It's wasteful and `sut` was not a name I would now understand. Change the tests that need a default config. The diff also shows that the tests would benefit a lot from a rewrite into a table-driven form but I do not want to shave the entire Yak. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

Add `New()` function to create a Config and deprecate `NewConfig` which is a) not extensible and b) broken in the sense that no external caller was actually using the argument. Many call sites use `Default()` which now has improved documentation and allows for interacting with `New(). Most call sites just need to access a pro-loaded config (via `Default()`). This config can overridden by `New()` if the caller sets the specific option - a requirement for an upcoming feature for Podman allowing to load user-specified configs via CLI flags. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

Add a new concept to containers.conf called "modules". A "module" is a containers.conf file located at a specific directory. More than one module can be loaded in the specified order, following existing override semantics. There are three directories to load modules from: - $CONFIG_HOME/containers/containers.conf.modules - /etc/containers/containers.conf.modules - /usr/share/containers/containers.conf.modules With CONFIG_HOME pointing to $HOME/.config or, if set, $XDG_CONFIG_HOME. Absolute paths will be loaded as is, relative paths will be resolved relative to the three directories above allowing for admin configs (/etc/) to override system configs (/usr/share/) and user configs ($CONFIG_HOME) to override admin configs. Also move some functions from config.go for locality. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

To return absolute paths to modules a config was loaded with. Knowing the modules is required for conmon's callback to Podman's cleanup. Returning them as absolute paths makes loading the modules a bit faster as it avoids the lookup. Also drop the attempted performance tune in `Default()` to accommodate for go's memory model. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

Support a new concept in containers.conf called "modules". A "module" is a containers.conf file located at a specific directory. More than one module can be loaded in the specified order, following existing override semantics. There are three directories to load modules from: - $CONFIG_HOME/containers/containers.conf.modules - /etc/containers/containers.conf.modules - /usr/share/containers/containers.conf.modules With CONFIG_HOME pointing to $HOME/.config or, if set, $XDG_CONFIG_HOME. Absolute paths will be loaded as is, relative paths will be resolved relative to the three directories above allowing for admin configs (/etc/) to override system configs (/usr/share/) and user configs ($CONFIG_HOME) to override admin configs. Pulls in containers/common/pull/1599. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

Luap99

LGTM

vrothberg · 2023-08-14T14:19:59Z

@n1hility @rhatdan PTanotherL

Thanks for the comment, everybody!

rhatdan · 2023-08-14T16:02:41Z

LGTM

rhatdan · 2023-08-14T16:02:53Z

/approve
/lgtm

openshift-ci · 2023-08-14T16:03:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: giuseppe, Luap99, rhatdan, vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Luap99,giuseppe,rhatdan,vrothberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Support a new concept in containers.conf called "modules". A "module" is a containers.conf file located at a specific directory. More than one module can be loaded in the specified order, following existing override semantics. There are three directories to load modules from: - $CONFIG_HOME/containers/containers.conf.modules - /etc/containers/containers.conf.modules - /usr/share/containers/containers.conf.modules With CONFIG_HOME pointing to $HOME/.config or, if set, $XDG_CONFIG_HOME. Absolute paths will be loaded as is, relative paths will be resolved relative to the three directories above allowing for admin configs (/etc/) to override system configs (/usr/share/) and user configs ($CONFIG_HOME) to override admin configs. Pulls in containers/common/pull/1599. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

openshift-ci bot added do-not-merge/work-in-progress approved labels Aug 7, 2023

vrothberg force-pushed the RUN-1873 branch 4 times, most recently from ff4843b to 56397b6 Compare August 8, 2023 08:14

Luap99 requested changes Aug 8, 2023

View reviewed changes

vrothberg force-pushed the RUN-1873 branch from 56397b6 to 95dd794 Compare August 8, 2023 09:31

Luap99 reviewed Aug 8, 2023

View reviewed changes

pkg/config/new.go Show resolved Hide resolved

pkg/config/new.go Show resolved Hide resolved

Luap99 reviewed Aug 8, 2023

View reviewed changes

pkg/config/new.go Outdated Show resolved Hide resolved

vrothberg mentioned this pull request Aug 10, 2023

add --module flag containers/podman#19567

Merged

vrothberg marked this pull request as ready for review August 10, 2023 13:32

openshift-ci bot removed the do-not-merge/work-in-progress label Aug 10, 2023

vrothberg mentioned this pull request Aug 11, 2023

Introduce (global) default config file locking #1610

Closed

vrothberg force-pushed the RUN-1873 branch from 18b6c47 to 7bb8170 Compare August 11, 2023 09:47

n1hility reviewed Aug 12, 2023

View reviewed changes

vrothberg added 6 commits August 14, 2023 09:40

pkg/config: s/defaultConfigFromMemory/defaultEngineConfig/

f27dae1

Because that's what it actually does. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

pkg/config: unexport DefaultConfig

9452f4a

It has no external user and should not be exported to avoid any API misuse; built-in defaults are an implementation detail. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

vrothberg force-pushed the RUN-1873 branch from 7bb8170 to 671ae57 Compare August 14, 2023 07:57

Luap99 approved these changes Aug 14, 2023

View reviewed changes

openshift-ci bot assigned rhatdan Aug 14, 2023

openshift-ci bot added the lgtm label Aug 14, 2023

openshift-merge-robot merged commit b70b0c4 into containers:main Aug 14, 2023
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

containers.conf: implement modules #1599

containers.conf: implement modules #1599

vrothberg commented Aug 7, 2023 •

edited

vrothberg commented Aug 8, 2023

Luap99 left a comment

vrothberg commented Aug 8, 2023

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023

Luap99 commented Aug 8, 2023

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023

vrothberg commented Aug 8, 2023

vrothberg commented Aug 8, 2023

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023

vrothberg commented Aug 8, 2023 •

edited

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023 •

edited

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023 •

edited

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023

Luap99 commented Aug 8, 2023

rhatdan commented Aug 8, 2023

Luap99 commented Aug 9, 2023

vrothberg commented Aug 9, 2023

vrothberg commented Aug 9, 2023

n1hility Aug 12, 2023

vrothberg Aug 14, 2023

vrothberg Aug 14, 2023

n1hility Aug 14, 2023

vrothberg Aug 14, 2023

Luap99 left a comment

vrothberg commented Aug 14, 2023

rhatdan commented Aug 14, 2023

rhatdan commented Aug 14, 2023

openshift-ci bot commented Aug 14, 2023

containers.conf: implement modules #1599

containers.conf: implement modules #1599

Conversation

vrothberg commented Aug 7, 2023 • edited

vrothberg commented Aug 8, 2023

Luap99 left a comment

Choose a reason for hiding this comment

vrothberg commented Aug 8, 2023

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023

Luap99 commented Aug 8, 2023

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023

vrothberg commented Aug 8, 2023

vrothberg commented Aug 8, 2023

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023

vrothberg commented Aug 8, 2023 • edited

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023 • edited

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023 • edited

Luap99 commented Aug 8, 2023

vrothberg commented Aug 8, 2023

Luap99 commented Aug 8, 2023

rhatdan commented Aug 8, 2023

Luap99 commented Aug 9, 2023

vrothberg commented Aug 9, 2023

vrothberg commented Aug 9, 2023

n1hility Aug 12, 2023

Choose a reason for hiding this comment

vrothberg Aug 14, 2023

Choose a reason for hiding this comment

vrothberg Aug 14, 2023

Choose a reason for hiding this comment

n1hility Aug 14, 2023

Choose a reason for hiding this comment

vrothberg Aug 14, 2023

Choose a reason for hiding this comment

Luap99 left a comment

Choose a reason for hiding this comment

vrothberg commented Aug 14, 2023

rhatdan commented Aug 14, 2023

rhatdan commented Aug 14, 2023

openshift-ci bot commented Aug 14, 2023

vrothberg commented Aug 7, 2023 •

edited

vrothberg commented Aug 8, 2023 •

edited

vrothberg commented Aug 8, 2023 •

edited

vrothberg commented Aug 8, 2023 •

edited