rgw: implement support for authentication using keystone for s3 and swift #13807

jklippel · 2024-02-22T09:17:55Z

This PR implements the authentication using keystone of swift and s3 as described in the already existing design document swift-and-keystone-integration (see #9444 for the design request and #9088 for the general issue).

The design document describes another feature namely using k8s custom resources to create swift sub-users which will be implemented in a another PR.

Resolves #9088

Checklist:

Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
Reviewed the developer guide on Submitting a Pull Request
Pending release notes updated with breaking and/or notable changes for the next minor release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

BlaineEXE · 2024-02-22T16:23:08Z

Documentation/Contributing/rook-test-framework.md

-export TEST_HELM_PATH=/tmp/rook-tests-scripts-helm/linux-amd64/helm
+export TEST_HELM_PATH=/tmp/rook-tests-scripts-helm/helm


I'm curious about this change. Was this fixing a CI bug, or is this related to a change here?
@subhamkrai do you have insights here?

After running helm.sh up, I have:

$ ls /tmp/rook-tests-scripts-helm/ helm helm.tar.gz LICENSE README.md

I suspect the layout in the Helm archive changed (again). Last time appears to have been in 2020, when --strip-components 1 was introduced in 27c7146. Maybe this needs to be reverted now instead of changing the path in the docs, but either way export TEST_HELM_PATH=/tmp/rook-tests-scripts-helm/linux-amd64/helm did not work for me.

BlaineEXE

I'm impressed how well this PR is put together. There are detailed docs, full test files, and good comments.

I've also really only started scratching the surface of this review. There is a lot here, and it will take time.

One question related to #13808 : Is one of these PRs needed as a basis for the other?

BlaineEXE · 2024-02-22T16:27:18Z

Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-swift.md

+!!! note
+This sample requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.


I think this note makes the most sense immediately before the yaml below. I was confused about what "sample" meant here until I read and re-read this a couple times.

BlaineEXE · 2024-02-22T16:27:51Z

Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-swift.md

+!!! note
+This example assumes that there already is an existing OpenStack Keystone instance ready to use for authentication.


Probably best to include this in the same note which will be moved below, to just above the yaml example.

BlaineEXE · 2024-02-22T16:31:45Z

Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-swift.md

+title: Object Store with Keystone and Swift
+---
+
+You can configure the Swift API and the Keystone integration of Ceph RGW natively via the Object Store CRD, which allows native integration of the Rook operated Ceph RGW into [OpenStack](https://www.openstack.org/) clouds.


This is a little pedantic, but when we had the Ceph foundation's doc expert review our docs, the biggest suggestion he gave was for us to avoid use of "you," "our," "we," etc.

He suggested that we use "imperative" language in our docs. So this doc line should be shifted around to read something like this:

"Ceph RGW can integrate natively with the Swift API and Keystone via the CephObjectStore CRD. This allows native integration of Rook-operated Ceph RGWs into OpenStack"

Please adjust the rest of the document as well. Doing a file search for "you" is usually sufficient to find which lines should be adjusted.

c6495fc

This is not pedantic but good advice! Thanks. I rephrased the document accordingly.
I would suggest to add a section on the imperative language in the Contributing/documentation.md file.

BlaineEXE · 2024-02-22T16:32:40Z

Documentation/Storage-Configuration/Object-Storage-RGW/object-storage.md

-Alternatively, if you have an existing Ceph cluster with Rados Gateways, see the
-[external section](#connect-to-an-external-object-store) to consume it from Rook.
+Most commonly, the object store will be configured in Kubernetes by Rook.
+Alternatively, if you have an existing Ceph cluster with [Rados Gateways](https://docs.ceph.com/en/quincy/radosgw/index.html), see the [external section](#connect-to-an-external-object-store) to consume it from Rook.


Please adjust based on "you" usage here.

I don't see that commit as part of the PR any more.

Also, as a preparation, we will ask you to squash all the commits before final merge. It's fine to keep each commit separate while we are working, however.

Seems as if this commit has already been squased already. Trying to cherry-pick it I ended up with an empty commit.
If you could please do a quick check whether the "you" based language is not gone.

Documentation/Storage-Configuration/Object-Storage-RGW/object-storage.md

BlaineEXE · 2024-02-22T16:34:53Z

pkg/apis/ceph.rook.io/v1/types.go

@@ -1440,6 +1440,16 @@ type ObjectStoreSpec struct {
 	// +nullable
 	Gateway GatewaySpec `json:"gateway"`

+	// The protocol specification
+	// +optional
+	// +nullable


Not nullable because not a pointer to a struct.

Addressed in 556765c

BlaineEXE · 2024-02-22T16:35:03Z

pkg/apis/ceph.rook.io/v1/types.go

+
+	// The authentication configuration
+	// +optional
+	// +nullable


Not nullable because not a pointer to a struct.

Addressed in 556765c

BlaineEXE · 2024-02-22T16:35:43Z

pkg/apis/ceph.rook.io/v1/types.go

+	Enabled         *bool `json:"enabled,omitempty"`
+	AuthUseKeystone *bool `json:"authUseKeystone,omitempty"`
+}
+
+// SwiftSpec represents Ceph Object Store specification for the Swift API
+type SwiftSpec struct {
+	AccountInUrl      *bool   `json:"accountInUrl,omitempty"`
+	UrlPrefix         *string `json:"urlPrefix,omitempty"`
+	VersioningEnabled *bool   `json:"versioningEnabled,omitempty"`
+}
+
+// AuthSpec represents the authentication protocol configuration of a Ceph Object Store Gateway
+type AuthSpec struct {
+	// +optional
+	// +nullable
+	Keystone *KeystoneSpec `json:"keystone,omitempty"`
+}
+
+// KeystoneSpec represents the Keystone authentication configuration of a Ceph Object Store Gateway
+type KeystoneSpec struct {
+	Url                   string                `json:"url"`
+	ServiceUserSecretName string                `json:"serviceUserSecretName"`
+	AcceptedRoles         []string              `json:"acceptedRoles"`
+	ImplicitTenants       ImplicitTenantSetting `json:"implicitTenants,omitempty"`
+	TokenCacheSize        *int                  `json:"tokenCacheSize,omitempty"`
+	RevocationInterval    *int                  `json:"revocationInterval,omitempty"`


Please add godoc comments to all fields, and mark pointer types as nullable.

Addressed in 556765c

BlaineEXE · 2024-02-22T16:36:22Z

pkg/apis/ceph.rook.io/v1/types.go

@@ -1763,6 +1826,7 @@ type ObjectUserQuotaSpec struct {
 	MaxObjects *int64 `json:"maxObjects,omitempty"`
 }

+// CephObjectRealm represents a Ceph Object Store Gateway Realm


Why is this line duplicated here?

That looks like a rebase mess up. Will fix.

BlaineEXE · 2024-02-22T16:36:37Z

pkg/operator/ceph/config/monstore.go

-// https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
+// https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#monitor-configuration-database


Were these links broken?

Yes, docs/master was broken when we tried to access them. I see it now redirects to en/latest. Shall we revert this change?

If the old link is working, I think it's best to revert this. If it's still broken, I'm not too concerned.
But most of our links are in this format, and if they all break, we would want to fix them all at once I think.

pkg/operator/ceph/object/config.go

rkachach · 2024-02-28T13:09:47Z

pkg/operator/ceph/object/config.go

+	if ks := rgwConfig.Auth.Keystone; ks != nil {
+
+		logger.Info("Configuring authentication with keystone")
+
+		configOptions["rgw_keystone_url"] = ks.Url
+		configOptions["rgw_keystone_accepted_roles"] = strings.Join(ks.AcceptedRoles, ",")
+		if ks.ImplicitTenants != "" {
+
+			// only four values are valid here (swift, s3, true and false)
+			//
+			// https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstack-keystone
+			if strings.ToLower(string(ks.ImplicitTenants)) != "true" &&
+				strings.ToLower(string(ks.ImplicitTenants)) != "false" &&
+				strings.ToLower(string(ks.ImplicitTenants)) != "swift" &&
+				strings.ToLower(string(ks.ImplicitTenants)) != "s3" {
+
+				errString := fmt.Sprintf("ImplicitTenantSetting can only be 'swift', 's3', 'true' or 'false', not %s", string(ks.ImplicitTenants))
+				logger.Errorf(errString)
+				return errors.New(errString)
+
+			}
+
+			configOptions["rgw_keystone_implicit_tenants"] = string(ks.ImplicitTenants)
+
+		}
+		if ks.TokenCacheSize != nil {
+			configOptions["rgw_keystone_token_cache_size"] = fmt.Sprintf("%d", *ks.TokenCacheSize)
+		}
+		if rgwConfig.KeystoneSecret == nil {
+			logger.Error("Cannot find keystone secret!")
+			return errors.New("Cannot find keystone secret")
+		}
+
+		var err error
+		configOptions, err = mapKeystoneSecretToConfig(configOptions, rgwConfig.KeystoneSecret)
+		if err != nil {
+			logger.Infof("error mapping keystone secret %s to config: %s", rgwConfig.KeystoneSecret.Name, err)
+			return err
+		}
+
+	} else {
+		logger.Info("Authentication with keystone is disabled")
+	}


this block seems as a good candidate for a proper function

rkachach · 2024-02-28T13:13:55Z

tests/integration/ceph_object_test.go


 	// test that a second object store can be created (and deleted) while the first exists
 	s.T().Run("run a second object store", func(t *testing.T) {
 		otherStoreName := "other-" + storeName
 		// The lite e2e test is perfect, as it only creates a cluster, checks that it is healthy,
 		// and then deletes it.
 		deleteStore := true
-		runObjectE2ETestLite(t, helper, k8sh, installer, namespace, otherStoreName, 1, deleteStore, tlsEnable)
+		// TODO: Why ist swiftAndKeystone always set to false?


please, address the TODO

0a9fdf9
(These TODOs were already dealt with and are now removed.)

tests/integration/ceph_object_test.go

mergify · 2024-02-28T17:54:24Z

This pull request has merge conflicts that must be resolved before it can be merged. @jklippel please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork

Documentation/CRDs/Object-Storage/ceph-object-store-crd.md

BlaineEXE

From an upstream maintainer perspective, I am really impressed by the implementation and thoroughness that I see in this PR. At the same time, I have some concerns about how this feature fits into our community, and I want to acknowledge and address those before we get too far along with reviewing and merging this code.

There do seem to be users that are interested in this feature, but that interest also seems pretty small. This is evidenced by the fact that the design doc for this feature is over 2 years old at this point, and the person who created it didn't see fit to continue to complete the work as designed.

I am concerned that this may be a feature that gets added but then doesn't have anyone sticking around to maintain and troubleshoot for other Rook users. As it is, none of the current Rook maintainers have much OpenStack experience that would allow us to (or motivate us to) keep this feature up-to-date.

While we could declare this feature "experimental," I am still concerned about the possibility of it becoming a long-term maintenance burden that could lead us to ultimately remove it down the road.

@jklippel and @Lykos153 could you tell us more about what your and your company's positions are on feature maintenance for Keystone/Swift integration? I'd like to hear more about what is driving you implement this functionality and what the time scale is for how long it will be used in production. It's also fine if you want to message me and @travisn directly on slack about this since I understand it could be slightly sensitive.

jklippel · 2024-03-01T13:23:49Z

@jklippel and @Lykos153 could you tell us more about what your and your company's positions are on feature maintenance for Keystone/Swift integration? I'd like to hear more about what is driving you implement this functionality and what the time scale is for how long it will be used in production. It's also fine if you want to message me and @travisn directly on slack about this since I understand it could be slightly sensitive.

@BlaineEXE @Lykos153 and I (@jklippel) just sat together and discussed the points you brought up.
All valid points that we would like to address:

Pretty small user interest / slow progress over the time of 2 years
We agree that user interest as provided on the design document seems to be low.
We see however that questions were raised occasionally as to when this feature will be implemented.
We are both from different companies and joined forces to implement this feature as both of our companies independent of each other saw the need to have this feature implemented for our products and within the Yaook oecosystem.
Unfortunately time and other factors slowed down and even stopped progress on this matter for a while, but both of our companies/products still see the need for this feature to integrate rook with keystone authentication in our OpenStack clouds. The person who created it is still involved in the company and still supports the implementation of this feature. He was also involved in bringing our companies together to form a joined effort.
regarding your concerns on low interest but no one supporting the feature
We and both of our companies are committed to support this feature as a contributor to this project with regards to occasional questions/problems in the issue tracker and with regards to adapting code to future changes if necessary.
regarding the long time maintenance burden
Both of our companies are committed to help you in this regards. Additionally the functional changes are not that big. The heavy lifting (if any) is done by Ceph. The OpenStack knowledge to support this feature is nearly limited to the integration tests and the tests only involve the OpenStack keystone component.
That said, we want to support and use these features in our products and will be here to maintain them if necessary. Just mention us (@Lykos153 and me @jklippel) in any issue if you think we should have a more active involvement in an issue.

Does this suffice as commitment to the cause from our side or do you need more/something else entirely?

jklippel · 2024-03-01T13:43:05Z

We created a list of TODO-issues in "our" forked repository to handle the mentioned issues. We linked back to this issue, so do not mind the long list of issues listed here.

horazont · 2024-03-01T15:57:21Z

Hi @travisn, @BlaineEXE,

I'm working at CLOUD&HEAT Technologies GmbH, as team lead of @Lykos153 and former rook contributor (#5483), and I'd like to add a few bits from our perspective.

We have been using Ceph RadosGW with OpenStack in production since 2018 (at least) and we intend to migrate the data from our main Ceph cluster (based on chef-ceph. Yes.) to rook. The lack of RadosGW/OpenStack integration has been a partial blocker (we are currently migrating block storage only because of this), so we are committed to getting this implemented. Likewise, we need this to work in the future, as our object store offering will depend on it.

With all data and information available to me to date, I expect that we offer OpenStack+Rook-based Object Store for the lifetime of the company, however long that may be ("the foreseeable future"). That also implies that we can invest some resources in keeping this feature going, if there turn out to be issues with it in the future.

Hope that helps :).

P.S.: This is my last day before a week of PTO, so if you're pinging me specifically, I may only reply from March 11th onward.

mergify · 2024-03-04T18:32:38Z

This pull request has merge conflicts that must be resolved before it can be merged. @jklippel please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork

BlaineEXE · 2024-03-05T19:12:38Z

Thanks @jklippel and @horazont! We will continue reviewing the PR with intent to merge it. Due to the size and because we are gearing up for kubecon, I think it could be several weeks until we all get a chance to give feedback.

mergify · 2024-04-15T17:37:01Z

This pull request has merge conflicts that must be resolved before it can be merged. @jklippel please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork

mergify · 2024-04-18T12:14:39Z

This pull request has merge conflicts that must be resolved before it can be merged. @jklippel please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork

mergify · 2024-04-18T12:20:02Z

This pull request has merge conflicts that must be resolved before it can be merged. @jklippel please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork

The design document for the swift and keystone integration contains incorrectly numbered annotations. Thie commit fixes the numeration of the annotations. Additionally some minor wording changes are made. Signed-off-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>

For the specification see: <https://github.com/rook/rook/blob/master/design/ceph/object/swift-and-keystone-integration.md> Signed-off-by: Sebastian Riese <sebastian.riese@cloudandheat.com> Signed-off-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>

* The parameter lists of the API call have changes, as parameters ignored by the RGW Admin Ops API are no longer serialized, therefore the mock has to be adapted. * There is now validation for the user keys that are passed to the User get API, therefore things failed when we had empty keys in our User proxy object. Co-authored-by: Jan Klippel <jan.klippel@uhurutec.com> Signed-off-by: Sebastian Riese <sebastian.riese@cloudandheat.com> Signed-off-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>

For the specification see: <https://github.com/rook/rook/blob/master/design/ceph/object/swift-and-keystone-integration.md> Co-authored-by: Jan Klippel <jan.klippel@uhurutec.com> Signed-off-by: Sebastian Riese <sebastian.riese@cloudandheat.com> Signed-off-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>

Minikube decides CPU cores and memory automatically based on the available resources on the machine which may be insufficient to run rook. This commit adds an environment variable to add arbitrary arguments to the minikube command, so both can be specified if desired. Signed-off-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>

The new integration of swift or s3 and keystone support by rook does not have any integration tests yet. This commit introduces integration tests for swift and keystone. The tests are done against a minimal keystone setup (keystone container image from Yaook-project (https://yaook.cloud), sqlite as database backend, cert-manager and trust-manager for test certificate setup). To prevent hardcoded credentials, passwords are generated by the tests. The integration tests use the openstack client (keystone- and swift-functionality) (https://docs.openstack.org/python-openstackclient/ latest/). This was a concious design decision to use client tooling as close as possible to the end user instead of using other go-libraries (such as gophercloud). Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

Currently there is no documentation on the use of Swift to access an object store as well as the use of OpenStack keystone for authentication. This commit adds documentation on the use of Swift and OpenStack keystone, as well as CRD-related documentation and an example setup. Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

This commit introduces integration tests for s3 and keystone. The tests are run against the same minimal keystone setup that the tests for swift and keystone use. The integration tests use the aws s3 client to use client tooling as close as possible to the end user instead of using other go-libraries. Signed-off-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>

Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

…gration Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

…gration Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

kayrus · 2024-05-07T10:24:39Z

deploy/charts/rook-ceph/templates/resources.yaml

+                        implicitTenants:
+                          description: Create new users in their own tenants of the same name. Possible values are true, false, swift and s3. The latter have the effect of splitting the identity space such that only the indicated protocol will use implicit tenants.
+                          type: string
+                        revocationInterval:


I don't see that this value is used anywhere to configure rgw. Also in my CEPH env I cannot set this option:

Error EINVAL: unrecognized config option 'rgw_keystone_revocation_interval'

Good catch! This parameter was removed sometime in the past (last seen in the pacific documentation).
We will remove it.
Thanks again!

I assume there isn't a replacement config either? I don't see anything similar in current Ceph docs.

It was removed with this commit ceph/ceph@6b4c985
If I understand the commit message correctly there was no need for a replacement.

jklippel mentioned this pull request Feb 22, 2024

Support Swift API and Keystone Integration #9088

Open

jklippel changed the title ~~Feature/swift and keystone~~ rgw: implement support for authentication using keystone for s3 and swift Feb 22, 2024

Lykos153 mentioned this pull request Feb 22, 2024

rgw: implement support for creating swift sub-users via the user CR #13808

Draft

6 tasks

jklippel force-pushed the feature/swift-and-keystone branch from abf98c0 to acc11c5 Compare February 22, 2024 09:42

Lykos153 force-pushed the feature/swift-and-keystone branch from acc11c5 to 2e9695d Compare February 22, 2024 09:51

BlaineEXE reviewed Feb 22, 2024

View reviewed changes

BlaineEXE requested changes Feb 22, 2024

View reviewed changes

BlaineEXE requested review from travisn and alimaredia February 22, 2024 16:40

rkachach reviewed Feb 28, 2024

View reviewed changes

BlaineEXE mentioned this pull request Feb 28, 2024

rgw: Design document for the Keystone and Swift integration #9444

Merged

10 tasks

BlaineEXE reviewed Feb 28, 2024

View reviewed changes

Documentation/CRDs/Object-Storage/ceph-object-store-crd.md Outdated Show resolved Hide resolved

BlaineEXE reviewed Feb 28, 2024

View reviewed changes

jklippel force-pushed the feature/swift-and-keystone branch 2 times, most recently from fa106d4 to c6495fc Compare March 5, 2024 15:16

jklippel force-pushed the feature/swift-and-keystone branch 4 times, most recently from ecaa4b0 to 5b8b850 Compare April 18, 2024 12:13

jklippel force-pushed the feature/swift-and-keystone branch 2 times, most recently from e188a29 to 5b8b850 Compare April 18, 2024 12:19

Lykos153 and others added 7 commits April 18, 2024 14:20

jklippel force-pushed the feature/swift-and-keystone branch from 5b8b850 to 904c5f7 Compare April 18, 2024 12:22

jklippel force-pushed the feature/swift-and-keystone branch 2 times, most recently from 176a755 to 30ece5d Compare April 18, 2024 14:13

jklippel added 2 commits April 29, 2024 14:59

fixup! docs: added documentation on swift and keystone

0c3e5cf

Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

fixup! rgw: expand the reconcile loop for the swift and keystone inte…

c4c6ff8

…gration Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

jklippel force-pushed the feature/swift-and-keystone branch from 9adb92c to c4c6ff8 Compare April 29, 2024 12:59

Lykos153 and others added 4 commits April 29, 2024 20:10

fixup! test: integration tests for swift and keystone

64eafba

fixup! docs: added documentation on swift and keystone

241bb57

Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

fixup! docs: added documentation on swift and keystone

f099307

Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

fixup! rgw: expand the reconcile loop for the swift and keystone inte…

32c00ca

…gration Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>

jklippel force-pushed the feature/swift-and-keystone branch from 1ced5e8 to 32c00ca Compare May 3, 2024 18:07

kayrus reviewed May 7, 2024

View reviewed changes

jklippel mentioned this pull request May 7, 2024

Remove unused/no longer implemented in ceph parameter rgw_keystone_revocation_interval jklippel/rook#26

Open

		export TEST_HELM_PATH=/tmp/rook-tests-scripts-helm/linux-amd64/helm
		export TEST_HELM_PATH=/tmp/rook-tests-scripts-helm/helm

		!!! note
		This sample requires at least 3 bluestore OSDs, with each OSD located on a different node.

		!!! note
		This example assumes that there already is an existing OpenStack Keystone instance ready to use for authentication.

		// https://docs.ceph.com/docs/master/rados/configuration/ceph-conf/#monitor-configuration-database
		// https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#monitor-configuration-database

rgw: implement support for authentication using keystone for s3 and swift #13807

Are you sure you want to change the base?

rgw: implement support for authentication using keystone for s3 and swift #13807

Conversation

jklippel commented Feb 22, 2024

Choose a reason for hiding this comment

Lykos153 Feb 22, 2024 • edited

Choose a reason for hiding this comment

BlaineEXE left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jklippel Mar 5, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jklippel Mar 5, 2024 • edited

Choose a reason for hiding this comment

BlaineEXE Mar 5, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lykos153 Mar 7, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Feb 28, 2024

BlaineEXE left a comment

Choose a reason for hiding this comment

jklippel commented Mar 1, 2024

jklippel commented Mar 1, 2024

horazont commented Mar 1, 2024 • edited

mergify bot commented Mar 4, 2024

BlaineEXE commented Mar 5, 2024

mergify bot commented Apr 15, 2024

mergify bot commented Apr 18, 2024

mergify bot commented Apr 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BlaineEXE May 7, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lykos153 Feb 22, 2024 •

edited

jklippel Mar 5, 2024 •

edited

jklippel Mar 5, 2024 •

edited

BlaineEXE Mar 5, 2024 •

edited

Lykos153 Mar 7, 2024 •

edited

horazont commented Mar 1, 2024 •

edited

BlaineEXE May 7, 2024 •

edited