object: add support for user s3 key for cephobjectstoreuser #14001

parth-gr · 2024-04-01T12:54:06Z

CephObjectStoreUser should optionally be able to reference a secret where S3 key is defined. This enables us to specify the accesskey and accesssecret rather than those values being randomly generated.

Closes: #11563

Checklist:

Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
Reviewed the developer guide on Submitting a Pull Request
Pending release notes updated with breaking and/or notable changes for the next minor release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

parth-gr · 2024-04-01T12:57:30Z

pkg/operator/ceph/object/user/controller.go

@@ -296,6 +296,20 @@ func (r *ReconcileObjectStoreUser) createOrUpdateCephUser(u *cephv1.CephObjectSt
 	user, err = r.objContext.AdminOpsClient.GetUser(r.opManagerContext, *r.userConfig)
 	if err != nil {
 		if errors.Is(err, admin.ErrNoSuchUser) {
+			namspacedName := types.NamespacedName{Namespace: u.Namespace, Name: "rook-ceph-object-user-" + u.Spec.Store + u.Name}


We should think on the name, as it can go more than 63 characters very eaisly

we can do something similar to how we truncate the node names in the TruncateNodeName method.

pkg/operator/ceph/object/user/controller.go

sp98 · 2024-04-01T13:48:55Z

pkg/operator/ceph/object/user/controller.go

@@ -296,6 +296,20 @@ func (r *ReconcileObjectStoreUser) createOrUpdateCephUser(u *cephv1.CephObjectSt
 	user, err = r.objContext.AdminOpsClient.GetUser(r.opManagerContext, *r.userConfig)
 	if err != nil {
 		if errors.Is(err, admin.ErrNoSuchUser) {
+			namspacedName := types.NamespacedName{Namespace: u.Namespace, Name: "rook-ceph-object-user-" + u.Spec.Store + u.Name}


we can do something similar to how we truncate the node names in the TruncateNodeName method.

pkg/operator/ceph/object/user/controller.go

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

BlaineEXE

Do we have a design doc for this? I know this seems really simple, but the complexity is large enough that I would like to discuss it in detail.

Specifically, I am aware that the code as it is written will enforce a particular workflow for managing user keys. User keys can only be rotated by modifying the k8s secret. However, there may be users who want to rotate keys using the RGW admin ops APIs (dashboard or CLI) instead.

If we want to support other ways of rotating keys, we will need to discuss this further and make a design for the intended behaviors.

pkg/operator/ceph/object/user/controller.go

travisn · 2024-04-02T22:26:25Z

Do we have a design doc for this? I know this seems really simple, but the complexity is large enough that I would like to discuss it in detail.

Note there is some discussion in #11563.

parth-gr · 2024-04-03T16:43:57Z

@travisn not sure but getting this,

  Warning  ReconcileFailed  10s (x4 over 29s)  rook-ceph-object-store-user-controller  (combined from similar events): failed to reconcile CephObjectStoreUser "rook-ceph/my-user1". failed to create/update object store user "my-user1": failed to set quotas for user "my-user1": NoSuchUser tx000006ba425179c6f75f2-00660d85e8-1203-my-store 1203-my-store-my-store

If used the custom secret

apiVersion: v1
kind: Secret
metadata:
  name: rook-ceph-object-user-my-store-my-user1
  namespace: rook-ceph
data:
  AccessKey: VFKF8SSU9L3L2UR03Z8C
  SecretKey: 5U4e2MkXHgXstfWkxGZOI6AXDfVUkDDHM7Dwc3mY
type: kubernetes.io/rook

Will look deeper tomorrow

BlaineEXE · 2024-04-03T17:02:04Z

Converting this to a draft while we wait for solicited user feedback from my proposal.
#11563 (comment)

pkg/operator/ceph/object/user/controller.go

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

BlaineEXE

I think this is looking like it's off to a good start based on our conversation from yesterday.

Because the changes in this PR are fairly straightforward, I think this is also a good PR to use as a basis to grow some more software design skills. I explain more in some of my comments, but I think we can start to focus on techniques for writing code that is clean and maintainable and lessens burden on future code edits.

I think it would also be nice to think about the end user as well. github.com/rook/rook/pkg/... can provide APIs for users to help them write their own tooling, and I think that structs, creation, and error handling features for filling in secret fields are things beneficial for users, as well as Rook internally. I also think this exercise is a good one for helping understand how to implement the single responsibility principle in the future, even when we aren't exporting APIs for users.

BlaineEXE · 2024-04-19T16:26:18Z

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

+[{"user":"","access_key":"IE58RNT71Y2F1EQE80RA","secret_key":"cULyMz5dCpX18dPsJhpIKay7vcDNRNJWJPu8VqUA","UID":"","SubUser":"","KeyType":"","GenerateKey":null},
+{"user":"","access_key":"IE58RNT71Y2F1EQE80RZ","secret_key":"cULyMz5dCpX18dPsJhpIKay7vcDNRNJWJPu8VqUE","UID":"","SubUser":"","KeyType":"","GenerateKey":null}]


Can we change this to a struct that has only access and secret key entries. Adding other, unused fields will only serve to confuse users and add risk for typos.

[ {"access_key":"BLAH","secret_key":"blah"} ]

We also need UID so updating with that

Or maybe I can update to object-user name for all the users

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

BlaineEXE · 2024-04-19T16:28:52Z

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

+data:
+  AccessKey: ***
+  SecretKey: ***
+  Endpoint: ***


I didn't realize the secret had an endpoint. We are going to have to think about this aspect a little. I think we're going to have to do 1 of 2 things:

Ignore the endpoint when rook.io/source-of-truth: secret is set, and tell users that they should use the CephObjectStore.Status.Endpoints list to retrieve endpoints for applications.

Break our rule that Rook doesn't edit the secret when rook.io/source-of-truth: secret is set so that Rook can set the endpoint itself.

Personally, I think it might be best to go with option 1, but I am open to other thoughts. @travisn, @parth-gr ?

We can add a note user should not update the endpoint manually in the secret it is controlled by rook.

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

pkg/operator/ceph/object/user/controller.go

BlaineEXE · 2024-04-19T16:52:33Z

pkg/operator/ceph/object/user/controller.go

 }

-func (r *ReconcileObjectStoreUser) createOrUpdateCephUser(u *cephv1.CephObjectStoreUser) error {
+func (r *ReconcileObjectStoreUser) createOrUpdateCephUser(u *cephv1.CephObjectStoreUser) (bool, error) {


I don't like including secret interpretation logic in this function. That starts to blur the lines about what this function does, and IMO it breaks the single responsibility principle. This makes it challenging to understand what the function does when reading the code and makes future changes harder.

It also makes unit testing harder. When functions are separated, it's easy to test complex behavior in unit tests to make sure the individual requirements are correct.

Practicing how to keep functions, structs, and methods focused on a single purpose helps us create code that is easier to test, easier to read, easier to maintain, and less likely to have bugs. We should all practice it and try to help each other do so in order to keep code quality high.

Writing quality code takes more time, but my experience has shown that it's worth it for the benefits I mentioned. As a maintainer of the Rook project, I have more interest in high-quality code that takes longer to write, versus code that is the quickest thing that can be written.

How I think of this is that I don't write code for the present day; I write code for 2+ years in the future, when I have forgotten what I talked about in meetings and when I am likely to need to extend its functionality or fix some obscure bug.

So, to help understand a bit, I can propose what would be a better change that follows the single responsibility principle.

Let's handle secret-interpretation logic separately, using structs and functions dedicated for that purpose.

Here's a high-level design of a struct that can store credentials for the k8s secret.

type Credentials struct { SecretKey string AccessKey string Keys []struct{ SecretKey string AccessKey string } }

This function (createOrUpdateCephUser()) has a singular purpose: "create (or update) an S3 user in the RGW."

With that in mind, we can consider how to alter its behavior to our current needs without changing its single purpose.

My suggestion is to use the Credentials struct as an input so that the create/update behavior can change, while the overall purpose remains the same.

We can also add a boolean parameter to control whether the function should overwrite existing S3 keys or not.

Because there is more complex behavior in the function, it would also be good to return the new credentials at the end of the function. This ensures that the calling routine will always get the most correct credentials after the function is returned, and it doesn't need to predict the output based on the input.

So here's the final function signature (with comments explaining behavior):

Suggested change

func (r *ReconcileObjectStoreUser) createOrUpdateCephUser(u *cephv1.CephObjectStoreUser) (bool, error) {

// behavior:

// - overwriteExistingCreds == false:

// - creds == <non-empty>

// - if creating a new user, the user should have these creds

// - if a user already exists, user's existing creds will be used, even if they are different

// - creds == <empty>

// - if creating a new user, generate random creds

// - if user already exists, user's existing creds will be used

// - overwriteExistingCreds == true:

// - creds = <empty> - ERROR: creds cannot be empty

// - creds = <non-empty> - always overwrite existing Creds with input creds

func (r *ReconcileObjectStoreUser) createOrUpdateCephUser(u *cephv1.CephObjectStoreUser, creds Credentials, overwriteExistingCreds bool) (Credentials, error) {

I'd encourage you to also take a look at the other 4 SOLID design principles in this article: https://stackify.com/solid-design-principles/

Single Responsibility Principle

Open/Closed Principle

Liskov Substitution Principle

Interface Segregation Principle

Dependency Inversion - Go really encourages this with Interfaces

The article talks in terms of object oriented design (like Java) whereas Golang doesn't have objects/inheritance. Instead, Golang has structs and interfaces, and it does builds behavior using "composition" rather than "inheritance".

Dave Cheney has an article explaining SOLID principles in Golang that I recommend as a follow-up: https://dave.cheney.net/2016/08/20/solid-go-design

Updated most of the code thanks for the feedback,

If you can take at the new push and can probably tell where I can introduce Credentials

Instead of creating a separate credential struct, I am using the UserKeySpec struct, which is defined in goceph.
Benefits:

Easy to marshal

All the rgw go ceph API calls need this, if we create something specific we then again need to update the UserKeySpec

In future if anything more required for the secret we can just update the credentials, and it will be auto handled

type UserKeySpec struct { User string `json:"user"` AccessKey string `json:"access_key" url:"access-key"` SecretKey string `json:"secret_key" url:"secret-key"` // Request fields UID string `url:"uid"` // The user ID to receive the new key SubUser string `url:"subuser"` // The subuser ID to receive the new key KeyType string `url:"key-type"` GenerateKey *bool `url:"generate-key"` // Generate a new key pair and add to the existing keyring }

We don't want to marshal all of this stuff into the Credentials info for users, who will find it unnecessary and confusing. Generally, things work best when the spec that users see via the API is a subset of the spec used internally. But APIs become "leaky abstractions" when the internal structs are exposed to users in API surfaces. These leaky abstractions are usually complicated for users to understand, and because they have fields that users aren't supposed to be allowed to interact with, their behavior is often undefined, or worse, broken.

We should only put stuff into Credentials that we want users to be able to modify. That means we should define a separate spec for what users see.

We can retain code reuse by using golang's struct inlining like this:

// secrets package type Credential struct { AccessKey string `json:"access_key" url:"access-key"` SecretKey string `json:"secret_key" url:"secret-key"` } // admin package type UserKeySpec struct { secrets.Credential User string `json:"user"` // Request fields UID string `url:"uid"` // The user ID to receive the new key SubUser string `url:"subuser"` // The subuser ID to receive the new key KeyType string `url:"key-type"` GenerateKey *bool `url:"generate-key"` // Generate a new key pair and add to the existing keyring }

// behavior:
// - overwriteExistingCreds == false:
// - creds ==
// - if creating a new user, the user should have these creds

Why user should have these creds until they have specific source of truth to these.

// - if a user already exists, user's existing creds will be used, even if they are different // - creds == <empty> // - if creating a new user, generate random creds // - if user already exists, user's existing creds will be used // - overwriteExistingCreds == true: // - creds = <empty> - ERROR: creds cannot be empty

We can still create them looking at the K8S secret access key and secret values and updating the cred with those values.
// - creds = - always overwrite existing Creds with input creds

Because the user (or CI test environments) may create the configmap before Rook is installed to pre-specify the credentials without wanting to specifically set the annotation.

In the documentation we have documented a proper section if you want to create a secret before rook is installed or update the secret later, we should do the following steps.

So that covers the above step.

So I believe users should always add an annotation if the secrets are managed by users, as it is being the source of truth.

That is a reasonable stance to take, I think. :)

BlaineEXE · 2024-04-19T16:54:58Z

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

+  AccessKey: ***
+  SecretKey: ***
+  Endpoint: ***
+  SecretKeys: ***


Now that we know that this is a list of secret-key/access-key pairs, let's not call this SecretKeys since that implies it is a list of only the secret-key parts. Instead, we can call this Credentials or Keys. @travisn thoughts on naming?

pkg/operator/ceph/object/user/controller.go

BlaineEXE · 2024-04-19T17:29:35Z

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

+### CephObjectStoreUser Reference Secret
+
+If a specific user key and secret is desired instead of randomly generated credentials, a specific user key and secret can be specified for an object store user.


It seems like we don't have pre-existing documentation that explains how users can read the secret. In this PR, the docs immediately jump to explaining how users can modify the secret themselves, but we expect this to be an advanced use-case. Most users will only want to read the secret. How can we structure this doc so that it explains how users can use the secret first, and then explains how users can modify it?

Should I add in this same document or create a separate file for this?

parth-gr · 2024-04-23T14:38:17Z

Testing:
Updating the secret

[rider@fedora examples]$ kubectl get secrets -nrook-ceph rook-ceph-object-user-my-store-my-user -o yaml
apiVersion: v1
data:
  AccessKey: **
  Credentials: **
  Endpoint: aHR0cDovL3Jvb2stY2VwaC1yZ3ctbXktc3RvcmUucm9vay1jZXBoLnN2Yzo4MA==
  SecretKey: **
kind: Secret
metadata:
  creationTimestamp: "2024-04-23T13:00:23Z"
  labels:
    app: rook-ceph-rgw
    rook_cluster: rook-ceph
    rook_object_store: my-store
    user: my-user
  name: rook-ceph-object-user-my-store-my-user
  namespace: rook-ceph
  ownerReferences:
  - apiVersion: ceph.rook.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: CephObjectStoreUser
    name: my-user
    uid: 6bc8cf02-4eb5-4779-bee0-3a899eff4bd9
  resourceVersion: "8256"
  uid: ce8cc7ee-3b5d-44b2-b0ad-e1c52ffe8c7d
type: kubernetes.io/rook

Updated credentials,

[{"access_key":"IE58RNT71Y2F1EQE80RA","secret_key":"cULyMz5dCpX18dPsJhpIKay7vcDNRNJWJPu8VqUA"}, {"access_key":"IE58RNT71Y2F1EQE80RC","secret_key":"cULyMz5dCpX18dPsJhpIKay7vcDNRNJWJPu8VqUB"}]

Updates the ceph user new creds and remove creds which are not in the secret

sh-4.4$ radosgw-admin user info --uid=my-user
{
    "user_id": "my-user",
    "display_name": "my display name",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "my-user",
            "access_key": "5IHXOX0PE4WXCZ1CW0CA",
            "secret_key": "ADSSDqgNGXjUbjSZ8QP1Nl9w0grvEfapkcCV5g7u"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

sh-4.4$ radosgw-admin user info --uid=my-user
{
    "user_id": "my-user",
    "display_name": "my display name",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "my-user",
            "access_key": "IE58RNT71Y2F1EQE80RA",
            "secret_key": "cULyMz5dCpX18dPsJhpIKay7vcDNRNJWJPu8VqUA"
        },
        {
            "user": "my-user",
            "access_key": "IE58RNT71Y2F1EQE80RC",
            "secret_key": "cULyMz5dCpX18dPsJhpIKay7vcDNRNJWJPu8VqUA"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md

pkg/operator/ceph/object/secret/secret.go

BlaineEXE · 2024-04-26T23:08:40Z

pkg/operator/ceph/object/secret/secret.go

+	return credentials, errors.Errorf("secret keys data is invalid, please update the secret with cvalid format")
+}
+
+func UnmarshalKeys(credentials []byte) ([]admin.UserKeySpec, error) {


There isn't a marshal equivalent to go back the other direction?

didnt get it

pkg/operator/ceph/object/secret/secret.go

pkg/operator/ceph/object/user/controller.go

CephObjectStoreUser should optionally be able to reference a secret where S3 key is defined. This enables us to specify the accesskey and accesssecret rather than those values being randomly generated. Closes: rook#11563 Signed-off-by: parth-gr <partharora1010@gmail.com>

parth-gr requested a review from travisn April 1, 2024 12:54

parth-gr commented Apr 1, 2024

View reviewed changes

parth-gr changed the title ~~object: add support for user secret for cephobjectstoreuser~~ object: add support for user s3 key for cephobjectstoreuser Apr 1, 2024

parth-gr force-pushed the object-secret branch 2 times, most recently from e041b45 to 1b33097 Compare April 1, 2024 13:15

sp98 reviewed Apr 1, 2024

View reviewed changes

parth-gr force-pushed the object-secret branch 2 times, most recently from df286e7 to 42d685f Compare April 1, 2024 15:11

travisn requested changes Apr 1, 2024

View reviewed changes

parth-gr force-pushed the object-secret branch from 42d685f to 2d0d3d6 Compare April 2, 2024 15:14

parth-gr requested a review from travisn April 2, 2024 15:15

BlaineEXE requested changes Apr 2, 2024

View reviewed changes

pkg/operator/ceph/object/user/controller.go Outdated Show resolved Hide resolved

parth-gr force-pushed the object-secret branch from 2d0d3d6 to 1618f54 Compare April 3, 2024 14:28

BlaineEXE marked this pull request as draft April 3, 2024 17:02

parth-gr force-pushed the object-secret branch from 1618f54 to 04bc74d Compare April 16, 2024 13:46

parth-gr marked this pull request as ready for review April 16, 2024 13:46

parth-gr requested review from BlaineEXE and sp98 April 16, 2024 13:46

parth-gr force-pushed the object-secret branch 6 times, most recently from c989e5a to 461ce29 Compare April 17, 2024 14:56

BlaineEXE reviewed Apr 17, 2024

View reviewed changes

pkg/operator/ceph/object/user/controller.go Outdated Show resolved Hide resolved

parth-gr force-pushed the object-secret branch from 461ce29 to 706e8b3 Compare April 17, 2024 16:17

parth-gr requested a review from BlaineEXE April 17, 2024 16:18

parth-gr commented Apr 18, 2024

View reviewed changes

Documentation/CRDs/Object-Storage/ceph-object-store-user-crd.md Show resolved Hide resolved

parth-gr force-pushed the object-secret branch 5 times, most recently from f83ccbe to 01a09ba Compare April 19, 2024 14:46

parth-gr requested a review from BlaineEXE April 19, 2024 14:46

BlaineEXE requested changes Apr 19, 2024

View reviewed changes

parth-gr force-pushed the object-secret branch 2 times, most recently from 1510c36 to bde4b6e Compare April 22, 2024 15:11

parth-gr requested a review from BlaineEXE April 22, 2024 15:12

parth-gr force-pushed the object-secret branch 4 times, most recently from 26ebbe6 to 5ab40eb Compare April 23, 2024 14:36

BlaineEXE requested changes Apr 26, 2024

View reviewed changes

parth-gr force-pushed the object-secret branch 10 times, most recently from 95134c3 to a80ee32 Compare April 29, 2024 16:47

parth-gr requested a review from BlaineEXE April 29, 2024 16:47

parth-gr force-pushed the object-secret branch from a80ee32 to 16f661f Compare April 30, 2024 06:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

object: add support for user s3 key for cephobjectstoreuser #14001

object: add support for user s3 key for cephobjectstoreuser #14001

parth-gr commented Apr 1, 2024

parth-gr Apr 1, 2024

sp98 Apr 1, 2024

sp98 Apr 1, 2024

BlaineEXE left a comment

travisn commented Apr 2, 2024

parth-gr commented Apr 3, 2024

BlaineEXE commented Apr 3, 2024

BlaineEXE left a comment •

edited

BlaineEXE Apr 19, 2024

parth-gr Apr 22, 2024

parth-gr Apr 22, 2024

BlaineEXE Apr 19, 2024

parth-gr Apr 22, 2024

BlaineEXE Apr 19, 2024

parth-gr Apr 22, 2024

parth-gr Apr 23, 2024

BlaineEXE Apr 26, 2024

parth-gr Apr 29, 2024

parth-gr Apr 29, 2024

BlaineEXE Apr 30, 2024

parth-gr May 2, 2024 •

edited

BlaineEXE May 3, 2024

BlaineEXE Apr 19, 2024

BlaineEXE Apr 19, 2024

parth-gr Apr 22, 2024

parth-gr commented Apr 23, 2024 •

edited

BlaineEXE Apr 26, 2024

parth-gr Apr 29, 2024

		[{"user":"","access_key":"IE58RNT71Y2F1EQE80RA","secret_key":"cULyMz5dCpX18dPsJhpIKay7vcDNRNJWJPu8VqUA","UID":"","SubUser":"","KeyType":"","GenerateKey":null},
		{"user":"","access_key":"IE58RNT71Y2F1EQE80RZ","secret_key":"cULyMz5dCpX18dPsJhpIKay7vcDNRNJWJPu8VqUE","UID":"","SubUser":"","KeyType":"","GenerateKey":null}]

-func (r *ReconcileObjectStoreUser) createOrUpdateCephUser(u *cephv1.CephObjectStoreUser) (bool, error) {
+// behavior:
+//   - overwriteExistingCreds == false:
+//     - creds == <non-empty>
+//       - if creating a new user, the user should have these creds
+//       - if a user already exists, user's existing creds will be used, even if they are different
+//     - creds == <empty>
+//       - if creating a new user, generate random creds
+//       - if user already exists, user's existing creds will be used
+//   - overwriteExistingCreds == true:
+//     - creds = <empty> - ERROR: creds cannot be empty
+//     - creds = <non-empty> - always overwrite existing Creds with input creds
+func (r *ReconcileObjectStoreUser) createOrUpdateCephUser(u *cephv1.CephObjectStoreUser, creds Credentials, overwriteExistingCreds bool) (Credentials, error) {

		### CephObjectStoreUser Reference Secret

		If a specific user key and secret is desired instead of randomly generated credentials, a specific user key and secret can be specified for an object store user.

object: add support for user s3 key for cephobjectstoreuser #14001

Are you sure you want to change the base?

object: add support for user s3 key for cephobjectstoreuser #14001

Conversation

parth-gr commented Apr 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BlaineEXE left a comment

Choose a reason for hiding this comment

travisn commented Apr 2, 2024

parth-gr commented Apr 3, 2024

BlaineEXE commented Apr 3, 2024

BlaineEXE left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parth-gr May 2, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parth-gr commented Apr 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BlaineEXE left a comment •

edited

parth-gr May 2, 2024 •

edited

parth-gr commented Apr 23, 2024 •

edited