PUT /v1/transform/{name}/meta #18236

oleiman · 2024-05-03T03:07:35Z

Introduce an Admin API endpoint for patching transform metadata

update environment vars with PUT semantics
pause/resume transform

Also adds rpk experience for pause/resume (no env manipulation for now)

TODO:

Probably split admin endpoint into two
- Went with a patch-like endpoint instead
rpk for pause/start
~~rpk for env mutation~~ (PUNT)

Docs issue: https://github.com/redpanda-data/documentation-private/issues/2437

Closes CORE-1653

Backports Required

Release Notes

Improvements

Adds the ability to pause a deployed data transform w/o removing it from the system

oleiman · 2024-05-03T03:08:01Z

/dt

rockwotj · 2024-05-03T14:26:34Z

Probably split admin endpoint into two

An alternative is to have an endpoint like: PATCH /transform/{name} where you can patch the transform object. Parameters would include environment variables, is paused and maybe some others? @mo-redpanda this patch endpoint could support updating environment variables only for console based transforms.

src/v/model/transform.h

src/v/redpanda/admin/api-doc/transform.json

src/v/transform/api.cc

src/v/transform/tests/transform_manager_test.cc

src/v/transform/transform_manager.cc

src/v/redpanda/admin/api-doc/transform.json

oleiman · 2024-05-03T14:57:55Z

Probably split admin endpoint into two

An alternative is to have an endpoint like: PATCH /transform/{name} where you can patch the transform object. Parameters would include environment variables, is paused and maybe some others? @mo-redpanda this patch endpoint could support updating environment variables only for console based transforms.

I like that idea. Unfortunately our seastar http mess doesn't support PATCH, so maybe PUT /transform/{name}/meta or something like that?

oleiman · 2024-05-03T14:58:31Z

(or finally go in and see how much work it would be to add PATCH support!)

rockwotj · 2024-05-03T15:15:57Z

I am not a REST purest so fine with PUT

oleiman · 2024-05-05T03:35:47Z

force push contents:

patch metadata instead of pause outright (allows update of transform env)
comments, minor CR changes

oleiman · 2024-05-05T03:41:17Z

force push fix swagger

oleiman · 2024-05-06T00:25:08Z

force push contents:

rpk transform pause
rpk transform start

rockwotj

🌶️

rockwotj · 2024-05-06T02:25:35Z

src/go/rpk/pkg/cli/transform/meta.go

+
+func newStartCommand(fs afero.Fs, p *config.Params) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "start [NAME]",


I would recommend creating an RFC and/or bringing this naming up with Mo + Rogger. Start makes me think I need to start transforms after deploying. Alternative name suggestion: resume (gcloud uses this terminology). Others to consider: unpause

yea, waffled on this a bit and went with what's in the ticket as a placeholder...I'm partial to suspend/resume in this context, but RFC seems prudent.

rockwotj · 2024-05-06T02:26:47Z

src/go/rpk/pkg/cli/transform/meta.go

+
+func newPauseCommand(fs afero.Fs, p *config.Params) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "pause [NAME]",


Alternatively consider the naming gcloud uses: suspend. Not sure if that is more clear or not to see what's happening under the hood.

rockwotj · 2024-05-06T02:29:05Z

src/v/redpanda/admin/api-doc/transform.json

+    },
+      "transform_metadata_patch": {
+          "id": "transform_metadata_patch",
+          "description": "A partial update to transform metadata",


Is there a way to document the semantics? I feel like patch requests are sometimes hard to understand. For example would be good to say things like:

environment map is overridden (so no deep patches)

If not present then it is unchanged

src/v/redpanda/admin/api-doc/transform.json

src/v/model/transform.h

oleiman · 2024-05-06T22:56:00Z

force push contents:

various comments/route descriptions
pause/start -> suspend/resume (tentative, pending input from devex & product)

src/v/redpanda/admin/transform.cc

tests/rptest/tests/data_transforms_test.py

oleiman · 2024-05-07T21:06:34Z

force push - rebase dev

oleiman · 2024-05-07T21:48:30Z

force push contents:

address CR comments (use new seastar path param function, add a patch env test case)
switch over from suspend/resume to pause/resume per discussion on slack and RFC

Also adds a couple of model types: is_transform_paused (bool) transform_metadata_patch - map<str, str> - Overwrite transform environment map (PUT semantics) - paused - Pause or unpause the transform Also adds ::paused to transform_metadata stream operator Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

r-vasquez

rpk changes LGTM, thanks!

src/v/transform/api.cc

src/v/redpanda/admin/api-doc/transform.json

tests/rptest/clients/rpk.py

- Paused transforms show `inactive` partitions in report - Metadata updates with is_paused::yes prevents new processor creation Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Proxy for existing plugin_table method. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Looks up a transform by name, modifies metadata accordingly with the environment map and paused state in the request, and upserts. includes a unit test for the pause update specifically, since it affects processor state. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

patch_transform_metadata From request body reads: - A paused state (optional) - A list of environment_variables (optional) - If present, overwrites the environment map in current metadata If neither field is present in the request body, return success but don't bother propagating it to the cluster. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2024-05-13T20:22:59Z

force push contents:

fix typo in rpk.py and harden DataTransformsTest.test_patch
fix swagger formatting, commit history

oleiman · 2024-05-14T03:49:42Z

CI Failure is test_raft_rpunit

Includes some code to parse a JSON request body into a transform_metadata_patch Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

PUT transform/{name}/meta Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Admin API returns environment variables as a list of objects, but rpk is clever enough to parse those out into a map before dumping json. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

- calls /v1/transform/{name}/meta - patchMetadataRequest body: - IsPaused *bool - Environment *[]EnvironmentVariable Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Pause and resume a transform via Admin API Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Includes both rpk and admin API usage. NOTE: env patches are not yet supported by `rpk transform` Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2024-05-15T04:30:55Z

force push adds json::validator for patch request

oleiman self-assigned this May 3, 2024

github-actions bot added area/redpanda area/wasm WASM Data Transforms labels May 3, 2024

rockwotj reviewed May 3, 2024

View reviewed changes

oleiman force-pushed the xform/core-1653/pause-transform branch from 2d26643 to 5d87f40 Compare May 5, 2024 01:14

oleiman changed the title ~~Pause a transform~~ PUT /v1/transform/{name}/meta May 5, 2024

oleiman force-pushed the xform/core-1653/pause-transform branch from 5d87f40 to f5d4848 Compare May 5, 2024 03:40

oleiman force-pushed the xform/core-1653/pause-transform branch from f5d4848 to 9072e0c Compare May 6, 2024 00:24

github-actions bot added the area/rpk label May 6, 2024

rockwotj reviewed May 6, 2024

View reviewed changes

oleiman force-pushed the xform/core-1653/pause-transform branch from 9072e0c to 0572830 Compare May 6, 2024 22:55

pgellert reviewed May 7, 2024

View reviewed changes

src/v/redpanda/admin/transform.cc Outdated Show resolved Hide resolved

tests/rptest/tests/data_transforms_test.py Outdated Show resolved Hide resolved

oleiman force-pushed the xform/core-1653/pause-transform branch from 0572830 to eb29a27 Compare May 7, 2024 21:06

oleiman force-pushed the xform/core-1653/pause-transform branch from eb29a27 to 2f90747 Compare May 7, 2024 21:45

oleiman marked this pull request as ready for review May 7, 2024 21:48

oleiman requested review from twmb, r-vasquez, gene-redpanda and Deflaimun as code owners May 7, 2024 21:48

oleiman dismissed pgellert’s stale review via 0074461 May 10, 2024 21:01

oleiman force-pushed the xform/core-1653/pause-transform branch from fd1ff3e to 0074461 Compare May 10, 2024 21:01

oleiman requested review from r-vasquez and pgellert May 10, 2024 21:02

r-vasquez previously approved these changes May 10, 2024

View reviewed changes

andijcr reviewed May 13, 2024

View reviewed changes

src/v/transform/api.cc Show resolved Hide resolved

src/v/redpanda/admin/api-doc/transform.json Outdated Show resolved Hide resolved

tests/rptest/clients/rpk.py Outdated Show resolved Hide resolved

oleiman added 4 commits May 13, 2024 11:36

transform: Respect paused state

abe2186

- Paused transforms show `inactive` partitions in report - Metadata updates with is_paused::yes prevents new processor creation Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

c/plugin_frontend: lookup_transform by name

8edb42d

Proxy for existing plugin_table method. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman dismissed r-vasquez’s stale review via b6f724d May 13, 2024 20:20

oleiman force-pushed the xform/core-1653/pause-transform branch from 0074461 to b6f724d Compare May 13, 2024 20:20

oleiman requested a review from andijcr May 13, 2024 20:21

andijcr previously approved these changes May 13, 2024

View reviewed changes

oleiman added 7 commits May 14, 2024 21:23

admin/transform: Wire up patch_transform_metadata

2e747cb

Includes some code to parse a JSON request body into a transform_metadata_patch Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

dt/admin: Introduce Admin.transforms_patch_meta

b7891fd

PUT transform/{name}/meta Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

dt/rpk: Fix bug in rpk transform list json output processing

e7c44ff

Admin API returns environment variables as a list of objects, but rpk is clever enough to parse those out into a map before dumping json. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/admin: Introduce PauseTransform, ResumeTransform

97d429c

- calls /v1/transform/{name}/meta - patchMetadataRequest body: - IsPaused *bool - Environment *[]EnvironmentVariable Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/transform: Introduce transform pause, transform resume

cb52295

Pause and resume a transform via Admin API Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

dt/rpk: Add Rpk.pause_wasm and Rpk.resume_wasm

1625924

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

dt/data_transforms: Test for patch_transform_metadata endpoint

e9fd450

Includes both rpk and admin API usage. NOTE: env patches are not yet supported by `rpk transform` Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman dismissed andijcr’s stale review via e9fd450 May 15, 2024 04:29

oleiman force-pushed the xform/core-1653/pause-transform branch from b6f724d to e9fd450 Compare May 15, 2024 04:29

oleiman requested a review from a team as a code owner May 15, 2024 04:29

andijcr approved these changes May 15, 2024

View reviewed changes

oleiman merged commit d0eedf7 into redpanda-data:dev May 15, 2024
25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PUT /v1/transform/{name}/meta #18236

PUT /v1/transform/{name}/meta #18236

oleiman commented May 3, 2024 •

edited

oleiman commented May 3, 2024

rockwotj commented May 3, 2024

oleiman commented May 3, 2024

oleiman commented May 3, 2024

rockwotj commented May 3, 2024

oleiman commented May 5, 2024 •

edited

oleiman commented May 5, 2024

oleiman commented May 6, 2024

rockwotj left a comment

rockwotj May 6, 2024

oleiman May 6, 2024

rockwotj May 6, 2024

rockwotj May 6, 2024

oleiman commented May 6, 2024

oleiman commented May 7, 2024

oleiman commented May 7, 2024

r-vasquez left a comment

oleiman commented May 13, 2024

oleiman commented May 14, 2024

oleiman commented May 15, 2024

PUT /v1/transform/{name}/meta #18236

PUT /v1/transform/{name}/meta #18236

Conversation

oleiman commented May 3, 2024 • edited

Backports Required

Release Notes

Improvements

oleiman commented May 3, 2024

rockwotj commented May 3, 2024

oleiman commented May 3, 2024

oleiman commented May 3, 2024

rockwotj commented May 3, 2024

oleiman commented May 5, 2024 • edited

oleiman commented May 5, 2024

oleiman commented May 6, 2024

rockwotj left a comment

Choose a reason for hiding this comment

rockwotj May 6, 2024

Choose a reason for hiding this comment

oleiman May 6, 2024

Choose a reason for hiding this comment

rockwotj May 6, 2024

Choose a reason for hiding this comment

rockwotj May 6, 2024

Choose a reason for hiding this comment

oleiman commented May 6, 2024

oleiman commented May 7, 2024

oleiman commented May 7, 2024

r-vasquez left a comment

Choose a reason for hiding this comment

oleiman commented May 13, 2024

oleiman commented May 14, 2024

oleiman commented May 15, 2024

oleiman commented May 3, 2024 •

edited

oleiman commented May 5, 2024 •

edited