Research: Dynamic Configuration

https://github.com/vmware/versatile-data-kit/issues/1201 outlines one possible solution for dynamic configuration

The solution is focused to allow administrators to set configuration options dynamically without needing to redeploy jobs. Basicallly also solve https://github.com/vmware/versatile-data-kit/issues/1722

The below research focuses on enabling users to do the same thing. Let's consider reusing Properties and Secrets API that already exists in the control service to enable the same thing This would simplify architecture as we will not be adding new components but reusing existing ones

Definition:

• User properties: arbitrary key/value pairs user can set , the value could be complex structure (with limited size 1KB) , user use them to keep state (last process time) or secrets (API keys)
• System configuration: Specific and documented configuration operations by Versatile Data Kit SDK. They control different aspects of the SDK
• Data Job - deployable software piece
• Team - a way to group jobs - multiple jobs can belong to the same team. A team could be analogous to LDAP Group.

Requirements

1. User should be able to user Properties and Secretes API to set arbitrary key/value pairs , the value could be complex structure (with limited size 1KB)
2. User should be able to set properties per team or system configuration if they have permissions for that team
3. Administrators should be able to set system configuration
4. Administrators should be able to set system configuration per team
5. Administrators should be able to set system configuration globally for all teams (and hence all data jobs)
6. Admins should have the option to lock specific system configurations so that they cannot be modified by users or simply any administration set setting has a higher precedence.
7. Changes to properties and configurations should take effect in real-time or near real-time without requiring a system restart or redeployment.

Other requirements that are not required for this research/design (so would likely not happen):

• Changes to user properties and system configurations should be versioned for auditing and rollback purposes.
• Users and administrators should be able to review the history of changes to configurations.
• Values set for system configurations should be subject to validation constraints, as defined by the SDK or system administrators.

To support both user settings and system settings, there are several approaches we can consider.

Option 1: Extend Existing Endpoints

Extend the existing API to include a flag indicating whether the secret is a system setting or user setting. PUT API

{
  "type": "user", // or "system"
  "secrets": {"my-custom-key": "value"}
}

GET API

{
  "user": {"my-custom-key": "value"},
  "system": {"db-password": "value"}
}

Option 2: Separate Endpoints

Create separate endpoints for managing system settings and user settings.

• /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/user-secrets
• /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/system-secrets

Option 3: URL Parameters

Use URL parameters to indicate the type of settings.

/data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/secrets?type=system

Option 4: Role-based Access

If system settings should only be editable by administrators, we can implement role-based access control. Any secret added by an admin could automatically be considered a system setting.

Option 5: Namespaced Secrets

Use a naming convention to distinguish between user and system settings. This could be implemented in the backend logic or required in the API request payload.

PUT API

{
  "secrets": {"user.my-custom-key": "value", "system.db-password": "value"}
}

GET API

{
  "user.my-custom-key": "value",
  "system.db-password": "value"
}

The scores range from 1 to 5, with 5 being the best

Criteria	Extend Existing Endpoints (1)	Separate Endpoints (2)	URL Parameters (3)	Role-based Access (4)	Namespaced Secrets (5)
Ease of Implementation	5	3	4	1	4
Backward Compatibility	4	2	5	4	5
User Experience	3	4	3	4	2
Flexibility	2	4	5	2	4
Security	3	4	3	5	3
Maintainability	3	5	4	3	3
Final Score	20	22	24	19	21