Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move s3 asset_storage settings to teraslice config #3614

Merged
merged 21 commits into from
May 17, 2024
Merged

Move s3 asset_storage settings to teraslice config #3614

merged 21 commits into from
May 17, 2024

Conversation

sotojn
Copy link
Contributor

@sotojn sotojn commented May 14, 2024 •
edited

This PR makes the following changes:

  • **Deprecated** all asset_storage configuration settings in terafoundation
    • These configuration settings now live in the teraslice config and will be removed from terafoundation in teraslice 2.0
    • The teraslice asset_storage config settings will take priority of the terafoundation asset_storage settings if both are set.
  • Updated documentation with information on deprecation.
  • Bumps the following packages:
    • (patch) @terascope/job-components to v0.74.2
    • (patch) @terascope/types to v0.17.2
    • (patch) @terascope/data-mate to v0.56.2
    • (patch) elasticsearch-store to v0.84.2
    • (patch) ts-transforms to v0.85.2
    • (patch) xlucene-parser to v0.58.2
    • (patch) xlucene-translator to v0.44.2
    • (patch) @terascope/elasticsearch-api to v3.20.2
    • (patch) @terascope/teraslice-state-storage to v0.53.2
    • (minor) terafoundation to v0.64.0
  • Bumps teraslice to from v1.5.1 to v1.6.0

Ref to issue #3601

@sotojn sotojn self-assigned this May 14, 2024
@sotojn sotojn requested review from godber, jsnoble and busma13 May 14, 2024 22:40
busma13
busma13 reviewed May 15, 2024
View reviewed changes
packages/teraslice/src/lib/config/schemas/system.ts Outdated Show resolved Hide resolved
packages/teraslice/src/lib/storage/assets.ts Outdated Show resolved Hide resolved
packages/teraslice/src/lib/storage/assets.ts Outdated Show resolved Hide resolved
packages/terafoundation/src/validate-configs.ts Outdated Show resolved Hide resolved
@sotojn sotojn marked this pull request as ready for review May 15, 2024 16:58
sotojn added 13 commits May 15, 2024 11:44
@sotojn
Add deprication labels to docs
3b2da96
@sotojn
add asset_storage settings to teraslice config types
0e7f434
@sotojn
add asset_storage settings into teraslice schema
13c14be
@sotojn
fix extractSchema func in terafoundation
0c86e0d
@sotojn
add more types
1d870e0
@sotojn
add logic that favors teraslice over terafoundation asset_storage con…
10f0563 
…figs
@sotojn
bump: (patch) @terascope/job-components@0.74.2, @terascope/types@0.17.2
e1630ff 
bump: (patch) @terascope/utils@0.59.2, @terascope/data-types@0.50.2

bump: (patch) @terascope/data-mate@0.56.2, elasticsearch-store@0.84.2

bump: (patch) terafoundation@0.63.2, ts-transforms@0.85.2

bump: (patch) xlucene-parser@0.58.2, xlucene-translator@0.44.2

bump: (patch) @terascope/elasticsearch-api@3.20.2, @terascope/teraslice-state-storage@0.53.2
@sotojn
release: (minor) teraslice@1.6.0
1c897da 
bump: (minor) terafoundation@0.64.0
@sotojn
fix assets_storage_spec test
a2c3540
@sotojn
refactor getBackendConfig() method
ecddba9
@sotojn
refactor validatorFn in teraslice config schema
3a9baec
@sotojn
refactor config_schema in teraslice
b8a03d5
@sotojn
Destructure assetConfig in AssetsStorage class
770a3ae
jsnoble
jsnoble requested changes May 15, 2024
View reviewed changes
packages/job-components/src/test-helpers.ts Outdated Show resolved Hide resolved
packages/job-components/src/interfaces/context.ts Outdated Show resolved Hide resolved
packages/teraslice/src/lib/cluster/services/assets.ts Outdated Show resolved Hide resolved
packages/teraslice/src/lib/config/schemas/system.ts Outdated Show resolved Hide resolved
packages/teraslice/src/lib/config/schemas/system.ts Outdated Show resolved Hide resolved
godber
godber requested changes May 17, 2024
View reviewed changes
Copy link
Member

@godber godber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix all the misspelled words.

docs/configuration/overview.md Outdated Show resolved Hide resolved
docs/configuration/overview.md Outdated Show resolved Hide resolved
packages/terafoundation/src/schema.ts Outdated Show resolved Hide resolved
packages/terafoundation/src/schema.ts Outdated Show resolved Hide resolved
packages/terafoundation/src/schema.ts Outdated Show resolved Hide resolved
packages/terafoundation/src/schema.ts Outdated Show resolved Hide resolved
jsnoble
jsnoble approved these changes May 17, 2024
View reviewed changes
Copy link
Member

@jsnoble jsnoble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@godber godber merged commit 718ec56 into master May 17, 2024
0 of 40 checks passed
@godber godber deleted the move-s3-asset-storage branch May 17, 2024 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@busma13 busma13 busma13 left review comments

@godber godber godber approved these changes

@jsnoble jsnoble jsnoble approved these changes

Assignees

@sotojn sotojn

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@sotojn @godber @jsnoble @busma13