Skip to content

Deployment Details

Jump to bottom
John Kellerman edited this page Mar 6, 2024 · 2 revisions

Deployment Details

https://github.com/eclipse/openvsx

Core implementation

  • Java/Spring server app
  • Postgres DB
  • ElasticSearch
  • Blob storage (Azure or Google Cloud)
  • React/TypeScript UI

https://github.com/EclipseFdn/open-vsx.org

Deployment specific details for open-vsx.org deployment

https://github.com/open-vsx/publish-extensions

Scripts and Github actions for publishing

  • extensions.json file for list of extensions to auto-publish
  • Github action for nightly auto-publishing
  • log of results
  • Other Github actions for publishing extensions

Publishing an Extension

Packaging

  • Create a .vsix file with the vsce package command
  • Minimal package includes extension.js and package.json

Publishing

  • Click 'Publish' and drag and drop .vsix file in the UI.
  • Use the ovsx publish --p [personal access token] command. Requires a personal access token one generates from profile settings.
  • Because file processing is asynchronous, extension will temporarily show as inactive until file processing completes, assuming no errors.

Deployment

Basic implementation architecture

Documentation on architecture of implementation and instructions for building and deploying.

Database tables - Core tables

user_data
extension
extension_review
extension_version
file_resource
namespace
namespace_membership
namespace_social_links
personal_access_token

Recurring Jobs

  • update-download-counts - hourly
  • MonthlyAdminStatistics - monthly
  • ElasticSearchUpdateIndex - daily
  • DataMirror, if enabled - configurable

Background processes

  • File processing on publish
  • Namespace rename/merge
  • DB migration
  • Extension file removal
  • Generate/delete keypair for extension signing
  • Create/delete extension signature

Open-vsx.org Deployment

Production: open-vsx.org

Staging: staging.open-vsx.org (creds required for access)

Deployment details

  • Production Kube Cluster
    • 2 server pods (with up to 5 CPUs and 8 GB of RAM)
    • 3 elastic search pods (with up to 4 CPUs and 6 GB of RAM)
  • Staging Kube Cluster
    • 1 server pod (with up to 1 CPU and 2 GB of RAM)
    • 1 elastic search pod (with up to 1 CPU and 2 GB of RAM)
  • Spring Boot
    • v3.1.0
  • Java
    • eclipse-temurin:17.0.7_7-jdk
  • Postgres
    • version?
    • 39GB
  • Elastic
    • v8.7.1
    • 5MB
  • Azure Blob storage
    • 1.5TB

Monitoring and Alerting

  • Better Uptime

Deployment Process

  1. SRE creates a pull request in the eclipse/openvsx repository.
  2. GitHub runs Eclipse Contributor Agreement action to check that the committer has signed the Eclipse Contributor Agreement.
  3. GitHub action runs unit tests and integration tests.
  4. SRE merges the pull request into the eclipse/openvsx master branch.
  5. SRE creates new release in the eclipse/openvsx repository:
    1. SRE merges @dependabot pull requests to fix vulnerabilities.
    2. SRE drafts a new release and publishes it.
    3. GitHub action creates server and webui Docker images.
    4. if webui has changed, then SRE publishes new openvsx-webui version to NPM.
    5. if cli has changed, then SRE publishes new ovsx version to NPM.
    6. SRE updates release post with links to server and webui Docker images and, if changed, openvsx-webui and ovsx NPM packages.
  6. SRE updates DockerFile in amvanbaren/open-vsx.org repository with new openvsx-server Docker image version and, if changed, new openvsx-webui package version.
  7. if openvsx-webui version has changed, then SRE updates the openvsx-webui version in the website/package.json file and runs yarn install to update the yarn.lock file.
  8. SRE commits and pushes the changes to a new branch in the amvanbaren/open-vsx.org repository.
  9. SRE creates pull request to merge the new amvanbaren/open-vsx.org branch into EclipseFdn/open-vsx.org repository main branch.
  10. GitHub runs Eclipse Contributor Agreement action to check that the committer has signed the Eclipse Contributor Agreement.
  11. GitHub action kicks off Jenkins job to verify the new Docker image.
  12. SRE merges the pull request into the main branch.
  13. GitHub action kicks off Jenkins job to build the Docker image and deploy it to staging.
  14. SRE checks the staging server and does some quick smoke testing.
  15. SRE creates pull request in EclipseFdn/open-vsx.org repository to merge the main branch into the production branch.
  16. Independent review and approval of the pull request.
  17. SRE merges the pull request into the production branch.
  18. GitHub action kicks off Jenkins job to build the Docker image and deploy it to production.
  19. SRE monitors production for ~15 min to make sure the deployment succeeded and the server is stable.

Common Error Situations and Remedies

  1. ElasticSearch unresponsive:
    1. Delete ElasticSearch pods.
    2. In server application.yml set ovsx.elasticsearch.clear-on-start: true.
    3. Wait until all ElasticSearch pods have started.
    4. Deploy server application.
    5. In server application.yml remove ovsx.elasticsearch.clear-on-start: true.
  2. Open VSX server unresponsive:
    1. Delete Open VSX server pod.
    2. Wait until new pod has started.
    3. Repeat the previous steps until all Open VSX server pods are renewed.
  3. Too much load:
    1. Delete pod that is under load.
    2. Wait until new pod has started.

Administration

  • Administration requests appear as issues.

Namespace ownership

  • Ensure there is not a compelling reason a person should not be granted ownership, e.g. a random person requesting ownership of 'redhat' namespace.
    • There is no current owner
    • There are no other contributors ('Openvsx', auto-publishing ID, is a special case)
    • If there are existing contributors to the namespace, attempt to contact them by Github ID in the namespace ownership issue.
  • If the namespace doesn't exist, create it.
  • Assign the user (by Github ID) as owner, by adding or by changing status from 'Contributor' to 'Owner'.
  • If extensions are already being auto-published in this namespace, alert the new owner

Un-publishing extension versions

  • This is an intensive process.
  • If un-publishing a larger number of versions, authentication will time out. Keep a second browser tab open to continually refresh.
  • One may need to break up a large number of versions into smaller sets

Renaming/merging namespaces

  • Background process that can take some time (exponential back off/retry)
  • Ensure all involved in the namespace are aware of the pending change

Questions about licenses and potentially malicious extensions

Merging Auto-publish PRs

Clone this wiki locally