Replace ajv with zod; simplified plugin API (#1441)

* Create zod schema for 1.0.16 * Move to separate package * Revert main package.json changes * Use builder functions * wip: previous versions * Upgrade functions * Initial tests * VitS as controlled component * Upgrade in <Vitessce/> * WIP: plugins registered via props (instead of window) * Update schemas * Joint file type schemas and via new plugin mechanism * Refactor plugin classes * Cleanup * Update * More cleanup * Cleanup register functions * Remove plugins file, use default coordination values from plugin objects * Comment * Use superRefine to check for deprecated coordination types * Revert * Fix tests * Extra imports * Try env.d.ts * Update * Update plugin examples * Linting for obsSets view * Comment * Allow ts in vite bundling config * Fix e2e tests * Update * Fix import * Allow empty props * Update * Remove ajv * Update * Update deps * Lint * Fix tests * Implement plugin classes with typescript * Add config uid property * use .length * Fix docs * Update plugin docs * Add config schema versioning docs * Add useCallback calls to obsSetsManager * Remove unnecessary fromEntries * Implement getSourceAndLoader function * Remove extra import * Eslint comment * Demo * Update dev-config-versioning.md * Update docusaurus.config.js * Lint * Make json schemas * Tsconfig * Lockfile * Update deploy script and readme * Simplify pnpm command * Update pnpm lock * Respond to PR feedback * Fix type issues, update docs * Refactor base plugins * Update * Pnpm install * Types in package.json * Fix tsconfig project references * Pnpm lock * Add dev-docs * Working types and linting * pnpm lock * Update * Fix test * Lint * Pnpm lock * React import * React import * More react imports * Update package.json exports in sub-packages, add new test for dynamic importmap * Add comment * Comment * Dev-docs * Add comment * Use dynamic-importmap * Try compressed-size-action * Update * Node options * env * Clean cmd * Revert unrelated github action change * Fix typo * Revert changed ObsSetsManager * Use .js extensions for imports to so that source is ESM (#1519) * Use .js * Comments * Revert webpack plugin * Update cypress e2e tests * Add/move dev-docs * Update design-guidelines.md * Fix meta-updater * Revert config * Update design-guidelines.md * Update README.md * Update design-guidelines.md * Update design-guidelines.md * Update design-guidelines.md * Update rollup.config.js * Revert unrelated change * WIP: clean up lodash, mui, uuid imports * More import fixes * De-duplicate imports * Dev-docs about js imports * Use publint * isEqual * Fix tests * Dev-docs * Dev-docs linked from main README * Demo * Feedback * Use same terminology in docs as code * Troubleshooting note * Upgrade guide
vitessce · May 23, 2023 · 33c30b5 · 33c30b5
1 parent 5d69698
commit 33c30b5
Show file tree

Hide file tree

Showing 417 changed files with 8,766 additions and 4,279 deletions.
diff --git a/.eslintrc.yml b/.eslintrc.yml
@@ -1,17 +1,24 @@
+# References:
+# - https://typescript-eslint.io/getting-started/#step-2-configuration
+# - https://stackoverflow.com/a/75108187
 root: true
 parser: '@typescript-eslint/parser'
 extends:
   - eslint:recommended
+  - plugin:require-extensions/recommended
   - plugin:react/recommended
   - plugin:react-hooks/recommended
   - plugin:cypress/recommended
-  - airbnb
+  - plugin:jsx-a11y/recommended
 plugins:
   - '@typescript-eslint'
+  - require-extensions
   - react
   - react-hooks
   - cypress
   - react-refresh
+  - import
+  - jsx-a11y
 settings:
   react:
     version: detect
@@ -26,20 +33,27 @@ rules:
   react/sort-comp: [0]  # Non-alphabetical groupings can make more sense.
   react/jsx-one-expression-per-line: [0]  # Makes punctuation after tab awkward.
   react/prop-types: [0] # Re-enable: https://github.com/vitessce/vitessce/issues/144
-  import/prefer-default-export: [0]  # Hit eslint error: SyntaxError: Unexpected token, expected {
+  react/require-default-props: [0]
   react/react-in-jsx-scope: [0]
   react/jsx-curly-newline: [0]
   react/jsx-no-bind: [0] # Allow passing functions as JSX props
   import/no-extraneous-dependencies: [1]
   react/jsx-props-no-multi-spaces: [0]
   no-plusplus: [0]
   react-refresh/only-export-components: [2, { "checkJS": true }]
+  import/no-duplicates: [2]
 overrides:
-  - files: 'sites/docs/src/theme/**'
+  - files: 'sites/docs/src/{pages,theme}/**'
     rules:
       # Allow @theme/ and @docusaurus/ imports
-      import/no-unresolved: [0]
-  - files: 'sites/docs/src/pages/**'
+      #import/no-unresolved: [0]
+      #import/extensions: [0]
+      '@typescript-eslint/no-var-requires': [0]
+      no-undef: [0]
+  - files: '**/*.{ts,tsx}'
+    extends:
+      - plugin:@typescript-eslint/eslint-recommended
+      - plugin:@typescript-eslint/recommended
     rules:
-      # Allow @theme/ and @docusaurus/ imports
-      import/no-unresolved: [0]
+      # Allow importing js/ts files without extension
+      import/no-unresolved: [0]
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
@@ -26,7 +26,7 @@ jobs:
       - id: get-options
         run: . ./set-node-options.sh --action
         shell: bash
-      - run: pnpm run build:ci && pnpm run bundle:ci
+      - run: pnpm run build:ci && pnpm run bundle:ci && pnpm run build-json-schema
         env:
           NODE_OPTIONS: ${{ steps.get-options.outputs.node-options }}
       - run: ./test.sh --action

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -39,6 +39,8 @@ jobs:
         env:
           NODE_OPTIONS: ${{ steps.get-options.outputs.node-options }}
       - run: ./test.sh --action
+      - run: pnpm run bundle:ci
+      - run: pnpm run publint
 
   test-e2e-demo:
     runs-on: ubuntu-20.04

diff --git a/.meta-updater/main.mjs b/.meta-updater/main.mjs
@@ -1,8 +1,7 @@
 import fs from 'node:fs';
 import path from 'node:path';
 import jsonDiff from 'json-diff-ts';
-import lodash from 'lodash';
-const { cloneDeep } = lodash;
+import { cloneDeep } from 'lodash-es';
 
 const isDryrun = process.env.META_UPDATER_MODE === 'dryrun';
 const isVersionOnly = process.env.META_UPDATER_MODE === 'versiononly';
@@ -13,11 +12,10 @@ const DECKGL_VERSION = '~8.8.6';
 const TURF_VERSION = "^6.5.0";
 const NEBULAGL_VERSION = "0.23.8";
 const OTHER_VERSIONS = {
-  "ajv": "^6.10.0",
-  'lodash': '^4.17.21',
+  'lodash-es': '^4.17.21',
   'react-grid-layout-with-lodash': '^1.3.5',
   "internmap": "^2.0.3",
-  "uuid": "^3.3.2",
+  "uuid": "^9.0.0",
   "zarr": "0.5.1",
   "zustand": "^3.5.10",
   "@hms-dbmi/viv": "~0.13.7",
@@ -33,6 +31,10 @@ const OTHER_VERSIONS = {
   "math.gl": "^3.5.6",
   "@math.gl/core": "^3.5.6",
   "mathjs": "^9.2.0",
+  "zod": "^3.21.4",
+  "semver": "^7.3.8",
+  "vite": "^4.3.0",
+  "@vitejs/plugin-react": "^4.0.0",
 
   // LumaGL
   "@luma.gl/constants": LUMAGL_VERSION,

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -43,6 +43,11 @@
   - Added ADDITIONAL_OBS_SETS, OBS_SET_COLOR, OBS_COLOR_ENCODING and OBS_SET_SELECTION coordination types to the Feature Histogram.
   - Added a new function called `onSelect`, passed as props to `ExpressionHistogram`. On selection on `ExpressionHistogram`, the function computes what cell ids belong to that selection. Then calls the pre-existing `setObsSelection` function to create a new cell set with the cell ids.
 - Added a signal to `ExpressionHistogram` component, which calls `onSelect`, after 1 minute of debounce.
+- Replace Ajv with Zod.
+  - Add generic config schema.
+  - Add builder function for generating plugin-specific config schema.
+  - Reimplement config version upgrades.
+- Provide plugins as React props rather than registering them globally on `window`.
 - Use hooks in `ObsSetsManagerSubscriber` to improve controlled-component performance.
 
 ## [2.0.3](https://www.npmjs.com/package/vitessce/v/2.0.3) - 2023-02-01

diff --git a/DEMOS.md b/DEMOS.md
@@ -130,3 +130,5 @@ Note that the S3 URLs contain the git hash, if you want to relate this to an exa
 - 2022-12-14: [release-v2.0.2](https://s3.amazonaws.com/vitessce-data/demos/2022-12-14/a8ee1542/index.html)
 - 2023-02-01: [release-v2.0.3](https://s3.amazonaws.com/vitessce-data/demos/2023-02-01/f2ddf311/index.html)
 - 2023-02-07: [release-v2.0.3](https://s3.amazonaws.com/vitessce-data/demos/2023-02-07/ca9494c3/index.html)
+- 2023-03-24: [keller-mark/zod](https://s3.amazonaws.com/vitessce-data/demos/2023-03-24/28cbfd99/index.html)
+- 2023-05-11: [keller-mark/file-exts](https://s3.amazonaws.com/vitessce-data/demos/2023-05-11/512964ad/index.html)
diff --git a/DOCS.md b/DOCS.md
@@ -52,3 +52,5 @@ Note that the S3 URLs contain the git hash, if you want to relate this to an exa
 - 2022-12-14: [release-v2.0.2](http://vitessce-data.s3-website-us-east-1.amazonaws.com/docs/2022-12-14/a8ee1542/)
 - 2023-02-01: [release-v2.0.3](http://vitessce-data.s3-website-us-east-1.amazonaws.com/docs/2023-02-01/f2ddf311/)
 - 2023-02-07: [release-v2.0.3](http://vitessce-data.s3-website-us-east-1.amazonaws.com/docs/2023-02-07/ca9494c3/)
+- 2023-03-24: [keller-mark/zod](http://vitessce-data.s3-website-us-east-1.amazonaws.com/docs/2023-03-24/28cbfd99/)
+- 2023-05-11: [keller-mark/file-exts](http://vitessce-data.s3-website-us-east-1.amazonaws.com/docs/2023-05-11/512964ad/)
diff --git a/README.md b/README.md
@@ -62,6 +62,8 @@ pnpm run start-demo
 
 The development server will refresh the browser as you edit the code.
 
+Further details for internal developers can be found within [dev-docs](./dev-docs/).
+
 ### Branches
 
 Please use one of the following naming conventions for new branches:
@@ -91,6 +93,14 @@ pnpm run build-demo
 - To run all the tests, both unit and e2e: `./test.sh`
 - To run just the unit tests: `pnpm run test`
 
+### Linting
+
+```sh
+pnpm run lint
+```
+
+To allow the linter to perform automated fixes during linting: `pnpm run lint-fix`
+
 ### Troubleshooting
 
 The following commands can be helpful in case the local environment gets into a broken state:
@@ -173,6 +183,8 @@ From local machine:
 
   ```sh
   pnpm run build
+  pnpm run bundle
+  pnpm run build-json-schema
   pnpm publish --filter='./packages/**' --no-git-checks --tag beta --access public --dry-run
   ```
 

diff --git a/consumer/src/index.js b/consumer/src/index.js
@@ -2,15 +2,15 @@
 import React from 'react';
 import ReactDOM from 'react-dom';
 import { VitS } from '@vitessce/vit-s';
-import { register as registerDescription } from '@vitessce/description';
+import { PluginViewType } from '@vitessce/plugins';
+import { CoordinationType } from '@vitessce/constants';
+import { DescriptionSubscriber } from '@vitessce/description';
 
 const e = React.createElement;
 
-registerDescription();
-
 const eng2019 = {
   name: 'Eng et al., Nature 2019',
-  version: '1.0.15',
+  version: '1.0.16', // Must be the latest version because using VitS rather than Vitessce
   description: '',
   datasets: [],
   initStrategy: 'auto',
@@ -29,6 +29,10 @@ const eng2019 = {
   ],
 };
 
+const viewTypes = [
+  new PluginViewType('description', DescriptionSubscriber, [CoordinationType.DATASET, CoordinationType.SPATIAL_IMAGE_LAYER]),
+];
+
 class App extends React.Component {
   constructor(props) {
     super(props);
@@ -37,7 +41,7 @@ class App extends React.Component {
   render() {
     return e(
       VitS,
-      { config: eng2019, height: 500, theme: 'light' },
+      { config: eng2019, height: 500, theme: 'light', viewTypes },
       null
     );
   }

diff --git a/dev-docs/README.md b/dev-docs/README.md
@@ -0,0 +1,31 @@
+# Developer documentation
+
+The developer documentation here is meant for usage by the internal development team, external contributors, and plugin developers.
+
+## For plugin developers
+
+- [View type implementation](./plugin-view-types.md)
+- [File type implementation](./plugin-file-types.md)
+
+## For internal developers
+
+We not only want to document implementation details, but also higher-level architectural details and development processes.
+However, documentation that would be relevant to a wider audience than noted above should instead be included in the main documentation website that is hosted at http://vitessce.io.
+
+### Architecture
+
+The diagram below highlights how Vitessce is composed of a top-level `<Vitessce/>` React component which encapsulates several individual visualization or control views such as `<Scatterplot/>` and `<Spatial/>`.
+
+<a href="https://docs.google.com/drawings/d/1vS6wP1vs5QepLhXGDRww7LR505HJ-aIqnGn9O19f6xg/edit" target="_blank">
+    <img
+        src="https://docs.google.com/drawings/d/e/2PACX-1vSoB3YGPxOTKnFOpYHeHX4JruHnibGXruM36uAZtuvPQNM3a7F4uS3q4b5jwGNQ6TJ7bQ9IPB32rdle/pub?w=650"
+        alt="Architecture diagram"
+        className="ar-16x9"
+    />
+</a>
+
+### Table of contents
+
+- [Design guidelines](./design-guidelines.md)
+- [Monorepo and bundling](./monorepo-and-bundling.md)
+- [Config schema versioning](./config-schema-versioning.md)
diff --git a/dev-docs/config-schema-versioning.md b/dev-docs/config-schema-versioning.md
@@ -0,0 +1,26 @@
+# Config schema versioning
+
+## Motivation
+
+In Vitessce, we make the following commitment related to backwards compatibility: JSON configurations defined against a particular config schema version must be compatible with the version of the JavaScript package in which that schema version was defined __and all future package versions__.
+
+
+In addition, we need to allow the config schema to continue to evolve as development continues in the form of new and improved file types, coordination types, and view types. Sometimes, we will even need to modify the behavior of particular views, file types, or coordination types (e.g., to fix bugs).
+
+For example, the developers of the HuBMAP Portal have written many Vitessce configs, which must remain valid despite updating the Vitessce JS package version to pull in new features or bug fixes.
+
+__Versioning of the config schema enables us (as developers in the present) to infer the behavior of Vitessce that was assumed by the author of a particular config (as users in the past).__
+
+## When
+
+A new config schema version should be added when:
+- behavior of a coordination type changes significantly
+- the value schema for a coordination type changes
+- coordination type(s) are added, removed, or renamed
+- view type(s) are added, removed, or renamed
+
+## How
+
+Config schemas are defined using [Zod](https://zod.dev/) in the `@vitessce/schemas` sub-package of the Vitessce monorepo.
+
+See the [README](../packages/schemas/README.md) for detailed instructions.
diff --git a/dev-docs/design-guidelines.md b/dev-docs/design-guidelines.md
@@ -0,0 +1,59 @@
+<!-- Inspired by https://github.com/visgl/deck.gl/blob/a58191a83c7d06526d9a7419db76d8442e83849c/dev-docs/deckgl-api-guidelines.md -->
+
+# Vitessce Design Guidelines
+
+An evolving set of guidelines to ensure that features remain maintainable, scalable, and consistent.
+
+## Design Guidelines
+
+### User experience decisions
+- When making decisions that affect user experience, prefer [consistency](http://vis.pku.edu.cn/research/publication/consistency.pdf).
+
+### Tests
+
+- Prefer unit tests over end-to-end tests - Keep React components relatively "dumb" and put complex logic into utility functions that can be easily unit-tested.
+- Test in [portal-ui](https://github.com/hubmapconsortium/portal-ui) with diverse data modalities prior to major releases
+
+### Data types
+
+- Prefer minimal data types - If you can imagine reasonably storing data in multiple files (e.g., using formats like CSV or JSON), then consider splitting into multiple simpler data types. [Joint file types](http://vitessce.io/docs/data-types-file-types/#joint-file-types) can be used in cases where data is stored in the same file.
+
+### Scripting for development
+
+- Prefer NodeJS scripts over complex Shell scripts.
+
+### Monorepo organization
+
+- Avoid [circular dependencies](https://github.com/vitessce/vitessce/issues/1490) - If you find the need to import functions across view type- or file type- sub-packages, consider moving to a utility sub-package (i.e., `packages/utils/`).
+- Avoid complex dependencies in utility sub-packages - Where possible keep dependencies in utility sub-packages small and simple (e.g., `lodash`). If a large/complex dependency is needed for a utility function, consider refactoring that function into a new utility sub-package.
+
+### TypeScript
+
+- Implement new sub-packages using TypeScript to avoid creating tech debt.
+
+### Parsing and validation of user input
+- Prefer [Zod](https://zod.dev/) schema over JSON schema - This improves the TypeScript development experience and follows the ["parse, don't validate"](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) mantra.
+
+### Coordination type schemas
+- Prefer primitive values like numbers and strings over objects and arrays - Coordination values must be used entirely. Therefore, usage of objects and arrays prevent linking views on a subset of the object/array values.
+
+
+## Code style guide
+
+> A good style guide defines not only superficial elements like naming conventions or whitespace rules but also how to use the features of the given programming language. JavaScript and Perl, for example, are packed with functionality — they offer many ways to implement the same logic. A style guide defines The One True Way of doing things so that you don’t end up with half your team using one set of language features while the other half uses a totally different set of features. -- [How to Do Code Reviews Like a Human](https://mtlynch.io/human-code-reviews-1/).
+
+This section contains an evolving set of code style guidelines that are not currently automated via linting.
+
+- Use MUI style utilities to define styles ([makeStyles](https://v4.mui.com/styles/api/#makestyles-styles-options-hook), at least until we migrate to MUI v5).
+- Use `false`, `null`, and `undefined` as false-y values unless the corresponding truth-y value is a number (to align with [JSX boolean handling](https://legacy.reactjs.org/docs/jsx-in-depth.html#booleans-null-and-undefined-are-ignored)).
+- Use `lodash/isEqual` for set path equality checks and comparisons.
+- Prefer more specific naming for utility functions (despite length/verbosity) to help readability.
+- Prefer event handlers to side effects (from React docs: [You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect))
+  - Related: exercise caution when removing effect dependencies (from React docs: [Removing Effect Dependencies](https://react.dev/learn/removing-effect-dependencies))
+- Web workers should be [inlined](../packages/workers/rollup.config.js) so that they do not depend on relative paths that consumer applications and libraries would need to configure.
+- Use `.js` extensions for relative imports, [even in TypeScript contexts](https://github.com/microsoft/TypeScript/issues/42151#issuecomment-914472944).
+- Do not use `.js` extensions for third-party packages, unless either:
+  - The function or variable that needs to be accessed is not exported from the main entrypoint of the package.
+  - The package is published as CommonJS or UMD rather than ESM (or the ESM build is broken as with [json2csv](https://github.com/zemirco/json2csv/issues/539) and [react-virtualized](https://github.com/bvaughn/react-virtualized/issues/1632)).
+
+
diff --git a/TESTING.md → dev-docs/manual-testing.md b/TESTING.md → dev-docs/manual-testing.md