Centralize list of states #26
Comments
|
In GitLab by @lxcat on Sep 7, 2022, 12:19 One more pull request was accepted yesterday, for a total of four: #1261, #1265, #1367, and #1376. These pull request should give us the means to cleanup the schema generation code. I will implement these changes once #1389 is resolved. By Daan Boer on 2022-09-07T10:19:31 (imported from GitLab) By LXCat Project on 2022-09-07T10:19:31 (imported from GitLab) |
Merged
daanboer
added a commit
that referenced
this issue
Nov 13, 2023
This pull request does a lot more than was originally planned. The core improvements include the switch to zod for parsing and validating, which is used in combination with zod-to-json-schema to generate the JSON schemas. The schema itself has also seen major changes. Most of these changes are aimed at generalizing the schemas to support other data types apart from LUT cross sections. In short, these changes allow for the effortless definition of new LTP process data types (potentials, rate coefficients, diff. cross sections, etc.), and for the ways in which they are presented (function parameter set, constant, expression, etc.), without the need for major infrastructural changes. The new zod schemas allow for additional, custom validation logic through refine calls. This functionality is used to ensure that referenced state and reference ids are actually present in the provided dictionaries. Fixes #47. The species schemas are polished and simplified. A new Unspecified class of states is introduced that removes the confusion around the definition of unspecified electronic states (e.g. ). Furthermore, singular and unspecified state descriptors do no longer have to be supplied as an array, making their description and intention more clear. Finally, unspecified state descriptors are now simple strings instead of objects that store the identifier in the somewhat arbitrary e, v, and J properties (for electronic, vibrational, and rotational states respectively). Fixes #57, #58. Closes #26. The @lxcat/database package has been reworked. All functionality is now encapsulated in an LXCatDatabase class. Previously, each encapsulated query call retrieved the global database singleton to run its query. This made the code very hard to test, as it was impossible to define multiple databases without spinning up a new instance of ArangoDB. This has now been solved and test speedups have been observed of ~40x. The @lxcat/schema and @lxcat/database now have a well-defined external API through the use of the exports field in their package.json files. Fixes #108. Cleanup of the @lxcat/database package, some unused or deprecated code has been removed. The user-facing @lxcat/app pages are updated to support the new schema structure. Started development on a new data set edit form for authors. A utility is included that is able to dynamically generate form components based on a given JSON schema, which is used to generate the species forms. A new RouteBuilder class is introduced that provides excellent TypeScript support and integrates well with the "new" Nextjs app router. Additionally, a compatible zod middleware is provided that allows for easy, elegant parsing of user-provided data. Important to note is that merging the pull request in its current state will leave the @lxcat/app package in a state where it does not compile. This is a deliberate choice, as (clearly) this pull request did not stick to its original scope and is already way too large for comfort. The idea is to complete the remaining tasks in dedicated feature branches. Squash commit log: * Eliminate the usage of the XOR type It is not necessary to use XOR as AnyAtom and AnyMolecule are already discriminated unions. * Apply `@internal` to internal types This results cleaner generated schemas, as `@internal` types are not explicitly included in the schema `definitions`, but are instead evaluated inline (which is beneficial as their generated names are often nondescript and verbose). * Annotate species union types with `@discriminator type` The `@discriminator` annotation will generate more performant `if-then-else` schemas for discriminated union types, instead of the default `anyOf` schemas. * Add `Constant` storage In preparation for rate coefficients. * Add `as` cast to fix type error * Split the set header into a separate type Annotate `InputDocument` with `@internal`. * Rename `data` to `value` in `LUT` storage type` * Add `Expression` storage type * Switch to more simple generators for species types Partially implements #58 * Annotate more types with `@internal` Add the `AnySpecies` type. * Add `stateIsAtom` type guard function Add `AnyParticle` to `AnySpecies`. * Reimplement `insert_state_tree` to support the new schema changes This function adds a state with all of its required parent states, and the corresponding relations to the database. * Update `State` database collection type * Introduce `KeyedSpecies` As the database type. Move AnySpecies to its own module. * Update CrossSection schema * Update `State` schema * Update state tests for database package * Allow unspecified state descriptors as part of compound state This is already required to support one of the state descriptions in the test set, e.g. CO2{X, {0,n,0|n,0,0}}. * Update `@lxcat/schema` test data * Regenerate database state schema * Update `@lxcat/schema` tests to comply with new schema version * Add `zod` dependency to `@lxcat/schema` * Add preliminary migrate script for schema * Move `SimpleParticle` and `AnyParticle` to dedicated module * Add restriction on `State` generic parameter and add @Discriminator tags to `State` types. * Add missing `immer` dep for `@lxcat/database` * Use `KeyedSpecies` * Update state query generic constraints * Remove unused `LutTable` component * Cleanup minor (type) errors in `EditForm` component * Added `Unspecified` class of states This class of states accepts a string identifier as their `electronic` value. This is useful for e.g. `He*` type states. * Fix build errors in `EditForm` component FIXME: The edit form is in a broken state and needs to be completely revamped. * Fix build errors in `Chart` * Update `State` collection schema * Remove redundant example code * Fix query generators for new schema Remove use of `immer` in insert state procedure. * Remove migrate file from version control * Start new edit set page using app router * Add preliminary `zod` version of schema * Add `General` tab to new edit form * Annotate new files * Fix `LS1` and `J1L2` schema definitions * Add script to generate `AnySpecies` schema from `zod` definitions * Add `complete` flag to `SetHeader` definition * Make species schemas more strict where appropriate * Initial pass at automatic form generation from species schemas Includes form factory functions that use the schema generated from `AnySpecies` to build an input form for each of the state types. The method is currently working fairly well, apart from one annoying error concerning `Select` component that function as a switch between `Single`, `Compound` and `Unspecified` state descriptors. The error is as follows: *Warning: A component is changing a controlled input to be uncontrolled. This is likely caused by the value changing from a defined to undefined, which should not happen. Decide between using a controlled or uncontrolled input element for the lifetime of the component.* The result is that the selected option is not rendered, but the functionality of the component is fine. Proper handling of arrays should still be implemented. * Add keys in `allOf` component and default value in `anyOf` component * Equalize schema and app `zod` versions * Add descriptions for `Singular`, `Compound`, and `Unspecified` options in schema * Update new `EditForm` component Use a native `select` component to specify the state component type. Add new query to grab a data set by its id. Add `NotFound` component. Infer `LTPDocument` type. Rename `scat-css-new` to `set`. * Update copyright statement * Add `drop_non_user` script Drops all DB tables except for the user table. * Fix warning in `Dialog` Concerning lifetime of `ref.current`. * Fix zod types for `CSLData` and `Reaction` * Update zod schema Add `parameters` and `threshold` properties for cross sections. Allow unspecified level descriptors in compound layers. * Update Zod types Use method to generate generic typescript types from generic zod schemas. Use output types instead of input types. * Start using Zod types * `package.json` formatting * Rework species parsing Each zod component is now transformed to supply `summary` and `latex` functions that respectively serialize the object to a summarized short form and a latex form. * Add `serialize` function to `State` This function takes a `State` and generates its `StateSummary`. * Add first state serializer tests * Fix `@lxcat/schema` vitest configuration * Add more state tests * Add regression test for LTPMixture schema * Start deprecation of old schema * Fix further `@lxcat/schema` references * Update JSON schema creation * Fix build errors * Fix state charge serialization * Fix inspect page For new schema. * Add note about selection page crash * Fix data selection page * Fix serializer tests * Continue fixing database tests * Fix database state queries Fix more database tests. * Testing external serializer functions Supplied through `Component` type. * Use separate function to serialize `ShellEntry` * Rename `LSTermImpl` -> `LSTermUncoupled` * Add LS1 coupled component * Push `SimpleParticle` down the tree Add `makeComponent` function to help in definition of state level components. * Add helper types for new serialization strategy These helper types allow for the simple construction of both `serializable` and `non-serializable` versions of components and atoms. The atomic types have already been reworked to use these helper types. * Use `makeComponent` for molecular components * Split molecular types into serializable and non-serializable * `@lxcat/schema`: Emit ES6 with ESM imports * Remove `State` types `AnySpecies` is now used instead. Start preparing `index` files for removal of `dist` in external import statements. * Use zod refine to check validity of `state` and `reference` keys in LTPDocument. Add corresponding tests. * Perform key checks on `LTPMixture` Add tests. * Move `SetHeader` and `SelfReference` to dedicated modules. * Encapsulate exports and use `nodenext` module resolution Use `exports` field in `package.json`. * Fix database build Use new `@lxcat/schema` imports and types. Fix `tsconfig` to work with new module resolution strategy. * Update `module` setting in `schema` and `database` `tsconfig` * Add additional fields to `CSLNameVariable` These are not present in the standard CSL schemas, but may be supplied by `citation-js`. * Fix `@lxcat/app` build errors * Update database cli setup script * Fix data select/inspect/compute routes These pages now all use the new schema. Database cli scripts have also been fixed. * Update state interface in edit form Use an accordion with latex state descriptions in the control. Add `+` button that adds an empty species. Add `Add from database` button that should allow the user to pick an already existing species from the database (TODO). * Create `electronic` property upon type switch For new state objects. * Fix `byIdJSON` function * Add new `RouteBuilder` class This class can be used to build routes (as the name suggests). It utilizes the builder pattern to build routes for the Nextjs app router using middlewares and route handlers, while maintaining type safety. * Add `async` api to `RouteBuilder` * Add `hasSessionOrAPIToken` and `hasDeveloperOrDownloadRole` middlewares * Remove `dom` lib from `schema` tsconfig * Pin `zod` version to `3.21.3` Previous version `3.22.4` caused OOM errors on transpilation. * Add simple species routes Provide generally useful API calls. Will be used for edit form `pick from database` option. * Pin `zod` version in `app` * Add model to pick species from database in set edit form * Update flake lock file * Add array component for species form generation * Fix type errors involving document self referencing * Allow for unspecified entries in rovibrational compounds * Add species picker in set edit form Allows users to pick existing species from the database. Also adds the corresponding api endpoints: - /api/species - /api/species/children * Update schema generation command Command is now: `pnpm json:set`. Add cli module that prints the generated `LTPMixture` schema. Add `dom` lib dependency for printing. Remove `ts-json-schema-generator` dependency. Remove old schema types. * Secure endpoints using middleware * Add species API tests * Fix `/scat-css/[id]` endpoint * Fix many database tests * Fix and restrict atom and molecule schemas Compound species should have at least two entries. Fix wrong label for unspecified entries in compound species. * Fix reaction test * Partially fix CS write tests * Fix more database tests The process `info` property is now always an array. * Overhaul database package Made code a lot more testable. Fix broken database tests. Tests now run in ~3s instead of ~120. * Fix database package build * Fix annotate script * Fix reuse compliance * Tidy database package API * Fix schema regression test * Remove flake files from version control * Fix `@lxcat/schema` and `@lxcat/app` REUSE compliance
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In GitLab by @lxcat on May 29, 2022, 16:23
Currently when we want to add a new type of atom or state, its type needs to be added in multiple locations:
Stateindatabase/src/shared/types/collections.ts,AtomListandAnyAtomJSONinschema/src/core/atoms/index.ts,MoleculeListandAnyMoleculeJSONinschema/src/core/molecules/index.ts,Instead, all required types should be generated from
AtomListandMoleculeList. However, the required type constructs rely on usage of theinferkeyword. Unfortunately,ts-json-schema-generatordoes not currently support type inference. Therefore, I have implemented this functionality and opened a pull request #1265. As this is quite a big feature, I expect that the process of merging might be a bit slow. In the meantime I have locally implemented the desired changes in the type library using thets-json-schema-generatorfork that supports theinferkeyword. These changes will pushed as soon as the pull request is merged.The text was updated successfully, but these errors were encountered: