Progress Updates & Discussion

[nonara]

Update (latest status)

State: Unfinished
Status: Paused, but will be continued

Latest Update:

Date: April 19th 2022

Much has happened in life and the world since beginning this project! I regret that this is one thing which ended up being set aside for awhile. To answer the immediate question — I still very much intend to finish this project and to see that it can thrive.

As the overall ambition of the project grew, so also went the knowledge and time dedication requirements. However, I still believe that it is well worth the effort.

At the moment, I am working to scale up my business to a point where I will be able to set aside some personal time and also to devote capital to helping fund the completion of this project. For this, I intend to contract some brilliant minds with decades of experience in language design, who can "check my work" to ensure that the AST model is correct. This has been one of the big worries for me — namely, getting it right before release. After this step is done, we will wrap up the development and ensure that it's enterprise ready.

I am hoping that this can be addressed this year, but if recent years have taught me anything, I will not make hasty promises, though I can say that I do remain quite optimistic, given the current momentum in business!

---

Background

Around the start of this year, I began work on a new project. It was intended to be a tool which used TypeScript's TypeChecker to extract all possible types in a TS source-base and compile it to JSON Schema.

As time progressed, it became something more, until where we've arrived today!

The parser and compiler are complete, along with a robust system of options that can be employed via JSDoc tags. Some even be cascade based on Node type. It can also be manipulated as it generates an AST.

The discussion that started it began here: vega/ts-json-schema-generator#101

Today

It occurred to us that the library can be de-coupled entirely from both TypeScript and JSON Schema.

By creating a universal, language agnostic AST, we're able to represent types from any language. As a result, we're able to make it a cross-compilation engine with a specific focus on data structure types. In other words, you can convert FROM any supported language TO any supported language. The design allows easily adding language support as separate package modules.

In additional to the previous work, the universal AST structure is now complete as well! It covers the majority of possible types across many different languages, but it's also structured in a way that should avoid breaking changes as we need to adjust or add new nodes.

The first languages supported will be TypeScript and JSON Schema, but Python and others should soon follow.

[nonara]

Update

LanguagePackage has been re-designed and I like the new structure.

Now, all language-specific data (including SpecificKinds) is external to @crosstype/core. The cool part is, with a bit of effort, I've managed to get the type system to dynamically load the config types and optionsets based on whether or not they are installed.

This means that intellisense will only show you languages that you have installed. The same is true for AST nodes when accessing options. You'll only see possible CompileOptions instead of all.

Next step is to finish classes for NodeObject and factories. After that, I'll write unit tests.

[nonara]

Question (AST)

Nodes

https://github.com/crosstype/crosstype/blob/master/packages/core/src/ast/node-types.ts

1. Are we missing any nodes?
2. Are the names accurate and do they make sense?

Flags, etc.

https://github.com/crosstype/crosstype/blob/master/packages/core/src/ast/enums.ts

1. Are the semantics right?
2. Do the flag categories seems scalable and accurate? (see application for most in Node in node-types.ast)

[nonara]

Update

Just pushed a few updates...

Abstract

1. Node iterable components created (a couple more static methods will be added to assist parsers later)
2. Several notable AST updates (inc. added Date related abstract data types)

Next?

• Complete NodeObject classes (implementations of methods on nodes) - close to completed
• Create factories & guards for each node
• Write tests for nodes

[nonara]

Update

Just completed AST Node code (guards, factories, class implementation, and utilities)

Node Implementation: 75c7d1b
Type Updates: 5965346

Next?

• Migrate & update build tools package (enables auto-generating node-group-aliases.ts, node-metadata.ts, and node-lookups.ts by reading node-types.ts)
• Create tests for Nodes
• Begin migrating TypeScript Parser

[nonara]

Update

• Took a much needed break last week! The rest was helpful.
• Two weeks ago I transitioned to a split work-cycle, as I couldn't neglect for-profit work any longer. I'm currently doing 3-days per week on CT (Mon - Weds)
• Lots more done!

Newly Completed

1. Build tools package complete
2. System tools package migrated and now integrated
3. Generator for AST source complete (generates node-aliases.ts, node-metadata.ts, node-lookups.ts based on ast/node-types.ts)
4. All Node factory tests complete
5. Project can now be compiled (no unpublished pieces)

Next

• Finish AST tests
  1. Utils
  2. NodeObject class methods / properties
  3. Node TypeGuards
  4. Node Components
• Migrate TypeScript Parser

We're getting close! 🎉

[nonara]

Update

Just wrapped up a very productive session for the week! The short version is, the AST is done!

Newly Completed

1. Added @mixin decorator ability to system
2. Centralized Node Iterables prototype behaviour as mixin + allows synthetic array-like access (ie nodeMap[0] works)
3. Finished all AST Tests (NodeObject, components, typeguards, & utilities)
4. Lots of fixes to ensure all behaviour works as intended

Next

• Quick fix to reference target behaviour
  • References have the ability to smart resolve by path if a reference's target is deleted and a new one with the same path is created, however, the logic isn't quite tuned for declaration merging situations yet. It should be an easy fix, but needs to happen next session!
• TS parser migration begins!

Note

The bulk of the work over the past year has has been spent on two areas - determining proper architecture and discovering all of the nuances in the TS type system. The latter of which has, by far, been the most time consuming. The culmination of that work is in the parser which has been, up to this point, unpublished.

The good news is that it is complete and already handles all type scenarios (except variable statement, which I will likely add). Additionally, the architecture of the new system is built in a way that will allow a quick and easy migration.

I'll begin migrating next session (Sun-Weds) and hope to have an exciting update to share!

[dvins]

Looking forward to trying out the new parser, it’s been a few weeks since your last update, will it be coming soon?

[nonara]

@dvins Thanks for checking in! Shortly after the update, I got ill and was down for two weeks. Fortunately, I'm feeling better now and have back at it this week. I'm doing a quick catch-up session this week on for-profit work, and I'll be on the parser starting Sunday.

Side Note: if anyone has some time and is interested in helping out by writing tests, we could probably expedite things a lot

[dvins]

Thanks for the update and good to hear you’re on the mend and getting better! Your approach on this has really peeked our attention. Once the core parser is stabile enough to adopt we’re considering some work to project to some other outputs, namely GraphQL and/or other DTO-like formats.

[nonara]

@dvins Sounds great! I will keep you posted. Anything which outputs from the AST we classify as a compiler. I'll have the format ready for parser and compiler shortly. That way when the TS parser is ready, if you wanted to work on alternate output formats, you can make a compiler which can be officially consumed.

[nonara]

Update

I'm still here - I've been swamped! I ended up having to do a lot of PR work to fix problems in some dependencies. Fortunately, that's wrapped up, and I've made progress within CT as well.

I finally have the LanguagePackage, Parser, and Compiler structures set. I agonized for far too long over figuring out how to structure them to perform well, scale, and to make as much optional as possible. The biggest challenge, structurally, was to enable the core library and other language packages to access all specific types and metadata other languages, without requiring the whole package to be installed.

I'm now moving in the TS Parser and Compiler, should have an update end of next week.

[nonara]

Update

I'm nearly finished migrating the TypeScript parser! It's about 80% there. Next up is porting the type-walker portion to use the new AST and migrating in the cascading options system.

Getting very close, now.

[domoritz]

🎉 How hard do you think it will be to generate JSON schema from the parsed representation?

[nonara]

🎉 How hard do you think it will be to generate JSON schema from the parsed representation?

It's being ported from finished code that does just that! 🙂 The former has an AST which is very similar to this. Each node has a method attached that does JSON compilation. (Supports json schema 7 as well as the latest draft) It works well and supports all TS types.

So, the logic is already written for JSON schema compilation; it just needs to be ported to the CT Compiler structure and the new AST, which should be a relatively easy transition.

[domoritz]

Sweet. Looking forward to taking it for a spin.

[cspotcode]

Are there any limitations to how we write our TypeScript code? Just as an example, will type Foo = Exclude<keyof Bar, 'notthisone'> generate the correct enum schema?

[domoritz]

That already works in https://github.com/vega/ts-json-schema-generator, no?

[cspotcode]

Yeah, I think so, I'm not sure if crosstype supports it as well. I'm a little unclear on exactly what is doing the semantic analysis to convert TS syntax into type information. Supposing my types refer to types from a third-party library, and supposing that library uses TS conditional types or stuff like that, I'm wondering if/when the schema generation will break.

[domoritz]

I think we are all looking forward to actually trying out the JSON schema generation (#4 (comment)).

[nonara]

Are there any limitations to how we write our TypeScript code? Just as an example, will type Foo = Exclude<keyof Bar, 'notthisone'> generate the correct enum schema?

No limitations outside of those imposed by the TS compiler (with exception for generics, more detail later)

We rely entirely almost entirely on the TypeChecker to get the type information. That has been the top consideration from starting off, because otherwise we end up having to re-create it and keep it up-to-date, which didn't seem a good route to go.

As far as the TS parser goes:

Initially, all files are walked to discover any module-level declarations, (or those which have been explicitly tagged). From there, the parser walks the types to convert TS Type information to CT AST. As for the issue with type-aliases and indexedaccess not giving us the immediate next target in the type checker, we do a little pre-transform magic to ensure that we get accurate type references. So, for the most part, this is a 1-to-1 conversion from the data represented by Type & Symbol.

You managed to hit on the two differences in CT AST and TypeScript, though! ☺ When looking into creating a universal AST that would work in a language-agnostic manner, I ended up deciding that we would not create nodes to represent conditional and mapped logic. That may change later, but for now it is the case.

So, for the most part, everything should behave as you'd expect (see example below). The one caveat being that generics (non-parameterized types which require a parameter) will not be included in the parsed results. Though, from the perspective of someone interested in using it solely for JSON-schema conversion, this would not matter and will match the behaviour of other tools.

Example:

// Will not be included in parsed types because we do not have nodes to represent mapped logic
type MyType<T> = { [K in T]: string }

// Will be parsed as `TypeDeclaration` > `ObjectNode` (a: `StringNode`, b: `StringNode`)
type MyTypeInstance = MyType<'a' | 'b'>

Supposing my types refer to types from a third-party library, and supposing that library uses TS conditional types or stuff like that, I'm wondering if/when the schema generation will break.

Because we're using the TypeChecker, the library being external or internal doesn't matter here. Again, the parser covers the full range of TS type information which is applicable to our AST, so nothing should get lost in translation.

As a side note, one of the issues to tackle was determining what gets 'resolved' (meaning no explicit dependency definition is created), this particularly applies in cases of external libraries.

For that, you can configure CT in a number of ways, including on individual nodes or files, but let's assume we have a project-wide parse option of externalDependencyDepth .

local.ts

import { MyType } from 'external-module'
export type A = MyType

[external-module]/index.ts

export type InternalType = string
export type MyType = InternalType

(The following are just pseudo-AST to demonstrate the concept)

externalDependencyDepth = 0

A = (value: StringNode)

externalDependencyDepth = 1

A = (value: ReferenceNode (target: MyType))
MyType = [dependency] (value: StringNode)

externalDependencyDepth = 2

A = (value: ReferenceNode (target: MyType))
MyType = [dependency] (value: ReferenceNode (target: InternalType))
InternalType = [dependency] (value: StringNode)

Hope that helps!

[nonara]

Just as an example, will type Foo = Exclude<keyof Bar, 'notthisone'> generate the correct enum schema?

Kind of left this part out, sorry about that!

The structure now is composed of language packs which have a Parser and Compiler. In this case, the TypeScript Parser will create a UnionNode. And during compilation the JSON-schema Compiler will translate that to the proper enum type.

JSON-Schema's fluid-nature is going to be great for handling declaration merging! It will be cool to see the output with some merged types.

[cspotcode]

Very cool, thanks for the info. I'm glad to hear the type info comes straight from TS's type-checker. I'm excited to give it a try on our project at work.

[andreialecu]

With that in mind, I'm committing to having a release ready by end of Fall. Barring catastrophic circumstance, you can expect this deadline to be hit!

Hopefully nothing catastrophic happened. 😟

Recently came across this project, looked very promising, and am wondering if it was abandoned or is still pending release.

[nonara]

Hopefully nothing catastrophic happened. 😟

I appreciate that! The answer to that is that it somewhat has, but this isn't abandoned.

The scope and impact of covid-19 since the last update has been greater than I, or perhaps anyone, assumed. Unfortunately, it also impacted my financial stability in a way that left me having to work very hard to quickly wrap up a long-running build cycle for a new company that will re-establish income. I've been doing 8-12hr days, 6-7 days per week, so open source work has been largely back-burnered. Fortunately, I'm just about to launch and hope to be in the green again soon.

With regard to CT, it's definitely not abandoned, and I do hope to get it wrapped soon. I'm a bit embarrassed to have missed my estimations so often. If this year has taught me anything, it's to be slower to make predictions on completion dates!

The current plan is to look over this again this January to see about getting this finished and stable. I'll post an update here when I do.

[dvins]

Hi @nonara, like @andreialecu said this has looked promising for past year but with circumstances unfortunately doesn’t seem like you’ve been able to return to it. Is this still remotely on your radar at all?

[nonara]

@dvins I hear you. Not being able to finish it has been bothering me for awhile. I'm back on it. I won't be giving updates until it's ready to beta - so anyone interested can follow the commits.

Also, anyone willing to help out would expedite it greatly. It turned into a big undertaking.

[aaronhuggins]

@nonara What areas do you need help with? We are using @domoritz ts-json-schema-generator for a number of internal projects and are very interested in crosstype as well.

[domoritz]

It's great to hear you're using my schema compiler. Are there any big limitations with it for you right now?

[aaronhuggins]

@domoritz Nothing really outstanding when it comes to limitations. There are some other custom type systems we're dealing with where we have to transpile to either typescript or JSON Schema (or both). In those situations we're rolling our own code internal to that project, where I'd like to see us plug-in to crosstype to reduce the number of transpiling stages and potentially perform the transpile in parallel by forking processes.

[nonara]

@aaronhuggins I appreciate that! I'll start adding some tickets for finishing things up, and I'll tag "help wanted" on any that are reasonable to jump in without background knowledge of the rest of the project. To start, it will likely be tests. Anyone who wants to help in a greater capacity after some initial work, we can look at integrating into the org.

Looking forward to getting this finally wrapped up.