Spec Evolution

[jdesrosiers]

Let's get on the same page about how the spec will evolve. We've previously agreed that spec releases will be compatible. In the rare occasion that an incompatible change is necessary, it must be deprecated for a reasonable amount of time before it can be removed. Other than the constraint that new features must be added in a compatible way, how do we want to introduce new features into JSON Schema?

Here are some of the options that have been put forward.

1. Staged Stability - There would be a set of stages that a feature has to go through to reach "stable" status. Until it's "stable", incompatible changes are allowed. The goal is to get feedback from implementers and users so we have enough data to be sure a feature is ready to be considered stable.
2. Deprecate and Replace - With this approach, anything we release for implementation and use is immediately considered "stable". If it needs changes, it must be deprecated and replaced with something else. The goal of this approach is to never subject users to the possibility of breaking changes (other than removal after deprecation).
3. External Spec - With this approach, any new feature is defined in a separate spec and implementations choose to implement the ones they like. The goal is that the most popular specs would eventually be folded into the official spec.

These options aren't mutually exclusive. Parts of each could be adopted. For example, we could choose a "Staged Stability" approach where the proposal starts out as an "External Spec", then moves into the spec as an explicitly "experimental" feature, and finally moves to stable status. Then once in stable status, it needs to follow the "Deprecate and Replace" approach.

Scope: I don't think we need to decide on details in this discussion. If possible, I'd like to agree on a general direction and work out the details later. However, getting into some of the details is often necessary to decide which direction is best.

Disclaimer: I did my best to accurately represent the approaches that have been proposed in various places. If I have misrepresented your idea or you have another idea, please let us know.

[gregsdennis]

My preference is a combination of (2) and (3).

A feature is developed in its entirety in an external specification. Once the feature is finalized, then it can be moved into the main spec. At that point, it follows the normal deprecation procedure.

It doesn't make sense to me to actively develop features in a spec that we claim to be stable.

[jdesrosiers]

To be clear, I don't think we should be actively developing a feature in the spec either. I don't think it should be moved to the spec until we've as sure as we can be without real usage that we aren't going to change it. Changes while in the spec are not intended, just allowed for because we don't have perfect foresight and sometimes it's necessary.

[gregsdennis]

I think it's necessary because I don't think anyone other than you and I will implement things that aren't in the spec.

If we can't get others to implement features in extension specs, then we're really bad at implementor relations. We don't need to put something in the spec in order to get people to build it. People already both ignore portions of the spec they don't want to implement (*cough* Visual Studio Code) and implement things that aren't in the spec (e.g. $data).

I think we need the feedback of people implementing and using features before we can be confident in calling them stable.

I agree with this.

---

This approach also allows third parties to develop their own spec extensions and then propose adding them to the spec.

[jdesrosiers]

We don't need to put something in the spec in order to get people to build it.

I'm not sure that's going to be true, but I'm willing to try it. Worst case is that it doesn't work and we pivot to something more integrated later. It would be easier to make that pivot than the other way around.

However, I'd still like to have a staged process with checkpoints and promotion criteria even if it's not in the spec.

[gregsdennis]

I don't think it should be moved to the spec until we've as sure as we can be without real usage that we aren't going to change it.

This approach guarantees that a feature won't be changed once it's in the spec.

without real usage

1. We have other means to get implementors involved in supporting these features.
2. I can't be sure that any specific part of my implementation is being used, stable or not. No implementor can know that without some kind of real feedback. I get notified of bugs or "can your implementation do X that's not in the spec?" But I don't get any information on whether $dynamic* is used in the wild. My only recourse would be to push stats to a server somewhere (if it even has online access), and I suspect my users would take exception to that.

If usage is a metric that you need for inclusion into the spec, then a means of obtaining that metric needs to be proposed.

[jdesrosiers]

We have other means to get implementors involved in supporting these features.

Do we?

If usage is a metric that you need for inclusion into the spec, then a means of obtaining that metric needs to be proposed.

I think the the most important thing is that a feature is supported and enabled by default by all or nearly all active implementations before it can be considered stable. This concept is borrowed from ECMAScript. We have to define what "nearly all" means, but getting the numbers of how many implementations support it will hopefully be as easy as opening bowtie.

The kind of feedback I'm expecting from implementers is about implementation. Is it difficult to implement? Does it fit into your architecture? Are there any tweaks that would make it easier to implement?

Feedback from users is more about the aggregation of questions that come in from Slack, SO, or Github and what that teaches us. For example, we observed a lot of questions about people being confused about the array form of items not validating all of their items. We used that as feedback that the items needed to be split.

It's impossible to measure how much something is being used, but if nearly all implementations support it and we see questions about it occasionally or see it in schemas occasionally, I think think that's enough to know it's getting used. I don't think we need more than that. It's pretty subjective, but I don't think this is as important as whether implementations support it.

[jdesrosiers]

I think a "Staged Stability" process is the best way to go.

I think "Deprecate and Replace" is too rigid. It assumes we're going to get it right the first time. Unless the feature is very simple, it's almost certainly going to go through some iteration to get it right. Dynamic References are a prime example of this. I think if we use this approach exclusively, we'll end up with a graveyard of deprecated keywords as some of the new features get worked out. Using this approach will also make it take longer for us to release new features and the feature will not be as good as they could be. We'll take longer thinking (and arguing) about the best way for something to work and when we do release it, we're still just guessing we'll likely get some things wrong. Complex features like the Vocabulary System might never get released because we'll never feel confident that it's "done" enough to be released.

I think "External Spec" is good, but too open and optional. As we've seen with vocabularies, implementations never support vocabularies that the implementer didn't define. I think we need stronger requirements for implementations to provide support for new features or nothing will ever get added. At best, we have a bunch of implementations with different supported feature sets.

The "Staged Stability" process I have in mind addresses the problems I have with each of those approaches. I think it allows us to move fast, get it right, and ensure interoperability. Loosely, this is what I had in mind.

Stage	In the spec	Required	Compatible Changes Only	Notes
Stage-0	❌	❌	❌	Requires multiple implementation and tests to move to next level
Stage-1	✔️	❌	❌	Putting it in the spec encourages implementation and use
Stage-2	✔️	✔️	❌	Making it required is necessary to get the wide spread implementation required to move to the next level.
Stable	✔️	✔️	✔️	Requires "Deprecate and Replace" for changes

Edit: I don't know why some of those check marks are rendering differently than others. They aren't intended to have different meanings.

[gregsdennis]

implementations never support vocabularies that the implementer didn't define

To my knowledge, Hyperschema, OpenAPI 3.1, and my two vocabs (data and uniqueKeys) are the only vocabs that have actually been specified. I'm not sure the dataset is big enough to make this claim.

---

I'm not completely against this approach, but I much prefer external docs.

[jdesrosiers]

We had a discussion on this topic at the last OCWM and here's summary of what we decided.

We decided that we will start off with developing new features external to the main spec documents. It will be optional, but strongly encouraged that implementations support these features. We will use community engagement and bowtie to encourage implementations to implement these features. We all agreed that these implementations provide important feedback that we need to make sure we get things right before declaring the feature stable. Once a feature is stable, it must be deprecated before it can be removed. What criteria it takes to be declared stable will be the topic of a different discussion.

There remains concern that implementers will shy away from implementing anything that isn't stable if it's not required to do so. If community engagement fails to get the results we hope for, we can pivot to being more strict about what implementations must support. It's easier to go from less strict to more strict than it would be to go the other way around.

Any features that are currently considered unstable will be moved out of the spec and into separate documents. What current features are considered unstable/experimental will be a topic of another discussion.

[gregsdennis]

This has been incorporated into https://github.com/orgs/json-schema-org/discussions/671.

Closing for now.