Project Background, Goals, and Non-Goals

[timburks]

Thanks @morphar, for pointing out a need for background on this project including goals and non-goals. As the author and maintainer, I'll use this issue to draft a perspective on that. Everything is negotiable, and I'll try to also reasonably represent constraints that affect some of the decisions here. This may take some time to fully write, but I'll do that with edits to this top-level comment and will respond to discussion below.

Origins

I started gnostic as a Noogler project in mid-2016. Originally called "OpenAPI compiler", its goal was to be a front-end for a Go-language replacement for an open-sourced internal tool called api-compiler that processed APIs for management by Google's internal API management platform. In discussions about that, someone suggested that instead of writing an OpenAPI reader in a particular language, it would be useful to have something that read OpenAPI docs into a protobuf representation that would be syntactically validated and easier to use for downstream tools. Another goal that we considered was client generation for OpenAPI APIs.

At that time, there was a JSON schema for OpenAPI (v2), and I found that it was possible to write code that read that and used it to drive a generator that produced a protobuf description of OpenAPI and "compiler" code that checked-and-converted YAML into this representation. When OpenAPI v3 emerged, there was some resistance from the OpenAPI core team to produce a similar schema, so I wrote a scraper that built a usable schema from the Markdown source of the spec, and then used that to generate protos and a compiler for OpenAPI v3.

The name change to "gnostic" came later (2016 or 2017) and was done to clarify that this wasn't an official OpenAPI Initiative project and to make the usual subtle jokes that engineers make (the silent g, the association of "gnosis" with "knowledge").

OpenAPI client generation

While this was an early interest of the project, it quickly became difficult to support. The OpenAPI specification (particular 3.0 and later) became very broad and emphasized documenting any of the ways people might structure their HTTP APIs. This made it much more difficult to use OpenAPI as a basis for client generation, particularly since Google already had generators for the API Discovery service and gRPC. The Discovery-based generators were deprecated but considered to be much-higher quality than third-party OpenAPI code generators and Google's API organization had focused on gRPC, which has many code generation advantages over OpenAPI. I personally stopped working on this in 2017 when I began leading a team that produced gRPC-based client generators that became known as GAPIC Generators. A small effort to produce a Go client generator is in google/gnostic-go-generator.

Non-goal: Generate OpenAPI clients. Currently this is not a project goal. Where automatically-generated clients are needed, we recommend that API teams use gRPC and for OpenAPI clients, we recommend exploring other open-source projects.

Kubernetes

In late 2016, the Kubernetes team discovered gnostic and began using it to solve a problem that they were having with kubectl, the Kubernetes CLI. Quite cleverly, kubectl dynamically-constructed its command interface from an OpenAPI description of the Kubernetes API, but this required parsing that API description at startup time every time kubectl was run. This OpenAPI description was a big (multi-megabyte) JSON file. Parsing it was taking over half a second, and with many frequent invocations of kubectl, concern for this was building. @mbohlool found that gnostic's compiled representation could be read in under 10ms, so now on startup kubectl reads a downloaded (possibly cached) serialized protobuf of the Kubernetes OpenAPI spec that uses gnostic protos and is available directly from the Kubernetes API.

Goal: Continue support for kubectl. This calls for a stable protobuf representation of OpenAPI that will also include OpenAPI 3.0 and any future versions that the Kubernetes team needs. In the near future, we may provide a separate distribution (as a Go module or separate repo) that minimizes the dependencies associated with use of gnostic-based packages.

gnostic-grpc

Beginning in late 2016, I've had many conversations with outside API leads who are interested in using gRPC transcoding to implement their HTTP APIs. A key question has been "how can I maintain compatibility with my existing APIs?" To address this, @LorenzHW worked with us on a Summer of Code project to create gnostic-grpc, a gnostic plugin that attempts to produce annotated protos for a gRPC API that, if transcoded, would produce an HTTP API described by a given OpenAPI document. In other words, gnostic-grpc takes an OpenAPI description as input and attempts to generate protos that would produce that API. Because of the limitations of transcoding, this wouldn't support all OpenAPI descriptions, but could be a useful starting point.

protoc-gen-openapi

In 2019, I joined the Apigee team and began work on API governance that led to the Apigee Registry API. This is a gRPC API with a transcoded HTTP API, and was the original target of the protoc-gen-openapi implementation that is in this project.

Goal: Continue support for protobuf-based methodologies for producing REST APIs. This includes gnostic-grpc, protoc-gen-openapi, and other tools as needed to fill gaps that we might effectively fill. It is important to ensure support for Google API practices, but other protobuf-based methodologies should be accommodated wherever practical.

Non-goal: Support generalized "round-tripping" between protobuf and OpenAPI representations. These formats have significantly different goals and scope, so it seems inappropriate to treat them as equivalent representations of common semantics.

JSON Schema support

The gnostic-generator reads JSON schemas and use them to generate protobuf models and "parsing" code that fills these models from YAML or JSON inputs. This seems generalizable to support other JSON schema structures, but so far we haven't seen a need for this. It would also probably involve updating our JSON schema use to current standards.

Goal/Non-goal?: Expand gnostic-generator to support similar compilation of arbitrary JSON schemas This really depends on someone finding a need for this and having availability to work on it, but it seems congruent with the overall goals of the project.

Contributors / Maintainers

Currently I (@timburks) am the only Googler who's actively associated with the project, and it's now more a personal commitment than an officially-supported project. The Kubernetes usage is pretty significant, and the kubectl dependency is big enough that I occasionally get check-ins from Kubernetes team members who need changes (or are trying to fix something that I've broken). But other than that, there's not a lot of internal dependency on the project - our main interests would be in using it to solve problems for external (non-Google) developers, so some of my Apigee team members are also exploring gnostic-related projects including mentoring intern projects such as Nicole Gizzo's 2020 project to analyze API vocabularies. And all that said, outside contributions are really great, extremely welcome, and could do a lot to help move the project in helpful directions. Thanks for reading!

[morphar]

Thank you for this @timburks.

Regarding JSON Schema support.
Can you elaborate on this:

Expand gnostic-generator to support similar compilation of arbitrary JSON schemas

gnostic does currently support generating JSON Schemas from protobuf descriptions.
The initial version I wrote, basically does the same as protoc-gen-openapi, but generates JSON Schema files matching the protobuf messages.
After @jeffsawatzky's great updates to protoc-gen-openapi, it probably needs some updates to match the correctness though.
And more interestingly, with OpenAPI v3.1.x actually supporting JSON Schema, there will be an overlap between generating / reading JSON Schema and generating / reading OpenAPI v3.1.x specs with gnostic.

Can this be summed up and put into goals and non-goals lists, that could be put in the README.md?

An initial attempt:

Goals:

• Generate OpenAPI specs from http annotated protobuf specs in a way that supports Google API practices and methodologies
• Accommodate other methodologies with options and parameters wherever practical
• Support tools for generating protobuf specs from e.g. OpenAPI and Google API Discovery Service specs

Non-goals:

• Generate REST clients (see tools: x1, x2, x3, ...)
• Generalized "round-tripping" between protobuf and OpenAPI

As far as I can see, some of this would practically mean that gnostic could support round-tripping, in some scenarios:

• When creating protobufs with that purpose in mind
• When the round-tripping is OpenAPI -> protobuf -> OpenAPI

A lot seems to point back to your question in #317.
If there is no "hard" goal of round-tripping, then your second suggestion seems more and more like the right path forward.
If you squint, gnostic could even start to look like a transpiler/generator with protobuf as intermediate representation, and everything else as generator tools using gnostic :)

[morphar]

You also mention this:

Currently I (@timburks) am the only Googler who's actively associated with the project, and it's now more a personal commitment than an officially-supported project.

Is it possible to have external co-maintainers? And is that something that would help you and/or the project?

[timburks]

@morphar I want to come back to your first question later, but to your second question, yes - let's discuss that offline (timburks@google.com)

[sagikazarmark]

Thanks a lot for writing this up.

It clarifies a lot, but I still have a couple questions, particularly around governance and how the largest players (Google, Kubernetes) affect the future of this projects.

Some of the project seems to be targeted at Google or Kubernetes specific use cases (I could be wrong, but that's the picture I see), while other parts (for example: protoc-gen-openapi) could be useful for a wider audience. Seeing all of under under one roof is a bit confusing to me as I'm not sure if they are really "independent" or that's just the appearance. To rephrase it: it's unclear who the audience is for protoc-gen-openapi for example in a repo that seems to be targeted at Google specific use cases.

Again, this is my perceived view of the library, but it's entirely possible I'm getting a wrong picture. I can't really tell you though what would make it easier to understand the goals and target audience of individual components better.

I saw there were attempts to split the repository up. It could help, but also make maintenance harder.

And again, tools like protoc-gen-openapi are actually quite sought-after in the community: grpc-gateway for instance has a swagger generator, but it doesn't support OpenAPI 3.x and the issue for that has been open for years now: grpc-ecosystem/grpc-gateway#441

(I'm testing grpc-gateway and gnostic generated OpenAPI schemas right now)

Anyway, thanks for writing this library!