Refactor go-runtime/encoding into a general purpose JSON package

[alecthomas]

Unfortunately the standard encoding/json is completely inflexible and doesn't support our requirements sufficiently. There may be a JSON package that does, so a good first step would be to do some research, but I wasn't able to find any.

Let's completely decouple this from FTL, so we can potentially spin it off into its own package at some point like we did with scaffolder.

We have two use cases for this package, so it will have to be flexible enough to be used in both scenarios:

1. Internally as the transport encoding we use for verb-to-verb calls.
2. For users of the Go FTL SDK.

For internal use these are the requirements:

• Each Go field name must be mapped to a JSON name via either json:"..." tag or via a mapping function (lowerCamelCase()).
• It must only support the types supported by FTL and must error on all other types.
• Must not support json.Marshaler/json.Unmarshaler
• The omitempty tag must be ignored.
• Must support registration of custom encoders/decoders (for sum types), eg .json.WithTranscoder[T](impl)

For Go FTL runtime users these are the requirements:

• Must support json.Marshaler/json.Unmarshaler
• The omitempty tag option must work with "zero" types, including ftl.Option
• Must support embedded structs.

The library should be designed such that all of these options are configurable per encoder/decoder instance. The transport encoding will instantiate a custom encoder/decoder. There should also be a global default encoder and decoder configured for "normal" use, with global functions that use them, and support for registering custom types.

Note that there may be additional requirements not listed initially, we'll discuss and update accordingly.

I think functional options make sense for this API. Something like:

type EncoderOption func(o *encoderOptions) error

type encoderOptions struct {
  encoders map[reflect.Type]func(any) ([]byte, error)
  renamer func(string) string
}

func WithEncoder[T any](enc func(v any) ([]byte, error)) EncoderOption
func WithFieldRenamer(renamer func(string) string) EncoderOption
// etc.

func NewEncoder(options...Option) *Encoder

[alecthomas]

@deniseli we think we need to reassess the difficulty of a bunch of quite a few issues, so let's talk next week or tomorrow

[deniseli]

@alecthomas sounds good!