Implement wrapper for google.protobuf.Timestamp, and correctly generate wrappers for static target. #1258

ntindall · 2019-06-26T01:49:17Z

@dcodeIO @alexander-fenster @JustinBeckwith can you help me understand what else is needed to address this?

Related: #351 #893 #916 #1108 #1304

ntindall · 2019-07-09T18:22:01Z

any update here? I'd like to get this in to the mainline if we can.

ntindall · 2019-07-26T08:20:11Z

@alexander-fenster @JustinBeckwith @dcodeIO any feedback on this would be appreciated.

ntindall · 2019-10-06T20:13:13Z

@nicolasnoble any chance you can take a look here?

kheyse-oqton · 2019-10-22T09:34:07Z

Maybe I do something wrong, but it seems that "fromObject" does not work with iso time strings.
So you can't to type.toObject(type.fromObject({myFieldTime: "1970-01-01T00:00:05.000Z"})).
I get the error: "object expected"

The simplest solution I could find is to replace

protobuf.js/src/converter.js

Line 36 in 4d490eb

("if(typeof d%s!==\"object\")", prop)

with

("if(false)")

But I have no idea what this may break.

ntindall · 2019-10-22T13:47:27Z

@kheyse-oqton could you provide a more complete example of the error scenario you are hitting? not sure I fully understand.

kheyse-oqton · 2019-10-22T13:59:41Z

I've created an extra test to illustrate it
ntindall/protobuf.js@nt-google.protobuf.Timestamp...kheyse-oqton:nt-google.protobuf.Timestamp

ntindall · 2019-10-22T14:01:40Z

Thanks @kheyse-oqton - thats helpful. I'll look into that.

gfecher · 2020-01-15T23:38:30Z

Any update on this? I would love to have this functionality.

untra · 2020-02-11T22:38:10Z

Serious +1 for this PR. Native support for google.protobuf.Timestamp fields would significantly help, considering the prevalence of this proto type and the number of +1s and associated issues related to this PR.

guyisra · 2020-03-19T10:34:57Z

any updates on this?

untra · 2020-03-19T13:54:43Z

we just bit the bullet replaced all of our google.protobuf.Timestamp fields with int64s. This worked for us because we have a relatively new codebase. I would advise new projects to do similar.

webmaster128

Great work. Would be nice to get that in.

Could you pull in the test case from #1258 (comment)?

Even if we don't hear from maintainers, this can be added by the caller of the library as follows (random playground code, TypeScript):

import * as proto from "protobufjs";

// ...

const root = proto
  .parse(source)
  .root.addJSON(proto.common.get("google/protobuf/timestamp.proto")!.nested!)
  .addJSON(proto.common.get("google/protobuf/any.proto")!.nested!)
  .resolveAll();
// const root = parsed.root;

// Obtain a message type
const AwesomeMessage = root.lookupType("awesomepackage.AwesomeMessage");

proto.wrappers[".google.protobuf.Timestamp"] = {
  fromObject: function (object) {
    return this.fromObject(object);
  },

  toObject: function (message, options) {
    console.log(message);

    if (!(message instanceof this.ctor) && message instanceof proto.Message) {
      const object = message.$type.toObject(message, options);
      object["@blub"] = "extra stuff";
      return object;
    }

    // (message as any).blub = 1233;
    return this.toObject(message, options);
  },
};

webmaster128 · 2020-04-27T07:00:45Z

src/wrappers.js

+            options = {};
+
+        // decode value if requested
+        if (options && options.json) {


options is always truthy here as it is normalized above

webmaster128 · 2020-04-27T07:02:47Z

src/wrappers.js

+
+        // decode value if requested
+        if (options && options.json) {
+            return new Date(message.seconds*1000 + message.nanos/1000000).toISOString();


It would be very cool to comply with the 0, 3, 6 or 9 fractional digits rule from https://developers.google.com/protocol-buffers/docs/proto3#json

Example implementation: https://github.com/gogo/protobuf/blob/v1.3.1/jsonpb/jsonpb.go#L250-L266

yeah, I think this needs to be fixed before this can be merged. We did this on our internal patch.

Suggested change

return new Date(message.seconds*1000 + message.nanos/1000000).toISOString();

// golang JSON serialization truncates extra points of precision.

// we do the same.

// https://github.com/golang/protobuf/blob/master/jsonpb/encode.go#L197

var isoDateString = new Date(message.seconds*1000 + message.nanos/1000000).toISOString();

isoDateString = isoDateString.replace(/[.]000Z$/, 'Z');

return isoDateString;

(needs to be updated to handle 6 and 9 points of precision).

webmaster128 · 2020-04-27T07:11:00Z

src/wrappers.js

+            return this.fromObject(object);
+        }
+        //Convert ISO-8601 to epoch millis
+        var dt = Date.parse(object);


This implementation cuts off information, since native Date can only store 3 fractional digits:

> new Date(Date.parse("1970-01-01T00:00:05.111822333Z")).toISOString() '1970-01-01T00:00:05.111Z'

It is not too hard to implement the parser manually, allowing to keep the precision.

chunshengji · 2020-06-29T17:27:30Z

this looks great, can we get this one updated and merged?

bangbang93 · 2020-07-01T06:04:30Z

+1 keep watching

Luabee · 2020-12-10T00:02:55Z

Still hoping this gets merged

kivancguckiran · 2021-03-24T17:06:35Z

Any update on the PR?

john-michael-murphy · 2021-07-13T12:14:50Z

Such a great feature! Any chance we can get this merged?

ntindall · 2021-07-13T12:21:24Z

I'm happy to help finish this / once there is better support from the maintainers. @alexander-fenster, maybe we can coordinate, I see you have a related PR out.

alexander-fenster · 2021-08-04T22:24:11Z

(copying my note from #1495 which was an attempt to add wrappers for several other types)

Folks, for those who follows this PR and, in general, follows the topic of proper proto3 JSON serialization/deserialization according to the protobuf spec, I wanted to post a quick update here. For reasons not directly related to anything in this PR, I won't be able to complete it here soon, so I released the proto3 JSON serialization/deserialization code as a separate npm package.

$ npm install proto3-json-serializer

At this moment, we only support instances of protobuf.Message which should be enough for most use cases. Feel free to try it and create issues in that repository in case if you need anything to be fixed; contributions are also welcome :)

This separate package will cover the workflows I need at this time; if I have some free time to backport its functionality directly to protobuf.js (e.g. by generating .fromProto3JSON / .toProto3JSON together with .fromObject / .toObject), I'll send a separate pull request here, but I cannot promise any ETA on that.

ideasculptor · 2021-10-28T03:43:57Z

Even if we don't hear from maintainers, this can be added by the caller of the library as follows (random playground code, TypeScript):

import * as proto from "protobufjs";

// ...

const root = proto
  .parse(source)
  .root.addJSON(proto.common.get("google/protobuf/timestamp.proto")!.nested!)
  .addJSON(proto.common.get("google/protobuf/any.proto")!.nested!)
  .resolveAll();
// const root = parsed.root;

// Obtain a message type
const AwesomeMessage = root.lookupType("awesomepackage.AwesomeMessage");

proto.wrappers[".google.protobuf.Timestamp"] = {
  fromObject: function (object) {
    return this.fromObject(object);
  },

  toObject: function (message, options) {
    console.log(message);

    if (!(message instanceof this.ctor) && message instanceof proto.Message) {
      const object = message.$type.toObject(message, options);
      object["@blub"] = "extra stuff";
      return object;
    }

    // (message as any).blub = 1233;
    return this.toObject(message, options);
  },
};

Just reinforcing the point, for those that missed the earlier message, that you can relatively easily get fromObject() and toObject() to convert Date instances to Timestamp protobufs by adding your own wrapper - a very convenient mechanism for converting from any arbitrary object to a particular protobuf that I don't believe is addressed in the docs anywhere.

My code looks like this, and always uses Date instances when converting between javascript objects and protos. Unfortunately, I don't think there is a convenient way to set up the same conversion when calling create() with a plain javascript object. If there is, I'd love to hear about it. Perhaps I can add a create() wrapper to the fromObject and toObject wrappers I already have? That thought just occurred to me.

protobuf.wrappers[".google.protobuf.Timestamp"] = {
  fromObject: function fromObject(object) {
    if (typeof object !== 'string') {
      if (object instanceof Date) {
        return this.fromObject({
          seconds: Math.floor(object/1000),
          nanos: (object % 1000) * 1000000
        });
      }
      return this.fromObject(object)
    }
    var dt = Date.parse(object);
    if (isNaN(dt)) {
      // not a number, default to the parent implementation
      return this.fromObject(object);
    }
    return this.create({
      seconds: Math.floor(dt/1000),
      nanos: (dt % 1000) * 1000000
    });
  },
  toObject: function toObject(message, options) {
    return new Date(message.seconds*1000 + message.nanos/1000000)
  }
}

Implement wrapper for google.protobuf.Timestamp

8931805

ntindall force-pushed the nt-google.protobuf.Timestamp branch from b390fe7 to 8931805 Compare June 26, 2019 01:49

ntindall added 2 commits June 25, 2019 19:58

do not use anonymous functions in wrapeprs

19d513d

change codegen to correctly respect src/wrappers.js

d71bc93

ntindall changed the title ~~Implement wrapper for google.protobuf.Timestamp~~ Implement wrapper for google.protobuf.Timestamp, and correctly generate wrappers for static target. Jun 26, 2019

add tests, respect the { json: true } option

114ecc9

ntindall mentioned this pull request Jun 27, 2019

google.protobuf.Timestamp deserialization incompatible with canonical JSON representation #893

Open

attempt to correct for the static code scenario

c3134e0

ntindall mentioned this pull request Oct 6, 2019

Automatically wrapping & unwrapping wrappers.proto types? #677

Open

ntindall mentioned this pull request Oct 22, 2019

Proto3 canonical JSON support #1304

Open

prevent infinite loop when static typing is used

f274f40

webmaster128 reviewed Apr 27, 2020

View reviewed changes

isherman mentioned this pull request May 26, 2020

Generate a JSON client stephenh/ts-proto#7

Open

alexander-fenster mentioned this pull request Oct 10, 2020

feat: support proper JSON for Value and Duration #1495

Closed

glenjamin mentioned this pull request Dec 17, 2021

Seamless Timestamp/Date encoding/decoding grpc/grpc-node#1990

Open

swyxio mentioned this pull request Feb 16, 2022

[Feature Request] Wrap WorkflowHandle.describe output in a JS friendly object temporalio/sdk-typescript#482

Closed

PhilipMantrov mentioned this pull request May 19, 2022

feat: added nestJsTimestampTypeWrapper stephenh/ts-proto#567

Merged

bulbaasd mentioned this pull request Aug 19, 2022

Doubt: Creating timestamp #1792

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement wrapper for google.protobuf.Timestamp, and correctly generate wrappers for static target. #1258

Implement wrapper for google.protobuf.Timestamp, and correctly generate wrappers for static target. #1258

ntindall commented Jun 26, 2019 •

edited

ntindall commented Jul 9, 2019

ntindall commented Jul 26, 2019

ntindall commented Oct 6, 2019

kheyse-oqton commented Oct 22, 2019 •

edited

ntindall commented Oct 22, 2019

kheyse-oqton commented Oct 22, 2019

ntindall commented Oct 22, 2019

gfecher commented Jan 15, 2020

untra commented Feb 11, 2020 •

edited

guyisra commented Mar 19, 2020

untra commented Mar 19, 2020

webmaster128 left a comment

webmaster128 Apr 27, 2020

webmaster128 Apr 27, 2020

ntindall Oct 21, 2020

ntindall Oct 21, 2020

webmaster128 Apr 27, 2020

chunshengji commented Jun 29, 2020

bangbang93 commented Jul 1, 2020

Luabee commented Dec 10, 2020

kivancguckiran commented Mar 24, 2021

john-michael-murphy commented Jul 13, 2021

ntindall commented Jul 13, 2021

alexander-fenster commented Aug 4, 2021 •

edited

ideasculptor commented Oct 28, 2021 •

edited

-            return new Date(message.seconds*1000 + message.nanos/1000000).toISOString();
+             // golang JSON serialization truncates extra points of precision.
+             // we do the same.
+             // https://github.com/golang/protobuf/blob/master/jsonpb/encode.go#L197
+             var isoDateString = new Date(message.seconds*1000 + message.nanos/1000000).toISOString();
+             isoDateString = isoDateString.replace(/[.]000Z$/, 'Z');
+             return isoDateString;

Implement wrapper for google.protobuf.Timestamp, and correctly generate wrappers for static target. #1258

Are you sure you want to change the base?

Implement wrapper for google.protobuf.Timestamp, and correctly generate wrappers for static target. #1258

Conversation

ntindall commented Jun 26, 2019 • edited

ntindall commented Jul 9, 2019

ntindall commented Jul 26, 2019

ntindall commented Oct 6, 2019

kheyse-oqton commented Oct 22, 2019 • edited

ntindall commented Oct 22, 2019

kheyse-oqton commented Oct 22, 2019

ntindall commented Oct 22, 2019

gfecher commented Jan 15, 2020

untra commented Feb 11, 2020 • edited

guyisra commented Mar 19, 2020

untra commented Mar 19, 2020

webmaster128 left a comment

Choose a reason for hiding this comment

webmaster128 Apr 27, 2020

Choose a reason for hiding this comment

webmaster128 Apr 27, 2020

Choose a reason for hiding this comment

ntindall Oct 21, 2020

Choose a reason for hiding this comment

ntindall Oct 21, 2020

Choose a reason for hiding this comment

webmaster128 Apr 27, 2020

Choose a reason for hiding this comment

chunshengji commented Jun 29, 2020

bangbang93 commented Jul 1, 2020

Luabee commented Dec 10, 2020

kivancguckiran commented Mar 24, 2021

john-michael-murphy commented Jul 13, 2021

ntindall commented Jul 13, 2021

alexander-fenster commented Aug 4, 2021 • edited

ideasculptor commented Oct 28, 2021 • edited

ntindall commented Jun 26, 2019 •

edited

kheyse-oqton commented Oct 22, 2019 •

edited

untra commented Feb 11, 2020 •

edited

alexander-fenster commented Aug 4, 2021 •

edited

ideasculptor commented Oct 28, 2021 •

edited