fix: add support for protobuf 5.x

[parthea]

Fixes #628 🦕
Fixes #641 🦕
Fixes #642 🦕

Regarding the including_default_value_fields argument of json_format.MessageToDict() in protobuf:

In protobuf version 3.19.6 - The default value of including_default_value_fields was False
https://github.com/protocolbuffers/protobuf/blob/5cba162a5d93f8df786d828621019e03e50edb4f/python/google/protobuf/json_format.py#L92

At the time that the argument was removed in protobuf 5.x - the default value of including_default_value_fields was False
protocolbuffers/protobuf@2699579#diff-8de817c14d6a087981503c9aea38730b1b3e98f4e306db5ff9d525c7c304f234

IOW, setting argument including_default_value_fields to False had no effect as it was the default behaviour.

The Unit tests / unit_prerelease-3.12 presubmit check was added to run tests against the latest pre-release version of protobuf as several dependencies have a constraint on protobuf<5

[anndro]

Hello @parthea is there any scheduled date for release this pull request?

[zhangskz]

@parthea Is there anything blocking this PR still?

[vchudnov-g]

A couple of suggestions, and some questions, but nothing major. This looks good!

[vchudnov-g]

You have this block repeated in four places. I suggest factoring it out into a helper function that only needs to take one parameter:

Suggested change

	# For backwards compatibility with protobuf 3.x 4.x
	# Remove once support for protobuf 3.x and 4.x is dropped
	# https://github.com/googleapis/python-api-core/issues/643
	if PROTOBUF_VERSION[0:2] in ["3.", "4."]:
	request_kwargs = json_format.MessageToDict(
	request,
	preserving_proto_field_name=True,
	including_default_value_fields=True, # type: ignore # backward compatibility
	)
	else:
	request_kwargs = json_format.MessageToDict(
	request,
	preserving_proto_field_name=True,
	always_print_fields_with_no_presence=True,
	)
	transcoded_request = path_template.transcode(http_options, **request_kwargs)
	transcoded_request = _transcode_request(request)

[vchudnov-g]

This is more succinct and clearer in highlighting the difference between the two branches, IMO:

Suggested change

	if install_grpc:
	session.install(
	"-e",
	".[grpc]",
	"-c",
	f"{constraints_dir}/constraints-{session.python}.txt",
	)
	else:
	session.install(
	"-e", ".", "-c", f"{constraints_dir}/constraints-{session.python}.txt"
	)
	session.install(
	"-e",
	".[grpc]" if install_grpc else ".",
	"-c",
	f"{constraints_dir}/constraints-{session.python}.txt",
	)

[vchudnov-g]

I'm confused: in line 206, where we call default(..., prerelease=True), the decorator constrains python=PYTHON_VERSIONS[-1]. But here we are referencing the constraints for PYTHON_VERSIONS[0]. Could you clarify?

[vchudnov-g]

Sorry, I have gaps in my mental model for testing prerelease;

This installs our dependencies at pre-release versions, right?

Is the only difference, then, the use of --pre below? If it's the same set of dependencies, it seems it would be clearer to have them loaded in the same place for pre-release and stable versions, and add the extra install parameters conditionally on each thing we install.

(It might be easier to talk synchronously about this.)

[vchudnov-g]

So this tests running against the pre-release versions of the dependencies, right? And we only bother to do this with the latest Python run-time.

A comment might be helpful. Also, my personal preference would to rename this (and similar code elsewhere) as unit_with_prerelease_deps. Every time I read something like unit_prerelease my mind first assumes we're testing the pre-release version of this library, not of its dependencies.

[vchudnov-g]

I think this more succinct code should work (I haven't tested it) and would be easier to maintain.
Ref: https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#expanding-or-adding-matrix-configurations

        option: ["", "_grpc_gcp", "_wo_grpc"]
        python:
          - "3.7"
          - "3.8"
          - "3.9"
          - "3.10"
          - "3.11"
          - "3.12"
        exclude:
          - option: "_wo_grpc"
            python: 3.7
          - option: "_wo_grpc"
            python: 3.8
          - option: "_wo_grpc"
            python: 3.9
         include:
          - option: "_prerelease"
            python: 3.12