New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a setuptools extension for generating Python protobufs #7783
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
8db9cef
Add a setuptools/distutils extension for generating Python protobuf c…
dlj-NaN 33a4e67
Tweak documentation in setup.py, and copy or move some of it to a new…
dlj-NaN a748602
Update copyright/license/author headers to match other Python sources.
dlj-NaN 6f9074b
- Fixed syntax highlight in README.md.
dlj-NaN abad144
Ignore the protobuf_distutils directory for cpp_distcheck.
dlj-NaN 8f7d467
Add Python 3.8, tweak wording, fix inconsistent filename for generate…
dlj-NaN bd98cc5
Add an option `proto_root_path`, which is the root for resolving impo…
dlj-NaN f86283b
Tweak and update options in README.md.
dlj-NaN 635e081
Tweak docstrings even more.
dlj-NaN File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
# Python setuptools extension | ||
|
||
This is an extension for Python setuptools which uses an installed protobuf | ||
compiler (`protoc`) to generate Python sources. | ||
|
||
## Installing | ||
|
||
To use this extension, it needs to be installed so it can be imported by other | ||
projects' setup.py. | ||
|
||
```shell | ||
$ python setup.py build | ||
$ python setup.py install | ||
``` | ||
|
||
(If you want to test changes to the extension, you can use `python setup.py | ||
develop`.) | ||
|
||
## Usage | ||
|
||
### Example setup.py configuration | ||
|
||
```python | ||
from setuptools import setup | ||
setup( | ||
# ... | ||
name='example_project', | ||
|
||
# Require this package, but only for setup (not installation): | ||
setup_requires=['protobuf_distutils'], | ||
|
||
options={ | ||
# See below for details. | ||
'generate_py_protobufs': { | ||
'source_dir': 'path/to/protos', | ||
'extra_proto_paths': ['path/to/other/project/protos'], | ||
'output_dir': 'path/to/project/sources', # default '.' | ||
'proto_files': ['relative/path/to/just_this_file.proto'], | ||
'protoc': 'path/to/protoc.exe', | ||
}, | ||
}, | ||
) | ||
``` | ||
|
||
### Example build invocation | ||
|
||
These steps will generate protobuf sources so they are included when building | ||
and installing `example_project` (see above): | ||
|
||
```shell | ||
$ python setup.py generate_py_protobufs | ||
$ python setup.py build | ||
$ python setup.py install | ||
``` | ||
|
||
## Options | ||
|
||
- `source_dir`: | ||
|
||
This is the directory holding .proto files to be processed. | ||
|
||
The default behavior is to generate sources for all .proto files found under | ||
`source_dir`, recursively. This behavior can be controlled with options below. | ||
|
||
- `proto_root_path`: | ||
|
||
This is the root path for resolving imports in source .proto files. | ||
|
||
The default is the shortest prefix of `source_dir` among `[source_dir] + | ||
self.extra_proto_paths`. | ||
|
||
- `extra_proto_paths`: | ||
|
||
Specifies additional paths that should be used to find imports, in | ||
addition to `source_dir`. | ||
|
||
This option can be used to specify the path to other protobuf sources, | ||
which are imported by files under `source_dir`. No Python code will | ||
be generated for .proto files under `extra_proto_paths`. | ||
|
||
- `output_dir`: | ||
|
||
Specifies where generated code should be placed. | ||
|
||
Typically, this should be the root package that generated Python modules | ||
should be below. | ||
|
||
The generated files will be placed under `output_dir` according to the | ||
relative source paths under `proto_root_path`. For example, the source file | ||
`${proto_root_path}/subdir/message.proto` will be generated as the Python | ||
module `${output_dir}/subdir/message_pb2.py`. | ||
|
||
- `proto_files`: | ||
|
||
A list of strings, specific .proto file paths for generating code, instead of | ||
searching for all .proto files under `source_path`. | ||
dlj-NaN marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
These paths are relative to `source_dir`. For example, to generate code | ||
for just `${source_dir}/subdir/message.proto`, specify | ||
`['subdir/message.proto']`. | ||
|
||
- `protoc`: | ||
|
||
By default, the protoc binary (the Protobuf compiler) is found by | ||
searching the environment path. To use a specific protoc binary, its | ||
path can be specified. |
Empty file.
145 changes: 145 additions & 0 deletions
145
python/protobuf_distutils/protobuf_distutils/generate_py_protobufs.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,145 @@ | ||
# Protocol Buffers - Google's data interchange format | ||
# Copyright 2008 Google Inc. All rights reserved. | ||
# https://developers.google.com/protocol-buffers/ | ||
# | ||
# Redistribution and use in source and binary forms, with or without | ||
# modification, are permitted provided that the following conditions are | ||
# met: | ||
# | ||
# * Redistributions of source code must retain the above copyright | ||
# notice, this list of conditions and the following disclaimer. | ||
# * Redistributions in binary form must reproduce the above | ||
# copyright notice, this list of conditions and the following disclaimer | ||
# in the documentation and/or other materials provided with the | ||
# distribution. | ||
# * Neither the name of Google Inc. nor the names of its | ||
# contributors may be used to endorse or promote products derived from | ||
# this software without specific prior written permission. | ||
# | ||
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | ||
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | ||
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | ||
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | ||
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | ||
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | ||
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | ||
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | ||
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
||
"""Implements the generate_py_protobufs command.""" | ||
|
||
__author__ = 'dlj@google.com (David L. Jones)' | ||
|
||
import glob | ||
import sys | ||
import os | ||
import distutils.spawn as spawn | ||
from distutils.cmd import Command | ||
from distutils.errors import DistutilsOptionError, DistutilsExecError | ||
|
||
class generate_py_protobufs(Command): | ||
"""Generates Python sources for .proto files.""" | ||
|
||
description = 'Generate Python sources for .proto files' | ||
user_options = [ | ||
('extra-proto-paths=', None, | ||
'Additional paths to resolve imports in .proto files.'), | ||
|
||
('protoc', None, | ||
'Path to a specific `protoc` command to use.'), | ||
] | ||
boolean_options = ['recurse'] | ||
|
||
def initialize_options(self): | ||
"""Sets the defaults for the command options.""" | ||
self.source_dir = None | ||
self.proto_root_path = None | ||
self.extra_proto_paths = [] | ||
self.output_dir = '.' | ||
self.proto_files = None | ||
self.recurse = True | ||
self.protoc = None | ||
|
||
def finalize_options(self): | ||
"""Sets the final values for the command options. | ||
Defaults were set in `initialize_options`, but could have been changed | ||
by command-line options or by other commands. | ||
""" | ||
self.ensure_dirname('source_dir') | ||
self.ensure_string_list('extra_proto_paths') | ||
|
||
if self.output_dir is None: | ||
self.output_dir = '.' | ||
self.ensure_dirname('output_dir') | ||
|
||
# SUBTLE: if 'source_dir' is a subdirectory of any entry in | ||
# 'extra_proto_paths', then in general, the shortest --proto_path prefix | ||
# (and the longest relative .proto filenames) must be used for | ||
# correctness. For example, consider: | ||
# | ||
# source_dir = 'a/b/c' | ||
# extra_proto_paths = ['a/b', 'x/y'] | ||
# | ||
# In this case, we must ensure that a/b/c/d/foo.proto resolves | ||
# canonically as c/d/foo.proto, not just d/foo.proto. Otherwise, this | ||
# import: | ||
# | ||
# import "c/d/foo.proto"; | ||
# | ||
# would result in different FileDescriptor.name keys from "d/foo.proto". | ||
# That will cause all the definitions in the file to be flagged as | ||
# duplicates, with an error similar to: | ||
# | ||
# c/d/foo.proto: "packagename.MessageName" is already defined in file "d/foo.proto" | ||
# | ||
# For paths in self.proto_files, we transform them to be relative to | ||
# self.proto_root_path, which may be different from self.source_dir. | ||
# | ||
# Although the order of --proto_paths is significant, shadowed filenames | ||
# are errors: if 'a/b/c.proto' resolves to different files under two | ||
# different --proto_path arguments, then the path is rejected as an | ||
# error. (Implementation note: this is enforced in protoc's | ||
# DiskSourceTree class.) | ||
|
||
if self.proto_root_path is None: | ||
self.proto_root_path = os.path.normpath(self.source_dir) | ||
for root_candidate in self.extra_proto_paths: | ||
root_candidate = os.path.normpath(root_candidate) | ||
if self.proto_root_path.startswith(root_candidate): | ||
self.proto_root_path = root_candidate | ||
if self.proto_root_path != self.source_dir: | ||
self.announce('using computed proto_root_path: ' + self.proto_root_path, level=2) | ||
|
||
if not self.source_dir.startswith(self.proto_root_path): | ||
raise DistutilsOptionError('source_dir ' + self.source_dir + | ||
' is not under proto_root_path ' + self.proto_root_path) | ||
|
||
if self.proto_files is None: | ||
files = glob.glob(os.path.join(self.source_dir, '*.proto')) | ||
if self.recurse: | ||
files.extend(glob.glob(os.path.join(self.source_dir, '**', '*.proto'))) | ||
self.proto_files = [f.partition(self.proto_root_path + os.path.sep)[-1] for f in files] | ||
if not self.proto_files: | ||
raise DistutilsOptionError('no .proto files were found under ' + self.source_dir) | ||
|
||
self.ensure_string_list('proto_files') | ||
|
||
if self.protoc is None: | ||
self.protoc = spawn.find_executable('protoc') | ||
|
||
def run(self): | ||
# All proto file paths were adjusted in finalize_options to be relative | ||
# to self.proto_root_path. | ||
proto_paths = ['--proto_path=' + self.proto_root_path] | ||
proto_paths.extend(['--proto_path=' + x for x in self.extra_proto_paths]) | ||
|
||
# Run protoc. It was already resolved, so don't try to resolve | ||
# through PATH. | ||
spawn.spawn( | ||
[self.protoc, | ||
'--python_out=' + self.output_dir, | ||
] + proto_paths + self.proto_files, | ||
search_path=0) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
# Protocol Buffers - Google's data interchange format | ||
# Copyright 2008 Google Inc. All rights reserved. | ||
# https://developers.google.com/protocol-buffers/ | ||
# | ||
# Redistribution and use in source and binary forms, with or without | ||
# modification, are permitted provided that the following conditions are | ||
# met: | ||
# | ||
# * Redistributions of source code must retain the above copyright | ||
# notice, this list of conditions and the following disclaimer. | ||
# * Redistributions in binary form must reproduce the above | ||
# copyright notice, this list of conditions and the following disclaimer | ||
# in the documentation and/or other materials provided with the | ||
# distribution. | ||
# * Neither the name of Google Inc. nor the names of its | ||
# contributors may be used to endorse or promote products derived from | ||
# this software without specific prior written permission. | ||
# | ||
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | ||
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | ||
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | ||
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | ||
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | ||
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | ||
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | ||
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | ||
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
||
"""Setuptools/distutils extension for generating Python protobuf code. | ||
This extension uses a prebuilt 'protoc' binary to generate Python types for | ||
protobuf sources. By default, it will use a system-installed protoc binary, but | ||
a custom protoc can be specified by flag. | ||
This command should usually be run before the 'build' command, so that the | ||
generated sources are treated the same way as the rest of the Python | ||
sources. | ||
Options: | ||
source_dir: | ||
This is the directory holding .proto files to be processed. | ||
The default behavior is to generate sources for all .proto files found | ||
under `source_dir`, recursively. This behavior can be controlled with | ||
options below. | ||
proto_root_path: | ||
This is the root path for resolving imports in source .proto files. | ||
The default is the shortest prefix of `source_dir` among: | ||
[source_dir] + self.extra_proto_paths | ||
extra_proto_paths: | ||
Specifies additional paths that should be used to find imports, in | ||
addition to `source_dir`. | ||
This option can be used to specify the path to other protobuf sources, | ||
which are imported by files under `source_dir`. No Python code will be | ||
generated for .proto files under `extra_proto_paths`. | ||
output_dir: | ||
Specifies where generated code should be placed. | ||
Typically, this should be the root package that generated Python modules | ||
should be below. | ||
The generated files will be named according to the relative source paths | ||
under `proto_root_path`. For example, this source .proto file: | ||
${proto_root_path}/subdir/message.proto | ||
will correspond to this generated Python module: | ||
${output_dir}/subdir/message_pb2.py | ||
proto_files: | ||
Specific .proto files can be specified for generating code, instead of | ||
searching for all .proto files under `source_path`. | ||
These paths are relative to `source_dir`. For example, to generate code | ||
for just ${source_dir}/subdir/message.proto, specify | ||
['subdir/message.proto']. | ||
protoc: | ||
By default, the protoc binary (the Protobuf compiler) is found by | ||
searching the environment path. To use a specific protoc binary, its | ||
path can be specified. | ||
recurse: | ||
If `proto_files` are not specified, then the default behavior is to | ||
search `source_dir` recursively. This option controls the recursive | ||
search; if it is False, only .proto files immediately under `source_dir` | ||
will be used to generate sources. | ||
""" | ||
|
||
__author__ = 'dlj@google.com (David L. Jones)' | ||
|
||
from setuptools import setup, find_packages | ||
|
||
setup( | ||
name='protobuf_distutils', | ||
version='1.0', | ||
packages=find_packages(), | ||
maintainer='protobuf@googlegroups.com', | ||
maintainer_email='protobuf@googlegroups.com', | ||
license='3-Clause BSD License', | ||
classifiers=[ | ||
"Framework :: Setuptools Plugin", | ||
"Operating System :: OS Independent", | ||
# These Python versions should match the protobuf package: | ||
"Programming Language :: Python", | ||
"Programming Language :: Python :: 2", | ||
"Programming Language :: Python :: 2.7", | ||
"Programming Language :: Python :: 3", | ||
"Programming Language :: Python :: 3.3", | ||
"Programming Language :: Python :: 3.4", | ||
"Programming Language :: Python :: 3.5", | ||
"Programming Language :: Python :: 3.6", | ||
"Programming Language :: Python :: 3.7", | ||
dlj-NaN marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"Programming Language :: Python :: 3.8", | ||
"Topic :: Software Development :: Code Generators", | ||
], | ||
description=('This is a distutils extension to generate Python code for ' | ||
'.proto files using an installed protoc binary.'), | ||
url='https://github.com/protocolbuffers/protobuf/', | ||
entry_points={ | ||
'distutils.commands': [ | ||
('generate_py_protobufs = ' | ||
'protobuf_distutils.generate_py_protobufs:generate_py_protobufs'), | ||
], | ||
}, | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The package installs as protobuf-disutils, with a hyphen not an underbar.
After running build and install:
python -m pip freeze|grep protobuf
protobuf-distutils==1.0