Reimplement rules in Starlark in rules_python #1069

rickeylev · 2023-02-14T07:05:45Z

This is the general tracking issue for re-implementing the Bazel-bundled Python rules in Starlark in the rules_python repo.

The Bazel-side issue is bazelbuild/bazel#15897

As of now (Feb 2023), the Bazel bundled rules (not providers or flags, though) have all been re-implemented in Starlark. They will used in Bazel 7. Hooks for low-level Bazel behaviors the Python rules need are exposed to the rules_python repo. So now this is mostly a matter of copying that code over here, cleaning it up, and flushing out any missing hooks.

TODO

Copy Bazel code over to rules_python
Re-implement Bazel tests in rules_python
Re-implement flags in Starlark
Setup a CI config that uses e.g. bazel=rolling to get access to pre-release bazel versions
Enable pystar by default

That's the high level gist, some lower level TODOs are:

Implement compatibility shim so rules_python and Bazel rules can interoperate to some degree
Deprecate and remove Bazel-bundled flags
Deprecate and remove Python-related parts in @bazel_tools

An optional bonus thing is to backport the low-level hooks to 5.x and/or 6.x. This makes them available sooner, so that we don't have to wait for Bazel 7. There isn't a ton to backport (I intentionally made them to be amenable to this idea), but there's several small pieces. The two general pieces are exposing the flags as fragments and the py_internal object:

py_internal object and it's backing pieces:
Configuration fields (Bazel builtin flags); Strictly speaking, only flags that don't have a fixed set of values (e.g arbitrary string or labels) need to be made available. Other flags can be "wrapped" in a regular Starlark-defined flag by just having an exhaustive listing of config_settings
- --python_path and --python_top from BazelPythonConfiguration; note, however, that --python_top is supposed to be unused (superceded by toolchains) and --python_path is probably unused (unclear; some comments indicate it might be used as part of windows support)
- --build_python_zip from PythonConfiguration.java; note, however, it, too, can be replicated in Starlark by using platforms or the py_internal.get_current_os_name() helper.

The text was updated successfully, but these errors were encountered:

This is a clean-copy without any changes to provide a baseline Work towards bazelbuild#1069

This is to provide some regression tests for the Starlark rewrite. Work towards bazelbuild#1069

This is to provide some regression tests for the Starlark rewrite. These tests are approximately the same as Bazel's Java-implemented tests. Work towards bazelbuild#1069

This is to provide some regression tests for the Starlark rewrite. These tests are approximately the same as Bazel's Java-implemented tests. Work towards #1069

This is so that the transitive dependencies of defs.bzl can be easily found and validated; some Google internal tooling does this validation. The old comment indicated bzl_library wasn't used to avoid a dependency on skylib, however, we've since added a dependency on skylib. Work towards bazelbuild#1069

This is so that the transitive dependencies of defs.bzl can be easily found and validated; some Google internal tooling does this validation. The old comment indicated bzl_library wasn't used to avoid a dependency on skylib, however, we've since added a dependency on skylib. Work towards #1069

This is basically a performance optimization. Bazel can only see that a bzl file changed, not the particular contents. This means that any downstream bzl file loading it is invalidated, even if it doesn't load any of the affected code. As an example, if a package only loads `py_library.bzl`, then changing `py_test.bzl` doesn't need to invalidate all libraries. Work towards bazelbuild#1069

This is basically a performance optimization. Bazel can only see that a bzl file changed, not the particular contents. This means that any downstream bzl file loading it is invalidated, even if it doesn't load any of the affected code. As an example, if a package only loads `py_library.bzl`, then changing `py_test.bzl` doesn't need to invalidate all libraries. * Also removes some more extraneous license comments Work towards bazelbuild#1069

This is basically a performance optimization. Bazel can only see that a bzl file changed, not the particular contents. This means that any downstream bzl file loading it is invalidated, even if it doesn't load any of the affected code. As an example, if a package only loads `py_library.bzl`, then changing `py_test.bzl` doesn't need to invalidate all libraries. * Also removes some more extraneous license comments Work towards #1069

This provides a public entry point for loading the underlying `PyCcLinkParamsProvider` provider that is built into Bazel. This provider isn't yet usable from Bazel, but adding a loadable way for it to migrate off the built-in rules is the first step. Work towards bazelbuild#1069

This provides a public entry point for loading the underlying `PyCcLinkParamsProvider` provider that is built into Bazel. This provider isn't yet usable from Bazel, but adding a loadable way for it to migrate off the built-in rules is the first step. Work towards #1069

This is to provide some regression tests for the Starlark rewrite. These tests are approximately the same as Bazel's Java-implemented tests. Work towards bazelbuild#1069

…nabled The PyCcLinkParamsInfo export wasn't respecting the `config.enable_pystar` setting. Work towards bazelbuild#1069

…s enabled (#1517) The PyCcLinkParamsInfo export wasn't respecting the `config.enable_pystar` setting. This would cause the Starlark implementation to look up the wrong provider symbol and result in an error later. Work towards #1069

This is to have better test coverage. Only workspace for them is used because of limited CI slots. Work towards bazelbuild#1069

This is to have better test coverage. Only workspace for them is used because of limited CI slots. Also switches from last_green to last_rc, since a Bazel 7 RC is now available. This will help catch any changes that need to be backported from Bazel head to the Bazel 7 branch. Work towards bazelbuild#1069

This is to have better test coverage. Only workspace for them is used because of limited CI slots. This also fixes the test_basic_windows test. The `--build_python_zip` flag built into Bazel has a different default depending on the host OS (not target platform): true for windows, and false otherwise. Updated the test to force the flag value for reliable behavior between platforms. Work towards #1069

This makes the rules_python Starlark implementation accept and return the builtin providers. This allows depending on, and being depended on by, the builtin rules, which enables the two rule sets to interoperate better. Work towards bazelbuild#1069

This makes the rules_python Starlark implementation accept and return the builtin providers. This allows depending on, and being depended on by, the builtin rules, which enables the two rule sets to interoperate better. Work towards #1069

This makes rules_python compatible with disabling using the native rules in Bazel. Work towards bazelbuild#1069

With the release of Bazel 7, support for 5 is no longer necessary. Work towards bazelbuild#1069

This makes rules_python compatible with disabling using the native rules in Bazel. While we're here, update some examples in docs to load py_runtime to better make clear the builtin py_runtime objects shouldn't be used. Work towards #1069

With the release of Bazel 7, support for 5 is no longer necessary. Also deletes all the integration tests of the examples. They were broken when run from the main repo, and CI was only building them, not running them, as part of the main repo due to having `--test_tag_filters=-integration-test`. CI runs them separately in their own workspace. Work towards #1069

Ryang20718 · 2023-12-29T18:30:05Z

wondering if pyc support is on the roadmap? #1761

… Bazel 7+ Respecting it for earlier versions was only done for testing purposes. The implementation requires the `py_internal` object, which only exists on Bazel 7 and higher. Work towards bazelbuild#1069

This makes the base rules use the Starlark code in rules_python ("pystar") instead of what is built into Bazel 7. The pystar implementation can be disabled by setting `RULES_PYTHON_ENABLE_PYSTAR=0` in your environment or using `--action_env`. Earlier versions of Bazel continue to use the rules built into Bazel regardless (the new implementation requires special APIs that only Bazel 7 has). Work towards bazelbuild#1069

… Bazel 7+ (#1698) Respecting it for earlier versions was only done for testing purposes. The implementation requires the `py_internal` object, which only exists on Bazel 7 and higher. Work towards #1069

This makes the base rules use the Starlark code in rules_python ("pystar") instead of what is built into Bazel 7. The pystar implementation can be disabled by setting `RULES_PYTHON_ENABLE_PYSTAR=0` in your environment or using `--action_env`. Earlier versions of Bazel continue to use the rules built into Bazel regardless (the new implementation requires special APIs that only Bazel 7 has). Work towards bazelbuild#1069

…7+ (#1699) This makes the base rules use the Starlark code in rules_python ("pystar") instead of what is built into Bazel 7. The pystar implementation can be disabled by setting `RULES_PYTHON_ENABLE_PYSTAR=0` in your environment or using `--action_env`. Earlier versions of Bazel continue to use the rules built into Bazel regardless (the new implementation requires special APIs that only Bazel 7 has). Work towards #1069

rickeylev · 2024-02-13T17:24:30Z

The 0.31.0 release of rules_python has enabled the rules_python Starlark implementation for Bazel 7+. 🎆 !

If you encounter issues, please file bugs in rules_python. I'm eager to identify and fix any issues. If you do encounter issues, setting RULES_PYTHON_ENABLE_PYSTAR=0 in your environment (directly or using --action_env) will disable the rules_python implementation.

The most likely problem to occur is using a combination of the Bazel-builtin rules/providers and the rules_python rules/providers. They should interoperate OK, but please report any issues.

rickeylev added type: feature request core-rules Issues concerning the stubs for native rules labels Feb 14, 2023

rickeylev self-assigned this Feb 14, 2023

rickeylev added a commit to rickeylev/rules_python that referenced this issue Feb 24, 2023

Copy Bazel's Starlark-based Python rules to rules_python.

0bad1ce

This is a clean-copy without any changes to provide a baseline Work towards bazelbuild#1069

rickeylev added a commit to rickeylev/rules_python that referenced this issue Feb 27, 2023

test(core): Add analysis tests for base Python rules.

2f1da4e

This is to provide some regression tests for the Starlark rewrite. Work towards bazelbuild#1069

rickeylev added a commit to rickeylev/rules_python that referenced this issue Feb 27, 2023

test(core): Add analysis tests for base Python rules.

fa9261e

This is to provide some regression tests for the Starlark rewrite. Work towards bazelbuild#1069

rickeylev mentioned this issue Feb 27, 2023

test(core): Add analysis tests for base Python rules. #1102

Merged

rickeylev mentioned this issue Mar 8, 2023

feat: add bzl_library for defs.bzl and its dependencies #1115

Merged

rickeylev mentioned this issue Mar 9, 2023

feat: bzl file per rule/provider #1122

Merged

rickeylev pinned this issue Mar 10, 2023

rickeylev mentioned this issue Mar 10, 2023

feat: add public entry point for PyCcLinkParamsInfo #1128

Merged

rickeylev mentioned this issue Oct 24, 2023

internal(pystar): use rules_python PyCcLinkParamsProvider if pystar is enabled #1517

Merged

rickeylev added a commit to rickeylev/rules_python that referenced this issue Nov 8, 2023

test(pystar): run pystar under Windows and Mac

677de7a

This is to have better test coverage. Only workspace for them is used because of limited CI slots. Work towards bazelbuild#1069

rickeylev mentioned this issue Nov 8, 2023

test(pystar): run pystar under Windows and Mac #1547

Merged

rickeylev mentioned this issue Nov 21, 2023

pystar: support builtin providers for compatibility #1573

Merged

rickeylev added a commit to rickeylev/rules_python that referenced this issue Dec 12, 2023

chore: use rules_python's py_runtime in generated toolchains

1a48a42

This makes rules_python compatible with disabling using the native rules in Bazel. Work towards bazelbuild#1069

rickeylev mentioned this issue Dec 12, 2023

fix: use rules_python's py_runtime in generated toolchains #1604

Merged

rickeylev added a commit to rickeylev/rules_python that referenced this issue Dec 14, 2023

chore: use rules_python's py_runtime in generated toolchains

f79ecad

This makes rules_python compatible with disabling using the native rules in Bazel. Work towards bazelbuild#1069

rickeylev added a commit to rickeylev/rules_python that referenced this issue Dec 14, 2023

chore: Update Bazel support: drop 5, minimum 6.2, current 7.0

5d02131

With the release of Bazel 7, support for 5 is no longer necessary. Work towards bazelbuild#1069

rickeylev mentioned this issue Dec 14, 2023

chore: Update Bazel support: drop 5, minimum 6.2, current 7.0 #1613

Merged

rickeylev mentioned this issue Jan 17, 2024

internal(pystar): Only respect RULES_PYTHON_ENABLE_PYSTAR env var for Bazel 7+ #1698

Merged

rickeylev mentioned this issue Jan 17, 2024

feat: enable rules_python Starlark implemention by default for Bazel 7+ #1699

Merged

junyer mentioned this issue Mar 12, 2024

How to set python interpreter within cc_binary pybind/pybind11_bazel#77

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reimplement rules in Starlark in rules_python #1069

Reimplement rules in Starlark in rules_python #1069

rickeylev commented Feb 14, 2023 •

edited

Ryang20718 commented Dec 29, 2023

rickeylev commented Feb 13, 2024 •

edited

Reimplement rules in Starlark in rules_python #1069

Reimplement rules in Starlark in rules_python #1069

Comments

rickeylev commented Feb 14, 2023 • edited

Ryang20718 commented Dec 29, 2023

rickeylev commented Feb 13, 2024 • edited

rickeylev commented Feb 14, 2023 •

edited

rickeylev commented Feb 13, 2024 •

edited