docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

dougthor42 · 2024-04-11T18:52:56Z

Add an example of using a src dir and separate tests dir and
having gazelle generate targets correctly so that tests can be run.

Fixes #1775.

dougthor42 · 2024-04-16T16:56:05Z

I can't seem to figure out why builds are failing. Any tips?

If I run bazel test //... from the new example directory, things run just fine.

$ cd examples/bzlmod_python_src_dir_with_separate_tests_dir/
$ bazel test //...
INFO: Analyzed 10 targets (4 packages loaded, 24 targets configured).
INFO: Found 7 targets and 3 test targets...
INFO: Elapsed time: 20.303s, Critical Path: 18.58s
INFO: 13 processes: 7 internal, 2 local, 4 processwrapper-sandbox.
INFO: Build completed successfully, 13 total actions
//:gazelle_python_manifest.test                                          PASSED in 0.2s
//:requirements_test                                                     PASSED in 16.2s
//tests:test_my_python_module                                            PASSED in 0.6s

Executed 3 out of 3 tests: 3 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

If I run from the git repo root, I get the same failures seen in CI (unknown repo 'pypi').

$ cd -
/c/dev/rules_python
$ bazel test //...
ERROR: Skipping '//...': error loading package under directory '': error loading package 'examples/bzlmod_python_src_dir_with_separate_tests_dir': Unable to find package for @@[unknown repo 'pypi' requested from @@]//:requirements.bzl: The repository '@@[unknown repo 'pypi' requested from @@]' could not be resolved: No repository visible as '@pypi' from main repository.
ERROR: error loading package under directory '': error loading package 'examples/bzlmod_python_src_dir_with_separate_tests_dir': Unable to find package for @@[unknown repo 'pypi' requested from @@]//:requirements.bzl: The repository '@@[unknown repo 'pypi' requested from @@]' could not be resolved: No repository visible as '@pypi' from main repository.
INFO: Elapsed time: 25.549s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Couldn't start the build. Unable to run tests

From what I can tell, the only differences between this example and bzlmod or bzlmod_build_file_generation is that this example:

Doesn't have a WORKSPACE file
Uses pypi for the hub name instead of pip.

But I tried making both of those edits locally and still got the same error.

Note: I'm ignoring the MacOS and Windows CI failures for now.

aignas · 2024-04-17T14:17:52Z

Please run the pre-commit hooks that will update .bazelignore, which should fix the errors seen in the tests running from the root.

Edit: or the .bazelrc, can't remember which one in this case.

dougthor42 · 2024-04-17T17:52:35Z

Thanks. Turns out it's both .bazelrc and .bazelignore.

I also noticed that the pre-commit hook update-deleted-packages doesn't appear to work correctly, so I've opened #1858 with details.

aignas

Sorry, I thought I had submitted these review comments, but it seems that I had not done this. Happy to move this forward.

I think we may want to add a bazel-in-bazel integration test somewhere here.

aignas · 2024-04-18T01:02:20Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

@@ -0,0 +1,69 @@
+# Define metadata about this repository/project.


Having an empty WORKSPACE file may fix some of the issues you saw with the pre-commit hook, I think.

Hmm... still no dice. Same issue.

Given that CI is passing, I'm not too concerned about the pre-commit check. My assumption is that it's something specific to my computer/setup rather than specific to this branch, as pre-commit run --all-files also fails on the latest main commit 3730803.

aignas · 2024-04-18T01:05:05Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+# In the old WORKSPACE file, this would be 4 items:
+#   1.  `load` the http_archive rule
+#   2.  run the http_archive rule, grapping rules_python from github
+#   3.  load the py_repositories target from rules_python
+#   4.  execute py_respositories()


We could just link to the relevant example (e.g. pip_parse or pip_parse_vendored).

Relative file references added, but I kept the original list because I find it easier to follow if I don't have to jump around to various other examples. LMK if you feel strongly about removing the list.

aignas · 2024-04-18T01:06:04Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+)
+
+# Install rules_python, which allows us to define how bazel should work with python files.
+# See https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/MODULE.bazel


Why not use relative file references? At least in vim users could use gf to go to the file instantly and I hope other editors might support something similar.

Suggested change

# See https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/MODULE.bazel

# See ../bzlmod/MODULE.bazel

+1, updated here and for gazelle/README.md below.

aignas · 2024-04-18T01:06:44Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+    # Use the bazel downloader for pulling pypi packages.
+    experimental_index_url = "https://pypi.org/simple",


aignas · 2024-04-18T01:08:09Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+    requirements_lock = "//:requirements.lock",
+)
+
+# Same as WORKSPACE install_deps() - actually install the python deps.


This is not technically correct. Here we just expose the repo to be used by the module and the install happens lazily by bazel. I do think the intention is good - to draw parallels between WORKSPACE and MODULE.bazel, but this example might be something that people may read without any WORKSPACE knowledge.

Ah, good to know! Updated, PTAL and check for correctness.

dougthor42

I thought I had submitted these review comments

I do the same things all the time. I really wish GitHub had some "you have an unsubmitted review pending" notification or something.

But anyway, thanks for the comments! And don't worry about the delay! You've already been super responsive elsewhere, and adding a new example is a low priority.

I think we may want to add a bazel-in-bazel integration test somewhere here.

Can you elaborate on this?

dougthor42 · 2024-05-18T20:53:39Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

@@ -0,0 +1,69 @@
+# Define metadata about this repository/project.


Hmm... still no dice. Same issue.

Given that CI is passing, I'm not too concerned about the pre-commit check. My assumption is that it's something specific to my computer/setup rather than specific to this branch, as pre-commit run --all-files also fails on the latest main commit 3730803.

dougthor42 · 2024-05-18T20:54:05Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+)
+
+# Install rules_python, which allows us to define how bazel should work with python files.
+# See https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/MODULE.bazel


+1, updated here and for gazelle/README.md below.

dougthor42 · 2024-05-18T20:57:59Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+# In the old WORKSPACE file, this would be 4 items:
+#   1.  `load` the http_archive rule
+#   2.  run the http_archive rule, grapping rules_python from github
+#   3.  load the py_repositories target from rules_python
+#   4.  execute py_respositories()


Relative file references added, but I kept the original list because I find it easier to follow if I don't have to jump around to various other examples. LMK if you feel strongly about removing the list.

dougthor42 · 2024-05-18T21:07:19Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+    requirements_lock = "//:requirements.lock",
+)
+
+# Same as WORKSPACE install_deps() - actually install the python deps.


Ah, good to know! Updated, PTAL and check for correctness.

dougthor42 added 3 commits April 11, 2024 11:50

Add python directory structure and code. No bazel yet.

53f8a26

Add bazel and gazelle config

472850e

bazel run: requiremens.update, manifest, gazelle

a5dad2e

dougthor42 marked this pull request as ready for review April 11, 2024 18:53

dougthor42 requested a review from rickeylev as a code owner April 11, 2024 18:53

dougthor42 added 2 commits April 11, 2024 14:06

Better comments in README.md

60a6482

Update changelog

df44f4b

dougthor42 added 2 commits April 17, 2024 09:18

run ./tools/update_deleted_pacakges.sh

fb5a495

Add new example to .bazelignore

0dc8f34

dougthor42 mentioned this pull request Apr 17, 2024

pre-commit hook for updating deleted packages removes items added by the script that it calls #1858

Open

aignas reviewed May 13, 2024

View reviewed changes

dougthor42 added 4 commits May 18, 2024 13:56

Use relative file references

4215fb7

Add links to other WORKSPACE examples

ec41cfe

Update comment about 'use_repo'

7303a70

Spelling, grammar, and capitalization of comments

4fac351

dougthor42 commented May 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

dougthor42 commented Apr 11, 2024

dougthor42 commented Apr 16, 2024

aignas commented Apr 17, 2024 •

edited

dougthor42 commented Apr 17, 2024

aignas left a comment

aignas Apr 18, 2024

dougthor42 May 18, 2024

aignas Apr 18, 2024

dougthor42 May 18, 2024

aignas Apr 18, 2024

dougthor42 May 18, 2024

aignas Apr 18, 2024

aignas Apr 18, 2024

dougthor42 May 18, 2024

dougthor42 left a comment

dougthor42 May 18, 2024

dougthor42 May 18, 2024

dougthor42 May 18, 2024

dougthor42 May 18, 2024

		@@ -0,0 +1,69 @@
		# Define metadata about this repository/project.

	# See https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/MODULE.bazel
	# See ../bzlmod/MODULE.bazel

		# Use the bazel downloader for pulling pypi packages.
		experimental_index_url = "https://pypi.org/simple",

docs(examples): Add example of using a src dir and separate tests dir with gazelle #1842

Are you sure you want to change the base?

docs(examples): Add example of using a src dir and separate tests dir with gazelle #1842

Conversation

dougthor42 commented Apr 11, 2024

dougthor42 commented Apr 16, 2024

aignas commented Apr 17, 2024 • edited

dougthor42 commented Apr 17, 2024

aignas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dougthor42 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

aignas commented Apr 17, 2024 •

edited