[Build] mach boostrap fails with: Exception: python/mozboot/**/*android* did not match anything

[yuripourre]

Configuration

Wolvic version: master
Wolvic build ID: -

Hardware: Dell Inspiron 15 7000 / Fedora 38

Steps to Reproduce

1. Follow the Developer workflow
2. Download the firefox-$VERSION.source.tar.xz
3. Apply the patches
4. Try to run ./mach bootstrap

Current Behavior

Build fails with the following error:

Exception: python/mozboot/**/*android* did not match anything

Expected Behavior

Command would succeed

Possible Solution

Context

Error Logs and Stack Traces

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html.

Please choose the version of Firefox you want to build (see note above):
  1. Firefox for Desktop Artifact Mode [default]
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
  5. SpiderMonkey JavaScript engine
Your choice: 4
Requirement already satisfied: glean-sdk==55.0.0 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (55.0.0)
Requirement already satisfied: semver>=2.13.0 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean-sdk==55.0.0) (3.0.2)
Requirement already satisfied: glean_parser~=10.0 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean-sdk==55.0.0) (10.0.3)
Requirement already satisfied: appdirs>=1.4 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean_parser~=10.0->glean-sdk==55.0.0) (1.4.4)
Requirement already satisfied: Click>=7 in ./third_party/python/click (from glean_parser~=10.0->glean-sdk==55.0.0) (8.1.6)
Requirement already satisfied: diskcache>=4 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean_parser~=10.0->glean-sdk==55.0.0) (5.6.3)
Requirement already satisfied: Jinja2>=2.10.1 in ./third_party/python/Jinja2 (from glean_parser~=10.0->glean-sdk==55.0.0) (3.1.2)
Requirement already satisfied: jsonschema>=3.0.2 in ./third_party/python/jsonschema (from glean_parser~=10.0->glean-sdk==55.0.0) (4.17.3)
Requirement already satisfied: PyYAML>=5.3.1 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean_parser~=10.0->glean-sdk==55.0.0) (6.0.1)
Requirement already satisfied: MarkupSafe>=2.0 in ./third_party/python/MarkupSafe/src (from Jinja2>=2.10.1->glean_parser~=10.0->glean-sdk==55.0.0) (2.0.1)
Requirement already satisfied: attrs>=17.4.0 in ./third_party/python/attrs (from jsonschema>=3.0.2->glean_parser~=10.0->glean-sdk==55.0.0) (23.1.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in ./third_party/python/pyrsistent (from jsonschema>=3.0.2->glean_parser~=10.0->glean-sdk==55.0.0) (0.16.0)
Requirement already satisfied: six in ./third_party/python/six (from pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0->jsonschema>=3.0.2->glean_parser~=10.0->glean-sdk==55.0.0) (1.16.0)
Requirement already satisfied: psutil<=5.9.4,>=5.4.2 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (5.9.4)
Requirement already satisfied: zstandard<=0.22.0,>=0.11.1 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (0.22.0)

Looks like you have the correct version of the Android NDK installed at:
/home/user/.mozbuild/android-ndk-r23c


Looks like you have the Android SDK installed at:
/home/user/.mozbuild/android-sdk-linux
We will install all required Android packages.


We are now installing the following Android packages:
emulator
platform-tools
build-tools;33.0.1
platforms;android-33
You may be prompted to agree to the Android license. You may see some of
output as packages are downloaded and installed.

Warning: Errors during XML parse:                                               
Warning: Additionally, the fallback loader failed to parse the XML.             
[=======================================] 100% Computing updates...             


Looks like you have the correct version of the Android NDK installed at:
/home/user/.mozbuild/android-ndk-r23c


Looks like you have the Android SDK installed at:
/home/user/.mozbuild/android-sdk-linux
We will install all required Android packages.


We are now installing the following Android packages:
emulator
system-images;android-24;default;x86_64
You may be prompted to agree to the Android license. You may see some of
output as packages are downloaded and installed.

Warning: Errors during XML parse:                                               
Warning: Additionally, the fallback loader failed to parse the XML.             
[=======================================] 100% Computing updates...             


Looks like you have the correct version of the Android NDK installed at:
/home/user/.mozbuild/android-ndk-r23c


Looks like you have the Android SDK installed at:
/home/user/.mozbuild/android-sdk-linux
We will install all required Android packages.


We are now installing the following Android packages:
emulator
system-images;android-24;default;armeabi-v7a
You may be prompted to agree to the Android license. You may see some of
output as packages are downloaded and installed.

Warning: Errors during XML parse:                                               
Warning: Additionally, the fallback loader failed to parse the XML.             
[=======================================] 100% Computing updates...             

Your version of Rust (1.78.0) is new enough.
Rust supports aarch64-linux-android, i686-linux-android, thumbv7neon-linux-androideabi, x86_64-linux-android, x86_64-unknown-linux-gnu targets.
Error running mach:

    mach --log-no-times artifact toolchain --from-build toolchain-linux64-clang

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: python/mozboot/**/*android* did not match anything

  File "/home/user/git/firefox-121.0.1/python/mozbuild/mozbuild/artifact_commands.py", line 393, in artifact_toolchain
    tasks = toolchain_task_definitions()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/python/mozbuild/mozbuild/toolchains.py", line 20, in toolchain_task_definitions
    toolchains = load_tasks_for_kind(params, "toolchain", root_dir=root_dir)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 449, in load_tasks_for_kind
    for task in tgg.full_task_set
                ^^^^^^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 160, in full_task_set
    return self._run_until("full_task_set")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 425, in _run_until
    k, v = next(self._run)
           ^^^^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 311, in _run
    new_tasks = kind.load_tasks(
                ^^^^^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in load_tasks
    tasks = [
            ^
  File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in <listcomp>
    tasks = [
            ^
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 2219, in check_run_task_caches
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 2129, in check_task_dependencies
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 2115, in check_task_identifiers
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 2096, in chain_of_trust
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1899, in build_task
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1872, in set_task_and_artifact_expiry
    for job in jobs:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1845, in try_task_config_routes
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1832, in try_task_config_chemspill_prio
    yield from tasks
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1812, in try_task_config_env
    yield from tasks
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1779, in add_index_routes
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1574, in validate
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1552, in task_name_from_label
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1542, in setup_raptor
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1486, in set_defaults
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1462, in set_implementation
    for task in tasks:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/cached_tasks.py", line 65, in cache_task
    for task in order_tasks(config, tasks):
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/cached_tasks.py", line 20, in order_tasks
    pending = deque(tasks)
              ^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 282, in use_fetches
    jobs = list(jobs)
           ^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 247, in use_system_python
    for job in jobs:
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 228, in make_task_description
    configure_taskdesc_for_run(
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 507, in configure_taskdesc_for_run
    func(config, job, taskdesc)
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/toolchain.py", line 247, in docker_worker_toolchain
    common_toolchain(config, job, taskdesc, is_docker=True)
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/toolchain.py", line 170, in common_toolchain
    digest_data = get_digest_data(config, run, taskdesc)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/toolchain.py", line 82, in get_digest_data
    data = [hash_paths(GECKO, files)]
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/util/hash.py", line 57, in hash_paths
    raise Exception("%s did not match anything" % pattern)

[svillar]

Could you please share the mozbuild file you're using?

[svillar]

Could you please share the mozbuild file you're using?

Also did mach bootstrap went well with no issues?

[yuripourre]

Could you please share the mozbuild file you're using?

Thanks for the reply! Sure, I can share but I haven't touched it. It would be the same as the one in the gecko-dev repo.

Also did mach bootstrap went well with no issues?

No, actually the error happens exactly when I run ./mach bootstrap.

[svillar]

Could you please share the mozbuild file you're using?

Thanks for the reply! Sure, I can share but I haven't touched it. It would be the same as the one in the gecko-dev repo.

Also did mach bootstrap went well with no issues?

No, actually the error happens exactly when I run ./mach bootstrap.

Right, again the problem of bootstrap ever changing the dependencies it downloads depending on the date you execute it :(. I am not sure there is an easy fix for that. I don't want to change my dev environment now that we're in the middle of a release but I can try to reproduce the issue after that

[yuripourre]

@svillar Thank you, I appreciate your efforts. I will try some ideas this weekend and I will update here if I have some progress.

[weller0]

The following methods can be used to build
Download gecko124.0.1 and then build the environment in gecko124.0.1

./mach bootstrap

Switch to gecko121.0.1 again to build

./mach build

[yuripourre]

Thank you @weller0! That worked! I could proceed after using your suggestion.

I had more errors, fixed some and currently struggling with libclang_rt.builtins-aarch64-android.a not found. I am trying to compile the compiler-rt but having issues in Fedora, apparently some internal packages cannot be found (Fedora problems, no worries).

I think we consider it closed now. Thank you very much!