Bazel Tests

Scope

This document describes how to define, build and run tests that exclusively exist in the Bazel graph.

Fuchsia supports Bazel host tests that meet the following requirements:

These tests must be buildable exclusively with Bazel; i.e. they do not depend on artifacts generated by a prior Ninja invocation.
These tests only include binaries and runtime files that can run on a compatible host system that is NOT connected to any Fuchsia device or emulator.

Fuchsia also supports Bazel device tests, declared with fx_test() and run on a Fuchsia device or emulator. These have the same requirement of being buildable exclusively with Bazel, and additionally:

Aside from environments (which defaults to the build's default test environments), build_only, max_log_severity (which defaults to WARN), and test_type (for running in a non-hermetic test realm such as "starnix" or "system"), test spec fields (timeouts, etc.) cannot be customized per test yet.

fxbug.dev/564574581 tracks the remaining work for target tests in Bazel.

Defining Fuchsia host tests in Bazel

Regular Bazel test rules (e.g. rustc_test, go_test) are not compatible with Fuchsia test runners. They will not be visible to fx test and will not be run by infra builders.

Hence, to define a Fuchsia-compatible host test in Bazel:

Use language-specific host test wrapper from //build/bazel/rules/host_tests:host_<xxx>_test.bzl, such as host_rustc_test(), host_go_test(), etc.

These are designed to be drop-in replacements for regular Bazel test rules, with extra arguments like test_data, test_args, so Fuchsia test runners can invoke them with the right runtime environment and arguments.
Only if no language-specific wrappers are available for your test, use the generic host_test() wrapper from //build/bazel/rules/host_tests:host_test.bzl. For example:
```
load("//build/bazel/rules/host_tests:host_test.bzl", "host_test")
load("@rules_sh//sh:sh_binary.bzl", "sh_binary")

sh_binary(
    name = "test_bin",
    ...
)

host_test(
    name = "shell_test",
    binary = ":test_bin",
)
```
Note: ALWAYS prefer using language-specific wrappers if available, as they also implement extra Fuchsia-specific features, which are critical for parsing test results in infra.
Do NOT assume the execution location of a test binary (e.g. from out/default), as it is launched from a directory whose exact path is not predictable.

Bazel host tests MUST be hermetic and list all their runtime dependencies explicitly, just like GN host tests. Runtime dependencies can be provided through one of the mechanisms:
- As test_data dependencies to a host_test_data_files() or host_test_data_map() target (see //build/bazel/rules/host_tests:host_test_data.bzl). These make the files available at runtime at fixed locations, relative to the test's execution directory, similar to the GN host_test_data() template.
- As data attribute on a host_xxx_test() target. In this case, the files will be added to the test's runfiles, and can be accessed using a runfiles library.
Use test_suite() instead of filegroup() to define groups of tests. Test suites can depend on tests and other test suites.

Warning: Bazel skips tests from test_suites if they are incompatible with the current configuration. See Bazel test compatibility for more details.

Exporting Bazel host tests to `fx test` and infra builders

Defining the host_xxx_test() target alone will NOT make it visible to fx test and, more importantly, to infra builders.

To make a Bazel host test visible to fx test and infra builders, write a bazel_test_suite target in GN that lists the Bazel labels of one or more Bazel host tests or suites, and add that target to a "tests" group in GN. For example:

//src/foo/bar/BUILD.bazel:

load("//build/bazel/platforms:constraints.bzl", "HOST_OS_CONSTRAINTS")

test_suite(
  name = "host_tests",
  host_tests = [
    "//src/foo/bar/lib:lib_tests",
    "//src/foo/bar/util:util_tests",
  ]
  target_compatible_with = HOST_OS_CONSTRAINTS,
)

//src/foo/bar/BUILD.gn:

import("//build/bazel/bazel_test_suite.gni")

bazel_test_suite("bazel_tests") {
  host_tests = [
    # This references the target in BUILD.bazel.
    "//src/foo/bar:host_tests",
  ]
}

group("tests") {
  deps = [
    ":bazel_tests",
    ...
  ]
}

This will cause entries for the Bazel tests to appear in out/default/tests.json. When in doubt, examine the content of this file after fx set or fx gen. The label field for Bazel tests begins with @.

During development, it is possible to build and run test targets once they are defined, before they are exported as described above. See the Running tests directly with Bazel section below.

Best practices

It's recommended to avoid deep nesting of Bazel test_suite targets. Instead, use just one test_suite Bazel target and one corresponding bazel_test_suite GN target per source code sub-area (e.g. host tool or Fuchsia component).

Defining Fuchsia device tests in Bazel

Device tests are components in a Fuchsia package, exactly like in the GN build. Declare the test component with fx_test_component(), package it with fx_package(), and expose it as a test with fx_test():

//src/foo/bar/BUILD.bazel:

load("//build/bazel/rules/packages:fx_packaged_binary.bzl", "fx_packaged_binary")
load(
    "//build/bazel/rules/components:fx_component.bzl",
    "fx_component_manifest",
    "fx_test_component",
)
load("//build/bazel/rules/packages:fx_package.bzl", "fx_package")
load("//build/bazel/rules/testing:fx_test.bzl", "fx_test")

fx_packaged_binary(
    name = "bar_test_packaged_bin",
    testonly = True,
    binary = ":bar_test_bin",
    binary_name = "bar_test",
)

fx_component_manifest(
    name = "bar_test_manifest",
    testonly = True,
    component_name = "bar_test",
    includes = ["//src/sys/test_runners/gtest:default.shard.cml"],
    manifest = "meta/bar_test.cml",
)

fx_test_component(
    name = "bar_test_component",
    compiled_manifest = ":bar_test_manifest",
    component_name = "bar_test",
    deps = [":bar_test_packaged_bin"],
)

fx_package(
    name = "bar_tests_package",
    package_name = "bar_tests",
    test_components = [":bar_test_component"],
)

fx_test(
    name = "bar_tests",
    package = ":bar_tests_package",
    # Optional: defaults to the build's default test environments when omitted.
    environments = [
        "//build/testing/environments:aemu_env",
        "//build/testing/environments:nuc11_env",
    ],
)

Every component in test_components becomes a separate test, whose name is its package URL, e.g. fuchsia-pkg://fuchsia.com/bar_tests#meta/bar_test.cm.

To customize which target environments a test runs in, pass environments to fx_test() as a list of environment targets. //build/testing/environments has one target for each individual environment in //build/testing/environments.gni, with the same name (e.g. :emu_env or :nuc11_env). When environments is omitted, the test runs in the build's default test environments (GN's basic_envs), exactly like a GN test that doesn't set environments. To build a test without running it, set build_only = True instead.

To add infra settings, such as tags, to one of these environments, use fx_test_environment_variant():

load(
    "//build/bazel/rules/testing:fx_test_environment.bzl",
    "fx_test_environment_variant",
)

fx_test_environment_variant(
    name = "nuc11_isolated_env",
    base = "//build/testing/environments:nuc11_env",
    env_tags = ["e2e-isolated"],
)

New environments should be rare; most tests should use one of the existing environments in //build/testing/environments. When a new environment is needed, add it to both //build/testing/environments.gni and //build/testing/environments/BUILD.bazel, rather than defining it with fx_test_environment() next to a test. Environments' Swarming dimensions are an interface with infra, so they're kept in one place.

To run a test in a non-hermetic test realm (such as "starnix", "system", or "vulkan"), pass test_type to fx_test_component() (or to fx_test() to apply to all test components in the package). Valid test_type values match type_moniker_map in //build/components/fuchsia_test_component.gni.

Exporting Bazel device tests to `fx test` and infra builders

Like host tests, device tests must be listed in a GN bazel_test_suite() target, using the target_tests parameter instead of host_tests:

//src/foo/bar/BUILD.gn:

import("//build/bazel/bazel_test_suite.gni")

bazel_test_suite("bazel_tests") {
  target_tests = [
    # This references the target in BUILD.bazel.
    "//src/foo/bar:bar_tests",
  ]
}

Device tests are tracked separately from host tests because they are queried and built in the fuchsia_platform Bazel configuration, rather than the host one.

In addition to the tests.json entries, this writes ${root_build_dir}/bazel_test_packages.list, which enumerates the package manifests of all Bazel test packages in the build graph. Full fx build invocations merge this list into all_package_manifests.list and publish its packages into amber-files, while fx test also merges it directly when building specific test targets on demand.

Testing libraries in both build systems

While a library is used by targets in both build systems, run its tests in both build systems. Once the library has users in only Bazel, run its tests only in Bazel. Tests for targets that are only built in Bazel—such as migrated host tools, libraries with no remaining GN users, and leaf integration test suites that don't export a library to GN—run only in Bazel.

A library migrated to Bazel stays defined in GN (usually generated by bazel2gn) until no GN targets depend on it. During that time, the GN and Bazel builds of the library can differ in ways that only tests catch. Compiler flags, build variants such as sanitizers, and GN-only bazel2gn overrides all differ between the two build systems. Some attributes are also dropped during translation, and the GN target is what unmigrated dependents actually link against. Running each test in both build systems costs extra build time and test capacity, but only while a library has users in both build systems.

In practice:

Define tests in Bazel and let bazel2gn generate the GN copy. When you migrate a library whose BUILD.gn is generated by bazel2gn, migrate its tests too, and don't put # @bazel2gn:skip on them. bazel2gn translates rustc_test(), go_test(), host_rustc_test(), host_go_test(), fx_unittest_package(), and the with_unit_tests, with_host_unit_tests, and test_deps attributes of Rust rules. Export the Bazel test with a bazel_test_suite() as described above, and list the generated GN copy in the GN test groups that listed the original GN test.
Device test packages get different names in GN. A GN test package and its Bazel copy can't share a package name, because both are published to the same package repository. bazel2gn appends -gn to the package name of the GN copy of an fx_unittest_package(), and the Bazel test keeps the original name, so its test URL doesn't change when you migrate it.
If bazel2gn can't translate a test's packaging yet and you want to keep running it in GN, translate the test binary with bazel2gn and write the GN test package by hand above the ## BAZEL2GN SENTINEL in BUILD.gn with package_name = "<name>-gn". This works for device tests that use fx_unittest_package(resources = [...]) (put # @bazel2gn:skip on fx_unittest_package() and fuchsia_package_resource_group(), not on the test binary) or explicit fx_package() and fx_test_component() targets for custom .cml manifests (which bazel2gn already ignores without # @bazel2gn:skip).
When to opt out with # @bazel2gn:skip. By default, keep running a library's tests in both build systems when bazel2gn can translate them automatically. It's fine to put # @bazel2gn:skip on a test target or test attribute (with a brief comment explaining why) and run it only in Bazel when:
- The target under test is only built in Bazel (for example, a binary or library in the same BUILD.bazel file that itself has # @bazel2gn:skip).
- bazel2gn doesn't support the test's rules or attributes (such as cc_test(), host_cc_test(), host_py_test(), host_shac_test(), host_test(), host tests with data / test_data / test_label, fx_unittest_package() with resources, or rustc_test() with cross-package srcs/crate or conflicting rustc_env), and maintaining a hand-written GN test target above the sentinel isn't worth the extra complexity.
- The cost of running or maintaining the test in both build systems outweighs the risk of a GN-only regression. Use these criteria to judge the tradeoff:
- Target parity: Does the library's BUILD.bazel use # @bazel2gn:raw_overwrite, # @bazel2gn:path_overwrite, or # @bazel2gn:skip on dependencies, configs, or compiler flags, or is it an IDK/SDK library compiled across multiple configurations in GN? If the generated GN target is a 1:1 translation of the Bazel target, the Bazel test already exercises the same code.
- Unmigrated GN dependencies: Does the library depend on GN-only targets that change frequently? Presubmit affected-test analysis doesn't trace from GN targets into Bazel tests, so changes to a GN-only dependency won't trigger a Bazel-only test in presubmit.
- Test weight: Fast, self-contained unit tests are cheap to mirror; slow or resource-heavy integration tests (or libraries whose remaining GN users are actively being migrated) are good candidates to run only in Bazel.
If Bazel can't express a test yet, keep it in GN above the ## BAZEL2GN SENTINEL and say why in a comment. Examples are e2e tests, tests that need Ninja-generated inputs, bootfs_test()s, and sdk_cpp_unittest_package()s that compile across multiple C++ standards.
Remove the GN copy with the library. When the last GN user of a library migrates to Bazel, delete the GN definition of the library and its tests, and remove the GN tests from the GN test groups that listed them.

Running Fuchsia Bazel device tests

Once exported, run a Bazel device test the same way as a GN one, by package URL or by package name:

fx test fuchsia-pkg://fuchsia.com/bar_tests#meta/bar_test.cm

fx test builds the test by invoking Bazel directly (as fx build --fuchsia_platform bazel_label), then publishes the package manifests listed in bazel_test_packages.list before running the test. A running package server and a connected device or emulator are required, exactly as for GN device tests.

Running Fuchsia Bazel host tests

Local testing

Running tests with `fx test`

To run Bazel host tests with fx test, they MUST be exported by following the instructions in the Exporting Bazel host tests to fx test and infra builders section above.

After the tests are exported, use fx test --host <name_or_label> to build and run a Bazel host test locally, same as with GN-defined host tests.

The only difference is that the test will be built on demand by invoking Bazel directly, skipping Ninja entirely.

Note that when using fx test:

<name_or_label> will be subject to fuzzy matching, and fx test will report multiple candidates in case of ambiguity.
<name_or_label> cannot reference a Bazel test_suite() label, just like it cannot reference a GN group() one.

See the User guide for fx test for more details about how to use fx test.

Running tests directly with `fx bazel test`

You can use fx bazel test --config=host <target_pattern> to build and run tests locally directly with Bazel. This does not require the extra plumbing to make tests visible to fx test as described in the previous section, and has the following advantages:

<target_pattern> is a set of Bazel target patterns that can match "all test targets in a given package", in which case Bazel will skip non-test targets automatically.
<target_pattern> can match test_suite() labels which can be convenient to run a set of host tests repeatedly during development.
bazel test caches test results, and will only re-run the tests impacted by your latest changes since the last invocation.

(Use --nocache_test_results flag to disable this).
<target_pattern> can reference labels that are not visible in tests.json. This can be handy during development when adding new tests.

Infra testing

Only host tests that are listed in tests.json can be launched on infra test bots. This applies equally to GN and Bazel host tests. So you MUST ensure they are visible in tests.json by following the Exporting Bazel host tests to fx test and infra builders section above, otherwise they will NOT be continuously tested on infra.

Infra builds Bazel tests using fint, which directly calls bazel build after running ninja, to build all Bazel tests that are listed in bazel_test_suite GN targets.

There is no explicit distinction between GN and Bazel-defined host tests when running them on infra test runners, or when collecting results. In particular, infra does not run tests using bazel test. Instead, it takes the binaries produced by bazel build and runs them directly, just like it does with host tests from ninja.

Debug symbol propagation

Host test binaries compiled with debug symbols must have their unstripped ELF binaries registered so that tooling can symbolize backtraces and correlate LLVM code coverage profiles (-profile-correlate=binary).

For general background on how Fuchsia manages Bazel debug symbols, see Technical Note on Debug Symbol Generation.

Authoring host test rules with debug symbols

The generic host_test() rule exposes the unstripped_binary field in its FuchsiaHostTestInfo provider:

If unstripped_binary is not explicitly set, host_test() inspects the underlying binary attribute:
- If binary provides DebugPackageInfo (C++ targets), it uses DebugPackageInfo.unstripped_file.
- If binary provides CrateInfo (Rust targets), it uses CrateInfo.output.
If the test uses a wrapper script launcher (such as rust_test_parser in host_rustc_test()), the macro MUST explicitly forward the original unstripped binary target via unstripped_binary:
```
host_test(
    name = name,
    binary = wrapper_script,
    unstripped_binary = ":" + binary_name,
    ...
)
```

Propagation to infra and debuginfod

When Bazel host tests are exported to the GN build:

Test Query: bazel_tests_utils.py runs a bazel cquery using //build/bazel/starlark/FuchsiaHostTestInfo.cquery, extracting the execroot path of unstripped_binary for all registered host tests.
Host Test Manifest: The paths are normalized relative to the Ninja build directory and written to ${root_build_dir}/bazel_host_tests.debug_symbols.json.
GN Build API Integration: //:bazel_host_test_suites in BUILD.gn attaches debug_symbol_manifests metadata pointing to bazel_host_tests.debug_symbols.json, which is consumed by the build_api_module("debug_symbols") target.
Artifactory Upload: In CI/CQ builds, Artifactory processes debug_symbols.json and uploads the unstripped ELF binaries to cloud storage / debuginfod servers, allowing tools like covargs to resolve Build IDs.

Local development and `fx coverage`

When running coverage locally at desk, uncommitted binaries have newly generated Build IDs that are not present on debuginfod servers.

To support local symbol correlation, fx coverage runs:

python3 ${FUCHSIA_DIR}/build/bazel/scripts/copy_bazel_debug_symbols.py ${FUCHSIA_BUILD_DIR}

This script reads bazel_host_tests.debug_symbols.json, computes the GNU Build ID for each ELF binary, and copies or symlinks the unstripped binary into ${root_build_dir}/.build-id/xx/yyyyyyyy.debug. Tooling such as ffx coverage and llvm-profdata then resolves these binaries via .symbol-index.json.

FAQ

Bazel test compatibility

If any test in a test_suite() (or any of its dependencies) is incompatible with the current configuration (e.g., using target_compatible_with), that test will be skipped during evaluation of the suite instead of failing the build.

The same is true when using patterns such as //... and :all on the command line.

For more information, see the handling of incompatible targets.

When running a test_suite() or using a target pattern, developers must verify that the test(s) of interest are reported as PASSED (and not SKIPPED).