When `SOONG_DELVE_STEPS` enviromnment variable is set in addition
to `SOONG_DELVE`, only given soong_build invocations ('steps') are
run in the debugger.
Also, make bootstrapBlueprint code table-driven.
Test: treehugger
Change-Id: Ia4016240ca9e88c2a85c6d06851c5bba30e7b2b5
Bug: b/258870895
Test: run `m --bazel-mode nothing && touch bionic/docs/BUILD.bazel && m --bazel-mode nothing` Then verify symlink_forest reran and `out/soong/workspace/bionic/docs/BUILD.bazel` does NOT exist
Change-Id: I49797fe52b3e0fb6ffe17ed57b8da0b69628fd7e
Bazel is expected to generate some files and symlinks that will not be
created by aquery, we explicitly ignore files that are in bazel output
paths but are not in a generated file directory.
Test: $ m --bazel-mode-staging checkbuild and verify it gets past
dangling rule test
Fixes: 258396112
Change-Id: I6dc45ea2613c9e1488ad3eda2313033c717217ff
Bug: b/256037411
Test: run mixed build twice and ensure soong_build_metric.pb file is absent on second run.
Change-Id: Ia11d850e1ad318e6383d8b392f20420259bc2f4d
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.
Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.
This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.
Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.
This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.
Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
OUT_DIR may not be in TOP (where the source tree is).
Test: presubmits
Test: set OUT_DIR=$HOME/out1; m --bazel-mode nothing
Fixes: 254022328
Change-Id: I8cf9fe2da3cdc0ba5489dfbe41d9d32f06ace83d
This is to use bazel to build targets that are being prepared for an
incipient release to the prod mode allowlist.
Bug: 254265047
Test: m nothing
Test: m nothing --bazel-mode-dev
Test: m nothing --bazel-mode-staging
Change-Id: Ic78a59cf51dba83ef1ac26483586560ea9b24aaf
The generated Bazel workspace will only contain api specific targets.
This is feasible since these targets do not have any cross dependencies
with the targets in the bp2build workspace
The advantages of a new mode are
1. Does not pollute bp2build workspace with api targets
2. Does not block api targets with the current allowlist conversion
mechansims in bp2build
(In the future we might want to combine these two workspaces)
A Soong module type will generate a Bazel target if it implements
ApiProvider interface
Test: m apigen
Test: m nothing
Change-Id: I69c57ca6539f932e0ad554ce84a87fb7936fdba0
Bazel at HEAD has a bug which causes Bazel to crash when HOME is set to
a relative path. We circumvent this bug by specifying an absolute path.
Also added some documentation on the significance of setting this HOME
value.
Bug: 248104019
Test: Presubmits
Test: Verified `m --bazel-mode-dev nothing` continues to pass with Bazel@HEAD
Change-Id: Id05c8dc322c8cca04f3d1caf44f5b04ffa96fd3b
This also shaves off a couple seconds from the build.
Test: `time m nothing` 2 times with and without the change
Bug: 243699430
Change-Id: I5cc5cba2b98c845ddd6ba7c3b31ad678451afe41
This allows "bazel mixed builds prod mode", in additional to reworking
the mechanism in which mixed builds dev mode is enabled.
As a followup, CI scripts will be migrated to use the new flags, as
USE_BAZEL_ANALYSIS=1 is deprecated.
Test: Manually ran --bazel-mode with an allowlist verifying that the
module alone was enabled
Test: Manually verified --bazel-mode and --bazel-mode-dev cause a build
failure
Change-Id: If0d34360e60452f428b05828f4ec7596b7cb619a
Per aosp/2183398, out/soong/soong.variables provide some useful information for debugging purpose.
This change enables copying out/soong/soong.variables to out/dist/soong_ui/soong/soong.variables in dist mode, similar to build.ninja.gz in the same directory.
Test confirmed that the content of the out/soong/soong.variables is
equal to that of out/dist/soong_ui/soong/soong.variables.
Test: m dist $target &&
diff out/soong/soong.variables out/dist/soong_ui/soong/soong.variables
Change-Id: I1bfa53a8e116c49fda2cc9a7252f4da0c59ce5b5
If we didn't need to run soong_build during the current run, we still
try to load the soong metrics. But in the case of `dist`, that's in a
directory that is not guaranteed to persist between runs.
Make loading the soong metrics optional if the file does not exist. Also
fixes a variable shadowing issue that meant we never passed it into
ctx.Metrics.
Test: treehugger
Change-Id: Ic836282f4d13e91daa0e7241ad7c488de3293d8b
Metrics are added to a materialized protobuf soong_build_metrics.pb
Recorded events have scoped names and recorded runtimes.
For example, mixedbuilds includes the following events:
mixedbuilds.prepare
mixedbuilds.analyze.write_files
mixedbuild (this is a total of all mixedbuild events)
Test: Manually verified materialized protos for mixedbuild, bp2build,
and legacy build.
Change-Id: I2e72d4502c0373bee0279bb1320b2e4dd586129c
Colin and Jin opined that we shouldn't change production in order to
make debugging easier. I originally thought it's better to have one
fewer knob but I don't have very strong feelings about it and their line
of reasoning makes total sense.
Test: Presubmits.
Change-Id: Id8a914aebbeffec6e8f860157c13f81093194f14
This behavior is enabled by setting the environment variable
SOONG_UNBUFFERED_OUTPUT=1 .
This is not the default behavior because only one process can be running
in the console pool and this would thus result in a loss of parallelism
if multiple soong_build invocations are needed.
Changing the value of this environment variable does not result in a
rebuild, since it does not affect the build outputs.
Test: Presubmits.
Change-Id: I49d15eb145966bd805f0a47b46866285203b122a
[ 0% 0/5 ] generating Soong docs at out/soong/docs/soong_build.html
0:03 analyzing Android.bp files and generating ninja file at out/soong/build.ninja
0:03 converting Android.bp files to BUILD files at out/soong/bp2build
0:03 generating the Soong module graph at out/soong/module-graph.json
0:03 generating the Soong module graph as a Bazel workspace at out/soong/queryview
0:03 generating Soong docs at out/soong/docs/soong_build.html
Test: m soong_docs json-module-graph bp2build queryview nothing
Change-Id: I33abb8828ead9ef999ffa1a6c9c4b016957c6dd1
The bpglob changes in build/blueprint submitted together with this
change make it necessary.
Test: Presubmits.
Change-Id: I3d19761ec1a199c2f77693a882f8744ce404c6de
It's better not to have state hidden in dotfiles (or rather,
dotdirectories) if we can help.
It's questionable whether the "linux-x86" path segment makes sense since
soong_build only ever runs on one operating system, but I didn't want to
rock the boat now.
Drive-by fixed some quoting fixes in rbc-run. Notably, I didn't wrap
`$@` into double quotes because I don't know whether the lack of double
quotes was intended or not.
Also drive-by fixed the fact that "out/soong" was added twice to the
directory name of bpglob. This turned out not to be a problem because
bpglob doesn't need to be explicitly built: if it's needed, it's
declared as an input of the glob files so it'll be built automatically
(at the cost of the first "null build" not actually being a null build)
Test: Presubmits.
Change-Id: I710d8d16cd8212059a0ca1ee95378505303eed83
ToolDir is going to become unstable when switching between KatiEnabled
and Soong-only builds while the duplication between out/soong/host and
out/host is resolved. bpglob gets executed very early during bootstrap,
before the primary builder has run to update the paths to match the
current configuration. Move it into SoongOutDir() so that its path
is more stable.
Moving bpglob causes incremental build issues, so add a bootstrapEpoch
constant that can be changed when making incompatible changes and will
wipe bootstrap intermediates.
Bug: 204136549
Test: m nothing
Change-Id: I7b1bd1ebfe1209d11db691b3ee00873ef92658cd
It was the same as SoongNinjaFile, except it used
shared.JoinPath instead of filepath.Join, which
doesn't matter here because the second argument
is never an absolute path.
Test: Presubmits
Change-Id: I95e64a8b8971bd168a1dc6670ae80f4d336045b5
- Remove useValidations: whenever it was used, it was turned on, so
there was no point in keeping that knob
- Use the new soong_build invocation name constants
Test: Presubmits.
Change-Id: Ia3e8928d30d3de1e0adaa741111771304e56b139
They were a bit repetitive.
In addition, make the environment checks more correct; they could
probably use an integration test or two to make sure that when the
environment changes, exactly those outputs are rebuilt that need to be,
but for now, this is an improvement already.
Test: Presubmits.
Change-Id: Idd79b81ca6975d57d00e5bf4699d266152505ff8
This makes soong_ui the only place where soong_build is invoked, thus
greatly simplifying the conceptual model of the build.
It comes with the slight limitation that now soong_docs (and queryview
and the JSON module graph) are not Make targets anymore, but I suppose
that's an acceptable loss.
The only place where someone depended on soong_docs from a Makefile is
removed in a separate change.
Test: Presubmits.
Change-Id: I3f9ac327725c15d84de725d05e3cdde1da3dcbe2
Its value is a function of the call site, so it doesn't make a lot of
sense to plumb it through the configuration.
Test: Presubmits.
Change-Id: If928b34de075969fd42932212ce9187808cbdf86
It used to be invoked from out/soong/build.ninja, which required two
soong_build invocations one after the other (ne to generate
out/soong/build.ninja, one to generate the queryview workspace). This
was slower and required some shell-quoted-in-ninja-quoted-in-Go .
Test: Presubmits.
Change-Id: Idda79c067606663b66e9f94626fa24f3b5af4114
This includes the JSON graph generator and bp2build.
Before:
GENERATE_BAZEL_FILES=1 m nothing
GENERATE_JSON_MODULE_GRAPH=1 m nothing
Now:
m json-module-graph
m bp2build
They can now also be combined with other targets or each other.
The longer-term goal is to run "m queryview" and "m soong_docs" using
the same infrastructure. There are two alternate approaches:
1. Call soong_build from within the main Ninja invocation. This requires
two sequential soong_build invocations and is thus slower.
2. Do everything requested in the same soong_build invocation. This
would be faster, but one AFAIU can't tell Ninja that multiple possible
actions can build the same output so that doesn't work.
(1) is somewhat more desirable because soong_docs seems to be built
from build/make/core/main.mk ; I assume that that can be worked around
although I haven't checked where the output of "m soong_docs" goes.
Test: Presubmits.
Change-Id: If5ba36490d9f3f60733e6d6be9286eb2b67c3ff5
It was confusing because bootstrapping uses two configurations: the
"global" config and the special-cased bootstrap one.
This change merges them.
Test: Presubmits.
Change-Id: I82b482cbe28a343ab6991374b2a28667e1a06b48
These are: TopFile and GeneratingPrimaryBuilder.
Also re-shuffle the list of flags to make a bit more sense and finish
the rename of BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively.
Test: Presubmits.
Change-Id: I103ff5f09f1c0d16f695a7da5dea13b55028e33e
Before, we piggybacked on the implementation of regular soong_build and
wrote a fake build.ninja file to satisfy Ninja.
Now, instead, the JSON module graph is a a separate action in the Ninja
output file. This has the pleasant side effect that one can flip back
and forth between generating the JSON file and regular Soong without
loss of incrementality.
Side cleanup: write .d files in a slightly cleaner way.
Test: Presubmits.
Change-Id: Ia853383567b9dd31c53f3bdf56cfc8d517b498ec
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: I999ca62162062f27e449f83ffb00fa580d4065b8
Merged-In: I2fcd80a9e376a6842a06aa104a9f6f5c5e16f8c7
Change-Id: Ib481d2aac40df8da1b74174dc8ec576c1cb48be0
