If we didn't need to run soong_build during the current run, we still
try to load the soong metrics. But in the case of `dist`, that's in a
directory that is not guaranteed to persist between runs.
Make loading the soong metrics optional if the file does not exist. Also
fixes a variable shadowing issue that meant we never passed it into
ctx.Metrics.
Test: treehugger
Change-Id: Ic836282f4d13e91daa0e7241ad7c488de3293d8b
Metrics are added to a materialized protobuf soong_build_metrics.pb
Recorded events have scoped names and recorded runtimes.
For example, mixedbuilds includes the following events:
mixedbuilds.prepare
mixedbuilds.analyze.write_files
mixedbuild (this is a total of all mixedbuild events)
Test: Manually verified materialized protos for mixedbuild, bp2build,
and legacy build.
Change-Id: I2e72d4502c0373bee0279bb1320b2e4dd586129c
Colin and Jin opined that we shouldn't change production in order to
make debugging easier. I originally thought it's better to have one
fewer knob but I don't have very strong feelings about it and their line
of reasoning makes total sense.
Test: Presubmits.
Change-Id: Id8a914aebbeffec6e8f860157c13f81093194f14
This behavior is enabled by setting the environment variable
SOONG_UNBUFFERED_OUTPUT=1 .
This is not the default behavior because only one process can be running
in the console pool and this would thus result in a loss of parallelism
if multiple soong_build invocations are needed.
Changing the value of this environment variable does not result in a
rebuild, since it does not affect the build outputs.
Test: Presubmits.
Change-Id: I49d15eb145966bd805f0a47b46866285203b122a
[ 0% 0/5 ] generating Soong docs at out/soong/docs/soong_build.html
0:03 analyzing Android.bp files and generating ninja file at out/soong/build.ninja
0:03 converting Android.bp files to BUILD files at out/soong/bp2build
0:03 generating the Soong module graph at out/soong/module-graph.json
0:03 generating the Soong module graph as a Bazel workspace at out/soong/queryview
0:03 generating Soong docs at out/soong/docs/soong_build.html
Test: m soong_docs json-module-graph bp2build queryview nothing
Change-Id: I33abb8828ead9ef999ffa1a6c9c4b016957c6dd1
The bpglob changes in build/blueprint submitted together with this
change make it necessary.
Test: Presubmits.
Change-Id: I3d19761ec1a199c2f77693a882f8744ce404c6de
It's better not to have state hidden in dotfiles (or rather,
dotdirectories) if we can help.
It's questionable whether the "linux-x86" path segment makes sense since
soong_build only ever runs on one operating system, but I didn't want to
rock the boat now.
Drive-by fixed some quoting fixes in rbc-run. Notably, I didn't wrap
`$@` into double quotes because I don't know whether the lack of double
quotes was intended or not.
Also drive-by fixed the fact that "out/soong" was added twice to the
directory name of bpglob. This turned out not to be a problem because
bpglob doesn't need to be explicitly built: if it's needed, it's
declared as an input of the glob files so it'll be built automatically
(at the cost of the first "null build" not actually being a null build)
Test: Presubmits.
Change-Id: I710d8d16cd8212059a0ca1ee95378505303eed83
ToolDir is going to become unstable when switching between KatiEnabled
and Soong-only builds while the duplication between out/soong/host and
out/host is resolved. bpglob gets executed very early during bootstrap,
before the primary builder has run to update the paths to match the
current configuration. Move it into SoongOutDir() so that its path
is more stable.
Moving bpglob causes incremental build issues, so add a bootstrapEpoch
constant that can be changed when making incompatible changes and will
wipe bootstrap intermediates.
Bug: 204136549
Test: m nothing
Change-Id: I7b1bd1ebfe1209d11db691b3ee00873ef92658cd
It was the same as SoongNinjaFile, except it used
shared.JoinPath instead of filepath.Join, which
doesn't matter here because the second argument
is never an absolute path.
Test: Presubmits
Change-Id: I95e64a8b8971bd168a1dc6670ae80f4d336045b5
- Remove useValidations: whenever it was used, it was turned on, so
there was no point in keeping that knob
- Use the new soong_build invocation name constants
Test: Presubmits.
Change-Id: Ia3e8928d30d3de1e0adaa741111771304e56b139
They were a bit repetitive.
In addition, make the environment checks more correct; they could
probably use an integration test or two to make sure that when the
environment changes, exactly those outputs are rebuilt that need to be,
but for now, this is an improvement already.
Test: Presubmits.
Change-Id: Idd79b81ca6975d57d00e5bf4699d266152505ff8
This makes soong_ui the only place where soong_build is invoked, thus
greatly simplifying the conceptual model of the build.
It comes with the slight limitation that now soong_docs (and queryview
and the JSON module graph) are not Make targets anymore, but I suppose
that's an acceptable loss.
The only place where someone depended on soong_docs from a Makefile is
removed in a separate change.
Test: Presubmits.
Change-Id: I3f9ac327725c15d84de725d05e3cdde1da3dcbe2
Its value is a function of the call site, so it doesn't make a lot of
sense to plumb it through the configuration.
Test: Presubmits.
Change-Id: If928b34de075969fd42932212ce9187808cbdf86
It used to be invoked from out/soong/build.ninja, which required two
soong_build invocations one after the other (ne to generate
out/soong/build.ninja, one to generate the queryview workspace). This
was slower and required some shell-quoted-in-ninja-quoted-in-Go .
Test: Presubmits.
Change-Id: Idda79c067606663b66e9f94626fa24f3b5af4114
This includes the JSON graph generator and bp2build.
Before:
GENERATE_BAZEL_FILES=1 m nothing
GENERATE_JSON_MODULE_GRAPH=1 m nothing
Now:
m json-module-graph
m bp2build
They can now also be combined with other targets or each other.
The longer-term goal is to run "m queryview" and "m soong_docs" using
the same infrastructure. There are two alternate approaches:
1. Call soong_build from within the main Ninja invocation. This requires
two sequential soong_build invocations and is thus slower.
2. Do everything requested in the same soong_build invocation. This
would be faster, but one AFAIU can't tell Ninja that multiple possible
actions can build the same output so that doesn't work.
(1) is somewhat more desirable because soong_docs seems to be built
from build/make/core/main.mk ; I assume that that can be worked around
although I haven't checked where the output of "m soong_docs" goes.
Test: Presubmits.
Change-Id: If5ba36490d9f3f60733e6d6be9286eb2b67c3ff5
It was confusing because bootstrapping uses two configurations: the
"global" config and the special-cased bootstrap one.
This change merges them.
Test: Presubmits.
Change-Id: I82b482cbe28a343ab6991374b2a28667e1a06b48
These are: TopFile and GeneratingPrimaryBuilder.
Also re-shuffle the list of flags to make a bit more sense and finish
the rename of BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively.
Test: Presubmits.
Change-Id: I103ff5f09f1c0d16f695a7da5dea13b55028e33e
Before, we piggybacked on the implementation of regular soong_build and
wrote a fake build.ninja file to satisfy Ninja.
Now, instead, the JSON module graph is a a separate action in the Ninja
output file. This has the pleasant side effect that one can flip back
and forth between generating the JSON file and regular Soong without
loss of incrementality.
Side cleanup: write .d files in a slightly cleaner way.
Test: Presubmits.
Change-Id: Ia853383567b9dd31c53f3bdf56cfc8d517b498ec
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: I999ca62162062f27e449f83ffb00fa580d4065b8
Merged-In: I2fcd80a9e376a6842a06aa104a9f6f5c5e16f8c7
Change-Id: Ib481d2aac40df8da1b74174dc8ec576c1cb48be0
The following files are forked:
- build-globs.ninja
- The glob list files
- .d files for the glob list files
- The output file (build.ninja or the bp2build marker file)
This makes bp2build and soong_build not overwrite each other's files
they need for proper incrementality.
Test: Presubmits.
Change-Id: I69d192cbd36ecd9677f46f3fa095dfce6f872227
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
This CL adds generateJsonModuleGraph bazelBuildMode and ensures that it
returns Soong early, before Kati and Soong metrics collection begin
(which causes errors, since they're looking for files that Soong in json
dump mode did not write).
Test: TH
Test: rm -rf out && SOONG_DUMP_JSON_MODULE_GRAPH=/tmp/soong.json m nothing
Change-Id: I264eadb3b3b0cd6b6e7a65adc7b39bb1c01ca136
1. Initialize an empty glob file for .bootstrap/build.ninja. Initializing
in soong_ui (and not soong_build) prevents inadvertently creating the
file inside the source tree
2. Remove soong-build-globs.ninja, which is not used during the build
Bug: 187194795
Test: Ran the following command locally for the target
art-target-arm:git_master-art
```
. ./build/envsetup.sh && lunch armv8-eng &&
art/tools/buildbot-build.sh --target
```
Change-Id: Ibe6eeff65ea1ab25136642299e9878d0da1cac42
I6dca478f356f56a8aee1e457d71439272351390b replaced calling
build/blueprint/bootstrap.bash with running minibp directly,
but didn't propagate the EMPTY_NINJA_FILE environment variable
to minibp. Since everything that uses EMPTY_NINJA_FILE is
executed directly from soong_ui now, replace the EMPTY_NINJA_FILE
environment variable with a config bool and pass it to minibp.
Bug: 189148777
Test: build/soong/build_test.bash --dist --incremental --shard-count=300 --shard=1 && du out/multiproduct
Change-Id: I4d64275ce02c5d68948012f71ac4dc3795af9e85
This CL fixes a typo in writeFakeNinjaFile to correctly write
a fake out/soong/build.ninja and its depfile. It also modifies bootstrap
phase to *not* run the main soong build phase (which takes more than a
minute) if GENERATE_BAZEL_FILES=1.
This change has the side effect that `GENERATE_BAZEL_FILES=1 m nothing`
no longer generates the real build.ninja, which is fine because one
shouldn't be using GENERATE_BAZEL_FILES=1 for that anyway (or, use
USE_BAZEL_ANALYSIS=1). This change has no effect on mixed builds.
Time on a change to Soong or any Android.bp files:
Before: bp2build_workspace_marker (~20 seconds) + build.ninja (1 min)
After: bp2build_workspace_marker (~20 seconds)
Time on the second of two consecutive `b build //bionic/...`: 2.070s
Test: TH
Test: Soong integration tests
Change-Id: I43720641815994caba97b8d165d7c3fc254cbd06
This refactors bazel-build mode determination logic in soong_ui so it's
clearer which of three possible modes are being used in a given
invocation (NO_BAZEL, GENERATE_BUILD_FILES, or MIXED_BUILDS).
Test: bootstrap tests
Change-Id: I41d2baebf8d560c2cc42db8daa8b936101d453e3
It now handles adding .bp files and changing globs.
In order to do this, depfiles are now written separately from RunBlueprint.
This is necessary due to the confluence of a number of seemingly
unrelated factors:
1. The glob filelist dependencies are discovered in globSingleton
2. Singletons need to be registered because otherwise singleton module
types panic
3. Singletons don't work because they require mutators bp2build does not
run
Due to (1), we would need to run the glob singleton. However, due to (2)
and (3), we can't run singletons and have to run Blueprint with
StopBeforeGeneratingBuildActions, which is when the build actions
writing glob files would be generated. So what happens is:
1. When bp2build is run, the glob singleton is disabled
2. At the end of bp2build, the list of glob files is artifically added
to the depfile of the workspace marker file
3. When build.ninja is generated, the Ninja file containing the glob
list file is written by the now-active glob singleton
Test: Presubmits.
Change-Id: I3c5898d8c57c554a93520276c64a952afc912dbe
This is done by setting the INTEGRATED_BP2BUILD environment variable
when invoking the build.
Even though the name of the marker file insinuates that a Bazel
workspace is already created, this is not the case yet.
An issue that remains is that a .d file is not written for the marker
file so it won't be rebuilt if a .bp file changes. Fixing this requires
delicate surgery because writing the .d file is the result of delicate
interplay between Soong and Blueprint.
There are also a number of semi-related fixes:
- The name of soong.environment.{used,available} is now on the command
line of soong_build (soong_docs is still special cased because its
command line in the Ninja file is taken from the os.Args of
soong_build so it's not trivial to remove the --{available,used}_env
from it
- bp2build writes a separate soong.environment.used file
- I had to call SetAllowMissingDependencies() separately when creating
the android.Context for bp2build so that bp2build runs in the
integration tests (it was not obvious how not to do this)
- Fixed a number of integration tests where a command with an expected
exit code of 1 was used as the last one in a test case, thereby
breaking the test suite
Test: Presubmits.
Change-Id: Ibeb61c26022cf801dcb98505b4039151b3409873
This is so that the way soong_build is invoked is simpler, which is in
turn useful so that it can be invoked multiple times within a single
build, which in turn will be used to invoke bp2build routinely (as
opposed to manually when needed)
Test: Presubmits.
Change-Id: Iddaebb05ff7bcedc0db8273192bb31284b739920
It's replaced with a flag in bootstrap.Args .
Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Ia0bfa2091013e389890e583c559006077806af99
This requires linking Blueprint into soong_ui. It lets us avoid the
complicated dance of Ninja files and shell scripts: now the information
as to how soong_build is built is passed directly to Blueprint using a
struct that contains all the information the command line arguments used
to contain.
The ability to run Blueprint from the command line is kept (for now).
Some variables in bootstrap/command.go needed public accessor functions
because soong_build reads them. This will be disentangled by moving the
flag parsing to soong_build.
The presence of the flag definitions in Blueprint means that soong_ui
now also accepts them. This is not a problem in practice because they
are ignored and because soong_ui itself is hidden behind a few layers of
shell scripts.
Test: Presubmits + the new bootstrap_test.sh .
Change-Id: I6dca478f356f56a8aee1e457d71439272351390b
This lets us not leak $PATH to soong_build. It was only needed for
Darwin tests anyway.
Test: Will ask jingwen@ who has a Mac.
Change-Id: I4647e41275b323fe6283580f8f92718c6229f23e
- Incorporate BAZEL_* variables in soong.environment.available
- Set SOONG_DELVE when requested
Test: Manual.
Change-Id: I3ac675f4876220b788cc48d288b402ac6016c37b
This is exclusively used by tests so it's awful, but Darwin builds are
currently broken so let's unbreak them first then start thinking.
Test: "m nothing" on Linux (I don't have access to a mac OS machine)
Change-Id: I1ea9ca71a5aff36a3454282a4c87ad368138baaa
This sandboxes Soong's tests better.
Setting "TOP" is slightly ugly, but doing it properly would conflict
with aosp/1609218.
Test: Manual ("m nothing")
Change-Id: Ie3d3cd17f116a5fe0ad25c670fc5ed9c33f8123b
This works by setting the SOONG_UI_DELVE= environment variable to the
port on which soong_ui should accept a Delve connection on.
This is achieved by reusing the Delve execution logic between soong_ui
and soong_build.
Test: Manual.
Change-Id: Id2c1d4b6faac1a4a3918c91030ce2239f7daf54f
This lets one avoid any decisions as to when to chdir there during its
execution and leads to better sandboxing because the pwd doesn't leak to
init() functions anymore.
Test: Manual.
Change-Id: I1560da8ed3a621249426f9e8908aa890c21e13ba
Also delete the now-unnecessary soong_env binary.
Test: Manually. Also checked what happens when a used environment
variable changes.
Change-Id: Ib393e7f444e94819198c6cce4bcd8719d9fd9441
bp2build is a Soong mode that returns as soon as the BUILD files are generated. This causes it not generate the build.ninja file, which the Ninja process executing soong_build itself expects to produce as an output. If there isn't an existing build.ninja file generated from a previous build, GENERATE_BAZEL_FILES=true m nothing will fail.
This CL generates the expected files as a workaround, and also makes GENERATE_BAZEL_FILES=true skip the Kati/Ninja steps in soong_ui since they aren't needed.
Test: rm -rf out/ && GENERATE_BAZEL_FILES=true m nothing && m libc
Fixes: 178683777
Change-Id: I2515ef7961682d2be5f096ed24831cc185165a67
