Commit Graph

298 Commits

Author SHA1 Message Date
Cole Faust
2fec4128e0 Make globs compatible with hash-based ninja semantics
Previously, globs worked by having soong_build rewrite a ninja file
that ran the globs, and then dependended on the results of that ninja
file. soong_build also pre-filled their outputs so that it wouldn't
be immediately rerun on the 2nd build.

However, the pre-filling of outputs worked for ninja, because although
it updated their timestamps, the soong ninja file was then touched
by soong_build after that, so the soong_build ninja file was newer
and ninja wouldn't rerun soong. But N2 reruns actions if their inputs'
mtimes change in any way, not just if they're newer. Similarly,
hashed-based ninja implementations could not enforce an order on
file contents, so they would have the same problem.

To fix this, lift the glob checking out of ninja and into soong_ui.
Soong_build will output a globs report file every time it's run, and
every time soong_ui is run it will check the globs file, and if any
globs change, update an input to soong_build. soong_ui is essentially
doing what was done in ninja with bpglob actions before.

Bug: 364749114
Test: m nothing, m nothing again doesn't reanalyze, create a new file under a glob directory, m nothing again reanalyzes
Change-Id: I0dbc5ec58c89b869b59cd0602b82215c4972d799
2024-09-09 17:42:49 -07:00
Taylor Santiago
c162dfac78 Undo filtering of HOME environment variable.
It an cause 'nonexistent' user errors in the sandbox.

Bug: 363037195
Change-Id: I5e17fa62029ba9bcd2422fa9aebef95223c68c0d
2024-09-05 19:43:27 +00:00
Taylor Santiago
8b0bed7187 Reapply "Clean environment variables to account for sandbox work directory."
with minor edits.

This reverts commit 9543d19b4b.

Bug: 363037195
Change-Id: I4cb10b312b7f468185bfad12a16f9b4b64e7a58a
2024-09-04 22:55:02 +00:00
PODISHETTY KUMAR (xWF)
cdd70d268a Merge "Revert "Clean environment variables to account for sandbox work directory."" into main 2024-09-02 06:57:34 +00:00
PODISHETTY KUMAR (xWF)
9543d19b4b Revert "Clean environment variables to account for sandbox work directory."
This reverts commit 0af8ea14fc.

Reason for revert: <Droidmonitor created revert due to b/363848580. Will be verifying through ABTD before submission.>

Change-Id: Iafba28897ad27df67ef8ae8904454196c482216d
2024-09-02 03:54:36 +00:00
Taylor Santiago
3cd732bcd1 Merge "Clean environment variables to account for sandbox work directory." into main 2024-08-31 04:04:22 +00:00
Taylor Santiago
0af8ea14fc Clean environment variables to account for sandbox work directory.
Unset HOME to prevent username leak.

Bug: 363037195

Change-Id: I7a8694f746d58de8f6e41adb6ad52abbaf955ef4
2024-08-29 20:12:35 +00:00
Qing Shen
713c54289c Revert "Revert "Add a coverage suffix to avoid Ninja file regene..."
Revert submission 3235973-revert-3207217-emma-flip-time-saver-BYWASVIMTD

Reason for revert: Rolling forward with a fix

Adding a coverage suffix for product_config.go too:

FAILED: ninja: 'out/soong/soong.sdv_core_cf.extra.variables', needed by 'out/soong/.intermediates/build/soong/product_config/product_config.json', missing and no known rule to make it
03:42:23 ninja failed with: exit status 1

Reverted changes: /q/submissionid:3235973-revert-3207217-emma-flip-time-saver-BYWASVIMTD

Change-Id: I74d0e3fda09b1367f3dbb939d4f47c4d09b3d51f
2024-08-23 21:41:52 +00:00
Cole Faust
4e58bba1fc Use musl build of n2
We're seeing bugs on the build server only, and previous similar bugs
were caused by issues in the older version of glibc the build server
uses, so try using the musl build of n2 to see if it resolves the
issues.

Bug: 318434287
Test: Presubmits
Change-Id: I416cd77f4dfb4ca04f6dd5b36fa73585a2904b55
2024-08-22 14:27:03 -07:00
Hugo Drumond Jacob
079871cd55 Merge "Revert "Add a coverage suffix to avoid Ninja file regeneration."" into main 2024-08-21 17:20:34 +00:00
Hugo Drumond Jacob
134fa5df00 Revert "Add a coverage suffix to avoid Ninja file regeneration."
Revert submission 3207217-emma-flip-time-saver

Reason for revert: b/361236607

Reverted changes: /q/submissionid:3207217-emma-flip-time-saver

Change-Id: I7b4dcd3f6e73e302a2c3ef7ed5c02f6345619463
2024-08-21 13:07:06 +00:00
Qing Shen
ff545caecd Merge "Add a coverage suffix to avoid Ninja file regeneration." into main 2024-08-20 01:16:53 +00:00
Qing Shen
9e05d1c5cd Add a coverage suffix to avoid Ninja file regeneration.
This CL uses the environment variable, EMMA_INSTRUMENT to determine whether or not coverage is enabled. If coverage is enabled, it adds a bunch of suffixes to the files generated in out/soong/ folders.

This change reduces the build time by avoiding ninja regeneration when users simply switches from atest to atest --experimental-coverage, or vice versa, and no Android.bp changes are made.

Bug: 331444846
Test: Locally run m libc, and then EMMA_INSTRUMENT m libc twice, no ninja regenerations required. Compared the hash of out/target/product/vsoc_x86_64/ between the before-change repo and after-change repo. Only diff is the after-change directory contains an extra file `out/target/product/vsoc_x86_64/.installable_files` while the original directory has only `out/target/product/vsoc_x86_64/.installable_files.previous`, both files have the same hash.
Change-Id: I972aec20a9d97b223cc833ee245016ac4b0f09a2
2024-08-13 02:04:53 +00:00
Yaowen Mei
d4da266b35 Fix RBE tmp dir
The RBE tmp dir holds all the re-client logs and socket files.

Currently, it is being created as the `out/soong/.temp/` dir, which is auto deleted by Soong. This is causing problems because while a build is running, user cannot open a second terminal and run `lunch`.

This CL fixes the issue by creating the RBE tmp dir in the `out/soong/rbe/` directory. The next RBE build will clean this folder.


Bug: 349664018
Change-Id: Ib238e48f5b19755b016263b45f02b1be3e8efb35
2024-07-30 14:21:04 +00:00
Taylor Santiago
3c16e61c57 Add ABFS (Android Build Filesystem) configuration option to Soong
Change-Id: Ibbe81004adc67b1212d7e619a35f7425bfee0d72
2024-07-17 14:50:25 -07:00
Cole Faust
bee030d36b Allow n2 as a replacement for ninja in builds
If `SOONG_USE_N2=true` is set in the environment, then n2 is used in
place of ninja.  Some ninja features are not available in n2 at this
time, but this enables bringup efforts to happen in parallel.

Bug: 352368206
Test: manual
Change-Id: I8455cb24eb640a4651782ee76e48a7d3a9932b93
2024-07-10 12:35:32 -07:00
Nathan Egge
978c934dd9 Add LEX to the list of env variables to unset.
Some linux environments export LEX and this variable must be unset to
 ensure Soong and Make LEX values match.

Change-Id: I8ff2b62a49e0108575ff88b1a88a83d650fc3768
2024-07-08 20:53:46 +00:00
Spandan Das
dd274aba5f Merge "Revert^2 "Enforce that output files are created in primary ninja execution"" into main 2024-07-01 23:28:56 +00:00
Spandan Das
28a6f19866 Revert^2 "Enforce that output files are created in primary ninja execution"
This reverts commit fddc3dc8d3.

Reason for revert: https://r.android.com/3148741 makes this compatible with art gtests

Change-Id: Ia85c60fdcfa3142fc464a35476170c88c172989c
2024-07-01 21:00:25 +00:00
Treehugger Robot
38a97b2c47 Merge "Revert "Enforce that output files are created in primary ninja execution"" into main 2024-06-26 20:25:07 +00:00
Santiago Aboy Solanes
fddc3dc8d3 Revert "Enforce that output files are created in primary ninja execution"
This reverts commit 8d8253f525.

Reason for revert: Breaks ART gtests e.g. https://android-build.corp.google.com/build_explorer/build_details/12016159/art-gtest/

Change-Id: Ie8e1a28c7d0a21ee6727954127a6ab00ed00e781
2024-06-26 14:52:52 +00:00
Treehugger Robot
e8d5c8a5b2 Merge "Enforce that output files are created in primary ninja execution" into main 2024-06-26 02:18:55 +00:00
Joe Onorato
c870a75fe1 Merge "RBE is only supported when the host is linux, so don't try on others." into main 2024-06-25 17:58:35 +00:00
Joe Onorato
86f50e752e RBE is only supported when the host is linux, so don't try on others.
Bug: 343589123
Test: m and verify
Change-Id: I872d36a40c980f58c5de2598c3fb5dcfcadabf4f
2024-06-24 14:28:25 -07:00
Spandan Das
8d8253f525 Enforce that output files are created in primary ninja execution
Missing output files in ninja execution will be treated as errors.
Products can bypass this using the newly introduced
BUILD_BROKEN_MISSING_OUTPUTS flag.

Test: m nothing # verified missingoutfile appears in out/soong.log
Test: checkbuild passes on presubmits

Change-Id: I4eebcd08aa57fc6ccf1688c32e0d1fe06d66ab9a
2024-06-24 19:26:45 +00:00
Yu Liu
88f4e331cc Merge "Experimental code to support build action caching." into main 2024-06-19 01:57:12 +00:00
Yu Liu
fa29764f9f Experimental code to support build action caching.
Bug: 335718784
Test: build locally
Change-Id: Icc1f1fb15f9fe305e95dd51e2e7aff1e9cbf340c
2024-06-19 00:09:32 +00:00
Inseob Kim
58c802f8e3 Add product_config module
product_config module will export productVariables struct (defined in
android/variable.go). Other modules can depend on product_Config module
to read product variables, without needing to update Soong.

Bug: 346214955
Test: build and see output of product_config module
Change-Id: I06e1d2716dc2a8b90984bb46c5fc901cd1ca5020
2024-06-18 09:56:45 +09:00
Spandan Das
aa9369b71d Drop IncludeTags
This was previously used in conjunction with blueprint_package_includes
to prune Android.bp files from soong analysis.

Test: m nothing
Bug: 308188212
Change-Id: Ie82e20eec63bd0be70e1cf0290c70f62d5621c76
2024-05-22 19:23:47 +00:00
Andus Yu
2b8928444d Merge "Make sure RBE socket address is under character limit" into main 2024-03-12 13:45:51 +00:00
Sorin Basca
7f4fa82732 Merge "Revert^2 "Switch to JDK 21"" into main 2024-03-11 18:40:04 +00:00
Sorin Basca
5dfa238f22 Revert^2 "Switch to JDK 21"
This reverts commit b832ae19d8.

Reason for revert: Reland Switch to JDK 21 when related issues are resolved.

Bug: 313924276
Change-Id: Ibb5031d9892d1ddc5bc4d2af9885c3192b8ac8ad
Test: TH
2024-03-11 17:23:06 +00:00
Sam Saccone
bfa20d7e15 Merge "Revert "Switch to JDK 21"" into main 2024-03-11 17:05:56 +00:00
Sorin Basca
b832ae19d8 Revert "Switch to JDK 21"
This reverts commit 5a2e36b755.

Reason for revert: Preparing revert to have it ready in case of breakages relating to using JDK 21.

Change-Id: I153c4933f6891c7fff43e522a4d128acad6b6d47
2024-03-07 21:22:00 +00:00
Treehugger Robot
092c3a56fb Merge "Switch to JDK 21" into main 2024-03-07 19:36:28 +00:00
Andus Yu
c917eb8207 Make sure RBE socket address is under character limit
Check that socket address paths are less than the character limit - 2 since absolute paths are prefixed with //, which is included in the character limit.

b/308986536

Change-Id: I3055b8bb8cdae00aafd66c51fdc8421b9cc5f21f
2024-03-06 21:54:57 +00:00
Sorin Basca
5a2e36b755 Switch to JDK 21
Bug: 313924276
Test: n
Change-Id: Ia9682181a6ce35ed0102d60bf7df1def8f876fd0
2024-02-29 11:44:09 +00:00
Cole Faust
331b27f511 Remove --multitree-build
Multitree has been discontinued.

Test: m nothing --no-skip-soong-tests
Change-Id: Ie4d96a11279bf0f20a5e1a49837d9df4b24b1662
2024-02-28 15:35:49 -08:00
Joe Onorato
e5ed34746e Add new soongdbg command and a big json file full of soong debugging info.
In order to use soongdbg, you must run analysis with GENERATE_SOONG_DEBUG=true set
in the environment.

Test: GENERATE_SOONG_DEBUG=true m nothing ; soongdbg ...
Change-Id: If43676fe2784f05cd87c0ecb4a46ab676b91023f
2024-02-03 14:44:11 -08:00
Colin Cross
8d411ff8f8 Remove bp2build and bazel from soong_ui
Bug: 315353489
Test: m blueprint_tests
Change-Id: I9df31b18caaae24e3cf2994e56bb90b50523f11e
2023-12-07 17:14:27 -08:00
Sorin Basca
0760c89882 Support experimental building with OpenJDK 21
Use EXPERIMENTAL_USE_OPENJDK21_TOOLCHAIN=true to build with OpenJDK 21
while still targeting java language version 17.

Bug: 313924276
Test: m EXPERIMENTAL_USE_OPENJDK21_TOOLCHAIN=true
Change-Id: Idc892bb7519e597f1e280ca0765c1a281bb29955
2023-11-29 19:13:55 +00:00
Jihoon Kang
2a929ad250 Revert^2 "Default from-text stub generation in build"
This change defaults Java stubs to be generated from API text files
during build. Using the `--build-from-source-stubs` flag, users can
toggle between the feature.

This change relands aosp/2617457.

Test: m nothing && verify ninja path exists between android_stubs_current and android_stubs_current.from-text, and does not exist between android_stubs_current.from-source, m nothing --build-from-source-stub && verify the opposite
Bug: 301522358
Change-Id: Ic129dafc3231e71470af982374739930947b69cb
2023-11-08 04:25:52 +00:00
LaMont Jones
9a9128610b Readd support for PRODUCT_RELEASE_CONFIG_MAPS
This allows product config to determine what build flags are present in
a product's build, as well as their value.

As product config moves to a declarative specification, this can be
handled with less unnecessary work.

Bug: b/302593603, b/308849337, b/309477343
Test: manual
Change-Id: Ic0551990864bba6d03b6bf49fcfd4ae2523d9c6d
2023-11-07 18:39:40 +00:00
Colin Cross
216ed6c918 Merge "Revert "Add support for PRODUCT_RELEASE_CONFIG_MAPS"" into main 2023-11-06 22:02:50 +00:00
Colin Cross
6a4fe68cc5 Revert "Add support for PRODUCT_RELEASE_CONFIG_MAPS"
Revert submission 2815850-product-flags

Reason for revert: b/309477343

Reverted changes: /q/submissionid:2815850-product-flags

Bug: 302593603
Bug: 309477343
Change-Id: I5b5a5ade1d85195c43377ed7639deba6eef86e23
2023-11-06 21:32:03 +00:00
LaMont Jones
4c01bb49da Merge "Add support for PRODUCT_RELEASE_CONFIG_MAPS" into main 2023-11-06 14:08:22 +00:00
LaMont Jones
127127b608 Add support for PRODUCT_RELEASE_CONFIG_MAPS
This allows product config to determine what build flags are present in
a product's build, as well as their value.

As product config moves to a declarative specification, this can be
handled with less unnecessary work.

Bug: b/302593603
Bug: b/308849337
Test: manual
Change-Id: If57622059bb7d1c33df7498321db621c0c2d30c0
2023-11-03 19:19:59 +00:00
Colin Cross
f2c51919ad Merge "Remove obsolete java version environment variables" into main 2023-11-02 17:05:06 +00:00
Treehugger Robot
87706098b0 Merge "Revert "Add support for PRODUCT_RELEASE_CONFIG_MAPS"" into main 2023-11-02 08:06:39 +00:00
Colin Cross
f77b2cb534 Revert "Add support for PRODUCT_RELEASE_CONFIG_MAPS"
Revert submission 2787001-product-build-flags

Reason for revert: Possible cause of b/308849337

Reverted changes: /q/submissionid:2787001-product-build-flags

Bug: 302593603
Bug: 308849337
Change-Id: Id3f27eca8d39e854dee232f072a8dc6c561bac63
2023-11-02 00:13:30 +00:00