Commit Graph

1067 Commits

Author SHA1 Message Date
Taylor Santiago
60fa202bb5 Merge "Use consistent directory for source and output inside the build sandbox for ABFS." into main 2024-07-31 17:55:41 +00:00
Taylor Santiago
ca30e08b9d Use consistent directory for source and output inside the build sandbox
for ABFS.

A consistent source directory addresses problematic full path inputs and
outputs, which can manifest as cache misses or build failures after an ABFS
patch application.

Bug: 347704053
Change-Id: Ib2297bb0b904b1b490e83c22fd468eee928e53fe
2024-07-30 13:52:31 -07:00
Yaowen Mei
d4da266b35 Fix RBE tmp dir
The RBE tmp dir holds all the re-client logs and socket files.

Currently, it is being created as the `out/soong/.temp/` dir, which is auto deleted by Soong. This is causing problems because while a build is running, user cannot open a second terminal and run `lunch`.

This CL fixes the issue by creating the RBE tmp dir in the `out/soong/rbe/` directory. The next RBE build will clean this folder.


Bug: 349664018
Change-Id: Ib238e48f5b19755b016263b45f02b1be3e8efb35
2024-07-30 14:21:04 +00:00
Christopher Ferris
cdd872a5be Merge "Update tests to expect RBE not supported on linux." into main 2024-07-26 17:58:26 +00:00
Wei Li
9f13555225 Merge "Generate SBOM of products in Soong." into main 2024-07-26 00:45:43 +00:00
Christopher Ferris
1db36e3888 Update tests to expect RBE not supported on linux.
Bug: 354280127

Test: Passes on linux
Test: Treehugger for the rest.
Change-Id: Ic1546a153e7e761a2132409743650cbf0a0b1ba6
2024-07-25 16:52:31 -07:00
Taylor Santiago
dd65d3e672 Merge "Add ABFS (Android Build Filesystem) configuration option to Soong" into main 2024-07-25 21:04:06 +00:00
Wei Li
b85a178b07 Generate SBOM of products in Soong.
Bug: 324465531
Test: CIs
Test: m soong-sbom
Change-Id: If76776851d49282829a79bfb1c33f05b8f57de31
2024-07-24 18:23:13 -07:00
Treehugger Robot
c7eb3bdae7 Merge "Highlight build failures in soong output" into main 2024-07-18 20:13:15 +00:00
Colin Cross
ea1b3b78b8 Merge changes from topic "elide_empty_variants" into main
* changes:
  Convert sdk mutator to TransitionMutator
  Convert link mutator to TransitionMutator
  Convert version mutator to TransitionMutator
  Convert python_version mutator to TransitionMutator
  Convert rust_libraries and rust_stdlinkage mutators to TransitionMutators
2024-07-18 17:37:38 +00:00
Treehugger Robot
865b877027 Merge "Generate .build-id directory tree after every build." into main 2024-07-17 23:13:24 +00:00
Colin Cross
8a49a3dd76 Convert rust_libraries and rust_stdlinkage mutators to TransitionMutators
Replace rust.LibraryMutator and rust.LibstdMutator with
TransitionMutators.

Bug: 319288033
Flag: EXEMPT refactor
Test: all soong tests pass
Test: no change to build.ninja
Change-Id: Ia24a582119d39889279d7b93bac9259685153619
2024-07-17 15:50:36 -07:00
Taylor Santiago
3c16e61c57 Add ABFS (Android Build Filesystem) configuration option to Soong
Change-Id: Ibbe81004adc67b1212d7e619a35f7425bfee0d72
2024-07-17 14:50:25 -07:00
cherokeeMeta
f7119b57ac Highlight build failures in soong output
Summary: When build failures occur, it can be hard to find where the error message begins. We now highlight "FAILURE" in red to make it easier to see what exactly has failed in the output.

Test: execute automated tests with m nothing --no-skip-soong-tests, can also manually test with a build failure and observe red text in the output

Change-Id: Iad3b94cc1c385f0afdebdf12c44843db04ff2bdc
Signed-off-by: Cherokee Toole <cherokee@meta.com>
2024-07-16 18:18:33 +00:00
Cole Faust
bee030d36b Allow n2 as a replacement for ninja in builds
If `SOONG_USE_N2=true` is set in the environment, then n2 is used in
place of ninja.  Some ninja features are not available in n2 at this
time, but this enables bringup efforts to happen in parallel.

Bug: 352368206
Test: manual
Change-Id: I8455cb24eb640a4651782ee76e48a7d3a9932b93
2024-07-10 12:35:32 -07:00
Nathan Egge
978c934dd9 Add LEX to the list of env variables to unset.
Some linux environments export LEX and this variable must be unset to
 ensure Soong and Make LEX values match.

Change-Id: I8ff2b62a49e0108575ff88b1a88a83d650fc3768
2024-07-08 20:53:46 +00:00
Spandan Das
dd274aba5f Merge "Revert^2 "Enforce that output files are created in primary ninja execution"" into main 2024-07-01 23:28:56 +00:00
Spandan Das
28a6f19866 Revert^2 "Enforce that output files are created in primary ninja execution"
This reverts commit fddc3dc8d3.

Reason for revert: https://r.android.com/3148741 makes this compatible with art gtests

Change-Id: Ia85c60fdcfa3142fc464a35476170c88c172989c
2024-07-01 21:00:25 +00:00
Treehugger Robot
38a97b2c47 Merge "Revert "Enforce that output files are created in primary ninja execution"" into main 2024-06-26 20:25:07 +00:00
Santiago Aboy Solanes
fddc3dc8d3 Revert "Enforce that output files are created in primary ninja execution"
This reverts commit 8d8253f525.

Reason for revert: Breaks ART gtests e.g. https://android-build.corp.google.com/build_explorer/build_details/12016159/art-gtest/

Change-Id: Ie8e1a28c7d0a21ee6727954127a6ab00ed00e781
2024-06-26 14:52:52 +00:00
Treehugger Robot
e8d5c8a5b2 Merge "Enforce that output files are created in primary ninja execution" into main 2024-06-26 02:18:55 +00:00
Treehugger Robot
51fc67bd20 Merge "Cleanup more bazel code" into main 2024-06-25 19:24:12 +00:00
Cole Faust
487b144fe0 Cleanup more bazel code
Bug: 315353489
Test: m nothing --no-skip-soong-tests
Change-Id: I6c6b4fcd559f4a45b1a96c70cbb5d6d1a615ccff
2024-06-25 11:02:38 -07:00
Joe Onorato
c870a75fe1 Merge "RBE is only supported when the host is linux, so don't try on others." into main 2024-06-25 17:58:35 +00:00
Joe Onorato
86f50e752e RBE is only supported when the host is linux, so don't try on others.
Bug: 343589123
Test: m and verify
Change-Id: I872d36a40c980f58c5de2598c3fb5dcfcadabf4f
2024-06-24 14:28:25 -07:00
Spandan Das
8d8253f525 Enforce that output files are created in primary ninja execution
Missing output files in ninja execution will be treated as errors.
Products can bypass this using the newly introduced
BUILD_BROKEN_MISSING_OUTPUTS flag.

Test: m nothing # verified missingoutfile appears in out/soong.log
Test: checkbuild passes on presubmits

Change-Id: I4eebcd08aa57fc6ccf1688c32e0d1fe06d66ab9a
2024-06-24 19:26:45 +00:00
Yu Liu
88f4e331cc Merge "Experimental code to support build action caching." into main 2024-06-19 01:57:12 +00:00
Yu Liu
fa29764f9f Experimental code to support build action caching.
Bug: 335718784
Test: build locally
Change-Id: Icc1f1fb15f9fe305e95dd51e2e7aff1e9cbf340c
2024-06-19 00:09:32 +00:00
Inseob Kim
58c802f8e3 Add product_config module
product_config module will export productVariables struct (defined in
android/variable.go). Other modules can depend on product_Config module
to read product variables, without needing to update Soong.

Bug: 346214955
Test: build and see output of product_config module
Change-Id: I06e1d2716dc2a8b90984bb46c5fc901cd1ca5020
2024-06-18 09:56:45 +09:00
Treehugger Robot
1571ad9605 Merge "Add following directories to Android.mk denylist to block new Android.mk files." into main 2024-06-13 23:43:25 +00:00
Cole Faust
1ae7b774eb Remove timestamp from local build number
The build_info.prop file was converted to soong, which then caused it to
have a stale build number due to not adding the build number file as a
dependency to avoid rebuilds. I added it as a dependency to fix the
staleness, but it always rebuilds locally now.

I think the way it worked before, it would get stale build numbers,
except that it was cleared with installclean which was always run on CI.
Now that it's a soong module and generated in out/soong/.intermediates
it's not cleared by installclean. I could make a system to register
files that should be installclean'd from a soong module, but I'd
eventually like to eliminate the need to run installclean entirely.

So as an alternative, just make the build number not change every build
when doing local builds, by removing the timestamp from it.

Bug: 346757289
Test: m repeatedly and observe no rebuilds the second time (of the system image, the build flags infrastructure seems to do some small rebuilding)
Change-Id: I0207feb739523dde3e89d1e5c4822865f641c313
2024-06-13 10:27:20 -07:00
Wei Li
8778d26c9d Add following directories to Android.mk denylist to block new Android.mk files.
bionic/
development/
device/sample/
prebuilts/
sdk/
test/
trusty/

Bug: 318428689
Change-Id: I327bf354df064a5249edc3e9c846199cde0ce00b
Test: CIs
2024-06-12 21:13:24 +00:00
Spandan Das
02f1e342cd Merge "Drop PRODUCT_INCLUDE_TAGS from banner vars" into main 2024-06-07 16:01:21 +00:00
Treehugger Robot
620b0c46d9 Merge "Regard out/soong/release-config as a source directory" into main 2024-06-07 05:12:31 +00:00
Spandan Das
f4c802bcd6 Drop PRODUCT_INCLUDE_TAGS from banner vars
PRODUCT_INCLUDE_TAGS was previously used alongside
blueprint_package_include to prune Android.bp files from soong analysis.
With the deprecation of blueprint_package_includes, this product config
variable bcomes a noop and does not provide useful information in
banner.

Test: NA
Bug: 333316620
Change-Id: Id7a1465c06a45f12b4ecb1bf80decddb657adce2
2024-06-07 00:03:20 +00:00
Justin Yun
605b5da192 Regard out/soong/release-config as a source directory
Files in out/soong/release-config are generated from lunch or kati
configuration phase. They don't have rules in ninja, but always
provided as source files.

Bug: 324996303
Test: m checkbuild
Change-Id: Ibfb76a8c58a81a4d768b29d80151fa47e2ebb9c1
2024-06-04 11:28:31 +09:00
Spandan Das
aa9369b71d Drop IncludeTags
This was previously used in conjunction with blueprint_package_includes
to prune Android.bp files from soong analysis.

Test: m nothing
Bug: 308188212
Change-Id: Ie82e20eec63bd0be70e1cf0290c70f62d5621c76
2024-05-22 19:23:47 +00:00
Yu Liu
1b2ddc8093 Write ninja file in parallel.
Bug: 335718784
Test: CI
Change-Id: Ie36d78478a60359b704767689846e76b1c2ba76d
2024-05-20 20:28:44 +00:00
Kousik Kumar
074e4cc9ad Remove warning about LOAS not enabled
RBE is now the default for all Google internal builds, so this warning
is no longer necessary.

Bug: n/a
Change-Id: I2da3e9b84c20f32b3e3f90f49a974512248db2ec
2024-05-03 13:50:10 +00:00
Nelson Li
ad3dae8780 Update Android.mk denylist to block frameworks/
All Android.mk files under `frameworks/` have been converted to
Android.bp. Add it to the block list to avoid using any newly added
Android.mk.

Bug: 322732227
Test: CIs
Change-Id: I1980b2f94704c15d60579b21915e51e0f4a35c1e
2024-04-03 15:17:49 +00:00
Elliott Hughes
34c93d8e15 Merge "We have a prebuilt getopt now." into main 2024-03-22 17:33:47 +00:00
Elliott Hughes
4e0ccf4f59 We have a prebuilt getopt now.
getopt doesn't appear to be used any more, but we've added the prebuilt anyway. Update soong's $PATH allowlist correspondingly.

Change-Id: Id416703697f464f4a3b73771611e087d2ec265d1
2024-03-22 16:08:21 +00:00
Peter Collingbourne
b805c61755 Generate .build-id directory tree after every build.
The .build-id directory tree is as described here:
https://fedoraproject.org/wiki/RolandMcGrath/BuildID

Tools such as llvm-symbolizer understand this tree and can use it
to look up symbols by build ID. This CL adds a post-build step that
updates the .build-id directory under symbols after every build if
necessary to contain symlinks to the corresponding symbols file.

I also considered adding this as a build step after copying the symbol
file to the symbols directory. However this would be complicated
by the fact that many binaries have the same build-id such as the
various copies of libc++ for the different APEXes. As a result it
would be difficult to implement a .build-id updater that produces a
correct symlink tree while taking into account concurrent updates to
the same symlink by different build steps. This situation is resolved
in the .build-id directory generator by having the symlink point to
the lexically smallest path out of all paths with the same build-id.

I measured the performance of the UpdateBuildIdDir function by writing
a standalone program that just calls the function and running it over
a copy of the symbols directory for a specific target. On my machine
the execution time of the program when simulating various scenarios
was as follows:

Null build [1]:             36.2 ms ±   3.4 ms
Initial build [2]:         162.3 ms ±   6.6 ms
Single update [3]:         128.0 ms ±   6.1 ms
Invalid .build-id dir [4]: 222.6 ms ±   8.6 ms

This is with some improvements that have been contributed to the Go
standard library [5,6]; without those improvements a null build is
37.9 ms ± 4.4 ms and a single update is 143.9 ms ± 4.5 ms.

[1] hyperfine './update-build-id ~/2/test-symbols2/'
[2] hyperfine -p 'rm -rf ~/2/test-symbols2/.build-id' './update-build-id ~/2/test-symbols2/'
[3] hyperfine -p 'dd if=/dev/urandom of=$HOME/2/test-symbols2/system/bin/init conv=notrunc seek=808 bs=1 count=16' './update-build-id ~/2/test-symbols2/'
[4] hyperfine -p 'touch ~/2/test-symbols2/.build-id/corrupt; sleep 0.1; touch ~/2/test-symbols2/system/bin/init' './update-build-id ~/2/test-symbols2/'
[5] https://go.dev/cl/570877
[6] https://go.dev/cl/571436

Bug: 328702178
Change-Id: I8fc0ea81bd31ec80d6b912ba477e2e24b6b05f68
2024-03-19 21:25:01 -07:00
Yu Liu
ee37914529 Merge "Store the changed_inputs field in build.trace." into main 2024-03-13 15:54:13 +00:00
Andus Yu
2b8928444d Merge "Make sure RBE socket address is under character limit" into main 2024-03-12 13:45:51 +00:00
Sorin Basca
7f4fa82732 Merge "Revert^2 "Switch to JDK 21"" into main 2024-03-11 18:40:04 +00:00
Sorin Basca
5dfa238f22 Revert^2 "Switch to JDK 21"
This reverts commit b832ae19d8.

Reason for revert: Reland Switch to JDK 21 when related issues are resolved.

Bug: 313924276
Change-Id: Ibb5031d9892d1ddc5bc4d2af9885c3192b8ac8ad
Test: TH
2024-03-11 17:23:06 +00:00
Sam Saccone
bfa20d7e15 Merge "Revert "Switch to JDK 21"" into main 2024-03-11 17:05:56 +00:00
Jihoon Kang
48872f5443 Merge "Fix missing credential error message" into main 2024-03-08 06:34:50 +00:00
Jihoon Kang
70de2c04a7 Fix missing credential error message
LOAS credential enforcement is already taking effect, not in the future.
Update the warning message accordingly.

Test: gcertdestroy && m and inspect error message
Bug: 318506452
Change-Id: I857f44b265c454beaffb9e662bd318aeb29f6cdb
2024-03-08 00:11:20 +00:00