844 Commits

Author SHA1 Message Date
Colin Cross
b63d7b3af7 Remove infrastructure to run bp2build
Bug: 315353489
Test: m blueprint_tests
Change-Id: Idcf6377d389b94c39e4e6ff4b8efa8a9f9e78b17
2023-12-08 13:51:07 -08:00
Colin Cross
e51c6e4109 Merge "Remove bp2build and bazel from soong_ui" into main 2023-12-08 21:43:26 +00:00
Cole Faust
14a2362183 Reapply "Handle symlinks in sbox"
This reverts commit 69b5a5f0bf.

Bug: 307824623
Test: Presubmits
Change-Id: Id7dd1518ae6b04232c48c0aa138cad2c59b228c3
2023-12-08 12:43:00 -08:00
Colin Cross
8d411ff8f8 Remove bp2build and bazel from soong_ui
Bug: 315353489
Test: m blueprint_tests
Change-Id: I9df31b18caaae24e3cf2994e56bb90b50523f11e
2023-12-07 17:14:27 -08:00
Cole Faust
69b5a5f0bf Revert "Handle symlinks in sbox"
This reverts commit dd9529050d.

Reason for revert: There are at least b/315337725 and b/315036698 issues, reverting this cl while I fix them.

Change-Id: I644bbbe594b9dd53a99a3b6513fbcba68afe0b4a
2023-12-07 21:40:08 +00:00
Cole Faust
dd9529050d Handle symlinks in sbox
Previously, sbox would follow the symlink and copy the symlink's target
into the sbox, instead of the symlink itself.

camera-its is a genrule that zips up a folder that includes a symlink,
so we want the symlink to still be there instead of a copy of the file
it points to.

Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py camera-its
Change-Id: Ib81f6fd6745ec2c58580082c73e33760d5051f5e
2023-11-28 17:34:38 -08:00
Jihoon Kang
8c4387e478 Merge "Revert^2 "Default from-text stub generation in build"" into main 2023-11-10 04:10:24 +00:00
Colin Cross
4ade19fe94 Merge changes I4d0b27ba,Ic48e9cf2 into main
* changes:
  Fix soong_ui file rlimits for go 1.21
  Fix TestReverseSlice for go 1.21
2023-11-09 18:47:40 +00:00
Colin Cross
611a2fbdd7 Fix soong_ui file rlimits for go 1.21
Go 1.21 modifies the file limit but restores the original when
execing subprocesses if it hasn't be overridden.  Call Setrlimit
even if it doesn't appear to be necessary so that the syscall package
considers it set.

Bug: 309895579
Test: m nothing
Change-Id: I4d0b27bac90a2a88bfc68a8491d54dc106e1ec13
2023-11-08 22:11:28 -08:00
Jihoon Kang
2a929ad250 Revert^2 "Default from-text stub generation in build"
This change defaults Java stubs to be generated from API text files
during build. Using the `--build-from-source-stubs` flag, users can
toggle between the feature.

This change relands aosp/2617457.

Test: m nothing && verify ninja path exists between android_stubs_current and android_stubs_current.from-text, and does not exist between android_stubs_current.from-source, m nothing --build-from-source-stub && verify the opposite
Bug: 301522358
Change-Id: Ic129dafc3231e71470af982374739930947b69cb
2023-11-08 04:25:52 +00:00
LaMont Jones
9a9128610b Readd support for PRODUCT_RELEASE_CONFIG_MAPS
This allows product config to determine what build flags are present in
a product's build, as well as their value.

As product config moves to a declarative specification, this can be
handled with less unnecessary work.

Bug: b/302593603, b/308849337, b/309477343
Test: manual
Change-Id: Ic0551990864bba6d03b6bf49fcfd4ae2523d9c6d
2023-11-07 18:39:40 +00:00
Colin Cross
6a4fe68cc5 Revert "Add support for PRODUCT_RELEASE_CONFIG_MAPS"
Revert submission 2815850-product-flags

Reason for revert: b/309477343

Reverted changes: /q/submissionid:2815850-product-flags

Bug: 302593603
Bug: 309477343
Change-Id: I5b5a5ade1d85195c43377ed7639deba6eef86e23
2023-11-06 21:32:03 +00:00
LaMont Jones
127127b608 Add support for PRODUCT_RELEASE_CONFIG_MAPS
This allows product config to determine what build flags are present in
a product's build, as well as their value.

As product config moves to a declarative specification, this can be
handled with less unnecessary work.

Bug: b/302593603
Bug: b/308849337
Test: manual
Change-Id: If57622059bb7d1c33df7498321db621c0c2d30c0
2023-11-03 19:19:59 +00:00
Colin Cross
fa24df6b23 Always reset timestamps in merge_zips
Jacoco uses ziptime and then merge_zips.  merge_zips is already rewriting
all the zip headers so its trivial to move the ziptime functionality
into merge_zips.

Bug: 308765940
Test: TestMergeZips
Change-Id: I8158c0b99878492233ac01fde3820e1951c9088e
2023-11-01 14:41:40 -07:00
Colin Cross
66d6409e06 Merge "Add performance counter metrics to build.trace.gz" into main 2023-10-30 22:21:14 +00:00
Colin Cross
46b0c75204 Add performance counter metrics to build.trace.gz
Start a background goroutine at the beginning of soong_build that
captures the CPU usage, heap size, and total system memory every
second.  Propagate the values through soong_build_metrics.pb back
to soong_ui, and then into build.trace.gz.

Test: m nothing, examine build.trace.gz
Change-Id: Iad99f8f1f088f4f7f7d5f76566a38c0c4f4d0daa
2023-10-30 12:39:07 -07:00
MarkDacek
39825ead17 Remove Bazel metrics/finalize-bazel-metrics.
Test: m nothing
Change-Id: Idd4610aca77f0a5deeadc3b895869879ded6c16e
2023-10-26 20:14:36 +00:00
Sebastian Pickl
1c4188c9da Revert "Default from-text stub generation in build"
This reverts commit 182b56b870.

Reason for revert: prime suspect for breaking boot tests b/307495247, b/307411752

Bug:307495247
Change-Id: Iea05703b767d2699ca3cf69377eb44b1d21697ad
2023-10-24 11:20:06 +00:00
Jihoon Kang
182b56b870 Default from-text stub generation in build
This change defaults Java stubs to be generated from API text files
during build. Using the `--build-from-source-stubs` flag, users can
toggle between the feature.

Test: m nothing && verify ninja path exists between android_stubs_current and android_stubs_current.from-text, and does not exist between android_stubs_current.from-source, m nothing --build-from-source-stub && verify the opposite
Bug: 274805756
Change-Id: I28834f92c1b1311e3fe0a71a6ea9e8ec2e278d7e
2023-10-20 23:11:19 +00:00
Sam Delmerico
285b66a117 Revert^4 "allow Ninja variables in RuleBuilder API"
789a7e0883

Change-Id: Iee1156adf0ad6e6d2ae7e22d90a447d4b564e59f
2023-09-25 12:13:17 +00:00
Colin Cross
40c7bae1df Merge changes from topics "revert-2629131-sandbox-rust-inputs-JRFPQTIPEY", "revert-2758566-WBNIADIEXA" into main
* changes:
  Revert^3 "add rust_toolchain_rustc_prebuilt module type"
  Revert^3 "add crate_root property to rust modules"
  Revert^3 "allow Ninja variables in RuleBuilder API"
  Revert "conditionally escape rule builder command"
  Revert "support sandboxed rust rules"
  Revert "fix failing rust_aconfig_library test"
  Revert "rustSetToolchainSource to use linux-x86 srcs"
  Revert "remove rust deps on clang prebuilts"
2023-09-23 05:19:13 +00:00
Wen-yi Chu
789a7e0883 Revert^3 "allow Ninja variables in RuleBuilder API"
0e2d63e4d2

Change-Id: I4a22fb398fc7bfdf86613829dfbc9c35fab11ee7
2023-09-22 22:05:54 +00:00
Ray Chin
0523ee828a Fix read file error in extract_apks
Bug: 301409551
Test: build
Change-Id: I353efea233e524c838298447b08743eeeedbb263
2023-09-22 13:20:50 +08:00
Sam Delmerico
52d1cc334b Merge changes from topics "revert-2746976-revert-2605644-rulebuilder-ninja-vars-OAAWYCDDLT-KMAGKVIXAT", "sandbox-rust-inputs" into main
* changes:
  support sandboxed rust rules
  conditionally escape rule builder command
  Revert^2 "allow Ninja variables in RuleBuilder API"
  Revert^2 "add crate_root property to rust modules"
  Revert^2 "add rust_toolchain_rustc_prebuilt module type"
2023-09-20 18:48:21 +00:00
Liz Kammer
00543dc61c Handle bad rename
Cleanspec wasn't working, so let's handle it in Soong

Test: create an invalid file, run m nothing, verify it is removed
Change-Id: I55d3c9344541b8e504887225a660044e14204427
2023-09-15 21:22:55 -04:00
Sam Delmerico
0e2d63e4d2 Revert^2 "allow Ninja variables in RuleBuilder API"
7b02d8159e

Change-Id: I9f2031162b019e51b869ea0db151c86db447b23d
2023-09-15 22:41:56 +00:00
Kousik Kumar
7a07b85fb5 Merge "Remove unused symlink code" into main 2023-09-15 17:49:07 +00:00
Christopher Parsons
3671c385c7 Merge "Add unit test for parsing build files in bp2build" into main 2023-09-15 00:04:09 +00:00
Kousik Kumar
4555802d20 Remove unused symlink code
It looks like hardware/qcom/sdm710 no longer exists in the source tree, so this symlink removal is should no longer be needed?

Bug: n/a

Change-Id: I8c9c6c0c0371039f910fc4290f81a692e4cd5cd4
2023-09-14 20:09:57 +00:00
Chris Parsons
5011e61c71 Add unit test for parsing build files in bp2build
This involves some minor changes to testing infrastructure.

This is a rollforward of aosp/2628496 with a couple of minor changes:

- In ParseBuildFiles, filter out all build files that are kept due
  to ShouldKeepExistingBuildFileForDir

- Add some minor test infrastructure for StubbedBuildDefinitions, with a
  couple of proof of concept tests used to demonstrate its usage. This
  pattern will become immensely more common as we implement allowlist v2
  (as we will need to update all tests which today simulate build
  definitions that have missing deps)

Bug: 285631638
Fixes: 286545783
Test: bp2build.sh
Test: m nothing
Change-Id: I7c3a03b02098e39dd8e51d327482b440f294478f
2023-09-14 19:19:28 +00:00
Sam Delmerico
1d3740a274 Merge changes from topic "revert-2605644-rulebuilder-ninja-vars-OAAWYCDDLT" into main
* changes:
  Revert "add rust_toolchain_rustc_prebuilt module type"
  Revert "add crate_root property to rust modules"
  Revert "allow Ninja variables in RuleBuilder API"
2023-09-08 20:46:55 +00:00
Sam Delmerico
7b02d8159e Revert "allow Ninja variables in RuleBuilder API"
Revert submission 2605644-rulebuilder-ninja-vars

Reason for revert: b/299568218

Reverted changes: /q/submissionid:2605644-rulebuilder-ninja-vars

Change-Id: Ia738c100b8431dea6151939a800b992f877b5bb8
2023-09-08 16:10:54 +00:00
Treehugger Robot
9abcf649a7 Merge changes from topics "rulebuilder-ninja-vars", "rustc_prebuilt_build_tool" into main
* changes:
  allow Ninja variables in RuleBuilder API
  add crate_root property to rust modules
  add rust_toolchain_rustc_prebuilt module type
2023-09-08 00:10:18 +00:00
Sam Delmerico
1f9bb26656 allow Ninja variables in RuleBuilder API
The RuleBuilder API would not expand Ninja variables because the
variables would be written verbatim to the sandbox manifest file. This
commit allows a rule to specify that the manifest file should be written
in an un-escaped format so that Ninja variables are expanded before
writing the manifest file.

Bug: 286077158
Test: rust sandboxing topic + go test
Change-Id: I1915431f6e24d04d343dacc213c9079674ec8251
2023-09-07 20:56:02 +00:00
Chris Parsons
73f411b72c Delete ApiBp2build soong_build mode
This feature is obsolete.

This makes a large number of codepaths "dead code" (such as
module-specific implementations of ApiBp2build functionality). These
will be deleted in a followup CL.

Bug: 284029211
Test: Presubmits
Change-Id: Ib53b99f1fe8c24380d219caf44e9bb3b96724fa0
2023-09-05 17:09:51 -07:00
Colin Cross
c2a62d40fa Strip zip64 extras after writing local header when copying
writeHeader generates zip64 extras that are correct for the local
header, but incorrect for the central directory header.  Strip the
extras again after writeHeader so that the central directory header
extras are recreated correctly.

Test: Zip2Zip64
Bug: 296314205
Change-Id: I1ca6a5745a9f97426df6c111db444facdfa25b2e
2023-08-22 14:24:25 -07:00
usta
40caf95004 cosmetic: unused argument
Test: m nothing
Bug: NA
Change-Id: I2a647e2e7c9c2fd3881c18e3cbb072b260cd9659
2023-08-04 16:53:31 -04:00
usta
aaf2fd13e1 add bp2build subevents
Test: verified new events in bp2build_metrics.pb
Bug: 286412441
Change-Id: Ib803c3a3d1a8e7eaae5fb4c01c7769a72f4664ff
2023-08-04 16:47:41 -04:00
Cole Faust
bc65a3fea8 Revert^2 "Initial implementation of the bazel sandwich"
c13fad8181

Change-Id: I478562c8fd89e62983feb5b52b62aad851d40f00
2023-08-01 10:27:57 -07:00
Jooyung Han
c13fad8181 Revert "Initial implementation of the bazel sandwich"
Revert submission 2651299-bazel_sandwich

Reason for revert: b/293883239, checking if this breaks the build.

Reverted changes: /q/submissionid:2651299-bazel_sandwich

Change-Id: I5f1d4bb13d21e6599f5c353dcaba2375f5ec234d
2023-08-01 05:15:17 +00:00
Cole Faust
a20d947329 Initial implementation of the bazel sandwich
The "bazel sandwich" is a mechanism for bazel to depend on make/soong
outputs. The name comes from the fact that bazel is now at the top
and bottom of the build graph. This is intended to allow us to work
on converting the partition builds to bazel while not all of the
dependencies of the partition have been converted.

It works by adding the bazel_sandwich_import_file rule, which emits a
dangling symlink that starts with bazel_sandwich:, and includes
information that the aquery handler in soong reads. The aquery handler
rewrites the symlink so that it points to a file generated by
make/soong, and adds a ninja dependency from the symlink to the file
it's targeting.

This allows us to depend on make-built files from bazel, but notably
it doesn't allow us to depend on analysis-time information from make.
This shouldn't be a problem for the partitions, but limits the use of
the bazel sandwich to similar, less complicated types of builds.

go/roboleaf-bazel-sandwich

Bug: 265127181
Test: m bazel_sandwich
Change-Id: Ic41bae7be0b55f251d04a6a95f846c50ce897adc
2023-07-31 11:53:41 -07:00
Colin Cross
7592d5a0bd Merge META-INF/services/* files in merge_zips -jar
kotlinx_coroutines_test and kotlinx_coroutine_android each provide a
META-INF/services/kotlinx.coroutines.CoroutineExceptionHandler with
different contents, and the final contents needs to be the combination
of the two files.  Implement service merging in merge_zips when the
-jar argument is provided.

Bug: 290933559
Test: TestMergeZips
Change-Id: I69f80d1265c64c671d308ef4cdccfa1564abe056
2023-07-20 18:59:35 +00:00
Spandan Das
958ca02582 Merge changes from topic "build_go_source_mixed_builds" into main
* changes:
  Delete aliases to prebuilts
  Add functionality to sandbox mixed build actions
2023-07-19 17:21:25 +00:00
Cole Faust
f8231dd0ea Platform mapping-based product config
This allows us to set product variables as build settings instead
of loading them from a target's provider, which further allows us
to read product config variables in transitions.

Bug: 287539062
Bug: 269577299
Test: Presubmits
Change-Id: I8497703f706162572ceb3486240e1eb02a37f5f6
2023-07-17 16:27:08 -07:00
Treehugger Robot
90d8f5514d Merge "Log the command line to soong.log when soong_ui starts" into main 2023-07-17 13:23:17 +00:00
Joe Onorato
010c6b67da Log the command line to soong.log when soong_ui starts
Change-Id: Ie39333e4771cddf4c828d5e3474e2e43001db807
Test: manual
2023-07-14 16:33:19 -07:00
Cole Faust
1269a74fad Make pom2bp output preprocessed: true on app imports
The only difference between preprocessed + presigned vs just presigned
is that just presigned will zip align the apk, and preprocessed +
presigned will instead check that the apk is already aligned and has
uncompressed jni and dex libs. This should be the default to ensure
we're getting valid apk imports from external sources.

Presigned alone also won't work for apps with target sdk >= 30, because
on those target sdks you have to use apk signature v2, and having the
build system align the zip file will break that signature.

Bug: 185811447
Test: Presubmits
Change-Id: Ie825ab2eab4feeda5c266233a9dca7bbf9559be7
2023-07-14 15:44:24 -07:00
Spandan Das
2a55cea4a0 Delete aliases to prebuilts
These tools will now be built from source using rules_go

Test: TH
Bug: 284483729
Change-Id: I6dde9f1418aaa516c4c6f8c8897dd20f4becfb51
2023-07-14 00:43:57 +00:00
Yabin Cui
f8622b732a Revert "Exclude prebuilts/clang/host/linux-x86/clang-dev/BUILD from the symlink forest."
This reverts commit bcfadce598.
Because now test_compiler.py copies clang-dev instead of creating a
symlink.

Bug: 288327832
Test: run test_compiler.py
Change-Id: Ib676a41c152046d710d73fdaac678d8a98de5af2
2023-06-26 22:18:09 +00:00
LaMont Jones
610ebf0312 Improve memory usage in multiproduct_kati
Start the subtasks at 15 second intervals, rather than all at once.

Bug: b/286423944
Test: manual
Change-Id: I21981460dcdd618f42abefcc221a05ce6aca84c1
2023-06-22 00:45:03 +00:00