Commit Graph

20780 Commits

Author SHA1 Message Date
Kelvin Zhang
49023a992f Merge "Fix zip64 reader when file size is < 4GB and 32 bit fields are -1" 2020-08-07 18:20:15 +00:00
Martin Stjernholm
9f381d5ce3 Merge "Only output make targets for uninstallable static libs in the APEX unavailable-to-platform case." 2020-08-07 11:30:29 +00:00
Thiébaud Weksteen
9bf3042493 Merge changes I329d0fe3,I8af9115b
* changes:
  rust: validate existence of library source
  rust: handle modules with same crate_name
2020-08-07 06:43:44 +00:00
Inseob Kim
eaebec1312 Merge changes from topic "cfi_vendor_snapshot"
* changes:
  Add cfi static libraries to vendor snapshot
  Refactor vendor snapshot modules
2020-08-07 01:48:10 +00:00
Treehugger Robot
8b073fc7f1 Merge "Reland "Support per-module MakeVars"" 2020-08-07 00:50:24 +00:00
Jingwen Chen
2106bf8458 Merge "Create a AOSP Bazel overlay workspace with Soong" 2020-08-06 23:56:54 +00:00
Dan Willemsen
6a6478d49e Reland "Support per-module MakeVars"
This allows setting per-module make variables earlier in the build,
particularly for prebuilt_build_tool users like LEX/M4/BISON (which will
come in a later CL). I moved filegroup over because it's a simpler
common interface, but it doesn't strictly need it.

Test: Inspect out/soong/make_vars-*.mk, out/soong/late-*.mk
Test: treehugger
Change-Id: I38631129d82b0aad2438e83610bb6a97568bf932
2020-08-06 15:56:09 -07:00
Martin Stjernholm
9e9bb7f223 Only output make targets for uninstallable static libs in the APEX
unavailable-to-platform case.

This removes the special case added in https://r.android.com/1274763
from SkipInstall(), so that it doesn't cause conflicting AndroidMk
entries when a cc_prebuilt_library_static module has prefer:true and
the corresponding source module exists.

Test: `m` in a tree with a snapshot created from art-module-sdk where
  the libartimagevalues module has prefer:true
Bug: 151303681
Change-Id: I651ae325753b707296892adb4cae80daaddb6af2
2020-08-06 22:52:58 +01:00
Matthew Maurer
217a1e406e Merge "[rust] Escape flags for bindgen" 2020-08-06 21:33:49 +00:00
Mitch Phillips
8dc2af873b Merge "Added write only sanitizer for ASAN and HWASAN" 2020-08-06 21:17:12 +00:00
Kelvin Zhang
45e2f1405d Fix zip64 reader when file size is < 4GB and 32 bit fields are -1
When CopyFrom writes a zipentry, it strips extra fields and generates
data descriptors. When writing data descriptors, it only writes 64 bit
values if the relevant sizes are >4GB. In some cases, the sizes are <4GB
but 32 bit sizes are set to -1. In this situation, CopyFrom will write
incorrect local file header, resulting in a zip file that can't be
parsed by standard zip tools.

Test: Unit Tests
Bug: 161922066
Change-Id: I64319a80647013eaf7693cf8bf5c6120016913a3
2020-08-06 15:48:04 -04:00
Inseob Kim
c42f2f2e9d Add cfi static libraries to vendor snapshot
CFI modules can't link against non-CFI static libraries, and vice versa.
So without capturing both CFI and non-CFI static libraries, vendor
modules won't be able to use CFI, which will be a critical security
hole.

This captures both CFI and non-CFI variants of all static libraries for
vendor snapshot, except for those whose cfi are explicitly disabled.

For example, suppose that "libfoo" is defined as follows.

cc_library_static {
    name: "libfoo",
    vendor_available: true,
}

As it doesn't have cfi disabled, two libraries "libfoo.a" and
"libfoo.cfi.a" will be captured. When installed, vendor snapshot module
for "libfoo" will look like:

vendor_snapshot_static {
    name: "libfoo",
    src: "libfoo.a",
    cfi: {
        src: "libfoo.cfi.a",
    },
}

The build system will recognize the "cfi" property, and will create both
CFI and non-CFI variant, allowing any modules to link against "libfoo"
safely, no matter whether CFI is enabled or not.

Two clarification:

1) The reason why we don't create separate modules is that DepsMutator
runs before sanitize mutators. CFI and non-CFI variant of a library
should exist in a single module.

2) We can't capture CFI variant if the source module explicitly disables
cfi variant by specifying the following.

sanitize: {
    cfi: false,
}

In this case, only non-CFI variant will be created for the vendor
snapshot module.

Bug: 65377115
Test: m dist vendor-snapshot && install && build against snapshot
Change-Id: Idbf3e3205d581800d6093c8d6cf6152374129ba4
2020-08-06 19:47:57 +00:00
Inseob Kim
2d34ad94f9 Refactor vendor snapshot modules
They have too many duplicated fields and functions.

Bug: 65377115
Test: manual
Change-Id: If3e2d67f02f3db7fa836c9550c978073ab7190e0
2020-08-06 19:47:43 +00:00
Steven Moreland
bd60c83271 Merge "allow empty phony modules" 2020-08-06 17:22:11 +00:00
Ulyana Trafimovich
910eb70891 Merge "Add structured representation for colon-separated jar lists." 2020-08-06 15:07:27 +00:00
Nicolas Geoffray
1f109cd376 Merge "Add platform-mainline-test-exports in list of modules to build." 2020-08-06 13:49:25 +00:00
Thiébaud Weksteen
83ee52f4cd rust: validate existence of library source
While rust_bindgen modules may be interpreted as libraries, they do not
have a reference to the generated source until bindgen has been
executed. For now, ignore these modules.

Update the unit tests to cover the rust_bindgen case.

Test: SOONG_GEN_RUST_PROJECT=1 m nothing
Bug: 162881856
Change-Id: I329d0fe3d94b77d395c3684f55ab01544ff7c18f
2020-08-06 15:17:46 +02:00
Thiébaud Weksteen
2f628baeae rust: handle modules with same crate_name
crate_name do not need to be unique. A library may depend on two
different versions of the same crate. Use the module name instead of the
crate name when indexing the modules for rust-project.json.

Test: SOONG_GEN_RUST_PROJECT=1 m nothing
Bug: 162896400
Change-Id: I8af9115b106fda2bed5b9fe96059cc29a33ac247
2020-08-06 15:17:42 +02:00
Thiébaud Weksteen
c810510898 Revert "rust: validate existence of library source"
Revert submission 1391076

Reason for revert: Broken downstream Darwin build (b/162975597)
Reverted Changes:
I275f04639:rust: handle modules with same crate_name
Ie736d7ebb:rust: validate existence of library source

Change-Id: I1732d50a08546864f17cd826c8af574c1064c670
2020-08-06 12:28:08 +00:00
Thiébaud Weksteen
246e69c586 Revert "rust: handle modules with same crate_name"
Revert submission 1391076

Reason for revert: Broken downstream Darwin build (b/162975597)
Reverted Changes:
I275f04639:rust: handle modules with same crate_name
Ie736d7ebb:rust: validate existence of library source

Change-Id: I995923153c11db26b4af985f2eabe94912fb04d3
2020-08-06 12:27:56 +00:00
Jingwen Chen
5ba7e479d1 Create a AOSP Bazel overlay workspace with Soong
The Bazel overlay is a directory at out/soong/bazel_overlay that
replicates the layout of the AOSP Soong module tree, but as a Bazel
workspace. Each Soong module variant is represented as a BUILD target
created with the `soong_module` rule.

To create this overlay, run `m bazel_overlay`.

A `soong_module` target can depend on other `soong_module` targets.
These dependencies replicate each module's `directDeps` in the Blueprint
graph, just before `PrepareBuildActions`.

This enables users to use bazel query as a way to introspect the Soong
module graph. For example,

- Direct reverse dependencies of //bionic/libc:generated_android_ids in
//bionic/libc/...:

$ bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
//bionic/libc:libc_bionic_ndk--android_recovery_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_ramdisk_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static_com.android.runtime
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static
//bionic/libc:generated_android_ids

- Why does com.android.runtime depend on lzma?

$ bazel query
'somepath(//bionic/apex:com.android.runtime--android_common_com.android.runtime_image,
//external/lzma/...)'
//bionic/apex:com.android.runtime--android_common_com.android.runtime_image
//bionic/libc/malloc_debug:libc_malloc_debug--android_arm_armv7-a-neon_shared_com.android.runtime
//system/core/libunwindstack:libunwindstack--android_arm_armv7-a-neon_shared_com.android.runtime
//external/lzma/C:liblzma--android_arm_armv7-a-neon_shared_com.android.runtime

- What does the dep graph of //bionic/libc:crtbegin_so look like?

$ bazel query
'deps(//bionic/libc:crtbegin_so--android_arm_armv7-a-neon)'
--output=graph > graph.in && dot -Tpng < graph.in > graph.png

https://photos.app.goo.gl/DfsdoFRNsRjGwTmy8

Test:  croot && m bazel_overlay && cd out/soong/bazel_overlay && bazel
query //... && bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'

Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I3bf40309bfb2d963bb8a688706385a57ee304c37#
2020-08-06 10:24:21 +00:00
Treehugger Robot
8c69770ff0 Merge changes from topic "soong_sanitize_fix"
* changes:
  Separate lists of soong sanitize modules
  Move image mutator things to image.go
2020-08-06 00:48:16 +00:00
Colin Cross
2eddd06879 Merge "Add libraryDependencyTag to track dependencies on static and shared libraries" 2020-08-06 00:06:28 +00:00
Elliott Hughes
325d73f260 Merge "Switch to toybox patch(1)." 2020-08-05 23:30:14 +00:00
Treehugger Robot
cf187bcbbc Merge "[cc/coverage] Pass directory to -fprofile-instr-generate" 2020-08-05 23:10:00 +00:00
Matthew Maurer
53bba5de8a Merge "Make Rust test harness optional for test binaries" 2020-08-05 21:26:45 +00:00
Ivan Lozano
d13b308960 Merge "Make source_stem required for Rust SourceProviders" 2020-08-05 19:38:19 +00:00
Steven Moreland
9cd45782e5 allow empty phony modules
In order to allow reservation of names in the build system.

Bug: 162275959
Test: build only
Change-Id: I8f1622bdbebe794dafd5c341d861da5468cd52bd
2020-08-05 19:31:54 +00:00
Ivan Lozano
80668b07d2 Merge "Add support for custom bindgen binaries." 2020-08-05 19:08:20 +00:00
Liz Kammer
9c55b0efa2 Merge "Add property api_levels_jar_filename to droidstubs" 2020-08-05 19:05:13 +00:00
Thiébaud Weksteen
6090575f63 Merge changes I275f0463,Ie736d7eb
* changes:
  rust: handle modules with same crate_name
  rust: validate existence of library source
2020-08-05 18:50:06 +00:00
Bill Peckham
6ed589039c Merge "Adds 'kernel/' to vendor proprietary dirs." 2020-08-05 18:14:52 +00:00
Ivan Lozano
ff3a5b31c0 Make source_stem required for Rust SourceProviders
This removes ambiguity for developers with the 'stem' property.

Bug: 159064919
Test: New Soong test; error is produced if this property is undefined.
Change-Id: I790fbcfa6cd7dbf6df63b5cf5c22983426ec072e
2020-08-05 13:31:34 -04:00
Pirama Arumuga Nainar
f45e152e15 [cc/coverage] Pass directory to -fprofile-instr-generate
Bug: http://b/157081822

This will be used if LLVM_PROFILE_FILE is not set at runtime.  Some
bionic tests are run without LLVM_PROFILE_FILE set and if they are run
from '/', they fail unable to write to '/default.profraw'.  This may
affect other tests as well.  Setting this to the value defined by
init.rc.

Test: Enable coverage for bionic, ensure relevant bionic-unit-tests
pass.

Change-Id: If1d0e16ce003149fb21d5472d10699ef6582e065
2020-08-05 10:23:07 -07:00
Liz Kammer
9eb51fc9d4 Merge "Refactor java compileDex" 2020-08-05 15:41:03 +00:00
Nicolas Geoffray
b0df23f201 Add platform-mainline-test-exports in list of modules to build.
Needed for ART testing.

Bug: 142935992
Test: build-mainline-modules.sh

Change-Id: I842feff85f88ecc2c7db1e84e803d9e8149923af
2020-08-05 15:56:21 +01:00
Ivan Lozano
c564d2d5a4 Add support for custom bindgen binaries.
In some cases customized logic is required to generate the expected
bindgen bindings. This adds support for rust_bindgen modules to define a
HostTool module to use instead of bindgen.

Bug: 161816141
Test: New Soong tests pass.
Test: Local test case shows custom_binary module being used for bindgen
      generation.

Change-Id: Id52aec4f25c38206d7e585d8e662be7836aa1d4b
2020-08-05 10:51:43 -04:00
Martin Stjernholm
42e6dabe81 Merge "Disable all host OS variants except those explicitly included." 2020-08-05 14:50:48 +00:00
Thiébaud Weksteen
ad8ac53581 rust: handle modules with same crate_name
crate_name do not need to be unique. A library may depend on two
different versions of the same crate. Use the module name instead of the
crate name when indexing the modules for rust-project.json.

Test: SOONG_GEN_RUST_PROJECT=1 m nothing
Bug: 162896400
Change-Id: I275f04639ef05f2b649c30168046e13b2efcacb9
2020-08-05 16:34:38 +02:00
Ivan Lozano
b14e141519 Merge "[rust] Disable clippy for rust_bindgen modules." 2020-08-05 12:47:14 +00:00
Andrei-Valentin Onea
fe58347aef Merge "Rename hiddenapi manual lists" 2020-08-05 12:42:18 +00:00
Thiébaud Weksteen
891cd578c7 rust: validate existence of library source
While rust_bindgen modules may be interpreted as libraries, they do not
have a reference to the generated source until bindgen has been
executed. For now, ignore these modules.

Update the unit tests to cover the rust_bindgen case.

Test: SOONG_GEN_RUST_PROJECT=1 m nothing
Bug: 162881856
Change-Id: Ie736d7ebb115f2c1f9a90be006c972ce24265c6a
2020-08-05 11:29:33 +02:00
Ulya Trafimovich
249386ad69 Add structured representation for colon-separated jar lists.
With the addition of apexes and /system_ext some of the bootclasspath
and system server jars have moved from /system to the new locations.
This has been implemented by using lists of colon-separated strings
called "apex-jar pairs" (although "apex" was misleading as it could
refer to "platform" or "system_ext", not necessarily a real apex).

Using the colon-separated string representation is inconvenient, as it
requires splitting and reassembling the list components many times,
which harms performance and makes error handling difficult. Therefore
this patch refactors the colon-separated lists into a struct that
hides the implementation details.

Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: Id248ce639a267076294f4d4d73971da2f2f77208
2020-08-05 09:59:30 +01:00
Inseob Kim
74d255698b Separate lists of soong sanitize modules
Dependencies of makefile modules are being redirected according to
SOONG_CFI_STATIC_LIBRARIES and SOONG_HWASAN_STATIC_LIBRARIES. But the
variables are shared among all variants (e.g. core, vendor, product,
arch), which can cause build error.

This splits the Makefile variables into several lists, one list per each
arch and each image variant, to correctly make the redirection.

Bug: 162476652
Test: build and inspect ninja
Change-Id: Icc753382f1c53de8468cc85243a6954e1986297a
2020-08-05 04:47:41 +00:00
Inseob Kim
e498dd9ac3 Move image mutator things to image.go
Bug: N/A
Test: m
Change-Id: I2d1a24fc067c376ed2e0741313d48248ad78ee78
2020-08-05 04:47:28 +00:00
Ivan Lozano
32267c88c4 [rust] Disable clippy for rust_bindgen modules.
bindgen generated code may not pass clippy checks, so preemptively
disable clippy for these.

Bug: 162828070
Test: clippy is not called when compiling rust_bindgen library variants.
Change-Id: I9632c889417bdfd1adf96d9cfbccbe6340824205
2020-08-04 16:27:16 -04:00
Stephen Crane
da931d4abd [rust] Disable lints for generated bindings
Bindgen generated bindings will not pass lints, so we need to disable
them when building bindings crates.

Test: m libbinder_ndk_bindgen
Change-Id: I4536f15f8505ff8d5436f1e6cbc2035e6801276d
2020-08-04 13:04:32 -07:00
Stephen Crane
12e2cb71c7 [rust] Escape flags for bindgen
Bindgen flags and cflags should be escaped, as they may contain shell
globs or other special characters.

Test: Updated Soong test passes.
Change-Id: I3df8ef25391f53a191f0494c5ff8c641d4d4b6f8
2020-08-04 12:29:54 -07:00
Yifan Hong
ea3574b5fa Merge "Support com.android.gki.* in apex_available." 2020-08-04 19:06:08 +00:00
Stephen Crane
02a623d8d3 Make Rust test harness optional for test binaries
Adds a test_harness field to Rust test blueprint properties. If this
field is explicitly set to false, do not build the test binary with
`--test`. This is necessary for Rust tests which cannot use the standard
test harness.

Test: atest rustBinderTest
Test: atest --host proc-macro2_tests_test
Change-Id: Icbcb54422cc716348feae56b2d71f013516b0ac0
2020-08-04 18:57:10 +00:00