Commit Graph

2447 Commits

Author SHA1 Message Date
Paul Duffin
103aaae715 Extract method to process bp file for androidmk_test.go
Test: m nothing
Bug: 174226317
Change-Id: Ifc9ee5088fe766d971718fc112d1e95719a2b825
2020-11-27 01:24:02 +00:00
Paul Duffin
d35d92a7b5 Merge "Improve TestGetDistForGoals debuggability" 2020-11-26 17:16:53 +00:00
Liz Kammer
6953adeb95 Merge "Convert csuite config to use AndroidMkEntries" 2020-11-26 00:05:30 +00:00
Paul Duffin
0cc047ad5d Improve TestGetDistForGoals debuggability
Previously, TestGetDistForGoals tested multiple test cases within a
single test so when it failed it was difficult to determine which test
case was the cause. This change runs each test case as its own
nested test.

It also corrects the order of expectedLine and line format parameters
to match the order in the message.

Test: m nothing
Bug: 174226317
Change-Id: I1408ec4125afc5c0b392cd7643dd3f630fe468e5
2020-11-25 17:02:32 +00:00
Jingwen Chen
c4d91bceba Remove soong.config, FileConfigurableOptions, MegaDevice config.
No current use cases for soong.config and user-editable local
configuration options for Soong, so let's remove it to reduce the API
surface.

The MegaDevice configuration is used for building every module, for
every cpu variant, for every architecture, but it apparently isn't
used for a while now.

Test: m nothing
Test: TH
Fixes: 174188200
Change-Id: I2550e70ff6f9c8b57e9a7cc517d6a119a032a27a
2020-11-24 23:42:17 -05:00
Jingwen Chen
65d5b7f0e9 Merge "Comment android/config.go" 2020-11-25 04:18:56 +00:00
Liz Kammer
e2a17a5a40 Convert csuite config to use AndroidMkEntries
This is to facilitate consolidation of compatibility suite logic.

Test: m nothing and compare soong Android mk files -- no diffs
Change-Id: Ibc60d14fdde93835124e18a5c484bbda496bdb9e
2020-11-24 13:41:28 -08:00
Colin Cross
38214f5ac5 Merge changes I8af00c8c,Ia526ae89
* changes:
  Reland: Rewrite sbox to use a textproto manifest
  Fix gensrcs subdirs
2020-11-24 19:27:35 +00:00
Colin Cross
e5218b6be6 Merge "Annotate dependency tags for dependencies of installed files" 2020-11-24 16:01:35 +00:00
JaeMan Park
ff71556a53 Add java sdk library enforcement flag
Add java sdk library enforcement for inter-partition library
dependency, for ensuring backward-compatible libraries for
inter-partition dependencies.

Test: m nothing
Bug: 168180538
Change-Id: I6bfac54c3499b03003a3bc6c2bb62b165b4ce5f9
2020-11-24 17:56:15 +09:00
Treehugger Robot
5bca812a4e Merge "Fail if non-allowlisted host system tool is used." 2020-11-24 07:02:30 +00:00
Jingwen Chen
f87a720101 Merge "Refactor 'in_make' to mean Kati is not skipped." 2020-11-24 02:09:21 +00:00
Colin Cross
e9fe2949b8 Annotate dependency tags for dependencies of installed files
Relands Ic22603a5c0718b5a21686672a7471f952b4d1017 with a minor
change to track libc++ dependencies for python hosts and after
a fix to an internal genrule that depended on transitively
installed java libraries (ag/13068670).

Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.

Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.

This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.

Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: I725871249d561428e6f67bba6a7c65b580012b72
2020-11-23 18:06:08 -08:00
Jingwen Chen
1b461e41bd Merge "Move bazel_module property to a common file, and add it to filegroup." 2020-11-24 01:54:43 +00:00
Jingwen Chen
2838c818da Fail if non-allowlisted host system tool is used.
This CL improves the clarity of the non-hermetic nature of using
HostSystemTools, and make the build fail if a caller tries to use a host
system tool that isn't allowlisted in config.go.

The only caller seems to be for xcrun, which is a special case that is
pretty hard to remove.

Test: TH presubmit
Change-Id: Icd3e7330406a08c00ac9682e37e2d42669f2e6b7
2020-11-23 20:43:41 -05:00
Jingwen Chen
c711fecfc1 Comment android/config.go
Fixes: 173973687
Test: TH presubmit
Change-Id: I4bab950e92a69b12cb7738515a6c516e5b273745
2020-11-23 20:27:45 -05:00
Jingwen Chen
1ad9dc00a0 Merge "Delete check for EXPERIMENTAL_JAVA_LANGUAGE_LEVEL_9." 2020-11-24 01:12:02 +00:00
Colin Cross
d2d8fd6f51 Merge changes I0f2f0e41,Ib034140c,I4b2ca283
* changes:
  Add java_data parameter to python modules
  Use local variations for python version splits
  Support SourceFileProducer in android.OutputFilesForModule
2020-11-23 23:20:06 +00:00
Colin Cross
74b1e2b880 Support SourceFileProducer in android.OutputFilesForModule
Add support to android.OutputFilesForModule to get paths from a
SourceFileProducer as well as an OutputFileProducer.

Bug: 173977903
Test: m checkbuild
Change-Id: I4b2ca2837342ddbb4210bee8f549a636d8b8b049
2020-11-23 18:35:17 +00:00
Jingwen Chen
17d690c8b0 Delete check for EXPERIMENTAL_JAVA_LANGUAGE_LEVEL_9.
This has been a no-op for a year now. a1c9e9da55

Test: TH presubmit
Change-Id: Id9cc56045cbca4ebae64170e5443501b9ccc8260
2020-11-23 05:00:42 -05:00
Jingwen Chen
cda22c9bb9 Refactor 'in_make' to mean Kati is not skipped.
In Nougat and before, Make wrote a marker file to indicate that
soong_build was invoked from Make to change certain behaviors of Soong
at build time.
https://cs.android.com/android/platform/superproject/+/android-7.1.2_r36:build/core/soong.mk;l=70-73;drc=ae18638b0406ad107b0882a02a13cdd8b92f2a4e

Things have changed, and now soong_build is invoked from soong_ui, which
supports a --skip-make configuration flag:
https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/build.go;l=31-33;drc=680387bf1d3ce7cbc77f535be7c42cec411b1687

Thus, the various remnants of 'EmbeddedInMake' and 'inMake'
configuration are misleading, since soong_build is no longer invoked
from Make. This CL refactors all instances to actually mean that
Kati is enabled (not skipped with --skip-make), and will run after
soong_build finishes, so Kati-specific behavior like the AndroidMk
singleton should run.

Test: TH presubmit

Change-Id: I576ab8e54f99f5c8ddf9feaf9a828019b279e266
2020-11-23 00:29:18 -05:00
Jingwen Chen
30f5aaaa77 Move bazel_module property to a common file, and add it to filegroup.
This enables prototyping against aosp/1441774 to demonstrate mixed
builds with converted BUILD files.

Test: TH Presubmit
Test: USE_BAZEL_ANALYSIS=1 m libc && prebuilts/build-tools/linux-x86/bin/ninja -f out/combined-aosp_cf_x86_auto.ninja -t commands libc | grep bazel-out | wc -l # 2 build actions
Bug: 171263886

Change-Id: I7b5cd0449d043ba26a339a0ef98b562fc62e13c8
2020-11-22 22:01:44 -05:00
Treehugger Robot
b26070efef Merge "Documenting android/apex.go" 2020-11-23 00:44:35 +00:00
Colin Cross
e16ce36818 Reland: Rewrite sbox to use a textproto manifest
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.

In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.

Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
2020-11-20 15:45:04 -08:00
Colin Cross
b893f8766c Merge "Revert "Rewrite sbox to use a textproto manifest"" 2020-11-20 18:45:35 +00:00
Colin Cross
619b9ab260 Revert "Rewrite sbox to use a textproto manifest"
This reverts commit 151b9ff0cf.

Reason for revert: broke builds

Change-Id: I69b3b8795d5a36b4fa0debb1af2d433be3c15d6c
2020-11-20 18:44:31 +00:00
Colin Cross
60d06cf8df Merge "Rewrite sbox to use a textproto manifest" 2020-11-20 17:43:29 +00:00
Jiyong Park
e4758ed844 Documenting android/apex.go
Mostly documentation changes, but includes a few refactorings like
changing the variable names, reording functions, reordering statements
in logical order, etc.

Bug: 173472337
Test: m
Change-Id: I000c76e818722ed06bac03d9de87588b23552b08
2020-11-20 10:07:19 +09:00
Colin Cross
11c89c0e9e Add more comments to path_properties.go
Bug: 173449605
Test: m checkbuild
Change-Id: I2638da9cc56726f5d646424e72f52289ad093781
2020-11-19 14:28:46 -08:00
Colin Cross
a684540945 Add more comments to arch.go
Make sure every exported function or type has a godoc comment.

Also makes minor changes like unexporting functions that are not used
outside the package and fixing minor style warnings.

Bug: 173449605
Test: m checkbuild
Change-Id: I533a595d02035aae8b2b603590be639826d2d4c8
2020-11-19 11:20:05 -08:00
Inseob Kim
4f4e2bd495 Revert^2 "Always turn on compatible property"
This reverts commit 7d13e5b964.

Reason for revert: Fixed broken targets

Change-Id: Ibfa2641575346675f8210297eaebfbe7d63cf5a5
2020-11-19 09:48:29 +00:00
Treehugger Robot
7a64f7e5b6 Merge changes from topic "package"
* changes:
  add android_filesystem
  Introduce PackagingBase
2020-11-19 07:49:00 +00:00
Colin Cross
5f8293417c Merge "Revert "Skip creating variants for disabled OSes"" 2020-11-19 02:34:36 +00:00
Colin Cross
08d6f8fd75 Revert "Skip creating variants for disabled OSes"
This reverts commit d976af0cb4.

Reason for revert: broke windows SDK build: missing bin/dexdump.exe

Bug: 173663545
Change-Id: Ibb541507650beabd2d94885dd8d66f724a358ca7
2020-11-19 02:33:19 +00:00
Jiyong Park
dda8f69e43 Introduce PackagingBase
PackagingBase is the base struct that implements the basic packaging
functionalities. Here, packaging means grouping dependencies into an
output file where the built artifacts from the dependencies are placed
in a directory-like structure in the output file. The exact format of
the output file is irrelevant; it could be a filesystem image, tar.gz,
zip, or whatever.

PackagingBase is responsible for traversing the dependencies and copying
their outputs under the package root directory, which is expected to be
provided by the module type that includes PackagingBase. Then the
concrete module type is expected to do final step of converting the
package root directory into the output file of specific format.

Bug: 159685774
Bug: 172414391
Test: m dist out/dist/cvd-host_package.tar.gz
Change-Id: I5446eee4834ce3b6f0f5843d93bb330a26d42fe3
2020-11-19 08:58:06 +09:00
Colin Cross
1357c11948 Merge changes Ic2daab29,I884ddd09
* changes:
  Skip creating variants for disabled OSes
  Use bitfield for HostOrDeviceSupported
2020-11-18 23:36:24 +00:00
Colin Cross
151b9ff0cf Rewrite sbox to use a textproto manifest
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.

Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
2020-11-18 13:02:22 -08:00
Treehugger Robot
b08a091502 Merge changes Iba57c949,Ief43ff51,Ib1809a4d,I2ab64f36
* changes:
  Store ndkKnownLibs in the config
  Register the kythe singleton on the Context instead of globally
  Store ninja file deps from PackageVarContext in the config
  Store SingletonMakeVarsProviders in the config
2020-11-18 20:00:31 +00:00
Colin Cross
d4a63eaeb7 Merge "Move genrule on top of RuleBuilder" 2020-11-18 18:44:19 +00:00
Christopher Parsons
2bb2017a30 Merge "Use packagepath and local_repository for mixed builds" 2020-11-18 16:06:35 +00:00
Treehugger Robot
ec40eb0abb Merge "add PackagingSpec" 2020-11-18 12:13:22 +00:00
Will Osborn
f0aa3b9527 Merge "Revert "Always turn on compatible property"" 2020-11-18 11:13:59 +00:00
Inseob Kim
7d13e5b964 Revert "Always turn on compatible property"
This reverts commit f6d606ebea.

Reason for revert: broken targets on the internal branch

Change-Id: I948280e2072789a2ee20e7fe4844c667f58b5182
2020-11-18 11:04:27 +00:00
Treehugger Robot
60447f1ce5 Merge "Always turn on compatible property" 2020-11-18 07:23:39 +00:00
Jingwen Chen
27a8719b0f Merge "Integrate bazelenv.sh environment variables into soong_ui environment." 2020-11-18 04:53:22 +00:00
Jiyong Park
073ea55fad add PackagingSpec
Currently, installation of a module is defined as an action of copying
the built artifact of the module to an install path like out/soong/host
(for host modules) and out/target/product/<device>/<partition> (for
device modules). After the modules are installed, the installed files
are further processed to create packages like system.img, vendor.img,
cvd-host-package.tar.gz, etc.

This notion of installation seems to have originated from the old time
when system.img is the primary product of the entire build process
(modulo a few more like root.img). Packaging the installed files as the
filesystem image was considered as a post-build step then.

However, this model doesn't seem to fit well to the current and future
environment where we have a lot more filesystem images (system, vendor,
system_ext, product, ...). The filesystem images themselves are even
grouped together to form a higher-level filesystem image like super.img.
Furthermore, things like cvd-host-package.tar.gz requires us to be able
to group some of the host tools in a format that isn't filesystem image.
Lastly, we are expected to have more filesystem images that are subsets
of system.img (and their friends) for the Android-like mini OS that will
be running on on-device virtual machines. These all imply that the
packaging (which we call installation today) is not a global post-build
step, but a part of the build rules for creating the package-like
modules.

A model better fits to the new sitatuation might be this; a module
specifies its built artifact and the path where it should be placed. The
latter path is not rooted at out/. It's a relative path to the root
directory which will be determined by another module that implements the
packaging. For example, cc_library will have ./lib (or ./lib64), not
out/target/product/<device>/<partition>/lib as the path. Then packages
like system.img, cvd-host-package.tar.gz, etc. are explicitly modeled as
modules and they have deps to other modules. Then the modules are placed
at the relative path under the package root, and the entire root
directory finally is packaged as the output file (be it img, tar.gz, or
whatever).

PackagingSpec is the first step to implement the new model. It abstracts
a request to place a built artifact at a certain path in a package. It
has extra information about whether the path should be a symlink or not,
and whether the path is for an executable. It currently is created when
InstallFiles (and its friends) are called, and can be retrieved via
the new method PackagingSpecs().

In this CL, no one is using PackagingSpec. The installation is still
done by the existing rules created in InstallFiles, etc. and the
structs are not used for the filesystem images like system.img.

Bug: 159685774
Bug: 172414391
Test: m

Change-Id: Ie1dec72d1ac14382fc3b74e5c850472e9320d6a3
2020-11-18 11:30:19 +09:00
Jingwen Chen
7c6089ad95 Integrate bazelenv.sh environment variables into soong_ui environment.
This removes the need to source bazelenv.sh for USE_BAZEL_ANALYSIS, and
unifies mixed builds to use the checked in tools/bazel and bazelrc.

It also unifies all bazel-related output to be in out/bazel.

Without aosp/1502095, this change still requires toplevel_output_directories to be an empty
list, otherwise there'll be this error:

ERROR: Directories specified with toplevel_output_directories should be
ignored and can not be used as sources.

Test: With aosp/1441774: rm -rf out/ && lunch aosp_cf_x86_auto && USE_BAZEL_ANALYSIS=1 m libc && prebuilts/build-tools/linux-x86/bin/ninja -f out/combined-aosp_cf_x86_auto.ninja -t commands libc | grep bazel-out | wc -l # 2 results
Change-Id: I69b217ec88da531415792bb6e04b6a194ca4718d
2020-11-17 18:58:03 -05:00
Colin Cross
d976af0cb4 Skip creating variants for disabled OSes
The documentation java_genrule_host states that it creates a single
variant, which would make it work with the single variant fallback
in AddDependency used by the data property, but it actually has a host
and a host-cross windows variant.  Modify osMutator to take the
OS-specific enabled properties into account to skip creating variants
that will immediately be disabled so there is a single variant.

Test: m checkbuild
Change-Id: Ic2daab29f4fa3a3797d7a08348fbfcf1036ec5dc
2020-11-17 14:24:24 -08:00
Colin Cross
34037c66ae Use bitfield for HostOrDeviceSupported
The HostOrDeviceSupported enums have become unwieldy, replace them
with bitfields to simplify checking if a specific trait is set.

Test: Soong arch tests
Test: m checkbuild
Change-Id: I884ddd092d4c6f740def611b6a055007dc24bd6e
2020-11-17 14:24:11 -08:00
Chris Parsons
8ccdb6358e Use packagepath and local_repository for mixed builds
This allows mixed builds to continue functioning even given the toplevel
WORKSPACE file containing toplevel_output_directories with out/.

Test: Manually verified on aosp_flame building libc.
Change-Id: I80fc4853421317e2d2c7f03d92d58286df1342ce
2020-11-17 15:42:36 -05:00