Commit Graph

2680 Commits

Author SHA1 Message Date
Colin Cross
1115ddc838 Merge "Revert "Refactor rate_limit.go for more clarify"" 2017-08-17 23:56:00 +00:00
Colin Cross
526416b1e4 Revert "Refactor rate_limit.go for more clarify"
This reverts commit 69f3b3e946.

Caused rare deadlocks.

Bug: 64536066
Bug: 64813447
Change-Id: Ieb1b931bb2c0afdd8bf8edbfc32c373df3c08d8d
2017-08-17 23:54:51 +00:00
Jeff Gaston
8cd393688b Merge "Run the Finder and make its results available to Kati" 2017-08-17 19:40:56 +00:00
Jeff Gaston
b90afd09d6 Merge "Refactor rate_limit.go for more clarify" 2017-08-17 01:31:21 +00:00
Treehugger Robot
8a55ec317d Merge "Export jar-args.sh script to make" 2017-08-17 00:48:37 +00:00
Jeff Gaston
69f3b3e946 Refactor rate_limit.go for more clarify
This wasn't intended to change the behavior, but it does slightly.

  Previously any requests to acquire memory wouldn't block; only
  (subsequent) requests for execution would block (if another
  caller had already consumed the memory quota). Now the requests
  for memory can also also block.

It turns out in a brief test on my computer that soong_zip
runs about 10 to 20% faster with this change than without it.

  The final step involving soong_zip decreased from about
  3.6 sec to about 3.3 sec in tests on my computer.

  When testing the process of re-zipping the contents of
  angler-target_files*.zip , the time decreased from about
  6.3 sec to about 5.3 sec in tests on my computer, and the
  peak memory usage reported by `top` decreased from about
  1.5g to 1g

Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: I0422e4c363c675bb7a93309fac4616c768bfbe8f
2017-08-16 15:54:25 -07:00
Jeff Gaston
b64fc1cab5 Run the Finder and make its results available to Kati
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.

Bug: 64363847
Test: m -j

Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
2017-08-16 14:56:00 -07:00
Jeff Gaston
16c04e0621 Merge "Have soong_zip add entries for ancestor directories" 2017-08-16 19:41:27 +00:00
Treehugger Robot
cefa501304 Merge "Fail the Finder in case of unexpected fs error" 2017-08-16 02:08:34 +00:00
Jeff Gaston
2d174132c4 Have soong_zip add entries for ancestor directories
Bug: 64536066

Test: When I run:

$ mkdir -p a/b/c
$ touch a/b/c/d
$ soong_zip -o test.zip -d -C . -f a/b/c/d
$ zipinfo -1 test.zip
a/b/c/
a/b/c/d

I should get:
a/
a/b/
a/b/c/
a/b/c/d

Change-Id: I779bb48062dfaf88a70fab08d8d98d1e366087b3
2017-08-15 18:57:39 -07:00
Colin Cross
972a160062 Merge changes I18acac41,Ib8baefed,I38e78bb9
* changes:
  Work around LLVM bug where TLS not emulated in LTO mode
  Add LTO support to soong
  Pass correct emulation string to gold
2017-08-16 00:13:00 +00:00
Jeff Gaston
fd4ce1bb68 androidmk no longer escaping escaped quotes in CFLAGS
Bug: 37547491
Test: m -j

Change-Id: I2f406be477f3990bf109778eb9adc9c33db2f2ad
2017-08-15 22:05:39 +00:00
Colin Cross
ded8a97c32 Export jar-args.sh script to make
Export jar-args.sh in JAR_ARGS so that make doesn't have to
duplicate the sed scripts.

Test: m -j checkbuild
Bug: 64634025
Change-Id: Ia5d0a303f23b32b8ff672fc882d51d5e99316ffa
2017-08-15 14:29:04 -07:00
Jeff Gaston
b629e184dd Fail the Finder in case of unexpected fs error
Permissions errors are ignored
Errors on pruned dirs are also ignored

Bug: 62455338
Test: m -j blueprint_tools # which runs unit tests
Change-Id: I8ba85fdd0295deb7dc374a851212e7c850e76b75
2017-08-15 13:18:24 -07:00
Stephen Crane
f5b9b95f01 Work around LLVM bug where TLS not emulated in LTO mode
In LTO mode the backend constructs a target exclusively from command line flags,
without opportunity to accept code gen arguments from Clang. Clang does not
currently pass the -emulated-tls parameter for Android, even though it defaults
to emulated TLS for Android in non-LTO builds. Until this is fixed upstream, we
can explicitly specify that the LTO output for Android should use emulated TLS.

Test: build libandroid with LTO, verified that it uses __emutls_get_address
rather than __tls_get_addr.

Change-Id: I18acac41aac885fc6635fbd55f96ba7c845eb5e7
2017-08-15 13:18:11 -07:00
Stephen Crane
ba090d16c2 Add LTO support to soong
Enabling the lto property for a module builds that module and all static
dependencies with LTO.

LTO (link-time optimization) allows the compiler to optimize and
generate code for the entire module at link time, rather than
per-compilation unit. LTO is required for Clang CFI and other
whole-program optimization techniques. LTO also allows cross-compilation
unit optimizations that should result in faster and smaller code, at the
expense of additional compilation time.

Test: make -j12 libc with lto: true for libc

Change-Id: Ib8baefedf60e02701d44673a7c473e0845730101
2017-08-15 12:54:12 -07:00
Stephen Crane
b3c2154f2a Pass correct emulation string to gold
Gold recognizes a different set of emulation strings to specify targets
than the bfd linker accepts. Clang only passes bfd emulations to the
linker, so we need to override these with the versions accepted by gold
when using gold.

Gold normally picks up the correct target from the first ELF input file
if it cannot parse the emulation parameter. However when using LTO, all
objects may be bitcode files, which causes gold to rely on the passed
--oformat or -m parameter to determine the proper target. If gold cannot
parse the emulation string passed by clang, it fails to link.

Test: build LTO version of libc

Change-Id: I38e78bb912fd3cc5fb7b4a762284f50ddd4f3998
2017-08-15 12:54:12 -07:00
Treehugger Robot
38d3d5d2e5 Merge "Convert soong java from soong_zip to jar" 2017-08-14 22:35:03 +00:00
Treehugger Robot
1d15c37766 Merge "Install java_binary wrappers in make" 2017-08-14 22:12:39 +00:00
Colin Cross
b852a58aa6 Convert soong java from soong_zip to jar
soong_zip produces jar files that are not compatible with java's
ZipInputStream.  Switch to jar using ugly sed scripts to munge
file lists into the alternating -C and file arguments required
by jar.

Bug: 64536066
Test: m -j checkbuild
Test: build/soong/scripts/jar_args.sh --test
Change-Id: Ifcc4bdab25e7d02342720eb246c673ff9a58bddb
2017-08-14 10:07:53 -07:00
Jiyong Park
a622de8498 cflags for vendor variants
cflags that are only applied when building the vendor variant of a lib
is now supported. This will be particularily useful when existing macro
should be turned off for the vendor variant:

cflags: ["-DENABLE_FEATURE_X"],

target: {
    vendor: {
        cflags: ["-UENABLE_FEATURE_X"],
    },
},

Without this, the library author has to undef the macro inside the
source code using __ANDROID_VNDK__, which is not desired especially for
external projects.

Bug: 64514237
Test: BOARD_VNDK_VERSION=current m -j libsqlite libsqlite.vendor
Merged-In: Ib484b5b12e14ab776276a30ed2763dffadc9d642
Change-Id: Ib484b5b12e14ab776276a30ed2763dffadc9d642
(cherry picked from commit 1e7495d973)
2017-08-14 11:46:42 +09:00
Jiyong Park
7447228e64 a shared lib header is not exported to vendors if it is in exclude_shared_libs
Since a lib in exclude_shared_libs are not used for the vendor variant,
it should also be removed from the export_shared_lib_headers.

Bug: 62471389
Test: BOARD_VNDK_VERSION=current m -j libxml2.vendor successful
(libxml2 has libicuuc in both exported_shared_lib_headers and
exclude_shared_libs for the vendor variant)

Merged-In: Ic9ddbd7be3c76df563e428ba2409a03bd2e1e113
Change-Id: Ic9ddbd7be3c76df563e428ba2409a03bd2e1e113
2017-08-14 11:40:54 +09:00
Colin Cross
10a0349d98 Install java_binary wrappers in make
Convert java_binary modules into two make modules, one for the
underlying java_library and one for the wrapper prebuilt.

Test: m -j checkbuild
Change-Id: I5ddf74f24f1e41fc1f39b3e8d254b7e191dbd47a
2017-08-11 22:22:48 -07:00
Treehugger Robot
b8245ea839 Merge changes Id9717132,I3e3b7251,I021c7971,I8a117a86,Ia5196f1f, ...
* changes:
  Allow AndroidMkData.Custom handlers to extend normal values
  Remove error from AndroidMkDataProvider.AndroidMk
  Remove error from AndroidMkData.Custom
  Remove error from AndroidMkData.Extra
  Bring java support closer to current version of make
  Rename java_prebuilt_library to java_import
2017-08-12 01:11:59 +00:00
Colin Cross
0f86d186b1 Allow AndroidMkData.Custom handlers to extend normal values
Pass accumulated AndroidMkData to AndroidMkData.Custom handlers
and expose WriteAndroidMkData so that Custom handlers can write
out the normal make variables and then add their own.

Test: No change to out/soong/Android-aosp_sailfish.mk
Change-Id: Id9717132bbd6c5cf3af8596f3eaa9bbb05d98e40
2017-08-11 15:24:11 -07:00
Colin Cross
a18e9cfa29 Remove error from AndroidMkDataProvider.AndroidMk
It's never anything except nil, and it unnecessarily complicates
the implementations.

Test: m -j checkbuild
Change-Id: I3e3b7251f32ffa84dbdfd0448faf248c306ca808
2017-08-11 15:24:11 -07:00
Colin Cross
91825d2279 Remove error from AndroidMkData.Custom
It's never anything except nil, and it unnecessarily complicates
the implementations.

Test: m -j checkbuild
Change-Id: I021c7971ede3e11bbe08cf1601f4690ed4d1a036
2017-08-11 15:24:11 -07:00
Colin Cross
27a4b05441 Remove error from AndroidMkData.Extra
It's never anything except nil, and it unnecessarily complicates
the implementations.

Test: m -j checkbuild
Change-Id: I8a117a86aa39aeb07d9d8d0686ef869c52784f19
2017-08-11 15:24:11 -07:00
Colin Cross
6416271a1f Bring java support closer to current version of make
Make the javac arguments match what is used by make, and export them
back to make.  A future change will switch make to use the the
exported ones.

This makes a dx.jar compiled with soong have identical class files
as one compiled with make.

Test: manual
Change-Id: Ia5196f1f42bc564e99de22e32e72fd2930e9fbae
2017-08-11 15:24:11 -07:00
Colin Cross
74d73e2bfb Rename java_prebuilt_library to java_import
And make it work like bazel's java_import, using a "jars" property
instead of "srcs", and allowing multiple jars to be listed.

Test: soong tests
Change-Id: Ida2ace6412bd77b4feb423646000a1401004e0ea
2017-08-11 15:24:11 -07:00
Colin Cross
6774e2893a Use --hash-style=both for NDK binaries
Old devices don't support gnu hashes, use --hash-style=both when
the sdk_version property is set.

Test: m -j checkbuild
Bug: 64608507
Change-Id: Iec5229a08fc192f1c8872466ee4bf866adf50443
2017-08-11 22:21:39 +00:00
Vishwath Mohan
c4cab48c26 Merge changes from topic 'revert_mutator_changes'
* changes:
  Revert "Fix ASAN mutator."
  Revert "Don't apply sanitizer mutators to host modules."
2017-08-11 14:47:17 +00:00
Treehugger Robot
c5d3abf107 Merge "List of VNDK-related libs are exported to make" 2017-08-11 01:38:15 +00:00
Vishwath Mohan
952293044f Revert "Fix ASAN mutator."
This reverts commit 12758d0cec.

Change-Id: I475e63d7e56ff166c2046fc423a1729ed098ff66
2017-08-11 00:56:44 +00:00
Vishwath Mohan
e615345f28 Revert "Don't apply sanitizer mutators to host modules."
This reverts commit bba1927bde.

Change-Id: Ie895d8eae5abb533468b836639e0835e7adbf8c9
2017-08-11 00:52:44 +00:00
Vishwath Mohan
e5d3ef4cc8 Merge "Don't apply sanitizer mutators to host modules." 2017-08-10 23:49:25 +00:00
Jeff Gaston
7943aebaa2 Merge changes Ib63e8e10,I9f730316
* changes:
  Darwin support for Finder
  Revert "Revert "Cacheable, multithreaded finder.""
2017-08-10 22:59:16 +00:00
Jeff Gaston
9f6da315aa Darwin support for Finder
Bug: 62455338

Test: m -j # while on Mac

Change-Id: Ib63e8e10bddcb3b507878364218273e8a5a8aa82
2017-08-10 14:22:16 -07:00
Jeff Gaston
f1fd45e784 Revert "Revert "Cacheable, multithreaded finder.""
Bug: 62455338
Test: m -j

This reverts commit d1abeb9d98.

Change-Id: I9f73031636157511b5f1c6ce8a205e9bc91669ff
2017-08-10 12:20:30 -07:00
Vishwath Mohan
bba1927bde Don't apply sanitizer mutators to host modules.
This CL blocks modules intended for the host from having the bottom up
sanitizer mutator applied. We only maintain a single copy of host
binaries, so generating variants doesn't make sense
anyway. Additionally, the existing logic was causing an error with
builds not embedded in make (eg: aosp-build-tools which uses
soong_ui.bash) where the wrong variant was installed on the host. This
change should fix that.

Bug: 64536751
Test: m -j40 && SANITIZE_TARGET="address" m -j40
Test: aosp-build-tools build breakage is fixed.
Change-Id: Ia79b4661f69ce26a7be17c4339c18ca0f397d760
2017-08-10 11:05:00 -07:00
Jiyong Park
d5b18a55bb List of VNDK-related libs are exported to make
LL-NDK, VNDK-core, VNDK-SP libraries are exported to make as
SOONG_LLNDK_LIBRARIES, SOONG_VNDK_CORE_LIBRARIES, and
SOONG_VNDK_SAMEPROCESS_LIBRARIES. This can be used to auto-generate
ld.config.txt from a template.

Bug: 64013660
Test: BOARD_VNDK_VERSION=current m -j successful
Test: check out/soong/make_vars*.mk and look for SOONG_*_LIBRARIES

Merged-In: I0f4c5d05d9cd28c3fc9fdcca6ce0e6eaeaacbe8d
Change-Id: I0f4c5d05d9cd28c3fc9fdcca6ce0e6eaeaacbe8d
2017-08-10 14:15:39 +09:00
Vishwath Mohan
46c6a171aa Merge "Fix ASAN mutator." 2017-08-10 02:42:31 +00:00
Jeff Gaston
c3f32070dc Merge "Revert "Cacheable, multithreaded finder."" 2017-08-10 01:21:09 +00:00
Jeff Gaston
d1abeb9d98 Revert "Cacheable, multithreaded finder."
This reverts commit b6d161bf16.

Reason for revert: New Build Breakage: aosp-master/sdk_mac @ 4260825

Change-Id: I8bda8c50c5e5c9f84621d11a4c15b168833bcd21
2017-08-10 01:16:34 +00:00
Vishwath Mohan
12758d0cec Fix ASAN mutator.
This CL fixes a bug in the ASAN top down mutator which was incorrectly
checking (and setting) SanitizeDep to true only for the parent module
and not for the visited child modules.

Fixing this also requires some changes to the variant creation logic
to ensure that the correct variant is passed to make for
installation. This will eventually be replaced by logic that appends
an appropriate suffix to sanitized libraries (eg: libc++.asan), which
will allow both variants to be passed to make.

Bug: 64536751
Test: m -j40 && SANITIZE_TARGET="address" m -j40
Change-Id: Id5e5a5946192adf07418dd433bca503047177007
2017-08-09 16:58:36 -07:00
Jeff Gaston
c17e048bad Merge "Cacheable, multithreaded finder." 2017-08-09 22:18:13 +00:00
Jeff Gaston
ccb58c1eb3 Merge "Move build system documentation" 2017-08-09 22:17:43 +00:00
Jeff Gaston
9ec0117915 Merge "Point root.bp at build/make/tools instead of build/tools" 2017-08-09 19:43:40 +00:00
Jeff Gaston
df4a081292 Move build system documentation
Bug: 62201421

Test: make help
Change-Id: I7d7b448d3f736e171ebfbdceee46eef7de748742
2017-08-09 19:40:01 +00:00
Dan Willemsen
e0879fc3ae Add --skip-make to replace Soong's blueprint wrapper with soong_ui
This way we only have one way to start a build, which always has logging
/ tracing / etc, even if we don't need Kati.

There's two ways to use this:

As a direct replacement for mkdir out; cd out; ../bootstrap.bash;
./soong -- as long as --skip-make is always passed, we'll never run
Kati, and Soong will run outside of it's "make" mode. This preserves
most of the speed, and allows full user control over the Soong
configuration.

A (experimental, dangerous) way to temporarily bypass the product
variable and kati steps of a build. As long as a user is sure that
nothing has changed from the last build, and they know exactly which
Ninja targets they want to build (which may not be the same as the
arguments normally passed to 'm'), this can lead to shorter build
startup times.

Test: rm -rf out; m --skip-make libc
Test: rm -rf out; m libc; m --skip-make libc
Test: rm -rf out; mkdir out; cd out; ../bootstrap.bash; ./soong libc
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: Ic0f91167b5779dba3f248a379fbaac67a75a946e
2017-08-08 22:49:28 -07:00