Allow filename collisions for directories, which may happen if
multiple globs of resources include the same directory names.
Continue to report errors if collisions occur between files, and
also add checks for collisions between files and directories.
Test: manual
Change-Id: Iac19fbd325c53fbb41552ea230d813c8dddf9439
Combining the results of a javac command into a jar needs to take
a whole directory, add -D to specify a directory to search for all
files.
Test: m -j checkbuild
Change-Id: Id4707189d57b664f0262a0782630f893af397900
This will enable writing a modified manifest without having to create a temporary file first
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
# make sure that the output is binary equal
# with and without this patch
Change-Id: I559d653e0e72e641e1ee6745924cb835bb0a355b
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
zipdetails /tmp/out.zip | less \
# and check that the first entry contains the \
# "CAFE" extra attribute as shown below:
0000001E Filename 'META-INF/'
00000027 Extra ID #0001 CAFE 'Java Executable'
00000029 Length 0000
Change-Id: I12c4078591f2ce2afc1af5b9db20393b26601bca
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
zipdetails /tmp/out.zip | less \
# and check that the META-INF entries are earlier in \
# the list than other entries
Change-Id: Id5c6ea9ce8c3a6fbfb8366db753e6603a076dbf8
This reverts commit 526416b1e4.
Figured out a fix for the deadlocks; resubmitting the patch.
The first version was deadlocking because the switch statement in
zipWriter.write would choose a specific zip entry to start writing,
but the individual chunks may not have all necessarily been compressed
yet. When each individual chunk was made to require to request its own
allocations, the compression of the chunks of the file being currently
written could be blocked waiting for memory to be freed by chunks from
other files that hadn't yet started being written.
This patch is much like the original except it preallocates the memory
for the entire file upfront (and happens to use the total file size
rather than the compressed size, but I didn't observe that to cause any
performance differences).
Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: Id1d7ff415e54d3a6be71188abbdbbbab5a719fcf
This wasn't intended to change the behavior, but it does slightly.
Previously any requests to acquire memory wouldn't block; only
(subsequent) requests for execution would block (if another
caller had already consumed the memory quota). Now the requests
for memory can also also block.
It turns out in a brief test on my computer that soong_zip
runs about 10 to 20% faster with this change than without it.
The final step involving soong_zip decreased from about
3.6 sec to about 3.3 sec in tests on my computer.
When testing the process of re-zipping the contents of
angler-target_files*.zip , the time decreased from about
6.3 sec to about 5.3 sec in tests on my computer, and the
peak memory usage reported by `top` decreased from about
1.5g to 1g
Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: I0422e4c363c675bb7a93309fac4616c768bfbe8f
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.
Bug: 64363847
Test: m -j
Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
Fix mutator registration for tests to allow different tests
in the same package to register different mutators.
Allow tests to track the resulting ModuleBuildParams objects
to use in assertions, and provide helpers for getting them.
For example:
config := android.TestConfig(buildDir)
ctx := android.NewTestContext()
ctx.RegisterModuleType(...)
ctx.MockFileSystem(...)
ctx.ParseBlueprintsFile("Android.bp")
ctx.PrepareBuildActions(config)
ctx.Register()
// Get the Inputs value passed to the javac rule for the foo module
inputs := ctx.ModuleForTests("foo".Rule("javac").Inputs
Test: java_test.go
Change-Id: I10c82967f5f3586d2c176f169906b571ed82fc73
This was causing android/soong/ui/build to rebuild sometimes when it
didn't need to.
Test: `m -j nothing; grep "B compile" out/.soong_ui.trace` repeatedly
Change-Id: I5d33271a7b037f53674a0d312d8da1746eed8eaa
It will cd to $TOP before running soong_ui itself, so that soong_ui
still runs from the top of the tree. ORIGINAL_PWD is saved so that we
can reference that later (for example, to move 'mma' implementation into
Go).
Test: cd system; ../build/soong/soong_ui.bash
Test: Set absolute / relative OUT_DIR or OUT_DIR_COMMON_BASE and repeat
Change-Id: Icb67a3ee6a3358cca50300755c8953419fc19437
Jars have a strange sorting order; the META-INF/ directory should
come first, then META-INF/MANIFEST.MF, then any other files in
META-INF/, and then any files outside META-INF. Add a -j argument
to zip2zip that sorts using jar ordering.
Test: zip2zip_test
Change-Id: I80e2bc7e284ef74f6561c26cb6541298834db1bc
When '-s' is passed, any globbed arguments will have their results
sorted. When there are multiple arguments, the files will still be
inserted in argument order.
A bare '**' is now special cased to mean every file in the input zip.
This allows zip2zip to sort entire zip files efficiently by using
`zip2zip -s -i <> -o <> '**'`. This can be useful if your original zip
program used filesystem ordering which was not reproducible.
Test: m -j blueprint_tools (new tests pass)
Change-Id: Ic3512c5fe14c94c6f3e134296905121d2ff8b58a
Cleanup of some failure messages
Also, this leaves the temp directory untouched if a declared output was not created
Bug: 35562758
Test: make
Change-Id: I8ef1315af80eb327752501f12a331dbdf52ba3e9
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.
Bug: 35562758
Test: make
Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
This is still fairly simplistic, but good enough to test some build
system changes.
Bug: 33381544
Test: run, inspect output
Change-Id: Ia5c19570493116dca01cb65605cdf20becf8c1d0
Adds a -dist argument that will use DIST_DIR to save logs.
Also writes a summary of each std.log to stderr on errors, so that the
error is more likely to show up in the error reporting. This output is
prefixed with "> " to differentiate it from the progress reports from
multiproduct_kati itself.
Test: multiproduct_kati -only-config
Test: DIST_DIR=dist build/soong/build_test.bash -dist (introducing errors)
Change-Id: I5005b5f3f200c876bc004dd9b0e01e7b6edf5be2
Move common microfactory functions to cmd/microfactory/microfactory.bash
so that they can be used to build both soong_ui for normal build and
multiproduct_kati for build system tests.
Test: m -j
Test: build/soong/build_test.bash
Change-Id: I9512642d846ce54d05a027b6d33a2b3029b3f90b
It now uses the same output style as ninja, overwriting status lines in
smart terminals.
Test: multiproduct_kati
Test: multiproduct_kati | cat
Change-Id: I8db5198ffdc5ebc5503241ac492379753d92978e
1. Added ability to keep the mixed "-f"/"-l" order as same as
command-line flags order.
2. Added "-s" flag to specify which target file within zip is stored uncompressed.
Test: manual
Change-Id: I338b25a7bd6bf1b7e9cc29ad3324575167630fb7
Piping the output of javac through a filter makes it hard to capture
the exit status. Convert it to a wrapper that executes javac and
propagates the exit status.
Bug: 36666657
Test: javac_wrapper_test
Change-Id: I9b56cc3794023aabc9328138a68830e26e980f97
Naming directories according to current date. This makes them easier to
parse when using the tool multiple time.
Also considered:
millis := time.Now().UnixNano() / 1000000
name := fmt.Sprintf("multiproduct-%d", millis)
Also considered putting separators in the number, but it makes for too
long of a directory name.
Test: multiproduct_kati
Change-Id: I10ac6754094653abf5bf11b04efc3c44905d3c8d
IsDir() doesn't handle the case where the file is a symlink to a
directory, which cause fileslist to crash.
The hash is used to validate whether system image is the same, so
hashing the content of symlink makes more sense.
Bug: 36274890
Test: joule builds
Change-Id: I6359418a5b28f8da13f85b01a30a72228fecf4ce
Added -P flag in soong_zip:
It provided a path prefix which will be added in front after -f/-l was
applied with -C.
eg. ./soong_zip -o /path1/../ -P a/c -C d/e -f d/e/test.txt
zip_dest: a/c/test.txt
Bug: b/31676493
Test: go build -a to build a real soong_zip, and test it with real
files.
Change-Id: Ic924ba252e963ca5847c370f733058888426b696
We can start removing out directories again in multiproduct_kati, since
the opendir bug has been fixed.
Add --color_warnings to the Kati command line. Since this is different
from Make, take this opportunity to reorder the command line to make
more sense. This wasn't done before because kati forces a regen whenever
the command line changes.
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I5ad03359fbc16db482722946202297c1ae0f2b90
That will be the go version at compile time. So read $GOROOT/VERSION, or
fall back to executing `$GOROOT/bin/go version` to find the go version
currently in GOROOT.
Test: Ensure everything rebuilds when switching between go1.8rc2 and go1.8
Change-Id: I8738a7aa249a088b1e0668af260fa3974844dab7
Some javac wrappers output the entire list of java files being
compiled on a single line, which can be very large, set the maximum
buffer size to 2MB.
Test: pipe the grok build output through soong_javac_wrapper
Change-Id: Ib208461dca02ced4959433b7695ec760dc134468
The soong_javac_filter tool will take the output of the javac tool
on stdin and produce a colorized version similar to what clang
produces on stdout. It also strips the useless "warning there are
warnings" messages:
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Modules that want the warnings can pass the appropriate -Xlint
options.
Test: javac_filter_test
Change-Id: Ic8d337d95ae6c48146771f2982fd1316cb7d82be
