Commit Graph

7575 Commits

Author SHA1 Message Date
Kelvin Zhang
67553739e3 Fix check_tf failure when duplicate package names are present
Test: check_target_files_signatures sdk_gphone64_x86_64-target_files-8069754.zip
Bug: 214388458

Change-Id: I0a8a0d0596c04a9da4283d4d4caeb3bb62762ccc
2022-01-13 12:30:22 -08:00
Kelvin Zhang
f2728d615e Add option to enable lz4diff
When lz4diff is enabled, we inject the host copy of liblz4.so in
target_file to delta_generator. This is to ensure that host tooling can
produce same compressed output as the device.

Test: th
Bug: 206729162

Change-Id: I2d8206f7ec54fadedea16bf3d811b8353bc9414d
2022-01-12 17:21:00 -08:00
Treehugger Robot
6095899cc6 Merge "Regression testcase for shipped vs nonshipped." am: 3a0656307c am: 192875fdfc am: 6300667501 am: b928cd6011
Original change: https://android-review.googlesource.com/c/platform/build/+/1946016

Change-Id: I54d0761adb5c3a34f028624ed2d0c3bae1d18910
2022-01-13 00:45:27 +00:00
Treehugger Robot
3a0656307c Merge "Regression testcase for shipped vs nonshipped." 2022-01-12 23:35:17 +00:00
Treehugger Robot
e74a41a8b9 Merge "Fix typo in doc: code segment for container dot" am: 710813e871 am: 1dd148f737 am: f79ab4b3a3 am: 00195a11bf
Original change: https://android-review.googlesource.com/c/platform/build/+/1946012

Change-Id: I7a4dc7026406c3e9adb8a17d79c369556a04349e
2022-01-12 23:27:18 +00:00
Treehugger Robot
710813e871 Merge "Fix typo in doc: code segment for container dot" 2022-01-12 22:31:40 +00:00
Treehugger Robot
adf5feeda2 Merge changes Ia2ec1b81,Ibc831ae8 am: 49380ebecf am: 5955f8a521 am: 4817ded07b am: 2e04c83b6b
Original change: https://android-review.googlesource.com/c/platform/build/+/1941454

Change-Id: I54efe335ac9d3ceea817c176872249be701900f4
2022-01-12 01:41:09 +00:00
Treehugger Robot
022e8e293f Merge "Fold annotations." am: b3f3e7b3f7 am: e27ecb85d4 am: b878b8ff56 am: 04b2d70747
Original change: https://android-review.googlesource.com/c/platform/build/+/1944765

Change-Id: Ie95025ef51be307a97b699d8659e91036fd0525d
2022-01-12 01:40:48 +00:00
Bob Badour
00bf3d9b29 Regression testcase for shipped vs nonshipped.
Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I17221ec8fe429b5f745c0733f8bb1ab0ea2f4567
2022-01-11 16:38:55 -08:00
Treehugger Robot
49380ebecf Merge changes Ia2ec1b81,Ibc831ae8
* changes:
  Performance and scale.
  Use struct{}
2022-01-12 00:27:41 +00:00
Treehugger Robot
b3f3e7b3f7 Merge "Fold annotations." 2022-01-12 00:27:35 +00:00
Bob Badour
99dba55b4b Fix typo in doc: code segment for container dot
Test: not applicable
Change-Id: Ibaadc6e8fd4a267b50a46feb2dd6278ff1f5ace1
2022-01-11 14:46:22 -08:00
Bob Badour
103eb0f9bc Performance and scale.
Defer edge creation.

Don't create edges until the count is known to avoid repeated allocate+
copy operatios.

Limit resolutions.

Allow only a single resolution condition set per target, and overwrite
intermediate results. Reduces memory and obviates allocations.

Propagate fewer conditions.

Instead of propagating notice conditions to parents in graph during
initial resolve, leave them on leaf node, and attach to ancestors in
the final walk. Reduces copies.

Parallelize resolutions.

Use goroutines, mutexes, and waitgroups to resolve branches of the
graph in parallel. Makes better use of available cores.

Don't accumulate resolutions inside non-containers.

During the final resolution walk, only attach actions to ancestors from
the root down until the 1st non-aggregate. Prevents an explosion of
copies in the lower levels of the graph.

Drop origin for scale.

Tracking the origin of every potential origin for every restricted
condition does not scale. By dropping origin, propagating from top
to bottom can prune many redundant paths avoiding an exponential
explosion.

Conditions as bitmask.

Use bit masks for license conditions and condition sets. Reduces maps
and allocations.

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Ia2ec1b818de6122c239fbd0824754f1d65daffd3
2022-01-11 10:40:50 -08:00
Bob Badour
5446a6f8e1 Use struct{}
Using struct{}{} as the payload for set maps reduces memory use for
large sets.

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Ibc831ae80fc50f35e1000348fb28fc0167d0ebed
2022-01-11 10:40:50 -08:00
Bob Badour
67d8ae390a Fold annotations.
Use annotation slices looked up from map instead of read from disk to
reduce copies.

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Ibf6f7d319092e0d54ab451ffbbd35dcd7d0080e0
2022-01-11 10:40:39 -08:00
Treehugger Robot
28e245f711 Merge "Do not append .meta_lic to meta_lic" am: 9d83209f12 am: 0f12de8954 am: 9212d765d1 am: 4250af6e75
Original change: https://android-review.googlesource.com/c/platform/build/+/1944764

Change-Id: I9f7a7d1824bd2ac198b34a26a61537e4c6242000
2022-01-11 07:17:58 +00:00
Treehugger Robot
9d83209f12 Merge "Do not append .meta_lic to meta_lic" 2022-01-11 06:11:14 +00:00
Bob Badour
63a281cc29 Do not append .meta_lic to meta_lic
Test: manual
Change-Id: I8d7162dc55f057d542faaa7feb546050d3a872e7
2022-01-10 18:00:03 -08:00
Victor Hsieh
9e12870560 Merge "Add min_sdk_version to BuildManifest.apk" am: 70ca7c6add am: 30afe79e30 am: 829fe6d623 am: 271ca249c3
Original change: https://android-review.googlesource.com/c/platform/build/+/1941946

Change-Id: I2c5281988a1bc9c14340d207c71275565eff866a
2022-01-10 21:17:34 +00:00
Jiyong Park
efbb6ffa00 Add min_sdk_version to BuildManifest.apk
sign_target_files_apks demands it.

Bug: 213823227
Test: run com.google.android.tradefed.ota.SigningToolTest#Testcase1_Signingtest
Change-Id: I1bba2b25a71449a57d284024e77fb76d1b79a0ee
2022-01-10 20:14:05 +09:00
Treehugger Robot
e7edcdc66d Merge "Add option to enable zucchini" am: dcd7e283d2 am: 3468085607 am: cd9f83eeb8 am: 2f5176647a
Original change: https://android-review.googlesource.com/c/platform/build/+/1939053

Change-Id: I0e155c9433a51c78a4fd63de68581ab0135f08db
2022-01-09 21:46:07 +00:00
Treehugger Robot
dcd7e283d2 Merge "Add option to enable zucchini" 2022-01-09 20:38:18 +00:00
Martin Stjernholm
6c0015b95e Merge "Add an option to specify the path to aapt2." am: 76537be0a9 am: 96cea0f44b am: a6e2ff4804 am: 948e8c78ab
Original change: https://android-review.googlesource.com/c/platform/build/+/1938648

Change-Id: Ie2c28d23ce90ac36cc2258192b6ea51cb7366511
2022-01-09 20:30:51 +00:00
Martin Stjernholm
76537be0a9 Merge "Add an option to specify the path to aapt2." 2022-01-09 19:11:49 +00:00
Treehugger Robot
82ad6d35f4 Merge "Write signature size even when it's empty" am: 555c3a7dc7 am: 77f7f00e3d am: 5c8ad6415b am: b7015798b9
Original change: https://android-review.googlesource.com/c/platform/build/+/1938523

Change-Id: I824c50ffea8cd8e122d473b068520d5faf3df014
2022-01-08 03:17:21 +00:00
Treehugger Robot
8d77820a48 Merge "Fix GetCareMap for small images" am: 8ef44309b2 am: 2dbcae4a5a am: 04d6278863 am: bcf6567964
Original change: https://android-review.googlesource.com/c/platform/build/+/1940646

Change-Id: Ibf3103142e5bb05c4e0e8d5df3b6916e2341d39e
2022-01-08 03:17:07 +00:00
Treehugger Robot
555c3a7dc7 Merge "Write signature size even when it's empty" 2022-01-08 02:20:11 +00:00
Martin Stjernholm
58472e8609 Add an option to specify the path to aapt2.
Thin manifests (e.g. master-art) may not have the aapt2 sources and
instead relies on the prebuilt in prebuilts/sdk/tools/linux/bin. In
that case there's no aapt2 in the search path, so it needs to be
specified.

Test: Heavy presubmit build on mainline_modules_bundles on git_master
Test: vendor/google/modules/ArtGoogle/build-art-module.sh
  with http://ag/16584845 on git_master-art, where only the prebuilt
  aapt2 is present
Bug: 212462945
Change-Id: I41a22e8146f5a88534c2721345b4d9d64f76698c
2022-01-07 23:08:53 +00:00
Kelvin Zhang
410bb389d8 Add option to enable zucchini
Change-Id: Id952826c9c5af912fba679af61f2a2e1645641dd
2022-01-07 14:59:01 -08:00
Kelvin Zhang
98ef7bb157 Fix GetCareMap for small images
Change-Id: Iac518e36dbb2fe9303e84d502cd7dfe74621c0af
2022-01-07 14:59:01 -08:00
Treehugger Robot
cfbdb4bd4a Merge "Move fsverity metadata generation to Makefile" am: 7c88c9c1e6 am: 2d6411ed06 am: eea503cf81 am: 9c4a78ceae
Original change: https://android-review.googlesource.com/c/platform/build/+/1937129

Change-Id: I8a3b44b5cd20af1089dccd61c041e8cd97638fda
2022-01-07 04:39:34 +00:00
Treehugger Robot
7c88c9c1e6 Merge "Move fsverity metadata generation to Makefile" 2022-01-07 03:37:16 +00:00
Treehugger Robot
1f087d2f37 Merge "Avoid packing ramdisk into GKI boot-*.img" am: bd5fd25265 am: a551c5d3d1 am: 2ba4030ac0 am: 447d5ec8ea
Original change: https://android-review.googlesource.com/c/platform/build/+/1937120

Change-Id: Ia1a40496d3bae1c969a403f63d54291718a0a25c
2022-01-07 03:35:54 +00:00
Treehugger Robot
bd5fd25265 Merge "Avoid packing ramdisk into GKI boot-*.img" 2022-01-07 02:35:06 +00:00
Inseob Kim
3d5ae80f50 Write signature size even when it's empty
Normally this has no effect, but when we generate metadata for small
files (<=4KB), merkle tree isn't generated. In such case, writing zero
will make the metadata format simpler and unconditional.

Test: manual
Change-Id: Ibe18175b580af3409c896a8bb97323792ad9c459
2022-01-07 10:56:17 +09:00
Inseob Kim
135c1f144f Move fsverity metadata generation to Makefile
build_image.py has been handling fsverity metadata generation in the
packing step, but it can cause issues because the metadata files are
missing in the $OUT directory, and they only exist in result system.img.
This change moves the generation logic into Makefile, and makes the
metadata tracked by ninja graph.

Bug: 206326351
Test: PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true and build
Change-Id: I1f910d8ac6e2cc3c54f35916871733c632f18e44
2022-01-07 09:16:42 +09:00
Bowgo Tsai
88fc2bdd19 Avoid packing ramdisk into GKI boot-*.img
If init_boot.img is present, the GKI boot-*.img should
not include the ramdisk because the ramdisk will be packed
into the init_boot.img instead.

The "has_ramdisk" flag incorrectly checks the condition of:
prebuilt_name != "boot.img" to see if it is a boot.img.
Because "has_ramdisk" was added before we packed multiple
GKI boot-*.img, e.g., boot-5.10.img, boot-5.10-lz4.img, etc.,
into the target files.

Fix this by checking the partition_name is "boot" or not.
Also moving the logic into a new function with comments for each
condition.

Bug: 203698939
Bug: 213028932
Test: sign_target_files_apks \
        --gki_signing_key=external/avb/test/data/testkey_rsa4096.pem \
        --gki_signing_algorithm=SHA256_RSA4096 \
        ./out/dist/*-target_files-eng.*.zip signed.zip, then
        unpack_bootimg to checks the signed boot-*.img has no ramdisk
Change-Id: I5354669feb54d547dbe797e51b1b1baa187cb7cf
2022-01-06 17:50:04 +08:00
Jacky Liu
0e3352a241 Merge "Add PRODUCT_EXTRA_OTA_KEYS to add extra ota keys in otacerts.zip" am: bff997fd1f am: 4f9eb6b9e4 am: 429e1f339b am: 5f1bdb32ae
Original change: https://android-review.googlesource.com/c/platform/build/+/1934214

Change-Id: Ic9e4bdd75e95d6df1595e1371977a532ae830db3
2022-01-06 07:37:19 +00:00
Treehugger Robot
8a3ac7b505 Merge "Fix the signing error on no-system-image targets" am: 790214a4c9 am: ba922579b1 am: 2a00929dec am: 50bf9c8a11
Original change: https://android-review.googlesource.com/c/platform/build/+/1936199

Change-Id: Ie9d4557587a1df9267d1a00ee97d4f46ec8d3ac2
2022-01-06 07:37:07 +00:00
Jacky Liu
bff997fd1f Merge "Add PRODUCT_EXTRA_OTA_KEYS to add extra ota keys in otacerts.zip" 2022-01-06 06:31:37 +00:00
Treehugger Robot
790214a4c9 Merge "Fix the signing error on no-system-image targets" 2022-01-06 06:27:17 +00:00
Treehugger Robot
c81279b860 Merge "Show directories and files with most warnings." am: b2610d7ecd am: 756adcf38b am: c7142be738 am: 152176c056
Original change: https://android-review.googlesource.com/c/platform/build/+/1927679

Change-Id: I63eccb78ac4c81b63371e017316a8cef8847737f
2022-01-06 04:13:51 +00:00
Chih-Hung Hsieh
d4691ecfb6 Show directories and files with most warnings.
* Now with 4 sections that can be shown/hidden independently.
* After the 'selected_projects' section, add two more sections:
  * 'top_directory_section' contains
    * a table of directories with at least 1% warnings.
    * a 'selected_directory_warnings' subsection to show all warnings
      in the clicked directory from the directory table
  * 'top_file_section' contains
    * a table of files with at least 100 or 1% warnings.
    * a 'selected_file_warnings' subsection to show all warnings
      in the clicked file from the file table
* Adjust button/section spaces and reduce button font size to 100%.
* Rename drawTable to genTables.

Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Change-Id: I765b09a46adc111cfe781719ba7aa0f917aa6ffc
2022-01-05 15:43:52 -08:00
Wei Su
cf2d0e0ef3 Merge "Write files in a temporary directory" am: 22ecce0658 am: 5d03df4516 am: 8623e17527 am: e6551e2973
Original change: https://android-review.googlesource.com/c/platform/build/+/1920697

Change-Id: I6b698f48bf6187b1873d46a707ff9e12f4efb24a
2022-01-05 21:04:45 +00:00
Wei Su
22ecce0658 Merge "Write files in a temporary directory" 2022-01-05 20:03:07 +00:00
Treehugger Robot
a95a2a82a7 Merge "Convert generate-self-extracting-archive.py to python 3" am: 7ca77ea994 am: ecf39825f0 am: 053bce1c76 am: bf0e5be139
Original change: https://android-review.googlesource.com/c/platform/build/+/1935067

Change-Id: I9e294ea8b3d5a59ef53076f7c053762f1796671b
2022-01-05 20:02:24 +00:00
Treehugger Robot
7ca77ea994 Merge "Convert generate-self-extracting-archive.py to python 3" 2022-01-05 18:58:58 +00:00
Jacky Liu
beb0b6927c Add PRODUCT_EXTRA_OTA_KEYS to add extra ota keys in otacerts.zip
Equivalent to PRODUCT_EXTRA_RECOVERY_KEYS but for A/B OTA.

Bug: 211848136
Test: set PRODUCT_EXTRA_OTA_KEYS and check otacerts.zip
Change-Id: I81e27d12a22b405f6227b09c01ed684dfcede19e
2022-01-05 17:54:40 +08:00
Bowgo Tsai
8d4b72405e Fix the signing error on no-system-image targets
Currently when running sign_target_files_apks on a no-system-image
target, it will raise the following error:

  ValueError: max() arg is an empty sequence

This is because there is no APK files in the target_files.zip.
Fixing this by setting maxsize to zero in this case.

Bug: 213028932
Test: lunch gki_arm64-userdebug; make dist
Test: sign_target_files_apks \
        --gki_signing_key=external/avb/test/data/testkey_rsa4096.pem \
        --gki_signing_algorithm=SHA256_RSA4096 \
        --gki_signing_extra_args="--prop gki:prop1 --prop gki:prop2" \
        ./out/dist/*-target_files-eng.*.zip signed.zip
Change-Id: I40daecbc2ff3f89d3e635d1a4a1c1dea31ba9a27
2022-01-04 17:55:19 +08:00
Cole Faust
547ca205ae Convert generate-self-extracting-archive.py to python 3
Test: Add a VENDOR_BLOBS_LICENSE entry to an existing product, then `m with-license`
Bug: 203436762
Change-Id: I4bbe77bda7789b6c44bea141518b1cd2e699d326
2021-12-29 13:58:28 -08:00