Commit Graph

7575 Commits

Author SHA1 Message Date
Tianjie Xu
c67a611103 Merge "Temporarily remove the care_map in the ota package" am: 3fcc52519e am: c8e5f1de93 am: 4cd4b0c572 am: 16b4beb644 am: edac9f8897
Original change: https://android-review.googlesource.com/c/platform/build/+/1883497

Change-Id: Ief5ef5829293048b82ebfdc3425fe91e9c7d74d9
2021-11-09 21:39:46 +00:00
Cole Faust
391cbc69b2 Merge "Use python 3 for release tools" am: ece722ca92 am: 9220d1a841 am: 454757ca83 am: 6df2faf751 am: f05bbe47e1
Original change: https://android-review.googlesource.com/c/platform/build/+/1883069

Change-Id: I1db1436cfef34ce4d0c4bff7e91556eb3cf564a1
2021-11-09 21:39:16 +00:00
Tianjie Xu
3fcc52519e Merge "Temporarily remove the care_map in the ota package" 2021-11-09 19:38:14 +00:00
Kelvin Zhang
4b588cf75a Skip signing key check in some cases
Test: th
Bug: 205657204

Change-Id: I3a2fc886cfb3fe3d53619bfb8fe84aa799d80de3
2021-11-09 08:43:01 -08:00
Tianjie
5856cfbf82 Temporarily remove the care_map in the ota package
The care_map has incorrect size for non-sparse images. Temporarily
removes it until the root cause is fixed

Bug: 205541521
Test: build
Change-Id: I76bcd2c0c778566b4a6c69b6d45428952225d406
2021-11-08 22:55:53 -08:00
Cole Faust
c76931fe8b Use python 3 for release tools
The release tools have already been updated in prior cls to
support python 3. To test this, I added code to print the script
name to a temp file at the beginning of every script, and then
ran various builds, ensuring that the scripts were run afterwards.

The builds run:
m brillo_update_payload checkvintf minigzip lz4 toybox unpack_bootimg deapexer (needed for releasetools_test)
m (as both sdk_phone_x86_64 and beagle_x15)
m target-files-package (as both sdk_phone_x86_64 and beagle_x15)
m releasetools_test && out/host/linux-x86/nativetest64/releasetools_test/releasetools_test (25 errors with and without python3)
m check_target_files_signatures apksigner target-files-package && out/host/linux-x86/bin/check_target_files_signatures out/target/product/emulator_x86_64/obj/PACKAGING/target_files_intermediates/sdk_phone_x86_64-target_files-eng.colefaust.zip
m dist
As aosp_cf_x86_64_phone: m dist && sign_target_files_apks  out/dist/aosp_cf_x86_64_phone-target_files-eng.$USER.zip /tmp/signed_target_files.zip && validate_target_files /tmp/signed_target_files.zip

This hit all the scripts except for make_recovery_patch,
merge_builds, and ota_package_parser.

I couldn't find anything that uses merge_builds, so it must
only be run manually.

make_recovery_patch and ota_package_parser are only run if
TARGET_OTA_ALLOW_NON_AB is true (among other things), which
is not the case for any product in aosp.

Test: Described in commit message + presubmits
Change-Id: I1a29eafa7ff1a69973b27055e311de77f7ee628b
2021-11-05 13:47:28 -07:00
Cole Faust
6f396911fc Merge "Prepare release tools for python 3" am: fdb8f17a83 am: ad943ea6df am: fecb7bb3a4 am: 48aac85b06 am: 67fdf2968b
Original change: https://android-review.googlesource.com/c/platform/build/+/1874167

Change-Id: I7a5aefb927ae70a86cb9d3f4b69593c37db536f6
2021-11-05 18:36:07 +00:00
Cole Faust
b820bcd829 Prepare release tools for python 3
- Sort dictionaries before looping over them
- Don't call sorted() on lists with Nones
- Open file in binary format when serializing protobufs

Change-Id: If5dbc908f7125f6184014b3c1c7891f833d1d8bf
Bug: 203436762
Test: Presubmits
2021-11-04 17:58:44 -07:00
Treehugger Robot
6ebf09d040 Merge "Fix a failure in signing test" am: 936e704999 am: 5c8fc606d4 am: aeba7359a0 am: 86250621a3 am: ce19509a15
Original change: https://android-review.googlesource.com/c/platform/build/+/1879016

Change-Id: Ib3106fa7f14873ad1b0d5d11b06a2e570bc7a3f4
2021-11-04 10:37:42 +00:00
Treehugger Robot
c62a01004a Merge "Add a compressor types option" am: 1e392720c2 am: a0c34f2c21 am: 44c33933c0 am: 99a9435e6c am: cd3ca41f20
Original change: https://android-review.googlesource.com/c/platform/build/+/1873795

Change-Id: I490cf27520ac8e85e20ca8e790c35ad512683610
2021-11-04 05:43:24 +00:00
Kelvin Zhang
2639048c21 Fix a failure in signing test
Pixel moved away from sparse images, so validate_target_files is failing
because it expects sparse images.

Test: th
Change-Id: I322ff10c2afbacfb4d78991be60c11aac92a6d4c
2021-11-03 21:43:22 -07:00
Treehugger Robot
1e392720c2 Merge "Add a compressor types option" 2021-11-04 04:13:13 +00:00
Treehugger Robot
ce15a031ce Merge changes from topic "tproto" am: cd128891ed am: 0f0e4d23e5 am: 40bd79e8b7 am: aa3348dd1b am: d1a921ec8b
Original change: https://android-review.googlesource.com/c/platform/build/+/1865920

Change-Id: I16f08374a3c9aa1d561fb9653cd2cbad515a1388
2021-11-03 20:15:35 +00:00
Bob Badour
380a1afb1a Add macros for non-module license metadata and deps am: 879cfa867d am: 1efd13a2a1 am: c1d98c6310 am: e6d1b1d160 am: d57a36fadc
Original change: https://android-review.googlesource.com/c/platform/build/+/1741493

Change-Id: I0c8fc45c6b8c7a76ed6411437513b6ea8997127d
2021-11-03 20:15:33 +00:00
Kelvin Zhang
9b6d4ec01b Add a compressor types option
Test: th
Change-Id: I1ff1bccf179bfd87ce9e08def0e5e38921d7d158
2021-11-03 11:17:04 -07:00
Colin Cross
1ee26cae73 Convert .meta_lic files to textproto
Make it easier to write tools against .meta_lic files and store complex
data by writing them in textproto.

Test: builds
Change-Id: Ibbb6cfbb1bdddd3d938a86d563673a049d826d66
2021-11-03 15:44:00 +00:00
Bob Badour
879cfa867d Add macros for non-module license metadata and deps
Gets rid of .meta_module files and instead defers emitting rules until
after all the non-module targets have been processed. Allows direct
dependency on .meta_lic files, which in turn depend on license text
files.

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m reportmissinglicenses

Change-Id: I2c467feac6e13a9366ff66f924889f1dbd48c3f1
2021-11-03 00:52:25 +00:00
Treehugger Robot
4865a3c4a7 Merge "Support custom APEX signing tool" am: 76a4c62f56 am: c24e277d03 am: 4cb46d6fc8 am: 690f6364e4 am: d6dfa2a9b4
Original change: https://android-review.googlesource.com/c/platform/build/+/1870245

Change-Id: Id1ad821e06c0bdeefd81a406bb2f7f29f0fcb41a
2021-10-30 01:06:39 +00:00
Jooyung Han
bac375bf57 Merge changes from topic "sign_apex-sign_tool" am: f531336c6f am: 01bb43478f am: f7c262f502 am: 0e7616f29b am: c7ccad5845
Original change: https://android-review.googlesource.com/c/platform/build/+/1870244

Change-Id: I6e5338dd181ae8c4a073b9d64a9de64d766d9c5a
2021-10-28 06:05:31 +00:00
Jooyung Han
ca275ba622 fix releasetools_test am: c4b7b34b4b am: 4197f3a4c6 am: c404ba2e2e am: 5f4eaf17e8 am: 24eba94ba0
Original change: https://android-review.googlesource.com/c/platform/build/+/1870249

Change-Id: Iac5e595781da028fa609cdff8f97b71e70c1bc9a
2021-10-28 06:05:30 +00:00
Rupert Shuttleworth
072daa5f98 Merge "Add ability to set --rotation-min-sdk-version for signapk." 2021-10-27 12:56:18 +00:00
Rupert Shuttleworth
a95ab436f6 Add ability to set --rotation-min-sdk-version for signapk.
Bug: 197787352

Test: m signapk

Change-Id: I3b5e9bb3985c2ae83381499a44b339a7b1deb05c
2021-10-27 12:53:16 +00:00
Jooyung Han
8caba5e14e Support custom APEX signing tool
When an APEX specifies its custom signing tool (custom_sign_tool:),
apexkeys.txt contains the info and sign_target_files_apks pass the value
to apex_util.

For now the Virt APEX has its own custom signing tool (sign_virt_apex),
which is added to OTATOOLS.

Bug: 193504286
Test: sign_target_files_apks invokes sign_virt_apex
Change-Id: Iba845723fe3e18f542963324b9c58cd00914c5ba
2021-10-27 16:56:22 +09:00
Jooyung Han
0f5a41da56 sign_apex --sign_tool
A new argument is a custom signing tool for APEX contents. When
specified, apex_util invokes the tool with payload's key and payload
directory.

For now, the Virt APEX has its own custom signing tool (sign_virt_apex)
to re-sign filesystem images in it.

Bug: 193504286
Test: atest releasetools_test
Test: m sign_apex sign_virt_apex
Test: sign_apex --sign_tool sign_virt_apex --payload_key ..
  --container_key .. resigned.apex
      adb install resigned.apex
      reboot & vm run-app
Change-Id: Ic4d369c2ba42a8295044a0a75e054dc8def93208
2021-10-27 16:56:22 +09:00
Jooyung Han
c4b7b34b4b fix releasetools_test
Some test cases are failing because tests are not updated accordinly.

Bug: n/a
Test: atest releasetools_test
Change-Id: I3df071f72f01dedd6df4fa462ca52b8a0b1ffd4e
2021-10-27 16:06:10 +09:00
Treehugger Robot
2992dccd43 Merge "Allow data wipe for full OTAs" 2021-10-27 04:53:16 +00:00
Kelvin Zhang
81641afae4 Allow data wipe for full OTAs
Test: th
Change-Id: I7a751b461a5442d22ebca07c98624f35c02dca15
2021-10-26 16:03:25 -07:00
Lingfeng Guan
8e7cdf65d1 SignApk - support loading private keys from pkcs#11 keystore
Summary:
Add two flags to load the keys from pkcs#11 keystore. When the option
-loadPrivateKeysFromKeyStore is specified, will load private keys from
the keystore with specified keyStoreName instead of load from file.

Test: make dist for arm_sunfish-user, which includes apk
and ota (wholefile) signing
Test:
- manually call signapk in Java11 (java9 may need additional
change to support), with statically registered pkcs#11 keystore, signed
both apk and ota-package.
- verified using apksigner and extracting otacert from ota-package, both
correct

Change-Id: I3efb8017f73d3d992c07ed4562acfef016a109fe
2021-10-26 12:13:40 -07:00
Sasha Smundak
2107a731b4 Rename cmd/main.go to cmd/canoninja.go
So that it can be built and installed with
````
(cd build/make/tools/canoninja && go install cmd/canoninja.go)
```

Bug: 201713929
Test: internal
Change-Id: I38133bf26ccfae5ebf8bc3c68bc595b7274576b9
2021-10-20 15:12:07 -07:00
Treehugger Robot
3bd9c7d36a Merge "A tool to facilitate large ninja files comparison." 2021-10-19 19:11:35 +00:00
Sasha Smundak
26c705f764 A tool to facilitate large ninja files comparison.
Bug: 201713929
Test: internal
Change-Id: Ifd976eed1e58b7409e3deacf99917206f0149ade
2021-10-19 09:48:04 -07:00
Treehugger Robot
d8f17b1fc4 Merge "Add pcluster and deduplication support for erofs images." 2021-10-16 01:56:43 +00:00
Treehugger Robot
777ef8384c Merge "sign_target_files_apks: adding --allow_gsi_debug_sepolicy" 2021-10-15 05:41:55 +00:00
David Anderson
fce20c249e Merge "Add blocklist support for EROFS." 2021-10-15 05:41:06 +00:00
David Anderson
333d256b39 Merge "Clean up code duplication in build_image.py." 2021-10-15 01:31:48 +00:00
Treehugger Robot
0a467339ca Merge "Fix formatting of rbcrun errors" 2021-10-14 22:22:10 +00:00
Bowgo Tsai
2a781692ae sign_target_files_apks: adding --allow_gsi_debug_sepolicy
https://android-review.googlesource.com/q/topic:gsi_debug_policy
adds userdebug_plat_sepolicy.cil into the GSI system.img to
reduce the steps of repacking a debug ramdisk.

This CL checks that the file userdebug_plat_sepolicy.cil shouldn't
exist before signing, unless the caller explicitly specifies
--allow_gsi_debug_sepolicy to allow it.

Note: also fixes the indentation around the block.

Bug: 201482141
Test: sign_target_files_apks *-target_files-*.zip signed.zip
Change-Id: I56ed328a9ae70cf49dbd3c6efb5a4a8c54e1b7a7
2021-10-14 16:54:37 +08:00
Cole Faust
e95122ea14 Fix formatting of rbcrun errors
Was using println instead of printf, and not expanding
the variable arguments.

Test: Manually
Change-Id: I70cd44c11a9b887e396dfa48ba445e2453d3da9b
2021-10-13 12:15:21 -07:00
David Anderson
64b351b882 Add pcluster and deduplication support for erofs images.
This adds BOARD_xIMAGE_EROFS_PCLUSTER_SIZE and
BOARD_EROFS_PCLUSTER_SIZE, which set the "pcluster size" of erofs images
for individual images or all erofs images respectively. The pcluster
size affects the maximum size of a physical compressed block.

This also adds BOARD_EROFS_SHARE_DUP_BLOCKS, which turns on chunk
support in EROFS.

Bug: 201685920
Test: manual test
Change-Id: I27ec0899f89890562796dd9fa567fc74182fbefb
2021-10-13 01:03:10 -07:00
David Anderson
d29e5371e1 Add blocklist support for EROFS.
This also fixes argument ordering for mkerofsimage.sh.

Bug: 201685920
Test: build OTA with erofs
Change-Id: Ic2ba794e37a0feb73a1028eb7522d35bffbf3af6
2021-10-13 01:03:08 -07:00
David Anderson
271dab6ceb Clean up code duplication in build_image.py.
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.

Bug: N/A
Test: m otapackage, treehugger
Change-Id: I4c71275303a9246b37c03b24f531925b90d26fc8
2021-10-11 21:20:35 -07:00
Christine Tsai
28b5a7c806 Revert "Clean up code duplication in build_image.py."
This reverts commit aac502fa31.

Reason for revert: Build breakage

Bug: 202674967
Change-Id: I331aef1896db236726b5313580a1172c08173702
2021-10-11 17:25:27 +00:00
David Anderson
aac502fa31 Clean up code duplication in build_image.py.
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.

Bug: N/A
Test: m otapackage, misc_info.txt is identical
Change-Id: I435987605fc66b78bfaf454556f1eaa498e3728f
2021-10-08 17:21:43 -07:00
Chih-Hung Hsieh
f36e01de61 Add new clang-tidy and clang warning patterns
* New "clang-tidy used ... seconds" warnings are reported when
  clang-tidy runs for more than 1/2 of TIDY_TIMEOUT seconds.
* Recognize other clang-tidy warnings in separate groups;
  they should not be used in normal Android builds.
* Add two more variants of C++ warnings.

Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Change-Id: I4a2e1c3c817c586cfe3da125b920cca77fcc63b6
2021-10-08 13:18:24 -07:00
Treehugger Robot
0b08b8dc14 Merge "Fix aosp_x86_64-user build failure" 2021-09-27 15:51:27 +00:00
Bowgo Tsai
aba5c9eacd Fix aosp_x86_64-user build failure
Commit Ia982eb2ee3d1eb64db72c1836e433bcc53e71e3f removes boot-5.4.img
and leaves only a boot-5.10.img file, which makes 'boot_container'
become false. This leads to the failure in AddVbmetaDigest() as it will
search a 'boot.img' based on the 'boot' descriptor from the vbmeta.img.

Add a condition that if boot_images[0] is not 'boot.img' then the
build is also a boot container.

Bug: 199807830
Test: build aosp_x86_64-user
Change-Id: I4a9487b075186f0abf2ba74d3a1cf78072352a05
2021-09-27 14:11:14 +08:00
David Anderson
684e74ae15 Merge "Allow changing the EROFS compressor." 2021-09-24 19:12:53 +00:00
Sasha Smundak
e8652d4cd1 Separate output from diagnostics in Starlark product configuration.
Some of the product configuration makefiles use `info` and `warning` Make's
builtins for diagnostics. As running Starlark configuration generates the makefile
as its output, this diagnostics has to go elsewhere. Implement `rblf_log` as
the functional equivalent of `print` that writes to stderr instead of stdout
and use it to implement `info` and `warning` counterparts.

Fixes: 201073196
Test: manual
Change-Id: Ib4d9c10566f9b20310fbee41eda67f0a621b0a84
2021-09-24 08:54:29 -07:00
Chih-Hung Hsieh
3bb6c9448b Add new clang and clang-tidy warning patterns
* New "clang-tidy aborted ..." warnings are reported when
  clang-tidy runs for more than TIDY_TIMEOUT seconds.

Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Change-Id: Iea62b06b2cf78737d7713dabbcf6d9ac8ba23898
2021-09-23 13:51:08 -07:00
David Anderson
40a821f600 Allow changing the EROFS compressor.
This adds BOARD_EROFS_COMPRESSOR to change the compression algorithm
globally, and BOARD_{x}IMAGE_EROFS_COMPRESSOR to change it for
individual partitions.

Bug: N/A
Test: manual test
Change-Id: I2ef831558242a4070ee96269140c33b66c689351
2021-09-22 18:04:27 -07:00