Commit Graph

7358 Commits

Author SHA1 Message Date
Cole Faust
b820bcd829 Prepare release tools for python 3
- Sort dictionaries before looping over them
- Don't call sorted() on lists with Nones
- Open file in binary format when serializing protobufs

Change-Id: If5dbc908f7125f6184014b3c1c7891f833d1d8bf
Bug: 203436762
Test: Presubmits
2021-11-04 17:58:44 -07:00
Kelvin Zhang
2639048c21 Fix a failure in signing test
Pixel moved away from sparse images, so validate_target_files is failing
because it expects sparse images.

Test: th
Change-Id: I322ff10c2afbacfb4d78991be60c11aac92a6d4c
2021-11-03 21:43:22 -07:00
Treehugger Robot
1e392720c2 Merge "Add a compressor types option" 2021-11-04 04:13:13 +00:00
Kelvin Zhang
9b6d4ec01b Add a compressor types option
Test: th
Change-Id: I1ff1bccf179bfd87ce9e08def0e5e38921d7d158
2021-11-03 11:17:04 -07:00
Colin Cross
1ee26cae73 Convert .meta_lic files to textproto
Make it easier to write tools against .meta_lic files and store complex
data by writing them in textproto.

Test: builds
Change-Id: Ibbb6cfbb1bdddd3d938a86d563673a049d826d66
2021-11-03 15:44:00 +00:00
Bob Badour
879cfa867d Add macros for non-module license metadata and deps
Gets rid of .meta_module files and instead defers emitting rules until
after all the non-module targets have been processed. Allows direct
dependency on .meta_lic files, which in turn depend on license text
files.

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m reportmissinglicenses

Change-Id: I2c467feac6e13a9366ff66f924889f1dbd48c3f1
2021-11-03 00:52:25 +00:00
Jooyung Han
8caba5e14e Support custom APEX signing tool
When an APEX specifies its custom signing tool (custom_sign_tool:),
apexkeys.txt contains the info and sign_target_files_apks pass the value
to apex_util.

For now the Virt APEX has its own custom signing tool (sign_virt_apex),
which is added to OTATOOLS.

Bug: 193504286
Test: sign_target_files_apks invokes sign_virt_apex
Change-Id: Iba845723fe3e18f542963324b9c58cd00914c5ba
2021-10-27 16:56:22 +09:00
Jooyung Han
0f5a41da56 sign_apex --sign_tool
A new argument is a custom signing tool for APEX contents. When
specified, apex_util invokes the tool with payload's key and payload
directory.

For now, the Virt APEX has its own custom signing tool (sign_virt_apex)
to re-sign filesystem images in it.

Bug: 193504286
Test: atest releasetools_test
Test: m sign_apex sign_virt_apex
Test: sign_apex --sign_tool sign_virt_apex --payload_key ..
  --container_key .. resigned.apex
      adb install resigned.apex
      reboot & vm run-app
Change-Id: Ic4d369c2ba42a8295044a0a75e054dc8def93208
2021-10-27 16:56:22 +09:00
Jooyung Han
c4b7b34b4b fix releasetools_test
Some test cases are failing because tests are not updated accordinly.

Bug: n/a
Test: atest releasetools_test
Change-Id: I3df071f72f01dedd6df4fa462ca52b8a0b1ffd4e
2021-10-27 16:06:10 +09:00
Treehugger Robot
2992dccd43 Merge "Allow data wipe for full OTAs" 2021-10-27 04:53:16 +00:00
Kelvin Zhang
81641afae4 Allow data wipe for full OTAs
Test: th
Change-Id: I7a751b461a5442d22ebca07c98624f35c02dca15
2021-10-26 16:03:25 -07:00
Sasha Smundak
2107a731b4 Rename cmd/main.go to cmd/canoninja.go
So that it can be built and installed with
````
(cd build/make/tools/canoninja && go install cmd/canoninja.go)
```

Bug: 201713929
Test: internal
Change-Id: I38133bf26ccfae5ebf8bc3c68bc595b7274576b9
2021-10-20 15:12:07 -07:00
Treehugger Robot
3bd9c7d36a Merge "A tool to facilitate large ninja files comparison." 2021-10-19 19:11:35 +00:00
Sasha Smundak
26c705f764 A tool to facilitate large ninja files comparison.
Bug: 201713929
Test: internal
Change-Id: Ifd976eed1e58b7409e3deacf99917206f0149ade
2021-10-19 09:48:04 -07:00
Treehugger Robot
d8f17b1fc4 Merge "Add pcluster and deduplication support for erofs images." 2021-10-16 01:56:43 +00:00
Treehugger Robot
777ef8384c Merge "sign_target_files_apks: adding --allow_gsi_debug_sepolicy" 2021-10-15 05:41:55 +00:00
David Anderson
fce20c249e Merge "Add blocklist support for EROFS." 2021-10-15 05:41:06 +00:00
David Anderson
333d256b39 Merge "Clean up code duplication in build_image.py." 2021-10-15 01:31:48 +00:00
Treehugger Robot
0a467339ca Merge "Fix formatting of rbcrun errors" 2021-10-14 22:22:10 +00:00
Bowgo Tsai
2a781692ae sign_target_files_apks: adding --allow_gsi_debug_sepolicy
https://android-review.googlesource.com/q/topic:gsi_debug_policy
adds userdebug_plat_sepolicy.cil into the GSI system.img to
reduce the steps of repacking a debug ramdisk.

This CL checks that the file userdebug_plat_sepolicy.cil shouldn't
exist before signing, unless the caller explicitly specifies
--allow_gsi_debug_sepolicy to allow it.

Note: also fixes the indentation around the block.

Bug: 201482141
Test: sign_target_files_apks *-target_files-*.zip signed.zip
Change-Id: I56ed328a9ae70cf49dbd3c6efb5a4a8c54e1b7a7
2021-10-14 16:54:37 +08:00
Cole Faust
e95122ea14 Fix formatting of rbcrun errors
Was using println instead of printf, and not expanding
the variable arguments.

Test: Manually
Change-Id: I70cd44c11a9b887e396dfa48ba445e2453d3da9b
2021-10-13 12:15:21 -07:00
David Anderson
64b351b882 Add pcluster and deduplication support for erofs images.
This adds BOARD_xIMAGE_EROFS_PCLUSTER_SIZE and
BOARD_EROFS_PCLUSTER_SIZE, which set the "pcluster size" of erofs images
for individual images or all erofs images respectively. The pcluster
size affects the maximum size of a physical compressed block.

This also adds BOARD_EROFS_SHARE_DUP_BLOCKS, which turns on chunk
support in EROFS.

Bug: 201685920
Test: manual test
Change-Id: I27ec0899f89890562796dd9fa567fc74182fbefb
2021-10-13 01:03:10 -07:00
David Anderson
d29e5371e1 Add blocklist support for EROFS.
This also fixes argument ordering for mkerofsimage.sh.

Bug: 201685920
Test: build OTA with erofs
Change-Id: Ic2ba794e37a0feb73a1028eb7522d35bffbf3af6
2021-10-13 01:03:08 -07:00
David Anderson
271dab6ceb Clean up code duplication in build_image.py.
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.

Bug: N/A
Test: m otapackage, treehugger
Change-Id: I4c71275303a9246b37c03b24f531925b90d26fc8
2021-10-11 21:20:35 -07:00
Christine Tsai
28b5a7c806 Revert "Clean up code duplication in build_image.py."
This reverts commit aac502fa31.

Reason for revert: Build breakage

Bug: 202674967
Change-Id: I331aef1896db236726b5313580a1172c08173702
2021-10-11 17:25:27 +00:00
David Anderson
aac502fa31 Clean up code duplication in build_image.py.
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.

Bug: N/A
Test: m otapackage, misc_info.txt is identical
Change-Id: I435987605fc66b78bfaf454556f1eaa498e3728f
2021-10-08 17:21:43 -07:00
Chih-Hung Hsieh
f36e01de61 Add new clang-tidy and clang warning patterns
* New "clang-tidy used ... seconds" warnings are reported when
  clang-tidy runs for more than 1/2 of TIDY_TIMEOUT seconds.
* Recognize other clang-tidy warnings in separate groups;
  they should not be used in normal Android builds.
* Add two more variants of C++ warnings.

Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Change-Id: I4a2e1c3c817c586cfe3da125b920cca77fcc63b6
2021-10-08 13:18:24 -07:00
Treehugger Robot
0b08b8dc14 Merge "Fix aosp_x86_64-user build failure" 2021-09-27 15:51:27 +00:00
Bowgo Tsai
aba5c9eacd Fix aosp_x86_64-user build failure
Commit Ia982eb2ee3d1eb64db72c1836e433bcc53e71e3f removes boot-5.4.img
and leaves only a boot-5.10.img file, which makes 'boot_container'
become false. This leads to the failure in AddVbmetaDigest() as it will
search a 'boot.img' based on the 'boot' descriptor from the vbmeta.img.

Add a condition that if boot_images[0] is not 'boot.img' then the
build is also a boot container.

Bug: 199807830
Test: build aosp_x86_64-user
Change-Id: I4a9487b075186f0abf2ba74d3a1cf78072352a05
2021-09-27 14:11:14 +08:00
David Anderson
684e74ae15 Merge "Allow changing the EROFS compressor." 2021-09-24 19:12:53 +00:00
Sasha Smundak
e8652d4cd1 Separate output from diagnostics in Starlark product configuration.
Some of the product configuration makefiles use `info` and `warning` Make's
builtins for diagnostics. As running Starlark configuration generates the makefile
as its output, this diagnostics has to go elsewhere. Implement `rblf_log` as
the functional equivalent of `print` that writes to stderr instead of stdout
and use it to implement `info` and `warning` counterparts.

Fixes: 201073196
Test: manual
Change-Id: Ib4d9c10566f9b20310fbee41eda67f0a621b0a84
2021-09-24 08:54:29 -07:00
Chih-Hung Hsieh
3bb6c9448b Add new clang and clang-tidy warning patterns
* New "clang-tidy aborted ..." warnings are reported when
  clang-tidy runs for more than TIDY_TIMEOUT seconds.

Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Change-Id: Iea62b06b2cf78737d7713dabbcf6d9ac8ba23898
2021-09-23 13:51:08 -07:00
David Anderson
40a821f600 Allow changing the EROFS compressor.
This adds BOARD_EROFS_COMPRESSOR to change the compression algorithm
globally, and BOARD_{x}IMAGE_EROFS_COMPRESSOR to change it for
individual partitions.

Bug: N/A
Test: manual test
Change-Id: I2ef831558242a4070ee96269140c33b66c689351
2021-09-22 18:04:27 -07:00
Romain Jobredeaux
73bddf4c72 Merge "Create Bazel BUILD targets and Bazel rule for logtag file conversion." 2021-09-22 17:28:31 +00:00
David Anderson
dfc9d7912f Merge "Avoid too much reserved_size for erofs image" 2021-09-16 19:36:29 +00:00
Daniel Norman
a529e0fd2e Merge "Build vendor images first when rebuilding with vendor_otatools." 2021-09-16 17:29:43 +00:00
Daniel Norman
78554ea568 Build vendor images first when rebuilding with vendor_otatools.
Only supports rebuilding vendor.img and odm.img currently.

This allows "other" images (e.g. vbmeta, super_empty, ...) to
be rebuilt using the latest tooling.

Bug: 188491126
Bug: 192422274
Test: Sign a GRF R+S merged target_files package using:
      sign_target_files_apks \
        --vendor_otatools=<otatools.zip from R> \
	--vendor_partitions=vendor,odm \
	merged-target_files.zip \
	signed-target_files.zip
Change-Id: Ib93aea9f083ee9b722c31a42dcc780b5222053b8
2021-09-15 21:53:25 +00:00
Treehugger Robot
50ec71eac8 Merge "Zipalign: Avoid unnecessary padding" 2021-09-15 00:34:44 +00:00
Fabien Sanglard
df73d1b473 Zipalign: Avoid unnecessary padding
Problem: Aligned entry always get padded to the next aligned slot.
Solution: Fix alignment test.

Bug: 199932309
Test: align_test.cpp
Change-Id: I1e61c67dcf45454dd782d0fcc6903f7e71954083
2021-09-14 15:20:42 -07:00
Christopher Ferris
fe9efa5a61 Merge "Make the script python3." 2021-09-14 21:03:01 +00:00
Greg Kaiser
d9c6d5fc91 Merge "Revert "Rebuild vendor images in sign_target_files_apks with vendor otatools."" 2021-09-14 19:38:29 +00:00
Greg Kaiser
e086f7222e Revert "Rebuild vendor images in sign_target_files_apks with vendor otatools."
This reverts commit a1094e9f0c.

Reason for revert: Likely cause for b/199911935

Change-Id: Ib20400cc2f740a26fbbfb0da34fdf20ed7dc620e
2021-09-14 19:32:27 +00:00
Fabien Sanglard
1011d45f5b Zipalign: Remove unused variables
Remove unused variable resulting in compilation warnings.
Remove unused temporary variable.

Bug: NA
Test: zipalign_tests
Change-Id: Ic2236b5878074f0c2537ab0b9f94dfd9c42225d8
2021-09-13 16:52:22 -07:00
Christopher Ferris
6a54633796 Make the script python3.
Test: Ran the python version of the script and the python3 version
Test: and verified the output is the same. This is for all three
Test: outputs (html, text, xml).
Test: Builds without an error.
Change-Id: I5138b0a1efbbc3ab764c644afc1783c79fd7e250
2021-09-13 14:54:58 -07:00
Daniel Norman
97e0c2831d Merge "Rebuild vendor images in sign_target_files_apks with vendor otatools." 2021-09-13 20:12:11 +00:00
Romain Jobredeaux
bbdd255ad7 Create Bazel BUILD targets and Bazel rule for logtag file conversion.
Test: b build build/make/tools/...
Change-Id: I0a12e6cfff553a927f86381358a3f5f58fec0c75
2021-09-13 14:48:50 +00:00
Huang Jianan
6552727e03 Avoid too much reserved_size for erofs image
After the erofs image is built, it won't be modified again. The reserve
room is only used for the hash footer. There is no need to set an
excessive margin.

Bug: 199248596
Test: erofs image size is smaller than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: If2535b9ad2c2d60bea74feeca576a168387ff341
2021-09-13 02:03:13 +00:00
Yifan Hong
1ab1c2ff96 Merge "Avoid repeated building of erofs image" 2021-09-11 00:05:45 +00:00
Yi Kong
8b50dea627 Fix arguments passed to llvm-readobj
Upstream llvm-readobj no longer accepts single dash prefix for long
options. Changed to double dash.

Also let the script to raise exception if rc is non-zero, to avoid
getting confusing error later in the program.

Test: build
Bug: 197230471
Change-Id: I364f51322904431d430549f79af58f455f47e028
2021-09-10 21:17:45 +08:00
Tianjie
e7ab38d964 Add option to override minor version
Pass the minor version override to delta_generator. So we can test
features in tradefed before releasing it to droidfoods.

Bug: 197361113
Test: TH
Change-Id: Ie8846935dfa13784f0745513be13811908f44ac3
2021-09-09 12:16:19 -07:00