Commit Graph

2985 Commits

Author SHA1 Message Date
Kelvin Zhang
68252b801a Move signing related options to a shared place
Move OTA signing flags to payload_signer.py so that we can re-use these
flags in multiple binaries.

Test: th
Bug: 293313353
Change-Id: I44f9910cee37c449397e174a5784f747ec9fb0d6
2023-11-14 12:41:58 -08:00
Kelvin Zhang
c68c6b9530 Allow ParseOptions to compose multiple option parsers easily
There are certain options which we need to share in multiple binaries,
for example, the signer options. Current options parsing function only
accepts 1 extra option handler, which is inflexible. Extend it to take a
list of extra option handlers.

Currently, to add a new CLI flag, caller must append the flag name to
`extra_long_opts`, then pass an extra option handler which can handle
that option. Define a new dataclass which contains both the CLI flag
name and the code to handle that flag for better composition.

Test: th
Bug: 293313353
Change-Id: I758db66dfd95934f5b2701454d97bfe7d37dc16d
2023-11-14 12:41:56 -08:00
Abhishek Nigam
1dfca46094 Revert "Move more non-AB code to separate files"
This reverts commit 513b86e5c2.

Change-Id: I6aae60642772a052404eb1773966b2e637864bbc
2023-11-08 22:29:18 +00:00
Abhishek Nigam
b148ac22f8 Revert "Fix non-AB ota generation failure"
This reverts commit 782b752889.

bug: 308932171
tested: locally generated and applied ota on nonab device
Change-Id: I67b26be01d6240b2cb332e13bc7c7959de128755
2023-11-08 22:29:18 +00:00
Kelvin Zhang
65c1162eb5 Generate incremental OTAs for 16K/4K boot option
Previously, the 16K/4K boot options OTAs are full OTAs, resulting in
file size of ~20MB each, and ~40M for both OTAs. To reduce the space
usage, use incremental OTAs instead.

Test: th
Bug: 302759296
Bug: 293313353
Change-Id: I61cc84c6c13f151dd6bc5ff37dd31daa5fb31abd
2023-11-06 12:37:49 -08:00
Kelvin Zhang
8c9166ae37 Convert partition size to int before CheckSize
Test: th
Change-Id: Ic312e40fdad99afd65b4fc259395b8952bb038c6
2023-11-01 10:14:13 -07:00
Kelvin Zhang
9f9ac4e8a0 Fix prebuilt boot image hermetic build
Break when we find a working salt, certain devices have boot image
containing both the kernel and ramdisk, resulting in the ramdisk being
used as salt. Since core/Makefile uses kernel as salt, update python
code to match.

Bug: 293313353
Test: th
Change-Id: I68b5a8c2f0747ddf68f775bc2a6c5db46eb7eb64
2023-11-01 10:14:13 -07:00
Kelvin Zhang
782b752889 Fix non-AB ota generation failure
Test: ota generation on pinot target
Bug: 308615740
Change-Id: I8b540516c0ba9e3acb23eebad8569a6dbf2ef9f3
2023-10-31 19:42:39 -07:00
Kelvin Zhang
513b86e5c2 Move more non-AB code to separate files
non-AB has different lifetime and development cycles from AB. To avoid
accidentally breaking each other when making changes, put non-AB code in
its own files.

Test: local non-AB OTA generation
Bug: 296485685

Merged-In: I0736ecef343834c746464666a454a8bfcd4a20ea

Change-Id: I0736ecef343834c746464666a454a8bfcd4a20ea
2023-10-30 23:43:15 +00:00
Kelvin Zhang
2ab6986dd9 Make non-ab code path compatible with extracted target files
Bug: 296485685
Test: ota_from_target_files -v -i non_ab.zip non_ab.zip ota.zip
Change-Id: I13ed54bfe7d14d2d1e87f45564da465a77bc2d1d
2023-10-27 11:01:27 -07:00
Kelvin Zhang
160762a276 Use deterministic salt for AVB footer of prebuilt boot img
When target specified a prebuilt boot.img, current build system will add
avb hash footer to it with a random salt. Use a deterministic salt
instead for more reproducible builds. To stay consistent with
non-prebuilt boot.img code path, we extract the kernel image from
prebuilt boot.img and uses sha256sum of kernel image as the salt.

Test: th
Bug: 293313353
Change-Id: I988999ddc4f18e0b8677b05a3165c847b6a11b52
2023-10-25 13:28:11 -07:00
Kelvin Zhang
9d74128194 Allow -v option to change log level
Setting logging level to 'NOTSET' would tell logging module to inherit
logging level from handlers. Since the verbosity of handlers is
controlled by -v option, this allows sign_apex.py to give verbose output
if -v is specified.

Test: th
Bug: 307191956
Change-Id: I9c1db107cf4eae80ab570d892b0465b3902298fd
2023-10-24 21:58:28 -07:00
Kelvin Zhang
d51332c82f Allow OEMs to bypass the on device SPL downgrade check
Certain OEMs bootloader implementation allow SPL downgrade. Allow on
device SPL downgrade check to be bypassed.

Test: th
Bug: 306271739
Change-Id: Iba991e12d36291e2d0547c94ef54c750d87f35fb
2023-10-23 10:10:33 -07:00
Thiébaud Weksteen
ed5abbdea4 Merge "Revert "Add sign_sepolicy_path for a binary to sign sepolicy.apex."" into main 2023-10-19 22:48:25 +00:00
Sandeep Dhavale
cc1a560241 Merge "Add support for specifying erofs blocksize" into main 2023-10-19 21:46:58 +00:00
Sandeep Dhavale
d7cc7e8f9a Add support for specifying erofs blocksize
This change allows us to specify erofs filesystem block size
which will be used to create erofs image.

Either it can be specified at board level using
BOARD_EROFS_BLOCKSIZE. Or per read only partition image
using BOARD_<PARTITION>IMAGE_EROFS_BLOCKSIZE.

Examples:
BOARD_EROFS_BLOCKSIZE := 16384 OR
BOARD_PRODUCTIMAGE_EROFS_BLOCKSIZE := 16384

Bug: 303316443
Test: m and file ${ANDROID_PRODUCT_OUT}/*.img

Change-Id: I9bf3d299ae02165d0bffc62f474a302408d8b9b9
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
2023-10-18 08:23:36 -07:00
Thiébaud Weksteen
62865caf1b Revert "Add sign_sepolicy_path for a binary to sign sepolicy.apex."
This reverts commit e0a977affd.

Test: atest --host releasetools_test
Bug: 297794885
Change-Id: I951277e4aa3ae1f90474a1f7a036fc8693453c53
2023-10-18 11:41:25 +11:00
Wei Li
6437bae07c Add zip2zip as a data dependency of releasetools_common so it is built with releasetools_common and other python_binary_host modules depend on releasetools_common.
Bug: 303082945
Test: CIs
Test: "m dev_sign_bundle" on internal udc branch and check zip2zip is built too.
Change-Id: Iad5aa401de39516876714c854dc07aeba2f234e5
2023-10-17 21:05:53 +00:00
Kelvin Zhang
d7924a5311 Add missing AB partitions to img.zip
Add all AB partitions to img.zip, so that we can verify integrity of a
build by running avbtool on img.zip .

Bug: 274923679
Test: img_from_target_files on a pixel build. Unzip the generated img
zip, run avbtool verify_image --follow_chain_partitions --image
vbmeta.img, make sure result is successful.

Change-Id: I7d586a6f2ad87d8ebf87b788947055339b1121a2
2023-10-12 10:31:44 -07:00
Kelvin Zhang
7d1b29bd5d Run validation on user specified vabc_compression_param
During OTA generation, we check the vabc compression param specified by
target files, and override compression param to gz if the specified
compression method is unsupported. Run the same validation on
compression param passed by --vabc_compression_param as well.

Test: th
Change-Id: If855a2ee8c3c280c74efd5a67df229ab6c5709c5
2023-10-09 16:47:19 -07:00
Treehugger Robot
44a7f9ccb3 Merge "Add zip2zip as a required module of releasetools_common." into main 2023-10-05 06:46:26 +00:00
Wei Li
9e37cf263b Add zip2zip as a required module of releasetools_common.
zip2zip is used in function ZipDelete in common.py.

Bug: 303082945
Test: CIs
Change-Id: I7ff1189122aac4915d411b50a6cc76bd76d658f6
2023-10-04 22:23:13 -07:00
Dennis Song
764585118f Merge "Resolve conflict AVB rollback index location" into main 2023-10-04 17:16:11 +00:00
Dennis Song
4aae62ee77 Resolve conflict AVB rollback index location
Add an `--avb-resolve-rollback-index-location-conflict`
option in merge_target_files. When this option is set,
the merge tool will resolve conflicting index locations
by assigning the smallest unused index location.

This is to support merging system and vendor target files
from two different targets. In this case, the two target
files may have conflict rollback index location because
they were built independently.

Test: atest releasetools_test
Test: validate_target_files *-target_files-*.zip
Test: merge_target_files &&
        add_img_to_target_files &&
        img_from_target_files &&
        flash device
Bug: 300604688
Change-Id: Ibd18ef2a9f3784157fe17966f5364c3c81c9bd9f
2023-10-04 06:45:15 +00:00
Kelvin Zhang
de53f7df43 Use deterministic salt for boot image avb footer
avbtool by default generates a random salt everytime, this makes builds
less reproducible. Use sha256 checksum of kernel image as the hex to
make the build reproducible.

Test: th
Bug: 293313353

Change-Id: I959b3dee77654098ab9fde475f11eaee8d40c790
2023-10-03 12:26:08 -07:00
Dennis Song
6e5e44df72 Minor refactor tools/releasetools/common.py
- Sort the import modules in alphabetical order.
- Define module variables for avbtool arguments name.

Test: atest releasetools_test
Bug: 300604688
Change-Id: I7990b94cfc1d7b3e977ceb0a3dc2317689891229
2023-10-03 02:46:51 +00:00
Kelvin Zhang
2f9a9ae64b Prevent VABC from being disabled on release-key builds
VABC is mandated now, and plain VAB will be unsupported later.

Test: th
Change-Id: Iad19433c06b25246d4f7bffe5599035836e36076
2023-09-27 09:35:21 -07:00
Kelvin Zhang
0fda62973b Fix custom image OTA generation with extracted target files
Bug: 301909132
Test: ota_from_target_files --custom_image vendor=vendor.img
target_files.zip ota.zip

Change-Id: I9db6e21d47174670e23f461b6107068cbfa35d0f
2023-09-26 16:08:17 -07:00
Kelvin Zhang
e14f2a59a3 Use symlinks instead of hard links
Test: th
Bug: 283172692
Change-Id: Idb91448677d94370e052755842a396cc0e1eb588
2023-09-25 15:24:20 -07:00
Dennis Song
36ce326f95 Respect framework partition set while merging meta files
The framework_partition_set would be generated from
framework_item_list and we should respect this partition
set while merging META/ab_partitions.txt.

This is to handle the case when we use a complete merged
target files package as the framework-target-files, where
its ab_partitions.txt may have non-framework partitions.
So we need to filter them out to prevent from the merged
meta file has some partitions that don't exist.

Test: merge_target_files &&
        add_img_to_target_files &&
	img_from_target_files &&
	flash device
Bug: 300193612
Change-Id: I0a76d706a20dcaffa2533278db7383563f03ec02
2023-09-15 06:20:19 +00:00
Cole Faust
98142f8446 Change default logging level to WARNING
To avoid spam in the build.

Test: Presubmits
Change-Id: I45db6cd92babda7f83582fd438210cae02c54708
2023-09-12 17:39:47 -07:00
Daniel Zheng
d6178ee812 Merge "Turn on fastboot-info for img_from_target_files" into main 2023-09-11 21:05:36 +00:00
Cole Faust
548775d247 Merge "Clean staging dirs in build_image.py" into main 2023-09-11 17:25:19 +00:00
Daniel Zheng
89a85ba933 Turn on fastboot-info for img_from_target_files
Use fastboot-info for fastboot update

Test: m update_package
Bug: 299854419
Change-Id: I9f7c93f8309a2c136ee81cad1391f93b2feb8156
2023-09-11 09:58:32 -07:00
Cole Faust
152cdfab7c Clean staging dirs in build_image.py
build_image.py now accepts a --input-directory-filter-file argument
that contains a list of files to consider from the input directory.
A new, temporary input directory will be created, and only the
considered files will be copied to it.

This removes most of the need for `m installclean`, as old files or
files built manually will no longer end up on the partition.

Bug: 205632228
Test: Presubmits
Change-Id: I07e0bf8e840abf0b51a2fdf67b758905fb3c5b5b
2023-09-08 16:24:26 -07:00
Firman Prayoga
44d86b138b Merge "Revert "Clean staging dirs in build_image.py"" into main 2023-09-08 04:06:57 +00:00
Firman Prayoga
df21706765 Revert "Clean staging dirs in build_image.py"
Revert submission 2736954-proper_installclean

Reason for revert: Potential culprit for build breakage b/299545346

Reverted changes: /q/submissionid:2736954-proper_installclean

Change-Id: I2b0671bd40b7a74ef02923a777ec899401691454
2023-09-08 01:24:45 +00:00
Cole Faust
c383825953 Merge "Clean staging dirs in build_image.py" into main 2023-09-07 16:41:45 +00:00
Kelvin Zhang
b5661d6ef5 Add per partition timestamp to 16K boot OTA
update_engine requiers per partition timestamp if this partition's build
prop includes build.date.utc.

Test: th
Bug: 293313353
Change-Id: I6bf3de8e855f0830b26bdbe2cbdc78ba891662a7
2023-09-06 13:02:19 -07:00
Cole Faust
48b81e7386 Clean staging dirs in build_image.py
build_image.py now accepts a --input-directory-filter-file argument
that contains a list of files to consider from the input directory.
A new, temporary input directory will be created, and only the
considered files will be copied to it.

This removes most of the need for `m installclean`, as old files or
files built manually will no longer end up on the partition.

Bug: 205632228
Test: Presubmits
Change-Id: I4c82ac7f1ed49a37271755749c8f72afeeb7f295
2023-09-06 11:51:13 -07:00
Kelvin Zhang
dfa0c85705 Merge "Use seek() instead of writing 0s" into main 2023-09-01 17:00:52 +00:00
Treehugger Robot
145ca3034f Merge "Mark boot OTAs as partial OTAs" into main 2023-09-01 16:58:26 +00:00
Kelvin Zhang
8aa6525ad7 Use seek() instead of writing 0s
test_ZipWrite_large_file need to write a 4GB file to zipfile for
testing. Instead of writing sequence of 0s to the test file, use seek().

Test: th
Bug: 298138109
Change-Id: I886bfa0c3f21664e529ba7de9688909518148f61
2023-08-31 12:17:14 -07:00
Kelvin Zhang
c4f737406a Mark boot OTAs as partial OTAs
Boot partition only OTA must be marked as "partial". This is required
for update_engine to correctly copy non-updated partitions to the target
slot.

Test: th
Bug: 293313353
Change-Id: Ic89d03f214e12fa43f686d37854d6fed1d78a6a3
2023-08-31 12:17:14 -07:00
Treehugger Robot
c20ef7ca4a Merge "Avoid calling MergeUpdateEngineConfig for non-ab target" into main 2023-08-31 14:30:17 +00:00
Kelvin Zhang
506a30851e Add timestamp to 16K boot OTA
Test: th
Bug: 297536338
Change-Id: Id4919d2437079e36899cd4c56da011a29aa0b470
2023-08-29 19:38:19 -07:00
Jooyung Han
ccec6ad2ec ota_from_target_files requires apexd_host
apexd_host is invoked in check_target_files_vintf.py
to prepare the apex directory before invoking checkvintf. ota_from_target_files uses check_target_files_vintf.py via (python_library) releasetools_check_target_files_vintf.

Since soong doesn't inherit "required:" property from library to binary, adding the dependency manually to ota_from_target_files.

Bug: n/a
Test: $ m ota_from_target_files # should build apexd_host
Change-Id: Ie7e6c3acf376c23960f242294f5a9bf99fec26aa
2023-08-29 01:28:40 +00:00
Kelvin Zhang
f6fe0a9c9c Reduce OTA tools's dependency on brillo_update_payload
brillo_update_payload is thin bash wrapper for delta_generator, and
OTA tools are wrappers around brillo_update_payload. This is
inconveinent, as adding a new flag requires changing 3 places. Since
brillo_update_payload is really thin, skip it and invoke delta_generator
directly.

Test: th
Bug: 293313353
Change-Id: I2fd5af95c571ea45aea62f9eec475f0f4e864953
2023-08-25 13:49:09 -07:00
Kelvin Zhang
c7441e5907 Add a tool to generate OTA from images
During build, we will need to generate an OTA for boot partition using a
16K boot image. Typically, OTA is generated from target_files.zip . To
avoid relying on target_files.zip as a dependency for 16K OTA, add a
tool to generate OTA directly from a raw image.

Test: th, ota_from_raw_img --partition_name boot --output ota.zip $OUT/boot_16k.img
Bug: 293313353
Change-Id: I2076332faf2a8dc573450597efd481e285a49545
2023-08-23 12:37:29 -07:00
Himanshu Jakhmola
21ef2c600b Avoid calling MergeUpdateEngineConfig for non-ab target
super image generation is failing for non-ab target as it is not able to find update_engine_config.txt while calling MergeUpdateEngineConfig.

META/update_engine_config.txt exists when AB_OTA_UPDATER is true.
Avoid calling MergeUpdateEngineConfig for non-ab target.

Bug: 290811619
Change-Id: Ie88b31adce15df87f73031e1f5bff0d825601e0c
2023-08-22 04:08:01 +00:00