Commit Graph

382 Commits

Author SHA1 Message Date
Treehugger Robot
ef9ba4d22b Merge "Check device's launch API level before using new compresison algo" into main 2023-08-17 00:35:36 +00:00
Kelvin Zhang
8f83000732 Check device's launch API level before using new compresison algo
If a T launch device later changes to LZ4(which is supported since U),
we will fallback to older compression algorithms, because if the full
OTA is applied on an T build, the update_engine on device won't support
lz4.

Bug: 295989519
Test: generate full OTA for T launched devices, make sure lz4 is not
used

Change-Id: I82fdf788e47e2a6daeaa4479bfecf317d8ebb5d5
2023-08-16 13:16:48 -07:00
Daniel Zheng
74955f8883 Adding option to configure compression level
Adding option to ota_from_target_files to configure compression level.
This option can be configured via the --vabc_compression_param flag.
e.g. --vabc_compression_param=gz,9 specifies gz compression algorithm
using level 9 compression

Test: ota_from_target_files
Change-Id: Ifc851faccbb3fba466d45c9695aaab322a362081
2023-08-08 08:09:15 -07:00
Daniel Zheng
3bff72fbc6 Adding supported compression methods
Adding in supported compression methods to help

Test: m ota_from_target_files
Change-Id: I1c3101f983eb8b38d43752530bd9f61a85fddba4
2023-08-03 15:32:29 -07:00
Kelvin Zhang
5b9ca88501 Support generating secondary OTAs from directories
Test: ota_from_target_files --include_secondary
Change-Id: I1004defdb4fe601a6ec733a7ceb213de7d540174
2023-08-01 10:04:08 -07:00
Kelvin Zhang
456c6d9a67 Adding an option to disable puffdiff
This is handy for quick OTA generation during development.

Change-Id: I878f7cb3479d6338c04aca44d16e48e83270252c
2023-08-01 10:03:54 -07:00
Kelvin Zhang
1e774245a4 Handle zip64 extra fields better
Test: check_target_files_signatures
Bug: 283033491
Change-Id: I7da89f8389c09cc99201cff342483c158bd7e9c1
2023-06-17 09:21:46 -07:00
Håkan Kvist
b8982fc538 fixup! Allow ota_from_target_file to work entirely on directories
The following log message would always be shown:
WARNING : Cannot find care map file in target_file package

Break out of the care map copying loop as soon a file has been
copied. This ensures that else statement is only executed if no
care map file exists.

Test: Manual. Run ota_from_target_files with target-zip with and
    without care map files.
Change-Id: Ia196aa182ed81f21424317a7005f5634866b4b99
2023-06-08 15:17:57 +02:00
Kelvin Zhang
b97416e09e Allow source build to be an extracted target_files
common.LoadInfoDict() already supports loading from extracted
directories, just use it.

Test: generate an incremental OTA where both inputs are directories
Bug: 227848550
Bug: 277028723
Change-Id: Iedba831bb4d65d971df6b2ac95279e3234a02e2f
2023-05-15 11:13:26 -07:00
Kelvin Zhang
b036007592 Fix non-AB ota generation for directories
Test: ota_from_target_files -v --block yukawa-target_files-10097950.zip
test.zip

Change-Id: I4291010d66cc28bdb6b6b334cdbed06daf3de837
2023-05-09 20:30:53 -07:00
Kelvin Zhang
6b10e15acc Support generating partial OTAs from extracted target_files
This allows the build system to potentially paralleize generation of OTA
package and zipping of target files

Bug: 262185376
Bug: 227848550
Change-Id: I90b6c25761683ebe3803b22fc8e23540a5282c66
2023-05-02 20:48:14 -07:00
Daniel Zheng
15c296958b Merge "Add --max_threads to OTA generation" 2023-04-26 21:53:53 +00:00
Daniel Zheng
a30989bafa Add --max_threads to OTA generation
Adding an option to configure max number of threads used during
ota_from_target_files

Bug: 277794445
Test: tested ota_from_target_files
Change-Id: I9bae92ee43c21926434ab04563a2b4f0678b73eb
2023-04-26 10:52:12 -07:00
Kelvin Zhang
9dbe2ce40f Allow ota_from_target_file to work entirely on directories
When building android, build system will first zip target_file directory
into a .zip file, and then invoke ota_from_target_files. We
ota_from_target_files can work directly on top of directories, we can
parallelize OTA generation and target_file zipping, making builds
faster.

Bug: 227848550
Bug: 277028723
Test: th
Change-Id: Id72bd7cca708af0e1bae2db754f39b27777cd601
2023-04-18 17:47:29 -07:00
David Anderson
23c2df71bc Merge "ota_from_target_files: Disable XOR on <T vendor." 2023-04-15 04:52:46 +00:00
Treehugger Robot
7b640806da Merge "Revert "Remove all ZIP64LIMIT hack"" 2023-04-15 01:48:58 +00:00
David Anderson
1c59617820 ota_from_target_files: Disable XOR on <T vendor.
Bug: N/A
Test: ota_from_target_files
Change-Id: Ia9093a3c3f79d88f98f0777e8a0f8c72741055b8
2023-04-14 16:01:55 -07:00
Kelvin Zhang
f92f7f046a Revert "Remove all ZIP64LIMIT hack"
This reverts commit 37a4290909.

Reason for revert: b/278156419

Change-Id: I67ea667619a9623be849d911993010ef0f0bfd88
2023-04-14 21:32:54 +00:00
Kelvin Zhang
fcd731e3d6 Invoke delta_generator directly
Currently, ota_from_target_files(a python script) calls
brillo_update_payload(a bash script), which then calls delta_generator(a
C++ binary) to do the actual diffing. Having brillo_update_payload in
the middle does not offer any additional flexibility, but it makes
maintaince more difficult. Bash code is less readable and harder to
debug. Further more, everytime we want to add a new flag to
delta_generator, we have to add the flag in three places:
ota_from_target_files, brillo_update_payload, delta_generator.

Historically, brillo_update_payload was there because we inherited from
ChromeOS side. This bash scripts extracts target_files.zip and invoke
delta_generator to generate the actual OTA. Any customizations we want
on OTA must be implemented by modifying the input target_files.zip ,
Manipuating big zip files is slow and inefficient.

To make thing simpler/faster, remove the intermdiary call to
brill_update_payload. ota_from_target_files will now extract target
files and call delta_generator directly.

Test: th
Bug: 227848550
Change-Id: I44b296e43bf7921bcf48ef6a1e4021a12669739c
2023-04-11 13:53:05 -07:00
Kelvin Zhang
bd111ccae2 Add comments explaining how secondary OTA works
Test: th
Change-Id: I72f890e3dcb6f8369dea8071da040b4c27877a2b
2023-04-07 09:47:04 -07:00
Treehugger Robot
cee4ebf1c3 Merge changes I674ad741,I908043ba
* changes:
  Disable VABC xor if COW isn't compressed
  Fix secondary payload generation
2023-03-08 02:03:33 +00:00
Kelvin Zhang
88043be8c3 Disable VABC xor if COW isn't compressed
XOR only saves space if COW data is compressed. If compression parameter
is set to none, disable XOR as it does not save anything but still costs
CPU cycles.

Test: th
Change-Id: I674ad74140570ce4740c290d1adb9e9b5a08426e
2023-03-07 14:42:18 -08:00
Kelvin Zhang
49fadf41eb Fix secondary payload generation
Secondary payload is already handled by creating a new instance of
PayloadGenerator, when creating PayloadGenerator for primary payload,
we should not include the OPTIONS.secondary parameter.

Bug: 270100752
Test: ota_from_target_files --include_secondary
Change-Id: I908043bad11aa6d47dc4dc9b61da460fcfa4f0b9
2023-03-07 14:35:42 -08:00
Gregory Montoir
10e0dec463 Update GetTargetFilesZipForCustomImagesUpdates zip2zip call
This changes the generation of the target files with oem images to
operate in two passes. 1/ zip2zip is called without any filespec.
2/ oem images are replaced.

This allows to generate the target-files faster (as entries do not
need to be iterated one by one) and removes the need to escape
special file names.

Bug: 269397842
Change-Id: Ic1cf398b90344e01b2f1658079903d45bcc00b46
2023-03-01 14:44:56 +08:00
Kelvin Zhang
3d05f68f3a Reject SPL downgrade OTAs on release-keys devices
Test: th
Bug: 270459308
Change-Id: I013324b5b97336366d8e8cbe05e6ade91778b747
2023-02-24 10:50:53 -08:00
Kelvin Zhang
a10528c7a8 Allow otatools to deduce partial partition list from input
Different devices/builds might have different constraints on what
partitions should be updated together in a partial OTA. Add a convenient
feature to auto detect.

Test: generate ota with --partial , make sure output OTA contains only 4
partitions

Change-Id: Id79e4bc4c5fb9d1225cb2aa7c1823afccc0b61f2
2023-01-03 16:04:49 -08:00
Kelvin Zhang
37a4290909 Remove all ZIP64LIMIT hack
In the old days, we hacked values of ZIP64LIMIT to get around size
limitations of non-zip64 supported zip files. Now that we switched to
python3 + zip64, there's no point in keeping those hacks.

Test: th
Bug: 255683436
Change-Id: I913db33dad5503736c68a7a1f1321aa952019f60
2022-10-26 13:22:52 -07:00
Baligh Uddin
968b961fb6 Allow passing in payload signer for ota signing.
BUG: 244632678
Test: TH
Change-Id: I12c45ac378a0d4c58b146115bc69b8e6d85a3075
2022-09-08 02:17:58 +00:00
Treehugger Robot
87d272251c Merge "Refactor ota_utils to reduce dependency on global OPTIONS" 2022-09-07 02:22:00 +00:00
Kelvin Zhang
bf01f8b3f0 Refactor ota_utils to reduce dependency on global OPTIONS
Move some depency on global OPTIONS to parameters. This makes it easier
for other modules to re-use these functions w/o having to modify OPTIONS
variables before/after calling.

Test: th
Bug: 227536004
Change-Id: I3bab292f65a4222f0c9502bcf04fa7a85f8124c6
2022-09-06 23:51:20 +00:00
Treehugger Robot
7aad536922 Merge "Add security patch level param to otatools" 2022-09-02 20:48:41 +00:00
Kelvin Zhang
fa9286943c Rename ota_utils.Payload to PayloadGenerator
There's also a Payload class in system/update_engine/scripts. To avoid
name collision, rename the one in releasetools to PayloadGenerator.
No functional changes in this CL.

Bug: 227848550
Test: th

Change-Id: Ib7d4c7ad9839d99416d965e3a3661b8cee7b7693
2022-08-30 19:46:49 +00:00
Kelvin Zhang
62a7f6e08e Move Payload/StreamProperty class to ota_utils.py
This allows other modules to import these classes w/o bring in tons of
depedency. No functional changes.

Test: th
Bug: 227848550
Change-Id: I98139b45c02eddefa8a26d032e759fa11cc4c694
2022-08-30 18:29:41 +00:00
Hung-Wei Chen
be469cf909 Merge changes Icc298256,I9268cb11,I446a0b16,I347447eb,Iaae39e4c, ...
* changes:
  Remove HashTreeInfo from verity_utils.py
  Remove verity keys from info_dict
  Remove replace verity key args
  Remove unused args from common.py
  Clean up unused code for VB in verity_utils.py
  Remove verity related props from build_image.py
2022-08-30 07:10:23 +00:00
Kelvin Zhang
0b7207cec5 Add security patch level param to otatools
Test: generate an OTA with --security_patch_level flag, install it, make
sure data wipe happened.
Bug: 242812845

Change-Id: Ic33c5c468dca050a966653e9233388371d8e9b1e
2022-08-26 18:03:27 +00:00
hungweichen
22e3b01d7b Remove verity keys from info_dict
We are removing VB support from release tools. This change aims to
remove verity keys in info_dict generated by common.LOADINFODICT

Bug: 242672222
Test: atest under build/make
Change-Id: I9268cb113c2c0bd11ce85b8485e5a4605bdbdca0
2022-08-24 06:37:12 +00:00
Kelvin Zhang
059bf6e50d Move some OTA signing functions to a separte file
This makes it easier for other otatools to re-use these logic without
having to pull in lots of dependencies.

Test: th
Bug: 227848550

Change-Id: I81ed01c5cea4b934a074650731b6f89752221de9
2022-08-22 18:20:14 +00:00
Kelvin Zhang
9a698e6a9f Temporary hack for making T->S downgrade work
Downgrade OTA was broken because generic_ramdisk was removed from boot
partition in android T. Neither does O6 have init_boot. O6 also doesn't
include generic ramdisk in vendor boot, so no way for otatools to locate
contents of generic ramdisk. As a hack, omit boot partition timestamp if
we can't find ramdisk.

Test: generate T->S downgrade OTA for O6
Bug: 231656318
Change-Id: I9f5359462332aadadc324348873a3a4b2b126068
2022-05-09 10:20:52 -07:00
Kelvin Zhang
a9a87ec204 Disable verity for VABC updates on android R
Bug: 230277030
Test: th
Change-Id: I2872c490c852fb7d3784721e191ece80b4b1fd58
2022-05-04 16:45:14 -07:00
Håkan Kvist
3db1ef61de Only enable zucchini if configuration file exists
Only enable zucchini if configuration file zucchini_config.txt exits
in target files and version in source and target is the same.

Without this patch zucchini would be enabled if configuration file
was missing from both target files.

Bug: 231204699
Test: Run ota package generation with zucchini_config.txt missing from
      both source and target zips. Confirm that zucchini is disabled.
Change-Id: Ia34998fd911d5860b38fe49fa5a88056a22d661e
2022-05-03 13:00:32 +02:00
Treehugger Robot
0cc4ae84ea Merge "Fix small typo in ota script options" 2022-03-09 03:06:26 +00:00
Kelvin Zhang
318a464a71 Fix small typo in ota script options
"enable_zucchin" was used instead of "enable_zucchini"

Test: th
Change-Id: Iaccb07bc6913a28c021fe84d3faeb59504ea1fbb
2022-03-08 15:13:01 -08:00
jiajia tang
c7c8697a62 Simplify codes when _LoadOemDicts
Use function LoadDictionaryFromFile(), instead of
LoadDictionaryFromLines(). Makes codes conciser.

Signed-off-by: jiajia tang <tangjiajia@xiaomi.com>
Change-Id: I15d6d4281a03be8a2f6f855054b1e4c1c3010fcd
2022-03-08 18:27:24 +08:00
Kelvin Zhang
84b3a28603 Add an option to set VABC compression algo
When testing on cuttlefish, space consumption isn't a concern, so
compression can be disabled to save time. We also want to test the
non-compressed code path, as we expect some OEMs to use userspace COW
w/o compression.

If no --vabc_compression_param option specified to
ota_from_target_files, we will use the compression algo specified in
target_files's dynamic_partition_info.txt.

Bug: 219966986
Test: th
Change-Id: Ie18064b15938550ad6fb4efc86fc100c12321dc0
2022-03-04 12:29:30 -08:00
Kelvin Zhang
e174c323e1 Add SPL downgrade option help text
Test: th
Change-Id: I71f916642f6327acad1d0c163a204718002f876d
2022-03-01 15:59:08 -08:00
Ramji Jiyani
13a4137929 core & tools: system_dlkm: add dynamic partition
Converts existing static partition support to a
dynamic partition.

Bug: 200082547
Test: TH
Signed-off-by: Ramji Jiyani <ramjiyani@google.com>
Change-Id: Ifd6d0c2a04e947b16f8b241e99cca594a1d315ae
2022-01-30 08:43:08 +00:00
Kelvin Zhang
f175801f62 Fix bug where VABC xor is used on unsupported builds
Test: th
Bug: 216426344
Change-Id: I352136eb5f261ee1ca1567530109e7623c5f978b
2022-01-27 10:23:10 -08:00
Kelvin Zhang
6fe2fc031f Add erofs compressor type param
Test: th
Bug: 206729162
Change-Id: I3ddd1a55e1544a8a4d24ed445562f05ad0aaad2c
2022-01-18 10:17:49 -08:00
Kelvin Zhang
f2728d615e Add option to enable lz4diff
When lz4diff is enabled, we inject the host copy of liblz4.so in
target_file to delta_generator. This is to ensure that host tooling can
produce same compressed output as the device.

Test: th
Bug: 206729162

Change-Id: I2d8206f7ec54fadedea16bf3d811b8353bc9414d
2022-01-12 17:21:00 -08:00
Kelvin Zhang
410bb389d8 Add option to enable zucchini
Change-Id: Id952826c9c5af912fba679af61f2a2e1645641dd
2022-01-07 14:59:01 -08:00