Commit Graph

2837 Commits

Author SHA1 Message Date
Treehugger Robot
96b51b54b6 Merge "parallelize add_partition_calls" 2023-05-12 04:54:11 +00:00
Daniel Zheng
717591dd4e Merge "Add fastboot-info to updatepackage" 2023-05-12 04:31:55 +00:00
Jeongik Cha
03d014a03f parallelize add_partition_calls
it made total time equal to the longest image build

1m10s->30s in local build

Bug: 281960217
Test: m dist
Change-Id: I13d4f45d9b46b39292a014e3b4e1913365d89b7a
2023-05-12 08:43:44 +09:00
Kelvin Zhang
b036007592 Fix non-AB ota generation for directories
Test: ota_from_target_files -v --block yukawa-target_files-10097950.zip
test.zip

Change-Id: I4291010d66cc28bdb6b6b334cdbed06daf3de837
2023-05-09 20:30:53 -07:00
Daniel Zheng
380cb8065a Add fastboot-info to updatepackage
Modifying img_from_target_files to also add fastboot-info to
updatepackage

Test: m updatepackage
Bug: 194686221
Change-Id: I2e08c4269f0d83417b9d7079633bc28796d1cdd6
2023-05-09 13:44:22 -07:00
Kelvin Zhang
6b10e15acc Support generating partial OTAs from extracted target_files
This allows the build system to potentially paralleize generation of OTA
package and zipping of target files

Bug: 262185376
Bug: 227848550
Change-Id: I90b6c25761683ebe3803b22fc8e23540a5282c66
2023-05-02 20:48:14 -07:00
Kelvin Zhang
e634bde6bd Use misc_info.txt to check for VABC
Change-Id: I77f23a51ddb027254ccda8d7ab17e64205522300
2023-04-28 23:59:43 -07:00
Daniel Zheng
15c296958b Merge "Add --max_threads to OTA generation" 2023-04-26 21:53:53 +00:00
Daniel Zheng
a30989bafa Add --max_threads to OTA generation
Adding an option to configure max number of threads used during
ota_from_target_files

Bug: 277794445
Test: tested ota_from_target_files
Change-Id: I9bae92ee43c21926434ab04563a2b4f0678b73eb
2023-04-26 10:52:12 -07:00
Treehugger Robot
dd62b0da0f Merge "Support to merge PREBUILT_IMAGES" 2023-04-25 04:24:20 +00:00
Kelvin Zhang
ca45d7a85d Fix incremental OTA generation failures
source_file might be a directory, use the _ReadApexInfo() helper
function to read it instead of go straight to zipfiles.

Test: build incremental OTA for pixel
Bug: 279142680
Bug: 227848550
Bug: 277028723

Change-Id: Ia0041e2c72c1a57123519c970f6e734194a81cd2
2023-04-21 17:30:50 +00:00
Kelvin Zhang
9dbe2ce40f Allow ota_from_target_file to work entirely on directories
When building android, build system will first zip target_file directory
into a .zip file, and then invoke ota_from_target_files. We
ota_from_target_files can work directly on top of directories, we can
parallelize OTA generation and target_file zipping, making builds
faster.

Bug: 227848550
Bug: 277028723
Test: th
Change-Id: Id72bd7cca708af0e1bae2db754f39b27777cd601
2023-04-18 17:47:29 -07:00
Po Hu
81c3f4a884 Support to merge PREBUILT_IMAGES
For example, pvmfw.img comes from framework target files package
and dtbo.img comes from vendor target files package.

Test: merge_target_files
Change-Id: If2e6fac5ce3671e3387e801808a195fb7b2c55b5
2023-04-19 08:23:40 +08:00
Treehugger Robot
67ca3c05c7 Merge "Fix duplicate super_empty.img" 2023-04-17 21:19:13 +00:00
Treehugger Robot
b7a45ea13d Merge "Try with search_path for some avb path args" 2023-04-17 20:43:12 +00:00
Kelvin Zhang
aecdc18222 Fix duplicate super_empty.img
When calling add_img_to_target_files twice, super_empty.img will be
written twice, and resulting zip would contain duplicate entries.

Test: th
Bug: 278522974
Change-Id: I2419d0c4f8b04f876a8cc69a1517f59cab94d75d
2023-04-17 09:52:39 -07:00
David Anderson
23c2df71bc Merge "ota_from_target_files: Disable XOR on <T vendor." 2023-04-15 04:52:46 +00:00
Treehugger Robot
7b640806da Merge "Revert "Remove all ZIP64LIMIT hack"" 2023-04-15 01:48:58 +00:00
Dennis Song
5da012adb0 Merge "Include META/* in the vendor item list when rebuild-sepolicy" 2023-04-15 01:15:44 +00:00
David Anderson
1c59617820 ota_from_target_files: Disable XOR on <T vendor.
Bug: N/A
Test: ota_from_target_files
Change-Id: Ia9093a3c3f79d88f98f0777e8a0f8c72741055b8
2023-04-14 16:01:55 -07:00
Kelvin Zhang
f92f7f046a Revert "Remove all ZIP64LIMIT hack"
This reverts commit 37a4290909.

Reason for revert: b/278156419

Change-Id: I67ea667619a9623be849d911993010ef0f0bfd88
2023-04-14 21:32:54 +00:00
Håkan Kvist
01e381954b Order entries in apex_info.pb
Adding the entries in apex_info.pb in sorted order makes it easier to
compare ota packages generated between the same source and target on
different computers/environments, ideally the generated ota zips should
be identical.

os.listdir() was used to find the apex files which should be
included in apex_info.pb. listdir() does not guarantee any order,
solution is to sort the result to ensure consistent order.

Bug: 278095305
Test: Manual. Confirm that apex_info.pb is generated sorted
    protoc --decode_raw < apex_info.pb
Change-Id: I2851d207f828a6ac31e63e20c9025965ac4b0eba
2023-04-14 10:36:04 +02:00
Dennis Song
09f5c5dc0b Include META/* in the vendor item list when rebuild-sepolicy
To ensure that `AddImagesToTargetFiles` can still be used with
vendor item list that do not specify all of the required META/
files, those files should be included by default.

Test: atest --host releasetools_test
Bug: 278133178
Change-Id: I5a04bfc43d405d6838028f2535ced2124dbd43aa
2023-04-14 12:13:42 +08:00
zhangyongpeng
707569711b Try with search_path for some avb path args
If we can't find relative signing_helper path, try with the OPTIONS.search_path dir prefix, and integrate the avb_*_key_path for the same logic.

Test: Build && releasetools_test

Signed-off-by: zhangyongpeng <zhangyongpeng@xiaomi.com>
Change-Id: Ifb1096ddea90693668f3344eb242bf9725113d11
2023-04-14 12:12:47 +08:00
Kelvin Zhang
629bc8dc1c Fix s to t OTA
Android S update_engine has a smaller minor version, we should use
update_engine version number from source build when generating an
incremental OTA.

Test: th
Bug: 277837157
Change-Id: Ic1ccdccd26c19bf9fb13d146a9151e699e481201
2023-04-11 21:30:42 -07:00
Kelvin Zhang
ea84d42199 Fix releasetools timeouts
Release tools test uses sleep(5) to make sure mtime on files change in a
visible way. Use hardcoded mtime instead of sleeps.

Improvement: 145.6s -> 65s

Test: atest releasetools_test
Bug: 277782284
Fixes: 277782284

Change-Id: I4d7b04707f3b6c5843cde883f43e95a1e7a69879
2023-04-11 13:53:05 -07:00
Kelvin Zhang
fcd731e3d6 Invoke delta_generator directly
Currently, ota_from_target_files(a python script) calls
brillo_update_payload(a bash script), which then calls delta_generator(a
C++ binary) to do the actual diffing. Having brillo_update_payload in
the middle does not offer any additional flexibility, but it makes
maintaince more difficult. Bash code is less readable and harder to
debug. Further more, everytime we want to add a new flag to
delta_generator, we have to add the flag in three places:
ota_from_target_files, brillo_update_payload, delta_generator.

Historically, brillo_update_payload was there because we inherited from
ChromeOS side. This bash scripts extracts target_files.zip and invoke
delta_generator to generate the actual OTA. Any customizations we want
on OTA must be implemented by modifying the input target_files.zip ,
Manipuating big zip files is slow and inefficient.

To make thing simpler/faster, remove the intermdiary call to
brill_update_payload. ota_from_target_files will now extract target
files and call delta_generator directly.

Test: th
Bug: 227848550
Change-Id: I44b296e43bf7921bcf48ef6a1e4021a12669739c
2023-04-11 13:53:05 -07:00
Dennis Song
5bfa43e5eb Support merging target files from directory
Expand `merge_target_files.py` API capabilities so that
`--framework-target-files` and `--vendor-target-files`
can be either zip archives or directories.

Test: Create a merged package by vendor target files folder
Test: atest --host releasetools_test
Bug: 276068400
Change-Id: I200be2a458ae59a61e05bfd7c78ab66093db32eb
2023-04-10 17:07:03 +08:00
Treehugger Robot
d8e23f9519 Merge "Add comments explaining how secondary OTA works" 2023-04-07 20:57:36 +00:00
Kelvin Zhang
bd111ccae2 Add comments explaining how secondary OTA works
Test: th
Change-Id: I72f890e3dcb6f8369dea8071da040b4c27877a2b
2023-04-07 09:47:04 -07:00
Oleg Lyovin
6d75a854ef Fix choosing wrong diff tool for recovery-from-boot patch
'bsdiff' tool should be used when recovery and boot image
have mismatching entries. This includes not only the case
when system_as_root enabled, but include_recovery_dtbo and
include_recovery_acpio too.

Add check if they are enabled to synchronize logic with
core/Makefile, which already takes this into account.

Test: make dist -j
Change-Id: Ie673403ae8eae446d006cc524dd48ac6c56194fc
Signed-off-by: Oleg Lyovin <ovlevin@sberdevices.ru>
2023-03-22 17:53:26 +03:00
Treehugger Robot
4fac9d776c Merge "Add an automated tools for generating brick OTA package" 2023-03-15 09:24:40 +00:00
Kelvin Zhang
f91d74bd52 Add an automated tools for generating brick OTA package
In the past, we have been generating brick OTA packages manually.
Automate this process.

Bug: 273561331
Test: create_brick_ota --product oriole brick.zip and apply the package
Change-Id: I7f4cf3cacaedd9d376b4cc07cbb388930bca643b
2023-03-14 16:58:36 -07:00
Rob Seymour
9492da59e3 Add test cases for APEX VINTF.
Add test cases to cover HAL definition found in embedded APEX.

Bug: 249086047
Test: releasetools_test
Change-Id: I5639b9b800a89164317740f64306ae10bf41343e
2023-03-09 17:00:06 +00:00
Treehugger Robot
cee4ebf1c3 Merge changes I674ad741,I908043ba
* changes:
  Disable VABC xor if COW isn't compressed
  Fix secondary payload generation
2023-03-08 02:03:33 +00:00
Treehugger Robot
135eb025f7 Merge "Fix sparse image detection" 2023-03-07 23:41:43 +00:00
Kelvin Zhang
88043be8c3 Disable VABC xor if COW isn't compressed
XOR only saves space if COW data is compressed. If compression parameter
is set to none, disable XOR as it does not save anything but still costs
CPU cycles.

Test: th
Change-Id: I674ad74140570ce4740c290d1adb9e9b5a08426e
2023-03-07 14:42:18 -08:00
Kelvin Zhang
49fadf41eb Fix secondary payload generation
Secondary payload is already handled by creating a new instance of
PayloadGenerator, when creating PayloadGenerator for primary payload,
we should not include the OPTIONS.secondary parameter.

Bug: 270100752
Test: ota_from_target_files --include_secondary
Change-Id: I908043bad11aa6d47dc4dc9b61da460fcfa4f0b9
2023-03-07 14:35:42 -08:00
Kelvin Zhang
0452128f09 Fix sparse image detection
Instead of relying on image build flags, check the actual file to
determine if input is sparse image.

Test: th
Change-Id: I6e58603535039eb3c4e799c61ed9f4783f1888dd
2023-03-02 09:42:52 -08:00
Gregory Montoir
10e0dec463 Update GetTargetFilesZipForCustomImagesUpdates zip2zip call
This changes the generation of the target files with oem images to
operate in two passes. 1/ zip2zip is called without any filespec.
2/ oem images are replaced.

This allows to generate the target-files faster (as entries do not
need to be iterated one by one) and removes the need to escape
special file names.

Bug: 269397842
Change-Id: Ic1cf398b90344e01b2f1658079903d45bcc00b46
2023-03-01 14:44:56 +08:00
Treehugger Robot
b9dd9dd417 Merge "Reject SPL downgrade OTAs on release-keys devices" 2023-02-24 22:17:01 +00:00
Kelvin Zhang
3d05f68f3a Reject SPL downgrade OTAs on release-keys devices
Test: th
Bug: 270459308
Change-Id: I013324b5b97336366d8e8cbe05e6ade91778b747
2023-02-24 10:50:53 -08:00
Dennis Song
16d6fe099d Allow rebuild sepolicy even though the image does not exist.
The `--rebuild-sepolicy` option should work when the target
files folder `ODM/` or `VENDOR/` exists. We should not
assume that there is always an image to remove.

Bug: 270529723
Test: m otatools; Create merged builds with rebuild_sepolicy.
Change-Id: I93aeea2cd084fdc7f8179732d77a78bb1d144616
2023-02-23 23:37:42 +08:00
Vincent Donnefort
6e861e9cf8 Fix missing dev nodes in init_boot.img ramdisk
Previously ("Add null and console dev nodes to the Android initramfs")
introduced into init_boot.img the /dev/console and /dev/null nodes.
Sadly, it missed changing the release tools which are also generating
images in their own way.

Bug: 254835242
Change-Id: I4d2893ac24c09cf0d117422cbe2bde66083e7118
2023-02-17 10:52:28 +00:00
Kelvin Zhang
c8ff84b219 Use zip2zip to copy zipfiles
ZipDelete() works by copying every non-deleted entry to a new zipfile.
Current implementation uses python's zipfile module to perform entry
copying, which is inefficient, as every entry must be decompressed and
then re-compressed. Instead, use zip2zip which avoid re-compression.

Improvement: deleting META/dynamic_partitions_info.txt from
raven-target_files-9465001.zip improved from 500+ seconds to 13 seconds.

Change-Id: I0548255bc29380303314763f6d81e74bf3dbb76e
2023-02-15 16:55:48 -08:00
Dennis Shen
0dfb3aea75 Fix signing test failure
BUG: b/266137486
TEST: manual run target file signing pass with change
Change-Id: I4b8127937ba726560a69c79619a7f1a2882c9972
2023-01-31 13:28:09 +00:00
Hongguang Chen
1a7323390b Overwrite prop.default in all ramdisk fragments
default.prop can be linked to prop.default. To rewrite the fingerprints,
the original prop.default should be overwritten.

Bug: 264853953
Test: sign_target_files_apks, check artifacts
Change-Id: I8d22b8391dbfe13e74880ffad0619ef7fc4e20e0
2023-01-29 19:43:04 +00:00
Kelvin Zhang
e8ce38450f Disable imgdiff for non-AB OTA
imgdiff assumes that libz produces identical output across different
platforms, which is sometimes not the case. This causes incremental OTA
on non-AB targets to fail ocassionally. For stability, disable imgdiff.

Test: th
Bug: 265895599
Change-Id: I92d09a50762ec69a79caa37ed4d6e0db1ec8a47a
2023-01-23 20:43:09 -08:00
Kelvin Zhang
f80e886273 Fix non-AB OTA signing issue
After aosp/2201756 , caller must explicitly pass signing key as a
parameter to FinalizeMetadata . We forgot to update non_ab.py ,
therefore causing non-AB OTAs to be unsigned.

Test: th
Bug: 265895599
Change-Id: Ia54b1105f37b6574742b9b55d715fc9c61c223eb
2023-01-20 10:18:11 -08:00
Baligh Uddin
66dbfde748 Switch check_targetfiles_vintf to use the correct blkid binary.
BUG: 264426585
Test: TH
Change-Id: Iefe86e78c3e607ac27a5197d45fa832fa281c09d
2023-01-20 00:56:54 +00:00