Commit Graph

589 Commits

Author SHA1 Message Date
James Wu
c5e321a06e Revise minSdkVersion to fix failed mainline_modules_bundles-userdebug
Test: abtd tests
Bug: 293456031
Change-Id: I674c50690bd04b7e4867c4aceb31d5bdcb53be84
2023-08-02 19:25:58 +00:00
Cole Faust
6833d7d47a Remove OPTIONS.host_tools
It appears to be unused.

Test: Presubmits
Change-Id: Idd7f9dda996458b93da77c783a546cdb3d1e0047
2023-08-01 18:00:37 -07:00
Kelvin Zhang
43df0805c2 Fix test failures in releasetools
new_path used before assignment, use `path` instead.

Bug: 280425458
Change-Id: I55086f92cf109af67032614cefde60e6478bb149
2023-07-24 13:16:03 -07:00
Kelvin Zhang
f3f6d0569b Merge "Mark extracted files as executable" into main 2023-07-11 17:39:31 +00:00
Elliott Hughes
c1577ff58b Merge "Switch from minigzip to gzip." into main 2023-07-11 15:52:27 +00:00
Kelvin Zhang
4cb28f6f80 Mark extracted files as executable
Python's zipfile doens't restore file permission by default, so we need
to manually restore permission.

Test: th
Bug: 290643514
Change-Id: I89c1e2ee178b534fa7e3f02afd04d170100d37e7
2023-07-10 13:03:07 -07:00
Treehugger Robot
59f99b31eb Merge "Fix test flake" 2023-06-28 01:18:55 +00:00
Elliott Hughes
97ad120adb Switch from minigzip to gzip.
Bug: http://b/288169261
Test: treehugger
Change-Id: I404cbcb850734ed8291c2215a4329fb372806d63
2023-06-27 15:39:05 -07:00
Kelvin Zhang
97a5afe0f3 Fix test flake
Test: th
Bug: 289013326
Change-Id: Ib233978ff861b010bdc3bd1a02ea9baefe6a3489
2023-06-27 10:30:48 -07:00
Xin Li
24c928cda8 Merge "Merge Android 13 QPR3" 2023-06-26 22:29:55 +00:00
Kelvin Zhang
e473ce9e80 Handle symlinks when extracting zipfiles
python3.11's zipfile implementation does not handle symlinks. This
causes important symlinks in ramdisk to be broken, and later causing a
boo failure.

Test: unzip a target files with symlinks, make sure symlinks are created
Bug: 287896098

Change-Id: Ia7d6ac8ffb03807680a36ff648aa11afafb7f481
2023-06-21 13:31:26 -07:00
Treehugger Robot
5c3a34b358 Merge "Handle zip64 extra fields better" am: 57c1aa7b16
Original change: https://android-review.googlesource.com/c/platform/build/+/2628431

Change-Id: Ieefcc8ee08c2d91f447fc5209967d43db7454c75
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-06-17 23:04:21 +00:00
Kelvin Zhang
1e774245a4 Handle zip64 extra fields better
Test: check_target_files_signatures
Bug: 283033491
Change-Id: I7da89f8389c09cc99201cff342483c158bd7e9c1
2023-06-17 09:21:46 -07:00
Treehugger Robot
8ac8ce2b16 Merge changes I9c1a5346,I0a9d2c58 am: 5c73eb33db
Original change: https://android-review.googlesource.com/c/platform/build/+/2627031

Change-Id: Ied665b5cbd2d1b207c184458eef0ef49259df129
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-06-15 16:36:23 +00:00
Kelvin Zhang
38d0c373ac Fix python3.11's support for zip64
Bug: 283033491
Test: check_target_files_signatures -v -l
Change-Id: I9c1a5346e3a5f3920242dc9a5268d999f50a4937
2023-06-14 12:54:57 -07:00
Treehugger Robot
976a36fb8e Merge "Use python based unzip function for portability" am: 3cda7834aa
Original change: https://android-review.googlesource.com/c/platform/build/+/2615433

Change-Id: Id10787e68ac4a0a6ea0ada47a24f46dd7e128915
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-06-06 19:08:34 +00:00
Treehugger Robot
4c357806a3 Merge "Always set a avb salt for hermetic build" am: fff48d788f
Original change: https://android-review.googlesource.com/c/platform/build/+/2613794

Change-Id: I3feffd1c65f8e92e7c6a6c6e0481e206b28ec8dd
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-06-06 19:07:16 +00:00
Kelvin Zhang
7c9205b008 Use python based unzip function for portability
Bug: 283033491
Test: th
Change-Id: Ief86b55c1d4a14220a0fb593c583a721d59cf86c
2023-06-05 15:54:12 -07:00
Kelvin Zhang
c819b29f46 Always set a avb salt for hermetic build
When building images via `m` , build_image.py is invoked directly
without going through add_img_to_target_files. To ensure images built in
either way are identical, move uuid/salt computation to build_image.py,
so that the same uuid/salt will be used.

Bug: 281960439
Test: m installclean && m && m target-files-dir , maks sure images in
$OUT and $OUT/obj/PACKING/target_files_intermediates are identical

Change-Id: Icdab29df84f5a0ec7c080f99f9fdbdc3c9b10b90
2023-06-05 09:59:54 -07:00
Treehugger Robot
74ad39125c Merge "Unsparse images before generating OTA" am: b4406d06f0
Original change: https://android-review.googlesource.com/c/platform/build/+/2598008

Change-Id: I40ac20dd7b30d11e1eda77688718566af4c1c416
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-23 21:22:54 +00:00
Kelvin Zhang
2268091521 Unsparse images before generating OTA
Test: th
Bug: 283172692
Change-Id: Ie6d3dc704fd9a8c107e2888222e4c2bf804dad3e
2023-05-22 10:22:59 -07:00
Treehugger Robot
5e4f59f9ca Merge "Support generating partial OTAs from extracted target_files" am: c3216a321b
Original change: https://android-review.googlesource.com/c/platform/build/+/2577071

Change-Id: I79fa834772dc12fe26678238bcf80ecffd2a34ce
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-04 00:54:58 +00:00
Kelvin Zhang
6b10e15acc Support generating partial OTAs from extracted target_files
This allows the build system to potentially paralleize generation of OTA
package and zipping of target files

Bug: 262185376
Bug: 227848550
Change-Id: I90b6c25761683ebe3803b22fc8e23540a5282c66
2023-05-02 20:48:14 -07:00
Treehugger Robot
8583eae383 Merge "Use misc_info.txt to check for VABC" am: ed1ab26f5d
Original change: https://android-review.googlesource.com/c/platform/build/+/2572975

Change-Id: I2ef6cf286ed259dafc20f8bb1f4b0388a32b5fda
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-01 22:41:40 +00:00
Kelvin Zhang
e634bde6bd Use misc_info.txt to check for VABC
Change-Id: I77f23a51ddb027254ccda8d7ab17e64205522300
2023-04-28 23:59:43 -07:00
Xin Li
d6b6d0102a Merge Android 13 QPR3 tm-qpr-dev-plus-aosp-without-vendor@9936994
Bug: 275386652
Merged-In: I4b2ea3956c34aa56cc6f54bab7148f116c34759e
Change-Id: I900022a8e8f104b6dc52745211c428601be53ba6
2023-04-21 09:49:08 -07:00
Kelvin Zhang
9dbe2ce40f Allow ota_from_target_file to work entirely on directories
When building android, build system will first zip target_file directory
into a .zip file, and then invoke ota_from_target_files. We
ota_from_target_files can work directly on top of directories, we can
parallelize OTA generation and target_file zipping, making builds
faster.

Bug: 227848550
Bug: 277028723
Test: th
Change-Id: Id72bd7cca708af0e1bae2db754f39b27777cd601
2023-04-18 17:47:29 -07:00
Treehugger Robot
b7a45ea13d Merge "Try with search_path for some avb path args" 2023-04-17 20:43:12 +00:00
David Anderson
23c2df71bc Merge "ota_from_target_files: Disable XOR on <T vendor." 2023-04-15 04:52:46 +00:00
Treehugger Robot
7b640806da Merge "Revert "Remove all ZIP64LIMIT hack"" 2023-04-15 01:48:58 +00:00
David Anderson
1c59617820 ota_from_target_files: Disable XOR on <T vendor.
Bug: N/A
Test: ota_from_target_files
Change-Id: Ia9093a3c3f79d88f98f0777e8a0f8c72741055b8
2023-04-14 16:01:55 -07:00
Kelvin Zhang
f92f7f046a Revert "Remove all ZIP64LIMIT hack"
This reverts commit 37a4290909.

Reason for revert: b/278156419

Change-Id: I67ea667619a9623be849d911993010ef0f0bfd88
2023-04-14 21:32:54 +00:00
zhangyongpeng
707569711b Try with search_path for some avb path args
If we can't find relative signing_helper path, try with the OPTIONS.search_path dir prefix, and integrate the avb_*_key_path for the same logic.

Test: Build && releasetools_test

Signed-off-by: zhangyongpeng <zhangyongpeng@xiaomi.com>
Change-Id: Ifb1096ddea90693668f3344eb242bf9725113d11
2023-04-14 12:12:47 +08:00
Treehugger Robot
766eaa6d78 Merge "Invoke delta_generator directly" am: 84031f7b49 am: 651e2a94c5
Original change: https://android-review.googlesource.com/c/platform/build/+/2529019

Change-Id: I7471c375cf5dc05680e71f64435325fb96159f2f
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-04-12 02:49:54 +00:00
Kelvin Zhang
fcd731e3d6 Invoke delta_generator directly
Currently, ota_from_target_files(a python script) calls
brillo_update_payload(a bash script), which then calls delta_generator(a
C++ binary) to do the actual diffing. Having brillo_update_payload in
the middle does not offer any additional flexibility, but it makes
maintaince more difficult. Bash code is less readable and harder to
debug. Further more, everytime we want to add a new flag to
delta_generator, we have to add the flag in three places:
ota_from_target_files, brillo_update_payload, delta_generator.

Historically, brillo_update_payload was there because we inherited from
ChromeOS side. This bash scripts extracts target_files.zip and invoke
delta_generator to generate the actual OTA. Any customizations we want
on OTA must be implemented by modifying the input target_files.zip ,
Manipuating big zip files is slow and inefficient.

To make thing simpler/faster, remove the intermdiary call to
brill_update_payload. ota_from_target_files will now extract target
files and call delta_generator directly.

Test: th
Bug: 227848550
Change-Id: I44b296e43bf7921bcf48ef6a1e4021a12669739c
2023-04-11 13:53:05 -07:00
Kelvin Zhang
3dc5fb7392 Merge "Fix choosing wrong diff tool for recovery-from-boot patch" am: 3aa4644117 am: c8e58a10a0
Original change: https://android-review.googlesource.com/c/platform/build/+/2501818

Change-Id: Ic11be84be27ee80860202f387961de1e518f6ff4
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-04-05 17:46:57 +00:00
Oleg Lyovin
6d75a854ef Fix choosing wrong diff tool for recovery-from-boot patch
'bsdiff' tool should be used when recovery and boot image
have mismatching entries. This includes not only the case
when system_as_root enabled, but include_recovery_dtbo and
include_recovery_acpio too.

Add check if they are enabled to synchronize logic with
core/Makefile, which already takes this into account.

Test: make dist -j
Change-Id: Ie673403ae8eae446d006cc524dd48ac6c56194fc
Signed-off-by: Oleg Lyovin <ovlevin@sberdevices.ru>
2023-03-22 17:53:26 +03:00
Treehugger Robot
679fe3fc75 Merge "Fix sparse image detection" am: 135eb025f7 am: 5a776518a1
Original change: https://android-review.googlesource.com/c/platform/build/+/2467460

Change-Id: I81a39cdba26ff44df1330fd7593a9f85a739620e
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-03-08 00:09:29 +00:00
Kelvin Zhang
0452128f09 Fix sparse image detection
Instead of relying on image build flags, check the actual file to
determine if input is sparse image.

Test: th
Change-Id: I6e58603535039eb3c4e799c61ed9f4783f1888dd
2023-03-02 09:42:52 -08:00
Vincent Donnefort
f74012fd97 Merge "Fix missing dev nodes in init_boot.img ramdisk" am: 65af9993b3 am: 07738c42b2
Original change: https://android-review.googlesource.com/c/platform/build/+/2442241

Change-Id: I09c97d6d099179f7760a7ab17467d08b19710455
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-02-20 11:36:17 +00:00
Vincent Donnefort
6e861e9cf8 Fix missing dev nodes in init_boot.img ramdisk
Previously ("Add null and console dev nodes to the Android initramfs")
introduced into init_boot.img the /dev/console and /dev/null nodes.
Sadly, it missed changing the release tools which are also generating
images in their own way.

Bug: 254835242
Change-Id: I4d2893ac24c09cf0d117422cbe2bde66083e7118
2023-02-17 10:52:28 +00:00
Treehugger Robot
aa651b93b0 Merge "Use zip2zip to copy zipfiles" am: a59e5b7a29 am: 8c53b320ed
Original change: https://android-review.googlesource.com/c/platform/build/+/2439632

Change-Id: Iec87d556be8d028464c81d377d65241fca254057
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-02-16 23:55:10 +00:00
Kelvin Zhang
c8ff84b219 Use zip2zip to copy zipfiles
ZipDelete() works by copying every non-deleted entry to a new zipfile.
Current implementation uses python's zipfile module to perform entry
copying, which is inefficient, as every entry must be decompressed and
then re-compressed. Instead, use zip2zip which avoid re-compression.

Improvement: deleting META/dynamic_partitions_info.txt from
raven-target_files-9465001.zip improved from 500+ seconds to 13 seconds.

Change-Id: I0548255bc29380303314763f6d81e74bf3dbb76e
2023-02-15 16:55:48 -08:00
TreeHugger Robot
8bdaa29ab1 Merge "Support SHA minSdkVersion in APK in APEX" into tm-dev am: 240c63fba3 am: 73214a06a4 am: 7b9f430cec
Original change: https://googleplex-android-review.googlesource.com/c/platform/build/+/21337794

Change-Id: I9a9f1c7a09d71defc019cbb0ce877b2730ae181e
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-02-11 04:44:58 +00:00
Paul Duffin
a03f1265fb Support SHA minSdkVersion in APK in APEX
Bug: 266903788
Test: rm -fr out/dist
      UNBUNDLED_BUILD_SDKS_FROM_SOURCE=true UNBUNDLED_BUILD_TARGET_SDK_WITH_API_FINGERPRINT=true TARGET_BUILD_APPS="com.google.android.rkpd" vendor/google/build/mainline_modules_bundles.sh
      # Without this change it fails, with this change it passes at
      # which point run the following.
      mkdir contents
      cd contents
      unzip ../out/dist/dev_keys_signed/com.google.android.rkpd/com.google.android.rkpd.apks
      deapexer extract standalones/standalone-x86.apex apex-contents
      aapt2 dump badging apex-contents/priv-app/rkpdapp.google@*/rkpdapp.google.apk  | grep Version
      # The sdkVersion and targetSdkVersion should both be set to SHA
      # versions.
Change-Id: I9ca54affd30f0a6cf561f1274b991d39e7cf1bce
(cherry picked from commit 730d0c437a)
Merged-In: I9ca54affd30f0a6cf561f1274b991d39e7cf1bce
2023-02-08 17:28:29 +00:00
Kelvin Zhang
b81b4e3621 Allow vendors to add custom vbmeta partitions
Currently we only have vbmeta_system and vbmeta_vendor. Add
BOARD_AVB_VBMETA_CUSTOM_PARTITIONS , which allows OEMs to define
their own vbmeta_*.img partitions. example: aosp/2381823

Bug: 263568805
Change-Id: Id671e2c3aee9ada90256381cce432927df03169b
2023-01-12 17:34:42 -08:00
Kelvin Zhang
37a4290909 Remove all ZIP64LIMIT hack
In the old days, we hacked values of ZIP64LIMIT to get around size
limitations of non-zip64 supported zip files. Now that we switched to
python3 + zip64, there's no point in keeping those hacks.

Test: th
Bug: 255683436
Change-Id: I913db33dad5503736c68a7a1f1321aa952019f60
2022-10-26 13:22:52 -07:00
Kelvin Zhang
eb147e051e Fix ota build error when using boot variable files
When performing a deep copy of info dict, input_file is serialized as
filename. Make ExtractFromInputFile support reading from filename of a
ZipFile.

Test: build OTA with partner provided target_files
Bug: 253549364
Change-Id: I366a076c32d638d61b83a1df9bf864807cf5257c
2022-10-21 10:53:21 -07:00
Kelvin Zhang
5ef25191f7 Fix picle error on ota_from_target_files
When trying to make a deep copy of PartitionBuildProps, execution fails
with picle error because PartitionBuildProps stores a ZipFile object,
which stores a lock object, which doesn't support pickling.

To fix it, pickle ZipFile object as a str to its path.

Test: generate ota with boot variable files
Bug: 253549364
Change-Id: I92c5d78ce65045495203c85862922d33d886e9ea
2022-10-19 11:25:22 -07:00
Wei Li
8895f9ecea Change function ZipDelete to use Python module zipinfo instead of command 'zip'.
'zip' is a host command, and it can not be provided as a dependency when some release tools are used in Bazel rules.

Test: atest --host releasetools_test
Bug: 243748589

Change-Id: Ie5f42eadbd316ccd018b19194c466a908971af82
2022-10-14 18:31:56 -07:00