Commit Graph

7575 Commits

Author SHA1 Message Date
Romain Jobredeaux
73bddf4c72 Merge "Create Bazel BUILD targets and Bazel rule for logtag file conversion." 2021-09-22 17:28:31 +00:00
David Anderson
dfc9d7912f Merge "Avoid too much reserved_size for erofs image" 2021-09-16 19:36:29 +00:00
Daniel Norman
a529e0fd2e Merge "Build vendor images first when rebuilding with vendor_otatools." 2021-09-16 17:29:43 +00:00
Daniel Norman
78554ea568 Build vendor images first when rebuilding with vendor_otatools.
Only supports rebuilding vendor.img and odm.img currently.

This allows "other" images (e.g. vbmeta, super_empty, ...) to
be rebuilt using the latest tooling.

Bug: 188491126
Bug: 192422274
Test: Sign a GRF R+S merged target_files package using:
      sign_target_files_apks \
        --vendor_otatools=<otatools.zip from R> \
	--vendor_partitions=vendor,odm \
	merged-target_files.zip \
	signed-target_files.zip
Change-Id: Ib93aea9f083ee9b722c31a42dcc780b5222053b8
2021-09-15 21:53:25 +00:00
Treehugger Robot
50ec71eac8 Merge "Zipalign: Avoid unnecessary padding" 2021-09-15 00:34:44 +00:00
Fabien Sanglard
df73d1b473 Zipalign: Avoid unnecessary padding
Problem: Aligned entry always get padded to the next aligned slot.
Solution: Fix alignment test.

Bug: 199932309
Test: align_test.cpp
Change-Id: I1e61c67dcf45454dd782d0fcc6903f7e71954083
2021-09-14 15:20:42 -07:00
Christopher Ferris
fe9efa5a61 Merge "Make the script python3." 2021-09-14 21:03:01 +00:00
Greg Kaiser
d9c6d5fc91 Merge "Revert "Rebuild vendor images in sign_target_files_apks with vendor otatools."" 2021-09-14 19:38:29 +00:00
Greg Kaiser
e086f7222e Revert "Rebuild vendor images in sign_target_files_apks with vendor otatools."
This reverts commit a1094e9f0c.

Reason for revert: Likely cause for b/199911935

Change-Id: Ib20400cc2f740a26fbbfb0da34fdf20ed7dc620e
2021-09-14 19:32:27 +00:00
Fabien Sanglard
1011d45f5b Zipalign: Remove unused variables
Remove unused variable resulting in compilation warnings.
Remove unused temporary variable.

Bug: NA
Test: zipalign_tests
Change-Id: Ic2236b5878074f0c2537ab0b9f94dfd9c42225d8
2021-09-13 16:52:22 -07:00
Christopher Ferris
6a54633796 Make the script python3.
Test: Ran the python version of the script and the python3 version
Test: and verified the output is the same. This is for all three
Test: outputs (html, text, xml).
Test: Builds without an error.
Change-Id: I5138b0a1efbbc3ab764c644afc1783c79fd7e250
2021-09-13 14:54:58 -07:00
Daniel Norman
97e0c2831d Merge "Rebuild vendor images in sign_target_files_apks with vendor otatools." 2021-09-13 20:12:11 +00:00
Romain Jobredeaux
bbdd255ad7 Create Bazel BUILD targets and Bazel rule for logtag file conversion.
Test: b build build/make/tools/...
Change-Id: I0a12e6cfff553a927f86381358a3f5f58fec0c75
2021-09-13 14:48:50 +00:00
Huang Jianan
6552727e03 Avoid too much reserved_size for erofs image
After the erofs image is built, it won't be modified again. The reserve
room is only used for the hash footer. There is no need to set an
excessive margin.

Bug: 199248596
Test: erofs image size is smaller than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: If2535b9ad2c2d60bea74feeca576a168387ff341
2021-09-13 02:03:13 +00:00
Yifan Hong
1ab1c2ff96 Merge "Avoid repeated building of erofs image" 2021-09-11 00:05:45 +00:00
Yi Kong
8b50dea627 Fix arguments passed to llvm-readobj
Upstream llvm-readobj no longer accepts single dash prefix for long
options. Changed to double dash.

Also let the script to raise exception if rc is non-zero, to avoid
getting confusing error later in the program.

Test: build
Bug: 197230471
Change-Id: I364f51322904431d430549f79af58f455f47e028
2021-09-10 21:17:45 +08:00
Tianjie
e7ab38d964 Add option to override minor version
Pass the minor version override to delta_generator. So we can test
features in tradefed before releasing it to droidfoods.

Bug: 197361113
Test: TH
Change-Id: Ie8846935dfa13784f0745513be13811908f44ac3
2021-09-09 12:16:19 -07:00
Daniel Norman
a1094e9f0c Rebuild vendor images in sign_target_files_apks with vendor otatools.
This is needed until image compilation tooling is backwards compatible.

Bug: 188491126
Bug: 192253131
Bug: 192422274
Test: Sign a GRF R+S merged target_files package using:
      sign_target_files_apks \
        --vendor_otatools=<otatools.zip from R> \
	--vendor_partitions=vendor,odm,vbmeta \
	merged-target_files.zip \
	signed-target_files.zip
Change-Id: Iec208f544b56f8ed577344d8d6ca904773d8baab
2021-09-09 17:32:22 +00:00
Huang Jianan
ffa1d57819 Avoid repeated building of erofs image
The image size of erofs is only affected by the content of the input
file, so it doesn't care about the input image_size. In addition, the
build time of erofs image is longer than ext4 due to compression. Only
execute mkfs once to reduce unnecessary time consumption.

Bug: 199248596
Test: erofs image build faster than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: I23a4e640d5066485de3fea0f808998045e716d1f
2021-09-09 14:18:59 +08:00
David Anderson
c01c341b6a Merge "Don't sparse right-sized ext4 and erofs images." 2021-09-08 22:25:37 +00:00
Chih-hung Hsieh
799e26e641 Merge "Handle warning lines from RBE" 2021-09-03 23:44:38 +00:00
Chih-Hung Hsieh
77e3146600 Handle warning lines from RBE
* Recognize USE_RBE special environment variable.
  * Remove RBE special file path prefix /b/f/w/
    and any leading characters.
  * Only do this for android now, maybe for chrome later.
* When finding android_root, do not use
  /b/f/w/ and /tmp/ file paths.
* Recognize BUILD_ID and add it into .html output.
* Style improvement with continue statements.

Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Bug: 198657613
Change-Id: I57746f026d5d8b71e792127b8d6ba15f6b28103c
2021-09-03 14:23:55 -07:00
David Anderson
9e95a02a87 Don't sparse right-sized ext4 and erofs images.
When we introduced Dynamic Partitions, we stopped giving readonly
partitions fixed sizes. In addition we introduced deduplication for
ext4. These two factors greatly reduce the impact of sparse images,
since there aren't many fill blocks to optimize.

This patch disables sparsing for images that are rightsized and do not
explicitly specify extra reserved space. This makes the images a little
easier to work with from an engineering perspective. They no longer have
to be unsparsed to interact with any tooling. It also eases a potential
source of bugs, as b/184225422 is not reproducible with sparsing off.

On Pixel, the difference between the sparsed partitions and unsparsed is
12M (out of roughly 4G).

Bug: 198001223
Test: make, treehugger, make target-files-package
      dynamic partitions are no longer sparse images
Change-Id: I74459f8abe74a15a24ba5a40cf701e6af2db8179
2021-09-01 13:12:39 -07:00
Kelvin Zhang
1250bcab0a Allow user to manually override enable_vabc_xor
--enable_vabc_xor=false to disable, --enable_vabc_xor=true or
--enable_vabc_xor to enable

Also the flag is now enabled by default. But update_engine will
still ignore XOR ops, as update_engine support for XOR is WIP.

Test: th
Change-Id: I038b0dda5cc9efcc1620e72c858626b56ffb1611
2021-08-31 18:52:47 +00:00
Kelvin Zhang
414ca429cf Check for vintf compatiblity early in generation process
Currently CheckVintfIfTrebleEnabled is called after generating payload.
Users have to wait for ~1 hour before vintf is checked. Do the check
early and fail early.

Test: th
Change-Id: I2e4a8c7eaa62c94c92973c127b1d825a26e63b35
2021-08-31 11:22:34 -07:00
Sasha Smundak
6b795dc6a5 Add find_files builtin, use it to fix find_and_copy implementation
The macro find-and-copy finds all the files in the given source tree that
match the given filename patten and create <source>:<dest> pair with the
same relative path in the destination tree.

Bug: 193540681
Test: rbcrun build/make/tests/run.rbc
Change-Id: Ic4315ce2fab7a7791ab55dd9eed039205a1c721a
2021-08-19 13:54:53 -07:00
Kelvin Zhang
ad427381b9 Respect VABC Xor prop in ota_from_target_files
Test: generate ota on a non-vabc xor target, make sure it's not used
Change-Id: Ibdbcfe3775c96848c37c7f580121583d4bd57fce
2021-08-12 17:55:57 -07:00
Sasha Smundak
3c569799ca Fix rblf_file_exists, it should return true for a directory, too.
Fixes: 184278019
Test: internal
Change-Id: I68c3274a4540dd36e047d6c2027766dbe75d38f8
2021-08-10 09:12:56 -07:00
Kelvin Zhang
3645e89d8e Merge "Make apex util detect if hashtree is present" 2021-08-06 23:31:59 +00:00
Kelvin Zhang
7cab7503e2 Make apex util detect if hashtree is present
Test: sign_target_fiels_apks oriole_target_files.zip
Bug: 195194430

Change-Id: I919d169ce4d66e6a1cdbfd15babde25005971a30
2021-08-06 11:51:37 -04:00
Yifan Hong
a4140d22df releasetools: better logs.
Bug: 195481978
Test: none
Change-Id: I86edfa653715dbbfab22ea29f4fd9e64fc57ef5a
2021-08-04 18:09:15 -07:00
Kelvin Zhang
25ab998d82 Fix a bug in computing streaming property of payload.bin
When computing the data offset of an entry in zip file, we used length
of extra field from central directory. That is correct most of the time
but wrong if the extra field in central directory has different length
than the one in local file directory. Since python's zipfile doesn't
provide an API to access local file header, we need to parse local file
header ourselves and extract length of extra field.

An incorrect offset will cause magic mismatch error from update_engine,
as update_engine expects to find uncompressed payload at the recorded
offset.

Test: th, partner verification
Bug: 191443484

Change-Id: Id670cd79b0bd65adffaaa5224ae4f8065d66b358
2021-07-28 11:40:18 -04:00
Jooyung Han
6c4e8d5f97 Merge "cleanup: remove unnecessary steps in apex_utils" 2021-07-28 00:56:41 +00:00
Kelvin Zhang
a19fb315e0 Skip vbmeta_system if it doesn't exist
Test: validate_target_files bonito-target_files-7582772.zip
Change-Id: I2da4f5a7897b1ef155e17441a8f5b97c0716d601
2021-07-27 09:17:05 -04:00
Kelvin Zhang
76e3fb0597 Merge "Add an option for VABC xor" 2021-07-24 00:25:03 +00:00
Jose "Pepe" Galmes
81005b13f7 Merge "Support for dexopt postprocessing in merge_target_files." 2021-07-23 04:42:43 +00:00
Jose Galmes
9c8f6eb7e1 Support for dexopt postprocessing in merge_target_files.
When using the VSDK, dexopt is not applied during the vendor build.
To avoid a first-boot time regression, dexopt is applied during the
merge stage, by running dexopt on the vendor apps and rebuilding
the vendor image.

Bug: 188179859
Test: Tested in keystone with VSDK target
Change-Id: Ie8e2d0a82850a2901fa6f250433bcbb43f0a97f2
2021-07-22 13:39:10 -07:00
Jooyung Han
f9be5eeb44 cleanup: remove unnecessary steps in apex_utils
`zipalign` before `signapk -a 4096` is unnecessary step since `signapk
-a 4096` does the same thing.

And, compressed APEX files don't need entries to be aligned.

Bug: 193362313
Test: atest --host releasetools_test
Change-Id: I0758d1162f7c3eb8cc9c73978b5ae5484f007a3a
2021-07-22 18:37:02 +09:00
Tianjie
35ec7561d0 Amend owner files for releasetool scripts
Test: th
Change-Id: Ia77586f634609e5f3a8868e7d1237776052aab30
2021-07-21 16:10:53 -07:00
Jooyung Han
ebe9afeb4a signapk: --align-file-size flag
A new flag tells that the resulting file size should be aligned as well.

For example, signapk -a 4096 --align-file-size ... generates the output
file of 4K-aligned sized.

Bug: 192991318
Test: check built APEX file size.
  APEX files should be sized as 4K-aligned.
Change-Id: I1c287e7219b4551bbb3e5957bdb64d80adfd3c39
2021-07-21 00:19:07 +00:00
Jooyung Han
28176b66ba Merge "zipalign: use getopt" 2021-07-19 22:53:21 +00:00
Daniel Norman
e62232d71a Merge "Adjust some behaviors for --rebuild-sepolicy" 2021-07-19 20:39:09 +00:00
Jooyung Han
b72009a8e9 zipalign: use getopt
Bug: 192991318
Test: zipalign_tests
Change-Id: I49f8ebc2cc2dbefe53c2eedda8d185eb658b4124
2021-07-17 07:19:09 +09:00
Treehugger Robot
c26dc73d81 Merge "Update language to comply with Android's inclusive language guidance" 2021-07-15 08:34:47 +00:00
Tianjie Xu
4772ec7583 Merge "Write the vbmeta digest to the output zipfile also" 2021-07-15 04:43:36 +00:00
Tianjie
c3bf3d00a3 Write the vbmeta digest to the output zipfile also
During signing, we write the entry directly to the output zip, intead
of a temp directory. Add the logic to write vbmeta_digest.txt to output
zipfile too. So the digest file will show up in the signed target files.

Bug: 189926233
Test: add_img_to_target_files -a <target-files.zip>
Change-Id: Ibf28a8f97512bda8c8c695e06190e1fb6573c53e
2021-07-14 16:03:08 -07:00
Justin Yun
9d3356b306 fs_config reads headers from snapshots, if available
fs_config requires the information from the header files in
system/core and bionic/libc. To build the vendor side fs_config,
use the header files in the vendor snashot if the required version
exists.

Bug: 187222756
Test: check ninja depedency with and without snasphots.
Change-Id: Ibf96eab4d9a129745be1a19b2aa2e4c8f57cf6bf
2021-07-13 11:42:18 +09:00
Kelvin Zhang
f66caeec58 Add an option for VABC xor
Test: th
Change-Id: Ia6ec01a834280cbaada82012ed35dfeeff53eba6
2021-07-12 09:44:42 -04:00
Po Hu
0e4403e846 Adjust some behaviors for --rebuild-sepolicy
1. system_ext or product may not be a standalone partition,
2. keep symbolic links in vendor image,
3. target-files package may not contain IMAGES folder.

Bug: 192422274
Change-Id: I77867408cc764abb975319a23c387ec63fcf87c3
2021-07-08 09:49:48 +08:00
Daniel Norman
571e182e9c Regenerate odm or vendor using combined sepolicy if --rebuild-sepolicy.
This allows merged devices to boot using a precompiled_sepolicy built
from merged sources, rather than recompiling this sepolicy at boot
time every boot.

Bug: 178727214
Test: Merge an R+S build using --rebuild-sepolicy and --vendor-otatools.
      Observe odm.img is rebuilt by the vendor otatools.zip
        when merging.
      Observe device boots using ODM's precompiled_sepolicy file.
Test: Same as above, for S+S.
Test: Merge an S+S build using --rebuild-sepolicy and *not*
        --vendor-otatools.
      Observe odm.img is rebuilt without using a separate otatools.zip.
      Observe device boots using ODM's precompiled_sepolicy file.
Change-Id: I9595b8a3296d6deec21db8f0c9bc5b7ec4debd57
2021-07-01 16:29:15 -07:00