Commit Graph

7925 Commits

Author SHA1 Message Date
Treehugger Robot
9e0472e9c7 Merge "Fix recovery OTA error on merged targets" 2022-11-15 18:33:53 +00:00
Treehugger Robot
5cf8eef25e Merge "Fix 'expected str instance, bytes found' error when compute patches" 2022-11-14 19:14:22 +00:00
luoqiangwei1
3e3456bb72 Fix 'expected str instance, bytes found' error when compute patches
When building an incremental OTA package, if the compress_target=True
parameter is passed in when calculating a block, an exception will occur
in the subsequent join operation.

The reason is that Python 3 no longer allows str type join bytes type.

Bug: None
Test: Build Incremental OTA package

Change-Id: I6d556f9905b7ab75b70d3785334d71d5a6e5479b
Signed-off-by: luoqiangwei1 <luoqiangwei1@xiaomi.com>
2022-11-11 00:26:19 +08:00
Ibrahim Kanouche
f0c6cf501c Merge "Optimized project selection for getProjectMetadata" 2022-11-10 15:28:12 +00:00
Dennis Shen
da42e6fedb Merge "Update deapexer call to explictly use blkid" 2022-11-10 14:48:41 +00:00
Kelvin Zhang
fa40c04cba Fix recovery OTA error on merged targets
Currently, when merging target files, system side's update_engine
version info is copied into the merged target_files. This causes
recovery OTA failure, because update_engine_sideload daemon in recovery
is from vendor side. To fix it, use the older version of update_engine
config when merging target files.

Test: th
Bug: 257426678
Change-Id: I5d3304366749995c99a61ca0645a22ee047f1465
2022-11-09 14:43:17 -08:00
Cole Faust
7a48527b95 Fix crash in check_elf_file.py
`lib` is already a string and doesn't need to be decoded.

Bug: 258395719
Test: On internal master: build/soong/soong_ui.bash --make-mode --bazel-mode-staging TARGET_PRODUCT=aosp_coral TARGET_BUILD_VARIANT=userdebug droid dist device-tests platform_tests
Change-Id: I83ff35339bea9f01c61d953e251788e0894005dc
2022-11-09 12:50:35 -08:00
Dennis Shen
a8d1143beb Update deapexer call to explictly use blkid
To support erofs apex extract via deapexer, we need blkid to tell the filesystem type of underlying payload image. If it is ext4, debugfs_static will be used, if it is erofs, then we should use fsck.erofs. Thus we now need explicit blkid input.

Also, remove dependency on fsck.erofs for other deapexer calls. Only extract deapexer call needs blkid and fsck.erofs.

BUG: b/255963179, b/240288941
Change-Id: I8cea0f2def664f9cabf8b14c9a7ecc47bbddfbdd
2022-11-09 18:22:14 +00:00
Treehugger Robot
8f6474af5a Merge "Support chained init_boot partition signing" 2022-11-08 02:41:53 +00:00
Hongguang Chen
0d6b727e03 Support chained init_boot partition signing
Bug: 256048561
Bug: 256237041
Test: sign_target_files_apks -d certs --avb_init_boot_algorithm \
        SHA256_RSA4096 --avb_init_boot_key init_boot_rsa4096.pem \
	xxx.zip signed.zip (Check signed.zip/META/misc_info.txt)
Change-Id: I65fc7913089ae318d90df55a533d3e2ebd93b029
2022-11-07 15:48:23 -08:00
Dennis Shen
f58e548943 add EROFS apex support.
Signing code uses deapexer to expand payload image, deapexer currently
only supports ext4 payload image expansion using debugfs or
debugfs_static. To support EROFS, need to supply deapexer with
fsck.erofs native binary for EROFS payload expansion.

cherrypick of ag/20154950BUG: b/195515562, b/240288941
Change-Id: Ibf707989f2502bfcf112202a3a6fdb74b6a8dbbc
2022-11-07 20:40:41 +00:00
Treehugger Robot
e32ad5e482 Merge "Convert soong_to_convert.py to python 3" 2022-11-05 03:34:03 +00:00
Cole Faust
5865420eb1 Convert soong_to_convert.py to python 3
Bug: 203436762
Test: m out/target/product/emulator_x86_64/soong_to_convert.txt
Change-Id: Ibf8d80a160315ed03a11fbeef66a630b224bea0a
2022-11-04 17:36:13 -07:00
Bob Badour
8eeff2b0b8 Merge "Fix final nits from noticeindex change." 2022-11-04 16:57:01 +00:00
Dennis Shen
44b714b7ca Update deapexer extract call with more input
BUG: b/255963179, b/240288941
Change-Id: Id7f5eb20e4c6f3c3e793860389f4be48f4f3938f
2022-11-03 17:39:04 +00:00
Ibrahim Kanouche
a68ed086ab Optimized project selection for getProjectMetadata
Test: m compliance_sbom

Change-Id: I9e60b7e138a4fca74170954d8bb83862677e750f
2022-11-03 16:01:13 +00:00
Bob Badour
3afc3004f2 Fix final nits from noticeindex change.
Test: m droid dist cts alllicensemetadata
Change-Id: Ie446e5693a09f4ae1c936cdae6f1e668105541f0
2022-11-02 10:24:04 -07:00
Bob Badour
88b02afbf0 Merge "Switch noticeindex from regex to proto." 2022-11-02 15:46:38 +00:00
Maciej Żenczykowski
458f90272c sbom_test - use UTC
Test: TreeHugger, m compliance_sbom
Signed-off-by: Maciej Żenczykowski <maze@google.com>
Change-Id: Id2abec8537a1dfa32d6816da4c34c86f46313086
2022-11-01 19:49:05 -07:00
Ibrahim Kanouche
1bb27903e8 Merge "Added SBOM generator module to implement the spdx utility bill of material" 2022-11-01 23:36:22 +00:00
Ibrahim Kanouche
bedf1a8a86 Added SBOM generator module to implement the spdx
utility bill of material

Added methods to graph.go to extract the annotation types

Test: m compliance_sbom

Test: tested against real targets, see go/paste/4932131502424064

Bug: 241591242
Change-Id: I344c15d64cd034d72076f9d9a677e593e288e3a6
2022-11-01 20:42:01 +00:00
Jihoon Kang
eb35b48df7 Merge "Create custom diff tool to compare stub contents" 2022-11-01 20:32:20 +00:00
Sam Delmerico
9fac7000ec Merge "move logtags rule from //build/make to //build/bazel" 2022-11-01 13:50:10 +00:00
Treehugger Robot
5667fe0a74 Merge "Allow RO build only for readonly mount points" 2022-10-31 22:58:32 +00:00
Bob Badour
5c12c66769 Replace nil-able *sync.Waitgroup with sync.Once
Simplifies synchronization and eliminates lock for nil waitroup.

Test: m droid

Test: m out/soong/.intermediates/packages/modules/StatsD/apex/com.android.os.statsd/android_common_com.android.os.statsd_image/NOTICE.html.gz

Change-Id: I381ee79e142214e7331241071f076db2f7960ba6
2022-10-29 22:48:13 -07:00
Bob Badour
d2c28ba897 Fix top-down resolve re-walking graph too much.
Firing off multiple concurrent walks of the same sub-tree with the same
conditions. Data race meant every walk would proceed. Move the logic to
entry of walk function and compare under lock.

Bug: 255526010

Test: m droid

Test: m out/soong/.intermediates/packages/modules/StatsD/apex/com.android.os.statsd/android_common_com.android.os.statsd_image/NOTICE.html.gz

Change-Id: Ie30edbb2ac9eaa9aa55badfc518d51eaadbb6be6
2022-10-29 22:27:49 -07:00
Jihoon Kang
3d38b6d9c8 Create custom diff tool to compare stub contents
Context:
- Create a tool to analyze loose equivalence of the stubs in two
  directories
- The tool can analyze strict equivalence of the directories of the
  stubs in two directories
- Analyze text to compare loose equivalence of the stub contents; Add a
  functionality to pass `skip_words` as arguments, which are optional
  list of words used to signal the tool which words are not considered
  diff
- The tool can be locally used to compare stub contents, and does not
  contribute to build process

Test: m
Change-Id: I74563a9a24ecdde939be2ce37b9096a9aeb4920a
2022-10-28 22:45:08 +00:00
Bob Badour
43daade15f Add some documentation and simplify data model.
Bug: 245562496

Test: m droid
Change-Id: Iae757a5767522e0734abbe3840ea0939620197e7
2022-10-28 12:06:08 -07:00
Bob Badour
ab5cfbd5b7 Switch noticeindex from regex to proto.
Bug: 254534552

Test: m droid dist cts alllicensemetadata

Test: repo forall -c 'echo -n "$REPO_PATH  " && $ANDROID_BUILD_TOP/out/host/linux-x86/bin/compliance_checkmetadata . 2>&1' | fgrep -v PASS

Change-Id: I34dfe143cdc88432d6e33abee096409a37ce3b7e
2022-10-27 18:13:24 -07:00
Bob Badour
d6574e5c97 Add AllMetadataFiles() method to track deps.
Improve METADATA parsing error message.

Example error message (for an unqoted string url starting with ssh:):

============
  error in project "<redacted>" METADATA "<redacted>/METADATA": proto: (line 4:12): invalid value for string type: ssh

METADATA and METADATA.android files must parse as text protobufs
defined by
   build/soong/compliance/project_metadata_proto/project_metadata.proto

* unknown fields don't matter
* check invalid ENUM names
* check quoting
* check unescaped nested quotes
* check the comment marker for protobuf is '#' not '//'

if importing a library that uses a different sort of METADATA file, add
a METADATA.android file beside it to parse instead
============

Bug: 254534552

Test: m droid dist cts alllicensemetadata

Change-Id: Ie5c0adc7362941e455dd522baaa31f2913d7db5f
2022-10-27 18:12:30 -07:00
Treehugger Robot
a2faaa8c45 Merge "Remove all ZIP64LIMIT hack" 2022-10-27 22:44:22 +00:00
Treehugger Robot
1fca7c5e7f Merge "Enable zip64 support in build image script" 2022-10-27 22:31:00 +00:00
Jooyung Han
e3862eacb6 Merge "Fix boot fail due to missing /apex in images" 2022-10-27 21:54:13 +00:00
Spandan Das
03b846ff37 Merge "Migrate check_elf_file.py to python3" 2022-10-27 16:46:18 +00:00
Jaegeuk Kim
551a2e66c3 Allow RO build only for readonly mount points
Should support RW for all other partitions.

Bug: 255454067
Change-Id: I9d87cf39195f8da0f6fe4153c703ba6a372aa2ea
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
2022-10-27 09:46:03 -07:00
Spandan Das
60b8195c83 Migrate check_elf_file.py to python3
Also, create a python_binary_host so that it runs using the hermetic
python toolchain.

Test: m check-elf-files
Bug: 203436762
Change-Id: I964342a27f6b6c9dcdbbe910d4dc6c9ec00c2213
2022-10-27 02:08:23 +00:00
Kelvin Zhang
37a4290909 Remove all ZIP64LIMIT hack
In the old days, we hacked values of ZIP64LIMIT to get around size
limitations of non-zip64 supported zip files. Now that we switched to
python3 + zip64, there's no point in keeping those hacks.

Test: th
Bug: 255683436
Change-Id: I913db33dad5503736c68a7a1f1321aa952019f60
2022-10-26 13:22:52 -07:00
Kelvin Zhang
1de1788dda Enable zip64 support in build image script
Images are getting bigger each year, we need zip64 support to store >4GB
files in .zip archive.

Test: th
Bug: 255683436
Change-Id: I9adbefa5ec72e357b02e46d6e629c054d466ddd7
2022-10-26 12:30:29 -07:00
Po Hu
4d2f64b6f0 Fix boot fail due to missing /apex in images
Do not move or rename any files in merge_target_files.py's temporary directory. They will be used to generate images further.

Bug: 254760332
Change-Id: I973a49b29234d49f5254a1403a0e236e69425d74
Test: check SYSTEM/apex in target-files.zip generated by merget_target_files.py
2022-10-26 02:34:12 +00:00
Treehugger Robot
1a6c0c5d86 Merge "Delete compare_fileslist.py" 2022-10-25 21:22:19 +00:00
Treehugger Robot
fe7277fdde Merge "The --why parameter in whichgit should be optional" 2022-10-25 20:08:59 +00:00
Cole Faust
29c032d260 Delete compare_fileslist.py
It was written in python 2, and is unused.

Bug: 203436762
Test: Presubmits
Change-Id: I3298ec06727ba1b72337753f52e3d09519fe0b39
2022-10-25 11:34:42 -07:00
Ibrahim Kanouche
bfa6c8e553 Merge "Added functions to projectmetadata to retrieve additional project info" 2022-10-25 17:25:11 +00:00
Cole Faust
3b9af315d2 Merge "Delete java-layers.py" 2022-10-25 17:00:52 +00:00
Cole Faust
c22b90c5d4 Merge "Delete parsedeps.py" 2022-10-25 17:00:11 +00:00
Treehugger Robot
f99edd9f67 Merge "Convert fileslist_util.py to python 3" 2022-10-25 15:58:19 +00:00
Treehugger Robot
aa3b322971 Merge "Convert findleaves.py to python 3" 2022-10-25 09:17:27 +00:00
Cole Faust
e0fab08a33 Convert fileslist_util.py to python 3
Also update it from getopt to argparse.

Bug: 203436762
Test: m $OUT/installed-files-root.txt
Change-Id: I006a36feab01cbbd76dc7d7c1c0675197848212a
2022-10-24 18:46:03 -07:00
Cole Faust
6a75c4f782 Delete java-layers.py
There are no usages of LOCAL_JAVA_LAYERS_FILE,
and java-layers.py was written in python 2.

Bug: 203436762
Test: Presubmits
Change-Id: I1be0f2547757a853e20811f28529479487037e94
2022-10-24 18:15:54 -07:00
Cole Faust
e995769d44 Delete parsedeps.py
This script is only ever manually run, and it's written
in python 2. Just delete it instead of updating to python
3 because it's unlikely anyone uses it anymore.

Bug: 203436762
Test: Presubmits
Change-Id: Ib55b3bf43a8066822308f49d1a948d27c520c5b2
2022-10-24 18:02:23 -07:00