Commit Graph

6708 Commits

Author SHA1 Message Date
Tao Bao
53af0f1a97 Merge "releasetools: Remove the mismatch map files when generating retrofit OTAs." am: 9814cb6b23
am: dc6d14b1c9

Change-Id: I2b838c52264ed1e38f1b386307cfa8fc788e7f25
2018-11-28 15:05:31 -08:00
Tao Bao
03fecb669c releasetools: Remove the mismatch map files when generating retrofit OTAs.
The map files contain the filename-block mapping for the original
images.

Bug: 120109184
Test: Generate an incremental OTA that was previously failing.
Change-Id: I1285d0b24a435477b958e6c2e4f474acf8f7578b
2018-11-28 10:59:23 -08:00
Yifan Hong
b57948b666 Merge "Infer prebuilt image size" am: 08d07f5aa9
am: f462760510

Change-Id: I3fc69f560370d30d4c87a4871c9f152d191ed979
2018-11-27 16:53:43 -08:00
Treehugger Robot
08d07f5aa9 Merge "Infer prebuilt image size" 2018-11-28 00:36:50 +00:00
Yifan Hong
70fc029ab8 Infer prebuilt image size
Test: call script twice; the second time uses prebuilts
Fixes: 119769527

Change-Id: Icd98a4dbfed91538cbc07aab957c7b042c05b330
2018-11-27 11:21:21 -08:00
Tao Bao
7a1617c529 Merge "Use common.RunAndCheckOutput() in add_img_to_target_files.py." am: f21a8f5376
am: 132f6c8a2e

Change-Id: I5a93a034ef486e9dde544e792780d073b317c0e9
2018-11-26 10:58:32 -08:00
Tao Bao
f21a8f5376 Merge "Use common.RunAndCheckOutput() in add_img_to_target_files.py." 2018-11-26 18:41:36 +00:00
Yifan Hong
50a144b7fe Merge "dynamic_partition_use -> use_dynamic_partitions." am: 9b024fe583
am: 0d69a1372a

Change-Id: I04da384285b4e30dc84f525c661c413534e2fb66
2018-11-21 16:41:32 -08:00
Yifan Hong
50611038e9 dynamic_partition_use -> use_dynamic_partitions.
Test: build OTA package
Change-Id: Ia43d23f49fc024ecc767da1c2186c4b446c034de
2018-11-21 13:00:57 -08:00
Tao Bao
2764aee664 Use common.RunAndCheckOutput() in add_img_to_target_files.py.
Test: `m dist` with aosp_marlin-userdebug
Test: python -m unittest test_add_img_to_target_files.py
Change-Id: Icff60a44e5c189c2443fbaba3fc6a2b106b409ea
2018-11-21 11:04:32 -08:00
Yifan Hong
6d5d0595a0 Merge changes from topic "dp_retrofit_gen" am: d585de5933
am: 34f75d3c7b

Change-Id: I91e442f9be6faf83ce3dba92435f26059b571d7d
2018-11-20 15:03:01 -08:00
Yifan Hong
50e7954ac9 Generate retrofit OTA.
Test: m otapackage -j, manual apply on top of P
Bug: 118506262

Change-Id: Iedc389a1058007a604de0482f2348a9ef0d5892a
2018-11-20 11:17:39 -08:00
Yifan Hong
c767f7cbfc Add split super images to target_files.
Test: build target files package
Bug: 118506262
Change-Id: Iee6143987c9bfa6f2d287e20c06ea5ef25e82985
2018-11-20 11:17:39 -08:00
Mark Salyzyn
eb958c388c Merge "fs_config: fix fs_config_*_system targets" am: 25d43708c8
am: bff53580b8

Change-Id: I028d4ef40237b486fdecfc8b66148f5c7e1b5d93
2018-11-19 07:31:44 -08:00
Mark Salyzyn
705b59e155 fs_config: fix fs_config_*_system targets
Content from fs_config designated for other partitions is leaking
into the system fs_config_* files.  Notably /vendor is showing up.

Rule expansion happens late, after all parsing.  This is the same in
make and kati.  build/make/tools/fs_config/Android.mk ends with
fs_config_generate_extra_partition_list :=, so by the time you get to
rule expansion the value is empty.

Added a PRIVATE_PARTITION_LIST variable to record the value of
fs_config_generate_extra_partition-list.

Test: inspect result of ${OUT}/product/*/system/etc/fs_config_files
Bug: 119677224
Bug: 119310326
Change-Id: I968b4936ab2c5d174f164c760820af13434654ca
2018-11-19 07:14:54 -08:00
Tao Bao
e316d24d69 Merge "releasetools: Temporarily disable verifying AVB-signed images." am: 810965d23c
am: 65df239dc1

Change-Id: I49888ea3fd69f9bc212c3f014f48ec273bab23a4
2018-11-15 18:23:27 -08:00
Tao Bao
9788b4ed31 releasetools: Temporarily disable verifying AVB-signed images.
Bug: 119624011
Test: Run validate_target_files.py on aosp_blueline-target_files.zip.
Change-Id: I0c3a93dd2ec9fe26c62be2ac2cc26780fb9fe1a6
2018-11-15 13:46:56 -08:00
Mark Salyzyn
91bd25f104 Merge "build: Split out mkfs in BuildImageMkfs (retry)" am: a81a534c58
am: bf95e44687

Change-Id: Ifbfe1b6d87b6734413201711dd8402b88d2fcfe9
2018-11-12 08:39:32 -08:00
Mark Salyzyn
3cd2460edc build: Split out mkfs in BuildImageMkfs (retry)
NB: second attempt, issue with scope for mkfs_output.

Separate out BuildImageMkfs from BuildImage, which just makes the
filesystem without any verity, avb, or other decisions.  BuildImage
does all the wrapping for such.  This will hopefully ease maintenance
and drop the issues surrounding BuildImage reentrancy.

Change right-size estimation path to use BuildImageMkfs, and thus do
so without verity or avb wrappings.  Add partition_headroom to space
consideration.  This makes the results of the estimation more
accurate and predictable.

Test: build
Bug: 111302946
Change-Id: I2549bd4e403c21290470b2fa1835492ae883f0fd
2018-11-08 15:42:22 -08:00
Mark Salyzyn
19667315d9 Merge "Revert "build: Split out mkfs in BuildImageMkfs"" am: c21c709f0b
am: ba3394e225

Change-Id: I19eb687615a0e4dfbbf5f8a28d6939fe2d9c4a44
2018-11-08 11:44:28 -08:00
Mark Salyzyn
533558ec96 Revert "build: Split out mkfs in BuildImageMkfs"
This reverts commit 2b72b7f01e.

out/target/product/generic_x86/installed-files.txt \${DIST_DIR}/installed-files-rescued.txt; exit 1 ) )" 
2018-11-08 19:12:28 - build_image.py - ERROR : Failed to build out/target/product/generic_x86/obj/PACKAGING/systemimage_intermediates/system.img from out/target/product/generic_x86/system 
Traceback (most recent call last): 
File "build/make/tools/releasetools/build_image.py", line 767, in <module> 
main(sys.argv[1:]) 
File "build/make/tools/releasetools/build_image.py", line 755, in main 
BuildImage(in_dir, image_properties, out_file, target_out) 
File "build/make/tools/releasetools/build_image.py", line 449, in BuildImage 
CheckHeadroom(mkfs_output, prop_dict) 
NameError: global name 'mkfs_output' is not defined 


Bug: 111302946
Reason for revert: Build problem

Change-Id: I786f232e07af653a7207509055df5a07a6d8bb9e
2018-11-08 19:26:50 +00:00
Mark Salyzyn
a263e00ad1 Merge "build: Split out mkfs in BuildImageMkfs" am: 5f2e64aaa4
am: cec322e782

Change-Id: I3485e6422909507115e018e1cfede76e170b640e
2018-11-08 10:01:11 -08:00
Mark Salyzyn
e3703eaeb9 Merge "build_image: right size image problems." am: 9da0bd95e5
am: f14c859afb

Change-Id: I025662a0d8d5627a40d438571dfeb061db0ec1ba
2018-11-07 17:35:44 -08:00
Mark Salyzyn
2b72b7f01e build: Split out mkfs in BuildImageMkfs
Separate out BuildImageMkfs from BuildImage, which just makes the
filesystem without any verity, avb, or other decisions.  BuildImage
does all the wrapping for such.  This will hopefully ease maintenance
and drop the issues surrounding BuildImage reentrancy.  Change
right-size estimation path to use BuildImageMkfs, and do so without
verity or avb wrappings.

Test: build
Bug: 111302946
Change-Id: I30e2e2b727f40ecca5164142f34139f5244f6424
2018-11-07 15:51:32 -08:00
Mark Salyzyn
41f781de9e build_image: right size image problems.
Neglected to correct for filesystem size in estimate for verity.

To keep build space optimized and make sure that the existing image
does not confuse the tools, remove the out_file after the test build
image phase.

Test: build
Bug: 111302946
Change-Id: I360775d862d02a721d061bbc43e8b4e2f03bd01e
2018-11-07 14:50:33 -08:00
Tao Bao
2bcdca3925 Merge "releasetools: Add tests for VerityImageBuilder." am: 68b081fd5a
am: c9a014e86d

Change-Id: Ia48ff477106d483d690c4a509aa9a03c8759db49
2018-11-07 12:51:29 -08:00
Tao Bao
68b081fd5a Merge "releasetools: Add tests for VerityImageBuilder." 2018-11-07 20:23:12 +00:00
Anton Hansson
ba124157d2 Merge "Make right-sizing less aggressive" am: 0f22d9e395
am: 3be47180c3

Change-Id: Ic3417fb29075259e4b19628e054c994c38bf8cd0
2018-11-07 06:42:50 -08:00
Mark Salyzyn
fbc685bd09 Merge "build_image still too aggressive with right sizing (part troix)" am: fcc971934b
am: 8d1b3a1366

Change-Id: I1992373368bc4fb2e22f88ea41ea1ccb9fa45bb5
2018-11-07 06:21:46 -08:00
Anton Hansson
0e60f8a42c Make right-sizing less aggressive
This is a rollback of Id5ae6645b9aa2d036e6fefe2fb17672e8f8ef6f0
and the followup I6a0f3919e8d45d0b3e7cd558a6ad4d3799012f2c

The magic constant introduced appears to be too aggressive.
Note that the new constant in I0f32c75e80b5da4d47671055ac274ccc2b485992
doesn't appear to fix the broken builds.

Bug: 111302946
Test: make
Change-Id: I29207265172d293c6fee98212e3266bde85895df
2018-11-07 14:02:42 +00:00
Tao Bao
21fbf6bd37 Merge "releasetools: Create VerityImageBuilder." am: d0b9758203
am: cebca02be1

Change-Id: Ie53185ba805fdd35c85186f367eb1735a8399488
2018-11-06 15:59:40 -08:00
Mark Salyzyn
2b191894c2 build_image still too aggressive with right sizing (part troix)
Regression introduced by c3d408e3c1

Found a case where 1 + 1/60 reduction in size based on free space
failed when right sizing an ext4 image. Reduced the math to 1 + 1/61.

Test: compile
Bug: 111302946
Change-Id: I0f32c75e80b5da4d47671055ac274ccc2b485992
2018-11-06 15:57:03 -08:00
Tao Bao
a4f058888e Merge "releasetools: Log the exceptions before rethrowing." am: 9c3a481b5f
am: e2dc3587ff

Change-Id: I9d4daafd9f1a65d16317e3ba209d6715006388b1
2018-11-06 15:46:34 -08:00
Tao Bao
d0b9758203 Merge "releasetools: Create VerityImageBuilder." 2018-11-06 23:37:15 +00:00
Treehugger Robot
9c3a481b5f Merge "releasetools: Log the exceptions before rethrowing." 2018-11-06 23:28:17 +00:00
Tao Bao
46901fb8ca releasetools: Log the exceptions before rethrowing.
In Append2Simg and Append in verity_utils.py, we catch and rethrow
exceptions as BuildVerityImageError. However, this suppresses the
traceback from the original exception which usually indicates the actual
cause. We can better handle this with the raise statement in Python 3,
which is however unavailable in Python 2.

This CL logs the exception before rethrowing to retain the useful bits.

Test: Inject an error to append2simg. `m -j systemimage` with
      aosp_marlin-userdebug. Check the output.
Change-Id: I0c2f57d6023fa1038256b85fa98d57ad0244a70d
2018-11-06 10:42:35 -08:00
Tao Bao
4a0d5134e1 releasetools: Add tests for VerityImageBuilder.
`testdata/verity_mincrypt` is generated with the following command.
  $ generate_verity_key \
        -convert testdata/verity.x509.pem \
        testdata/verity_mincrypt

Test: python -m unittest test_verity_utils
Change-Id: I3f2f0cd89da0fd5d2420e0e2b09761cc77f893c5
2018-11-06 10:00:51 -08:00
Tao Bao
7549e5e919 releasetools: Create VerityImageBuilder.
Test: `m dist` with aosp_marlin-userdebug (VB 1.0)
Test: `m dist` with aosp_taimen-userdebug (VB 2.0)
Test: `python -m unittest discover .`
Change-Id: I8c016b7621b73f220fb2a4586cc45a21eeee438d
2018-11-06 10:00:28 -08:00
Anton Hansson
80df117a12 Merge "Move ro.build.characteristics to /product props" am: 0625f9a524
am: 005d3c197b

Change-Id: I99161a7ce70fd2f2fed726a916171e1254ed5e5e
2018-11-06 01:13:14 -08:00
Anton Hansson
0625f9a524 Merge "Move ro.build.characteristics to /product props" 2018-11-06 08:52:02 +00:00
Anton Hansson
5ac408badd Move ro.build.characteristics to /product props
This property is not device-generic, as it's used for things like
whether the product is phone/tablet/other and whether it has an
sdcard or not.

Bug: 118618261
Test: adb shell getprop ro.build.characteristics for aosp_blueline
Change-Id: I6bbf1d4c36d7f37f92c67bebde58d72aba9d0852
2018-11-05 17:29:28 +00:00
Mark Salyzyn
f8e40f2154 Merge "build_image too aggressive with right sizing" am: 1f50f75988
am: 19d237de3d

Change-Id: I45de70512b79a6ffac08062b6a304b8211634f7c
2018-11-01 15:51:06 -07:00
Mark Salyzyn
4a6ce93f27 build_image too aggressive with right sizing
Regression introduced by c3d408e3c1

Found a case where 1 + 1/59 reduction in size based on free space
failed when right sizing an ext4 image. Reduced the math to 1 + 1/60.

Test: compile
Bug: 111302946
Change-Id: I6a0f3919e8d45d0b3e7cd558a6ad4d3799012f2c
2018-11-01 12:45:29 -07:00
Tao Bao
d31580571e Merge "Revert "Work around e2fsck issue for building sparse GSIs"" am: 757dfe8792
am: 6a18892b92

Change-Id: I5da761417d80cbf9362b72796b9417f2995f5d18
2018-10-30 08:33:23 -07:00
Tao Bao
757dfe8792 Merge "Revert "Work around e2fsck issue for building sparse GSIs"" 2018-10-30 15:18:58 +00:00
Mark Salyzyn
408186174e Merge changes Ia8fb5696,Id5ae6645 am: 6b38c9bb4c
am: d1eda9937a

Change-Id: Iecc5f662736791d824cb9aa45ef4719d7eb6d9d6
2018-10-29 16:50:02 -07:00
Tao Bao
27809667b4 Revert "Work around e2fsck issue for building sparse GSIs"
This reverts commit ec7fa1cd09.

The earlier e2fsck issue has been addressed by [1].

[1] https://android-review.googlesource.com/c/platform/external/e2fsprogs/+/725259/

Bug: 112062612
Test: m PRODUCT-aosp_x86_64-eng \
          SANITIZE_HOST=address ASAN_OPTIONS=detect_leaks=0 \
          -j e2fsck systemimage
Change-Id: I7287cb8639ae45abdf753821421bd070b819a30e
2018-10-29 16:02:47 -07:00
Mark Salyzyn
f0cef8d8bf build_image setup ext4 dedupe on more images.
Add ext4 dedupe to system_other, product, product_services, odm and
oem images.  Experimental savings without any other configuration
changes was 0%, .75%, 3.4% respectively for the raw image file size
for the first three.

Test: manual
Bug: 111302946
Change-Id: Ia8fb5696151acad59bb144ea93f2c2ddac962bbd
2018-10-29 13:37:10 -07:00
Mark Salyzyn
c3d408e3c1 build_image tune right-size image
To more quickly settle on a no free space result, recognize that
there is roughly 1/58.5 overhead managing the free space pool as
determined experimentally.  This algorithm carries with it some
risk of being sensitive to any alterations in ext4.

This also addresses the issue of a much larger apparent free space
available calculated on the sample pass when deploying ext4 dedupe,
and resolves the poorer estimation that happens without accounting
for the overhead.

The alternative of adding a second pass works to a similar, and
likely more comforting algorithm and result.  But doing so adds
a minute to the typical incremental build time, or +50% to the
time it takes to determine ext4 right sized images.

Test: manual
Bug: 111302946
Change-Id: Id5ae6645b9aa2d036e6fefe2fb17672e8f8ef6f0
2018-10-29 13:37:04 -07:00
Tao Bao
9a7e1cbdb0 Merge "releasetools: Fix the error message." am: fea4263ec1
am: 664873bd44

Change-Id: I495650de44e12dc546d4115d20bd1f9993e527af
2018-10-25 15:52:07 -07:00