Commit Graph

4234 Commits

Author SHA1 Message Date
Sasha Levitskiy
4a34533e11 build: tools: Add json collection of installed files with hashes.
Adds installed-files.json in addition to installed-files.txt
Further sorts the file list to be ordered within the same size bucket.

Test: manual, cross-checked checksums with sha256sum utility
      checked build outputs.

Bug: 19988819
Merged-in: Ifb632eb4df65ec48645c8f93e36bae44ccc52ba8
Change-Id: Ifb632eb4df65ec48645c8f93e36bae44ccc52ba8
Signed-off-by: Sasha Levitskiy <sanek@google.com>
2016-10-14 16:29:16 -07:00
Alex Deymo
3b54b69205 Merge changes Ie3d82cfa,I2f808896,I587f400d
* changes:
  fat16copy: Fix allocation logic when extending directories.
  fat16copy: Sort new directory entries.
  Make fat16copy.py add . and .. entries to directories
2016-10-14 03:19:55 +00:00
Clay Murphy
f8d8458edd Merge "Update droiddoc templates for build/make move" 2016-10-12 18:34:23 +00:00
Treehugger Robot
6949e7ecba Merge "Better pattern matching and dump of warning messages." 2016-10-12 16:35:34 +00:00
Dan Willemsen
82be5468c3 Update droiddoc templates for build/make move
Since build/ was moved to build/make/, the droiddoc templates now need
another ../ to reach the android root.

Test: m online-sac-docs
Change-Id: I2393f6c791df9b67ccafb4eb047fd8ccc52481cb
2016-10-12 00:10:33 -07:00
Chih-Hung Hsieh
5722f9247b Better pattern matching and dump of warning messages.
* Warning messages must start with source file path.
  This will reject lines containing only 'warning:'
  but not a source file path prefix.
* Escape both backslash and quotation marks in the
  dump of strings to JavaScript string literals.

Bug: 32060052
Test: run through build.log files
Change-Id: Ib064768b1ba2954f974604ea054f7a6d2ad15ae7
2016-10-11 15:33:19 -07:00
Chih-Hung Hsieh
a45c5c1cf9 Normalize warning source file path to relative path.
Remove Android root directory path from long
absolute source file path.

Test: run warn.py with build.log
Bug: 32059187
Change-Id: Ib630cebd8c56a9fef443139ca502d97e59c7b42e
2016-10-11 15:30:26 -07:00
Treehugger Robot
b37e81d3cd Merge "Fix a nullness warning." 2016-10-11 19:29:37 +00:00
Treehugger Robot
f943e138d7 Merge "Fix an uninitialized value warning." 2016-10-11 02:43:47 +00:00
Yunlian Jiang
221c1c027a Fix an uninitialized value warning.
Warning from the static analyzer:

build/tools/zipalign/ZipFile.cpp:503:5: warning: Function call argument
is an uninitialized value
pEntry->setDataInfo(uncompressedLen, endPosn - startPosn, crc,

Specifically, it's referencing `crc`, which would be uninitialized if we
hit either of these two error cases, since we'd return `NO_ERROR`.

Note that the warning is still there, but that's only because the static
analyzer can't see the asserts. If we #undef NDEBUG in the file, then
the warning disappears.

Bug: none
Test: With NDEBUG undefined, the warning is gone.
Change-Id: Iaed66127746c38add2c842ab027f2e1982d0e2fd
2016-10-10 17:31:06 -07:00
Chih-hung Hsieh
89244619ad Merge "Add a new ANALYZER severity group." 2016-10-05 23:31:43 +00:00
Dan Willemsen
4c364620ac Add tool to run kati for all products
Test: get_build_var all_named_products
Test: build/tools/kati_all_products.sh
Change-Id: I3496cb4c59a1edb11c9149b95ecabd3a828b8972
2016-10-05 14:04:21 -07:00
Chih-Hung Hsieh
1eabb0ecb2 Add a new ANALYZER severity group.
* Separate clang-tidy's clang-analyzer-* warnings from
  other clang-tidy warnings.

Bug: 31559947
Test: build with WITH_TIDY=1 and DEFAULT_GLOBAL_TIDY_CHECKS=*,-readability-*,-google-readability-*
Change-Id: If03c421c9319741ef6588bc0d803b6da0cc4433a
2016-10-05 11:53:20 -07:00
Yunlian Jiang
4f1a91cbac Fix a nullness warning.
This silences a warning null pointer passed as an argument to a
'nonnull' parameter

BUG=None
Test: The warning is gone.

Change-Id: I10a17d4674bfb373d7fe8666863abd5f10a4d11f
2016-10-05 10:52:35 -07:00
David Zeuthen
70209ad9f0 Merge "Make room for AVB hashtree and metadata." 2016-10-04 22:32:27 +00:00
David Zeuthen
4014a9daa7 Make room for AVB hashtree and metadata.
While the system.img images currently built with AVB support verify
correctly, mounting the filesystem content fails. This is because
'avbtool add_hashtree_footer' used to claim some of the unused /
DONT_CARE space for stashing the verity tables and this resulting in the
mapped device ending up being smaller causing the mount failure.

Fix this by leaving enough room for AVB hashtree and metadata before
building the image. This is achieved by moving the AVB hashtree support
into build_image.py and using a just added '--calc_max_image_size'
option to 'avbtool add_hashtree_footer' to figure out how much space to
leave out.

This depends on https://android-review.googlesource.com/#/c/281821/

Bug: 31264226
Test: Mounting dm-verity set up from system.img now works.

Merged-In: I4c5de1004c1059f8c582e76b3b8517d427aa1a87

Change-Id: I945a5f1f6782791736cd319f216cfa6b448fb04d
2016-10-04 17:32:16 -04:00
Przemyslaw Szczepaniak
a8c8dadd09 Merge "Fix for non-empty a[name] tags in openJdk based javadocs." 2016-10-04 11:35:58 +00:00
Przemyslaw Szczepaniak
4bce2866fb Fix for non-empty a[name] tags in openJdk based javadocs.
openJdk based android javadoc contains <a name=... tags
that are non-empty. Currently they are being hidden,
causing badly formatted pages for some java.* classes
(missing text and random whitespace blocks).

This change makes the existing a[name] hiding rule
apply only to empty tags.

Test: make docs
Bug: 31700998
Change-Id: Ifa75bbabe308d7aed0cce6165c1113e5a382a121
(cherry picked from commit a7edd92c45)
2016-10-04 10:31:54 +00:00
Colin Cross
49c45aecc4 Propagate signals through makeparallel
Set up a signal handler in makeparallel that will forward SIGHUP,
SIGINT, SIGQUIT, and SIGTERM to the child process.

Bug: 31907490
Test: m -j & killall make; pgrep -a ninja
Test: make makeparallel_test
Change-Id: I306e5335ed1b2c7056804d5da377a2f283877f30
2016-10-03 15:33:11 -07:00
Tao Bao
9aa4b9b1b6 releasetools: Clean up sign_target_files_apks.py.
Remove the obsolete workaround for API 24. Also make it pylint clean.

Test: 1. Sign a target_files.zip and get identical results.
      2. `pylint --rcfile=pylintrc sign_target_files_apks.py` gives 10.00/10.

Change-Id: I21736f959c5182486fd8ccebea9bbc594edef9fb
2016-09-30 15:29:09 -07:00
Tao Bao
b31b94e159 releasetools: Fix the reference to OPTIONS.info_dict.
sign_target_files_apks.py calls common.GetBootableImage() but without
calling 'OPTIONS = common.OPTIONS' first. In common.GetBootableImage(),
we should use the local info_dict parameter instead of OPTIONS.info_dict.

Test: sign_target_files_apks.py generates signed-TF.zip successfully.

Change-Id: Ia3d32b88691c26e5fb98feea709e3e3c3eb70fdb
2016-09-29 22:04:50 -07:00
David Zeuthen
8ee6d119d1 Merge "Update for new Android Verified Boot (AVB)." 2016-09-29 17:48:40 +00:00
David Zeuthen
2ce63edab7 Update for new Android Verified Boot (AVB).
This updates the build system for the new Android Verified Boot
codebase. As this is based on Brillo Verified Boot, this change replaces
the existing BVB support.

Android Verified Boot is enabled by the BOARD_AVB_ENABLE variable

 BOARD_AVB_ENABLE := true

This will make the build system create vbmeta.img which will contain a
hash descriptor for boot.img, a hashtree descriptor for system.img, a
kernel-cmdline descriptor for setting up dm-verity for system.img and
append a hash-tree to system.img.

Additionally, the descriptors are left in boot.img and system.img so a
third party can create their own vbmeta.img file linking - using the
option --chain_partition - to these images. If this is not needed
footers can be erased using the 'avbtool erase_footer' command. It's
also harmless to just leave them in the images.

By default, the algorithm SHA256_RSA4096 is used with a test key from
the AVB source directory. This can be overriden by the
BOARD_AVB_ALGORITHM and BOARD_AVB_KEY_PATH variables to use e.g. a
4096-bit RSA key and SHA-512:

 BOARD_AVB_ALGORITHM := SHA512_RSA4096
 BOARD_AVB_KEY_PATH := /path/to/rsa_key_4096bits.pem

To prevent rollback attacks, the rollback index should be increased on a
regular basis. The rollback index can be set with the
BOARD_AVB_ROLLBACK_INDEX variable:

 BOARD_AVB_ROLLBACK_INDEX := 5

If this is not set, the rollback index defaults to 0.

The variable BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS can be used to specify
additional options passed to 'avbtool make_vbmeta_image'. Typical
options to be used here include '--prop', '--prop_from_file', and
'--chain_partition'.

The variable BOARD_AVBTOOL_BOOT_ADD_HASH_FOOTER_ARGS can be used to
specify additional options passed to 'avbtool add_hash_footer' for
boot.img. Typical options to be used here include '--hash_algorithm' and
'--salt'.

The variable BOARD_AVBTOOL_SYSTEM_ADD_HASHTREE_FOOTER_ARGS can be used
to specify additional options passed to 'avbtool add_hashtree_footer'
for systems.img. Typical options to be used here include
'--hash_algorithm', '--salt', and '--block_size'.

BUG=31264226
TEST=Manually tested on edison-eng by inspecting {boot, system,
  vbmeta}.img in out/ directory as well as their counterparts in
  the IMAGES/ directory of edision-target_files-eng.zeuthen.zip

Merged-In: Ic9a61cfc65c148b12996e57f04da5432eef6b982

Change-Id: I97042655bca15e7eac899f12c5bada2f6184d307
2016-09-29 11:19:36 -04:00
Treehugger Robot
312c310f7c Merge "Only run warn.py's main function if it is the main program." 2016-09-29 03:58:07 +00:00
Meike Baumgärtner
ac9d5df349 Only run warn.py's main function if it is the main program.
Test: run warn.py --byproject build.log
Change-Id: I155cce8c952d6d8a02c8e1809bf61a46c847c26e
2016-09-28 17:59:08 -07:00
Treehugger Robot
6784a3fcdd Merge "Fix pylint warning and refactor project_list." 2016-09-28 17:47:41 +00:00
Tao Bao
6e12b72966 Merge "Analyze unchanged blocks in odex files." 2016-09-28 17:08:10 +00:00
Chih-Hung Hsieh
9f766239f0 Fix pylint warning and refactor project_list.
Bug: 31377083
Test: run warn.py --byproject build.log
Change-Id: Id442cf6608bf5c14784063c05f5ff0b731f17486
2016-09-27 21:13:52 -07:00
Chih-Hung Hsieh
b426c54e8a Fix Google pylint warnings.
* Rename 'severity' to 'Severity'.
* Fix bad line indentations and space.

Test: run warn.py --byproject build.log
Change-Id: I33d701cca408b5ca4715be8d44c890942b7d132a
2016-09-27 18:15:40 -07:00
Chih-Hung Hsieh
2a38c37d57 Add buttons to group warning by project or severity.
* Add more project patterns.
* Add more top level comments for global variables and functions.
* Resequence severity numbers to match the dump order.
* Emit warning messages and tables to static HTML JavaScript arrays.
* Replace old static HTML table dumper functions with
  new dynamic HTML JavaScript to generate sections of warnings.
* Warning messages are grouped into sections by severity or projects.
* Better descriptions for SKIP warning patterns.
* Replace output function with print.

Bug: 31377083
Test: run warn.py --byproject build.log
Change-Id: I7b44ef6223d5b2f1aa31655a5a47d854f9a1dedc
2016-09-26 16:02:12 -07:00
Treehugger Robot
bcd229277a Merge "Fix Google pylint warnings." 2016-09-26 21:18:08 +00:00
Chih-Hung Hsieh
6c0fdbb84d Fix Google pylint warnings.
* Rename variables and functions to follow Google coding style.
* Fix line indentations to follow Google coding style.
* Add module and function docstrings.
* Change shebang line to avoid pylint warning.
* Suppress some pylint warnings to avoid too many changes at once.
* Fix typos.

Test: run warn.py --byproject build.log
Change-Id: I569961981b562e35cb5609a96cd5ebb40a80829c
2016-09-26 12:24:54 -07:00
Alex Deymo
a1c977735b fat16copy: Fix allocation logic when extending directories.
When extending a directory, empty directory entries were left behind
signaling the end of a directory.

Bug: 31570546
Test: make dist; mdir shows all the files.

(cherry picked from commit d4516300b010c8155861e310dac871e8c10d3382)

Change-Id: Ie3d82cfaad51be73911293a54d18746e8adf62a2
2016-09-23 14:21:43 -07:00
Alex Deymo
567c5d0f95 fat16copy: Sort new directory entries.
Sort the entries returned by os.listdir to give a consistent ordering
across build.

Bug: None
Test: `make dist`

(cherry picked from commit ef34e5dca216e61fd81020e818f247f6ce47d26e)

Change-Id: I2f80889684f108208f07e13d725acaa7bc0ab099
2016-09-23 14:21:38 -07:00
Casey Dahlin
df71efe378 Make fat16copy.py add . and .. entries to directories
Test: fsck no longer complains
Bug: None

(cherry picked from commit 7d38ec37432c307d729b9a8fdb78dbf820427538)

Change-Id: I587f400df2f57b987df8b92d551c99f2f42a9af6
2016-09-23 14:21:31 -07:00
Tao Bao
08c8583aea Analyze unchanged blocks in odex files.
In BBOTA, we generate patches based on _all_ the blocks of a pair of
input files (src and tgt). For security incremental OTAs, one common
pattern is that only a few blocks are changed in odex files (e.g.
headers). We don't really need to stash/patch the unchanged blocks.

This CL analyzes the unchanged blocks in odex files and computes the
diff for the changed blocks only. It reduces the OTA install time by
about 25% to 40% in our experiments, by paying an increase of 5% to 30%
OTA generation time cost.

Bug: 31570716
Test: Generate an incremental and apply on device.

Change-Id: If842c1afeff6894a3d27eb60b7e8f65a179b7977
2016-09-23 12:15:23 -07:00
Chih-Hung Hsieh
0a19207f7b Recognize new warning messages.
* -Wunnamed-type-template-args and -W#pragma-messages

Test: run warn.py with build.log.
Change-Id: I36277e001081604f92a4d9c92f9a0b5d3f0ee5e5
2016-09-22 17:44:38 -07:00
Chih-Hung Hsieh
99459fc0d0 Simplify warn.py and output of warning count table and tablerow.
* Use a loop to initialize empty 'members' and 'option' of warnpatterns.
* Create severity.{color,columnheader,header} from an attributes list.
* Compute totalbyproject and totalbyseverity from a constructor.
* Skip all-zero rows and columns in the warning count table.
* Remove redundant spaces and newlines in tablerow output.

Test: run through build.log files
Change-Id: I4d3fa4ecd92e1afab91d85b7535d03f8696e83d8
2016-09-22 15:15:43 -07:00
Dan Willemsen
665cda08f0 Merge "Fix link_type checking" 2016-09-19 21:54:09 +00:00
Tao Bao
9c63fb59bd Add a script that verifies OTA package signature.
Currently it supports verifying packages signed with RSA algorithms
(v1-v4 as in bootable/recovery/verifier.cpp). No support for ECDSA (v5)
signed packages yet.

$ ./build/tools/releasetools/check_ota_package_signature.py \
    bootable/recovery/tests/testdata/testkey_v1.x509.pem \
    bootable/recovery/tests/testdata/otasigned_v1.zip

Package: bootable/recovery/tests/testdata/otasigned_v1.zip
Certificate: bootable/recovery/tests/testdata/testkey_v1.x509.pem
Comment length: 1738
Signed data length: 2269
Use SHA-256: False
Digest: 115e688ec3b77743070b743453e2fc6ce8754484

VERIFIED

Bug: 31523193
Test: Used the tool to verify existing packages (like above).

Change-Id: I71d3569e858c729cb64825c5c7688ededc397aa8
2016-09-16 15:08:40 -07:00
Dan Willemsen
121e284b46 Fix link_type checking
This was printing "error:", but not actually triggering an error.
Instead of trying to write a single line bash script to handle this,
move the actual check into python. This allows us to print all of the
errors for a single module before triggering the failure.

Also updates the warning format and the warn.py script to properly parse
these warning. Many of the java:sdk -> java:platform warnings are false
positives due to the lack of LOCAL_SDK_VERSION markings on prebuilts.

Individual tags can be marked as warnings now, which lets us check for
system libraries linking against vendor libraries (which won't work on
AOSP). I'm not sure this is a completely valid check, which one reason
that it's just a warning.

Test: m all_link_types (with some missing libs commented out)
Change-Id: I333e418c9a4511b7c7e826891ae481da08fbf6f9
2016-09-15 14:40:39 -07:00
Chih-hung Hsieh
029ced528b Merge "Emit a table of warnings by project and severity." 2016-09-14 16:39:27 +00:00
Chih-Hung Hsieh
e41c99b721 Emit a table of warnings by project and severity.
* Use 't1' CSS class for main warning tables,
  and default table style for new warning count table.
* Count total skipped warnings.
* Append count of warnings in each severity header.
* Fix some skipped warning patterns and count/dump skipped warnings.
* Add dumpskippedanchors function to fix expand_collapse
  error when --byproject flag is used and severity.SKIP
  is not emitted.
* Minor coding style changes:
  * Loop through severity.kinds.
  * Remove unused import.
  * Add spaces around assignment operators.
  * Adjust indentation.

Bug: 31377083
Test: run warn.py with build.log
Change-Id: I68949edc4601b10ea2f7ac84e48e6c3da4ab1e4f
2016-09-13 10:18:01 -07:00
Dan Willemsen
8e6e6fdba9 Remove incremental workaround for aapt overlays
This workaround to clean package intermediates is no longer necessary,
as ninja reruns commands whose command line changes.

Test: switch between product overlays that use add-resource, verify
      R.java is regenerated.
Change-Id: Icb1f76360e39882e4401180414a031956777b020
2016-09-12 20:30:12 -07:00
Treehugger Robot
6e85eeaf53 Merge ""SignApk" in Created-By header" 2016-09-08 17:10:37 +00:00
Alex Klyubin
969e354b1f "SignApk" in Created-By header
The switch to apksig changed the Created-By header value in .SF file
from "1.0 (Android SignApk)" to "1.0 (Android apksigner)". This commit
reverts the value back to "1.0 (Android SignApk)".

Change-Id: I2fc462cade40a5b31bb6191996fd6f18fabbf08f
2016-09-07 13:21:00 -07:00
Treehugger Robot
bfea84dabc Merge "Add tool to copy files in to fat16 images" 2016-09-07 19:51:06 +00:00
Chih-hung Hsieh
0c9a2ee864 Merge "Recognize new clang-tidy and Java warnings." 2016-09-07 05:27:45 +00:00
Chih-Hung Hsieh
0862c48770 Recognize new clang-tidy and Java warnings.
Test: run warn.py with build.log
Change-Id: I11ff844bcfd7fb13aef8909307bc7eb1f5ee9b2a
2016-09-07 05:27:16 +00:00
Casey Dahlin
29e2b21c0a Add tool to copy files in to fat16 images
fat16copy.py can be used to copy files or directories in to the root of
a fat16 image.

Usage: fat16copy <image> <file> [<file> ...]

Test: Used to compose a Raspberry Pi 3 boot image
Bug: 28912590
Change-Id: I29c9eec3786e2c5cc94f9a160360bac850809a93
2016-09-02 14:13:47 -07:00