In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.
We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.
payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.
Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
entry in the metadata file.
Change-Id: I04504e834eb36e18876c5f5a5a09289ee05c6f9a
Passing an empty argument in argv to ninja results in an error:
ninja: error: empty path
Don't add jarg to the argument list if it is empty.
Bug: 34392351
Test: m -j
Change-Id: I39c8998c79b09835650628e1413ae7625cdbc555
When running makeparallel in non-ninja mode, which is used when running
kati with USE_GOMA=true, convert -j to a reasonable parallelism value
the same way ninja does.
Bug: 34392351
Test: make -C build/make/tools/makeparallel makeparallel_test
Change-Id: I9aee4dd2a3b0f2b1c2c10087be83f7b2b06f4368
It was added in commit 96be7205dc
("Working ASLR implementation.") in 2010, and removed in commit
1807e700a5 ("don't generate retouch
commands in OTA scripts") in 2012.
Remove the obsolete --aslr_mode flag.
Test: ota_from_target_files.py still works (by generating incremental
and full OTAs respectively).
Change-Id: I6d8e62730ac192f3574d484c4a4b9b43b4ee0a9e
This information can be used to tune ext4 stripe and stride in the
userdata partition for better performance
Test: Build & flash userdata, confirm correct stripe & stride values
Bug: 33243520
Merged-In: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Change-Id: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Signed-off-by: Connor O'Brien <connoro@google.com>
Fix pylint warning:
W:292,47: Cell variable lineno defined in loop (cell-var-from-loop)
For the closure function that prints the error message, we actually
want the value of modified lineno within the closure.
Test: run unit tests
Change-Id: I7d08184563d347c826ca1af7558fb8e378f151e1
Signed-off-by: William Roberts <william.c.roberts@intel.com>
A few changes got introduced that yapf re-styles. To make it easy,
keep this file formated to:
yapf -i --style=google fs_config_generator.py
Test: run unit tests
Change-Id: I3160b92d7fd07a4d315dde59f1d20fe4c7587da1
Signed-off-by: William Roberts <william.c.roberts@intel.com>
Addresses the following build error:
"value" not in valid range [], got: 2901 for: "AID_ABC" file:\
"device/somc/abc/config.fs"
that happens when parsing device/somc/abc/config.fs:
[AID_ABC]
value: 2901
The problem is that the AID.PREFIX is 'AID_' so the _OEM_RANGE
expression is searching for AID__OEM_RESERVED, which never
exists. Drop the extra '_'.
Change-Id: I8a1556731e14e49cef66e3e91121e4be23a308e0
Test: ./build/tools/fs_config/fs_config_generator.py fsconfig --aid-header=system/core/include/private/android_filesystem_config.h device/somc/abc/config.fs
Only BBOTA v2 needs to maintain a pool of available 'stash slot id'.
BBOTA v3+ uses the hash of the stashed blocks as the slot id, which
doesn't need the id pool anymore.
Bug: 33694544
Test: Generate v2 and v4 incrementals w/ and w/o the CL. They produce
the same packages respectively.
Change-Id: I8121af5b6b1bee98c3639d54a00b06fd12e378e8
This reverts commit fad4b4b715.
Incorporating the following fixes:
1.
fsconfig: fix fs_config_* build for discovered headers
When android_file system_config.h is picked up from the device
directory, neither TARGET_FS_CONFIG_GEN or TARGET_ANDROID_FILESYSTEM_CONFIG_H
are specified. Thus, the build is not generating the required fs_config_files
and fs_config_dirs.
Test: Ensure that make fs_config_files works, and produces the same output as before
Build the system image and mount it as a loop back and dump the file system
capabilities with getcap. Verify that output to the supplied
android_file system_config.h
From the loopback of the system.img mount, from CWD system/bin:
$ getcap *
cnss-daemon = cap_net_bind_service+ep
hostapd = cap_net_admin,cap_net_raw+ep
imsdatadaemon = cap_net_bind_service+ep
ims_rtp_daemon = cap_net_bind_service+ep
logd = cap_setgid,cap_audit_control,cap_syslog+ep
mm-qcamera-daemon = cap_sys_nice+ep
pm-service = cap_net_bind_service+ep
run-as = cap_setgid,cap_setuid+ep
surfaceflinger = cap_sys_nice+ep
webview_zygote32 = cap_setgid,cap_setuid,cap_setpcap+ep
webview_zygote64 = cap_setgid,cap_setuid,cap_setpcap+ep
Compared to the android_filesystem_config.h:
{ 00700, AID_CAMERA, AID_SHELL, (1ULL << CAP_SYS_NICE), "system/bin/mm-qcamera-daemon" },
{ 00755, AID_SYSTEM, AID_SYSTEM, (1ULL << CAP_NET_BIND_SERVICE), "system/bin/pm-service" },
{ 00755, AID_SYSTEM, AID_SYSTEM, (1ULL << CAP_NET_BIND_SERVICE), "system/bin/imsdatadaemon" },
{ 00755, AID_SYSTEM, AID_RADIO, (1ULL << CAP_NET_BIND_SERVICE), "system/bin/ims_rtp_daemon" },
{ 00755, AID_SYSTEM, AID_SYSTEM, (1ULL << CAP_NET_BIND_SERVICE), "system/bin/cnss-daemon"},
2.
fsconfig: fix error message for duplicate AID
Fixes:
raise ValueError('Duplicate aid value "%u" for %s' % value,
TypeError: %u format: a number is required, not str
and
raise ValueError('Duplicate aid value "%s" for %s' % value,
TypeError: not enough arguments for format string
3.
fsconfig: add test for duplicate ranges
Add a test for duplicate range detection.
4.
fsconfig: skip AID_APP, AID_USER and all ranges
Do not output AID_APP, AID_USER and ranges. A range
is defined as ending with AID_ and ending in _START or
_END.
5.
fsconfig: test for skip AID_APP, AID_USER and all ranges
Test against AIDs that caused the bionic tests to fail.
Change-Id: I95569a9ccc83bd3231f8a6f395532cc2de316bd2
Signed-off-by: William Roberts <william.c.roberts@intel.com>
We compute the max stashed_blocks in ReviseStashSize(), prior to calling
WriteTransfers(), to avoid running out of space due to stashing.
There is a bug when computing the to-be-freed stashed blocks, where we
wrongly free the space _before_ executing the transfer command. This leads
to a script failure where the max stash size violates the max allowed
size in WriteTransfers().
Note that this bug doesn't affect already generated packages. It's only
an underestimate in ReviseStashSize(). The check in WriteTransfers() has
been correct to ensure the max stash size.
Bug: 33687949
Test: Successfully generated incremental OTA which failed previously.
Change-Id: I4f4f043c6f521fce81ca5286e6156f22d99bf7f7
Prior to this change, when signing APKs, the build system invoked
'aapt dump badging' on each APK, to detect the value to pass into
signapk as --min-sdk-version. Now that signapk uses the apksig
library, it can auto-detect that value on its own, thus avoiding the
need to invoke 'aapt dump badging' and thus speeding up the build
process.
The semantics of signapk's --min-sdk-version flag is changed by this
commit from having the default value of 0 to having the default value
of "auto-detect from APK".
P.S. The get-package-min-sdk-version-int is not removed from
core/definitions.mk in this commnit, because this function is used in
another project's .mk file and thus that .mk file needs to be modified
first.
Test: rm -Rf out/ && make
Change-Id: I0972fcf0abbde9cbf6794e6c05c743c77c8a78f9
We used to dump "Source: <fingerprint>" in update logs. The "Source: "
prefix was unintentionally dropped out.
Test: Check the generated incremental BBOTA script.
Change-Id: I4de62333aa38e3fb09a76df0e769b62af48e0313
* changes:
fs_config: add unit tests
fs_config: drop fs_config_files/dirs PRODUCT_PACKAGES requirement
fs_config: add group to build
fs_config: introduce group generator
fs_config: add passwd to build
fs_config: introduce passwd generator
fs_config: generate friendly in AID class
fs_config: limit characters for AID_<name> sections
fs_config: generate oem AID header file
fs_config: android_id header generator
fs_config: support parsing android_filesystem_config.h
fs_config: modularize fs_config_generator
Add support for specifying number of inodes when creating
system, vendor, oem partitions. These are all read-only
and have no use for extra inodes. Removing extra inodes
saves a lot of space.
Bug: 32246383
Change-Id: I13f1d4614b64a4abc752c42a1c65d3d151481c21
(cherry picked from commit b59eca3586)
In two-step OTAs, we write recovery image to /boot as the first step so
that we can reboot from there and install a new recovery image to
/recovery. However, bootloader will show "Your device is corrupt"
message when booting /boot with the recovery image. Because the recovery
image encodes the path of "/recovery" as part of the signature metadata,
which fails the verified boot.
This CL generates a special "recovery-two-step.img" in addition to the
regular recovery.img. This image encodes "/boot" when being signed,
which will be flashed to /boot at stage 1/3 in a two-step OTA.
Here are the desired changes:
- 'IMAGES/recovery-two-step.img' exists in target_files.zip for non-A/B
targets (e.g. bullhead). The image should not exist for targets that
don't have a recovery partition (e.g. A/B devices like sailfish).
- <device>-img.zip should not contain 'recovery-two-step.img'.
- Nothing should change when building non-two-step OTAs. For two-step
OTAs, 'recovery-two-step.img' should be included in the OTA package;
'updater-script' should flash this image to /boot at stage 1/3.
- When building a two-step OTA with an input TF.zip that doesn't have
IMAGES/recovery-two-step.img, it should use the existing
IMAGES/recovery.img instead.
Bug: 32986477
Test: Tested the steps above on bullhead and sailfish.
Change-Id: I34e6c599bcf2011d4cd5c926999418b3975d6d0f
The 'system_img_path' parameter was introduced in commit
d995f4b04d, but became obsolete since
commit 2ce63edab7.
Test: m dist
Change-Id: Iffd496d929db5cc3dfc955a48bfc1b1317bd012f
Add some initial unit tests.
The unit tests themselves are inlined into the tool and
can be executed by running the "test" commandlet.
Example:
$ python -m unittest test_fs_config_generator.Tests
.............
----------------------------------------------------------------------
Ran 13 tests in 0.004s
OK
Test: run the test commandlet and observe for failures.
Change-Id: I1bada385fa841fd50fa958997d440f1198e15198
Signed-off-by: William Roberts <william.c.roberts@intel.com>
When configuring fs_config_files or fs_config_dirs for file_system
capabilities, drop the requirement that OEMs must add the target
to PRODUCT_PACKAGES. This limits the configuration requirement
to only needing to set the new and preferred TARGET_FS_CONFIG_GEN
or the older TARGET_ANDROID_FILESYSTEM_CONFIG_H method.
Test: That only setting TARGET_FS_CONFIG_GEN results in passwd and
group in the build image.
Change-Id: I818854fa1b3e94edaff59a32bd7cf23cf9b504aa
Signed-off-by: William Roberts <william.c.roberts@intel.com>
Introduce a generator that outputs group files per man(5) group.
Succinctly, the output is a colon delimited string containing the following
fields:
* group name
* encrypted password (optional)
* gid (int)
* userlist (str,...)
Multiple colon delimited lines may exist, but will not be separated
across lines.
Sample generator output:
foo::2900:
foo_bar::2901:
custom_oem1::2902:
Test: That make group produces the group file.
Change-Id: Idd3fe925a09a227c6e894e1b5d2b3873b01531c6
Signed-off-by: William Roberts <william.c.roberts@intel.com>
Introduce a generator that outputs passwd files per man(5) passwd.
Succinctly, the output is a colon delimited string containing the following
fields:
* login name
* encrypted password (optional)
* uid (int)
* gid (int)
* User name or comment field
* home directory
* interpreter (optional)
Multiple colon delimited lines may exist, but will not be separated
across lines.
When run, produces:
foo::2900:2900::/:/system/bin/sh
foo_bar::2901:2901::/:/system/bin/sh
custom_oem1::2902:2902::/:/system/bin/sh
Note that this generator allows for 0 or more config.fs files. This allows for:
* Unconditional inclusion of /system/etc/passwd in the generated image
* A blank passwd file if no config.fs files are specified.
This ensures that when OEMs add config.fs files, there is no additional steps
for proper functionality (simpler for OEMs).
The one draw back is the additional inode consumption on system for a possible
blank file.
Test: That it produces a valid passwd file.
Change-Id: I19691c8260f02147ed861f8a319aeab3f5b1738e
Signed-off-by: William Roberts <william.c.roberts@intel.com>
Generate the friendly name in one location. This prepares
the tool for generatting passwd and group files.
Also support mapping friendly names to identifiers.
Test: That output files stay the same as before.
Change-Id: I12198611126613eae81ca61614ed269c2439b72b
Signed-off-by: William Roberts <william.c.roberts@intel.com>
When generating files restrict the characters allowed in
AID_<name> to upercase, numbers, and underscores.
This detects errors ahead of time for generated C files as
well as handles seperation characters for passwd/group files.
This also lends itself to automatic detection of collisions
on friendly names, since freindly names are the lowercase
version of <name>.
Test: That invalid aid values result in a build failure.
Test: That the output files are consistent with ones before
this change (hash and diff checks)
Change-Id: Ie8ec44c1157ba9c22100e9169d9187f615e71280
Signed-off-by: William Roberts <william.c.roberts@intel.com>
Generate an OEM AID_<name> header file seperate from fs_config
header file and provide details on how to export this interface
into native code.
Test: That ls, ps, chown and services function for built in
services as before.
Change-Id: Ie8ce6585e0721b52633ee50d62dcfe796e178f65
Signed-off-by: William Roberts <william.c.roberts@intel.com>
Implement an android_id generator that takes the data
acquired from parsing private/android_filesystem_config.h
and generates the android_id friendly name to uid mapping
for consumption in Bionic.
Test: That ls, ps, mkdir, chown, chgrp and services for built
in names work.
Change-Id: I1e55a401be0fca0ad162f8dc1e072e6afde7b927
Signed-off-by: William Roberts <william.c.roberts@intel.com>
Rather than hardcode the OEM ranges, parse and extract
AID values from android_filesystem_config.h.
An AID is defined to the tool as:
* #define AID_<name>
An OEM Range is defined to the the tool as:
* AID_OEM_RESERVED_START
* AID_OEM_RESERVED_END
or
* AID_OEM_RESERVED_N_START
* AID_OEM_RESERVED_N_END
Where N is a number.
While parsing, perform sanity checks such as:
1. AIDs defined in the header cannot be within OEM range
2. OEM Ranges must be valid:
* Cannot overlap one another.
* Range START must be less than range END
3. Like the C preproccessor, multiple matching AID_<name> throws
en error.
The parser introduced here, prepares the tool to output android_ids
consumable for bionic.
Note that some AID_* friendly names were not consistent, thus a small
fixup map had to be placed inside the tool.
Test: tested parsing and dumping the data from android_filesystem_config.h
file.
Change-Id: Ifa4d1c9565d061b60542296fe33c8eba31649e62
Signed-off-by: William Roberts <william.c.roberts@intel.com>
This internally structures fs_config_generator.py to be able
to plug in generators to produce different outputs. This
prepares this tool for group and pwd file outputs.
Test: Checked diff and hash of before and after files.
Change-Id: Ie558518ac227dd946d70ab48027698b72a9bc94a
Signed-off-by: William Roberts <william.c.roberts@intel.com>
We are investigating replacing make_ext4fs with the upstream tool mke2fs.
To mitigate the trouble that may arise if the new tool behave differently
compared to the old one, there will be a transition period.
Devices that want to use the new way of creating ext4 images can set the
variable "TARGET_USES_MKE2FS" to true in their BoardConfig.mk
By default, the build system will choose the old tool 'make_ext4fs'.
Test: m otapackage with TARGET_USES_MKE2FS={,false,true}
Change-Id: I282bcb9efe335a86c53986283090ca947d65c7f8
Test: Checked diff and hash of before and after output files, same.
Test: Tested build for basic boot and functionality.
Change-Id: If7806427e3a2a9ddb7a2c9aa14e1e4f9bf696acf
Signed-off-by: William Roberts <william.c.roberts@intel.com>
Testing on earlier versions of Python indicated an error in the
README. Python's ConfigParser prior to version 3.2 allows for
sections within the same file to override later sections.
Correct the README.
Test: Checked diff and hash of output file from before and after.
Change-Id: Ifd3f78414b4803e4b74decb1fdf6816d4902c4a2
Signed-off-by: William Roberts <william.c.roberts@intel.com>
