There were a few issues with the output packaging process that were
found during testing of the general-tests optimization.
First and foremost is that the packaging commands were trying to be
created before the build ran, when the outputs don't exist. I've changed
the logic to just collect the methods themselves which will then be run
during build plan execution after the build has completed.
A few other smaller issues include fixing the path to the soong_zip
binary, incorrect execution of the soong dumpvars command, and not
building the shared libs zip.
Test: atest build_test_suites_test; atest optimized_targets_test
Bug: 358215235
Change-Id: I8a3f54738f8bb5d871aadf7423844076c38b54a6
Implement the output packaging step in the GeneralTestsOptimizer. This
step takes all the outputs built by the general-tests target (in the
case where it was optimized) and packages them into the necessary zips
generated by the target normally.
Test: atest build_test_suites_test; atest optimized_targets_test
Bug: 358215235
Change-Id: I5d27eef4e37137cc9b6e235f52f3856ba0b30460
Refactor package_outputs in the TestOptimizer so it just returns a list
of soong_zip commands to be run by build_test_suites.
Since we already have a tested implementation for running subprocesses
in build_test_suites.py there's no reason to reimplement it in
optimized_targets.py. Because any packaging will ultimately use
soong_zip to package its final outputs change the code to just do
whatever prep it needs to and return a list of soong_zip commands.
This way the code is simpler to test without requiring subprocesses and
no reimplementation of subprocess running code is necessary.
Test: atest build_test_suites_test; atest optimized_targets_test
Bug: 358215235
Change-Id: I3025aefeeb7186f537266a72d8422211ca9835ba
Implement functionality in GeneralTestsOptimizer to find what targets to
build.
This logic is fairly complex and involves checking for test
configs that download general-tests.zip. Then the configs are checked to
see if they're proper test mapping tests (if they use the
'test-mapping-test-group' option). If they are, then TEST_MAPPING
modules are scanned to see if the list of changed files would cause any
test mapping modules to run. The tests are then further filtered by
test-mapping-test-groups used in the test configs.
In case that a test uses general-tests.zip but does not specify
'test-mapping-test-group' then all bets are off and general-tests.zip
is built in its entirety.
package_outputs is still unimplemented so this will need to be
implemented before the optimization can be enabled.
Test: atest build_test_suites_test && atest optimized_targets_test
Bug: 358215235
Change-Id: I6a7eebfd1b06b380799292eb2019ac17c9af5367
Add a BuildContext class to simplify parsing the build context dict, and
fix the parsing of enabledBuildFeatures, which was being parsed as a
list of strings when it's actually a list of dicts like:
[{'name': '<feature_name>'}]
Test: atest build_test_suites_test
Bug: 361605425
Change-Id: I6424c444daf1582e92313c39f43207cb274aa78f
The current artifact matching logic will cause any test target that uses
something like .*-tests.zip to be way too broad in what it causes to
build. Change the logic to look for the target name in the artifact
matching regex to make it so that only tests that look for a specific
target to match.
Test: atest build_test_suites_test
Bug: 348489774
Change-Id: Ia75e38b676607f45f2b1c8fcf948045c248f1729
Refactor the OptimizedBuildTarget base class to more simply handle
enabling/disabling build optimizations based on flags. Now all
optimizers (aside from the no-op NullOptimizer will have their enabling
built into the base class, and also necessitate their get_build_targets
and package_outputs functions to be defined in order to not error out.
Add the GeneralTestsOptimizer class as well, unimplemented for now.
Test: atest build_test_suites_test
Bug: 358215235
Change-Id: Ide22c64fc238d754db8d8e76484947401e54e680
Add functionality in build_test_suites.py to not build targets if
their outputs are not used in the test configurations saved in the build
context. If none of the tests reference the targets' outputs they will
not be built at all.
Note that the corresponding flags will need to be enabled for these
optimizations to take place.
Test: atest optimized_targets_test
Bug: 348489774
Change-Id: I8f0ac90e75552ae80073f13229b026c7f23476a6
Set up logging properly in build_test_suites.py so that the log file is
directed to the dist dir (and /tmp if it is not defined)
Test: atest build_test_suites_test
Bug: 355074139
Change-Id: I16ce644f1c717179377c8ca68c74d9af5f2e61df
Ensure that output and errors from the underlying build command are
streamed for real-time build progress updates and debugging.
This change strips out all code that is currently unecessary and adds
tests for the remaining functionality.
Test: atest --host build_test_suites_local_test build_test_suites_test
Bug: 330365727
Change-Id: I7ef98d6654fe1435cf67c15e2c516a0967e03a75
the dist dir should not be passed in as an argument, it should be picked
up from the environment.
Test: ./build/make/ci/build_test_suites --target_product aosp_x86_64 --target_release trunk_staging --with_dexpreopt_boot_img_and_system_server_only
Bug: 314171817
Change-Id: I32c007677cf1d81bb7b263b5c69e5510bf8dccfe
Due to branch constraints, remove the extra targets defined inside the
build script and revert to having them be passed in via command line.
Test: ./build/make/ci/build_test_suites_x86_64-trunk_staging --target_product aosp_x86_64 --target_release trunk_staging --with_dexpreopt_boot_img_and_system_server_only <extra_targets>
Bug: 314171817
Change-Id: I89a82e969d294104f587d778ad319a41905320f8
In preparation of switching over the git_main test suites target to
using this script change it to a no-op first to insure there's no
initial impact.
Test: ./build/make/ci/build_test_suites-trunk_staging --target_product aosp_x86_64 --target_release trunk_staging --with_dexpreopt_boot_img_and_system_server_only
Bug: 314171817
Change-Id: I7afab933143e4b617866b1975b1cbc71107f5b2f
Build general-tests_configs.zip and general-testlist.zip as part of the
build_test_suites step, which general-tests also outputs.
Test: BUILD_NUMBER="P0" ./build/make/ci/build_test_suites --target_product aosp_x86_64
--target_release trunk_staging --with_dexpreopt_boot_img_and_system_server_only --dist_dir <dist_dir> --change_info <change_info_file> <extra_targets>
Bug: 314171817
Change-Id: Ic324ed4e5a05e1308dee3af7f15c067f88018b2d
The current logic to determine if a build is pre or postsubmit is the
existence of the change_info file but this file also exists in
postsubmit, causing the build to break. Use the build ID as the
heuristic instead which should be more reliable.
Test: abtd run
Bug: 314171817
Change-Id: I9871eee71c0574583b528135dadd42e3f485a780
Two fixes:
soong dumpvars commands require TARGET_RELEASE to be set in the env, so
set that.
Added option to print command output which is useful for actually
logging build output not only in the case of command failures.
Test: ABTD run with this change
Bug: 314171817
Change-Id: Ia3ad8ebfa58959c13417abaeeaa42a354c4c61de
Add shebang line to make build test suties script callable via command
line without needing python.
Test: ./build/make/ci/build_test_suites --target_product aosp_x86_64
--target_release trunk_staging --with_dexpreopt_boot_img_and_system_server_only --dist_dir <dist_dir> --change_info <change_info_file> <extra_targets>
Change-Id: I94422993e864e021b036a4eb1673b463ad4b86d4
Move the test build command to a script that lives in-source and
intended to be called by the Android CI build system.
This script optimizes the build when change information is available by
only building relevant test modules. More specifically, only 'affected'
test modules listed in Test Mapping configuration files are built.
Using this script is expected to reduce build times and generate a
smaller `general-tests.zip` file which had increased to over 30 GB.
Test: ./build/make/ci/build_test_suites --target_product aosp_x86_64
--target_release trunk_staging --with_dexpreopt_boot_img_and_system_server_only --dist_dir <dist_dir> --change_info <change_info_file> <extra_targets>
Bug: 314171817
Change-Id: Ie2f8c066f5dd1cd13454141e55956328303fd79d
