diff --git a/dexpreopt/DEXPREOPT_IMPLEMENTATION.md b/dexpreopt/DEXPREOPT_IMPLEMENTATION.md new file mode 100644 index 000000000..c3a17307d --- /dev/null +++ b/dexpreopt/DEXPREOPT_IMPLEMENTATION.md @@ -0,0 +1,258 @@ +## Dexpreopt implementation + +### Introduction + +All dexpreopted Java code falls into three categories: + +- bootclasspath +- system server +- apps and libraries + +Dexpreopt implementation for bootclasspath libraries (boot images) is located in +[soong/java] (see e.g. [soong/java/dexpreopt_bootjars.go]), and install rules +are in [make/core/dex_preopt.mk]. + +Dexpreopt implementation for system server, libraries and apps is located in +[soong/dexpreopt]. For the rest of this section we focus primarily on it (and +not boot images). + +Dexpeopt implementation is split across the Soong part and the Make part. The +core logic is in Soong, and Make only generates configs and scripts to pass +information to Soong. + +### Global and module dexpreopt.config + +The build system generates a global JSON dexpreopt config that is populated from +product variables. This is static configuration that is passed to both Soong and +Make. The `$OUT/soong/dexpreopt.config` file is generated in +[make/core/dex_preopt_config.mk]. Soong reads it in [soong/dexpreopt/config.go] +and makes a device-specific copy (this is needed to ensure incremental build +correctness). The global config contains lists of bootclasspath jars, system +server jars, dex2oat options, global switches that enable and disable parts of +dexpreopt and so on. + +The build system also generates a module config for each dexpreopted package. It +contains package-specific configuration that is derived from the global +configuration and Android.bp or Android.mk module for the package. + +Module configs for Make packages are generated in +[make/core/dex_preopt_odex_install.mk]; they are materialized as per-package +JSON dexpreopt.config files. + +Module configs in Soong are not materialized as dexpreopt.config files and exist +as Go structures in memory, unless it is necessary to materialize them as a file +for dependent Make packages or for post-dexpreopting. Module configs are defined +in [soong/dexpreopt/config.go]. + +### Dexpreopt in Soong + +The Soong implementation of dexpreopt consists roughly of the following steps: + +- Read global dexpreopt config passed from Make ([soong/dexpreopt/config.go]). + +- Construct a static boot image config ([soong/java/dexpreopt_config.go]). + +- During dependency mutator pass, for each suitable module: + - add uses-library dependencies (e.g. for apps: [soong/java/app.go:deps]) + +- During rule generation pass, for each suitable module: + - compute transitive uses-library dependency closure + ([soong/java/java.go:addCLCFromDep]) + + - construct CLC from the dependency closure + ([soong/dexpreopt/class_loader_context.go]) + + - construct module config with CLC, boot image locations, etc. + ([soong/java/dexpreopt.go]) + + - generate build rules to verify build-time CLC against the manifest (e.g. + for apps: [soong/java/app.go:verifyUsesLibraries]) + + - generate dexpreopt build rule ([soong/dexpreopt/dexpreopt.go]) + +- At the end of rule generation pass: + - generate build rules for boot images ([soong/java/dexpreopt_bootjars.go], + [soong/java/bootclasspath_fragment.go] and + [soong/java/platform_bootclasspath.go]) + +### Dexpreopt in Make - dexpreopt_gen + +In order to reuse the same dexpreopt implementation for both Soong and Make +packages, part of Soong is compiled into a standalone binary dexpreopt_gen. It +runs during the Ninja stage of the build and generates shell scripts with +dexpreopt build rules for Make packages, and then executes them. + +This setup causes many inconveniences. To name a few: + +- Errors in the build rules are only revealed at the late stage of the build. + +- These rules are not tested by the presubmit builds that run `m nothing` on + many build targets/products. + +- It is impossible to find dexpreopt build rules in the generated Ninja files. + +However all these issues are a lesser evil compared to having a duplicate +dexpreopt implementation in Make. Also note that it would be problematic to +reimplement the logic in Make anyway, because Android.mk modules are not +processed in the order of uses-library dependencies and propagating dependency +information from one module to another would require a similar workaround with +a script. + +Dexpreopt for Make packages involves a few steps: + +- At Soong phase (during `m nothing`), see dexpreopt_gen: + - generate build rules for dexpreopt_gen binary + +- At Make/Kati phase (during `m nothing`), see + [make/core/dex_preopt_odex_install.mk]: + - generate build rules for module dexpreopt.config + + - generate build rules for merging dependency dexpreopt.config files (see + [make/core/dex_preopt_config_merger.py]) + + - generate build rules for dexpreopt_gen invocation + + - generate build rules for executing dexpreopt.sh scripts + +- At Ninja phase (during `m`): + - generate dexpreopt.config files + + - execute dexpreopt_gen rules (generate dexpreopt.sh scripts) + + - execute dexpreopt.sh scripts (this runs the actual dexpreopt) + +The Make/Kati phase adds all the necessary dependencies that trigger +dexpreopt_gen and dexpreopt.sh rules. The real dexpreopt command (dex2oat +invocation that will be executed to AOT-compile a package) is in the +dexpreopt.sh script, which is generated close to the end of the build. + +### Indirect build rules + +The process described above for Make packages involves "indirect build rules", +i.e. build rules that are generated not at the time when the build system is +created (which is a small step at the very beginning of the build triggered with +`m nothing`), but at the time when the actual build is done (`m` phase). + +Some build systems, such as Make, allow modifications of the build graph during +the build. Other build systems, such as Soong, have a clear separation into the +first "generation phase" (this is when build rules are created) and the second +"build phase" (this is when the build rules are executed), and they do not allow +modifications of the dependency graph during the second phase. The Soong +approach is better from performance standpoint, because with the Make approach +there are no guarantees regarding the time of the build --- recursive build +graph modfications continue until fixpoint. However the Soong approach is also +more restictive, as it can only generate build rules from the information that +is passed to the build system via global configuration, Android.bp files or +encoded in the Go code. Any other information (such as the contents of the Java +manifest files) are not accessible and cannot be used to generate build rules. + +Hence the need for the "indirect build rules": during the generation phase only +stubs of the build rules are generated, and the real rules are generated by the +stub rules during the build phase (and executed immediately). Note that the +build system still has to add all the necessary dependencies during the +generation phase, because it will not be possible to change build order during +the build phase. + +Indirect buils rules are used in a couple of places in dexpreopt: + +- [soong/scripts/manifest_check.py]: first to extract targetSdkVersion from the + manifest, and later to extract `` tags from the manifest and + compare them to the uses-library list known to the build system + +- [soong/scripts/construct_context.py]: to trim compatibility libraries in CLC + +- [make/core/dex_preopt_config_merger.py]: to merge information from + dexpreopt.config files for uses-library dependencies into the dependent's + dexpreopt.config file (mostly the CLC) + +- autogenerated dexpreopt.sh scripts: to call dexpreopt_gen + +### Consistency check - manifest_check.py + +Because the information from the manifests has to be duplicated in the +Android.bp/Android.mk files, there is a danger that it may get out of sync. To +guard against that, the build system generates a rule that verifies +uses-libraries: checks the metadata in the build files against the contents of a +manifest. The manifest can be available as a source file, or as part of a +prebuilt APK. + +The check is implemented in [soong/scripts/manifest_check.py]. + +It is possible to turn off the check globally for a product by setting +`PRODUCT_BROKEN_VERIFY_USES_LIBRARIES := true` in a product makefile, or for a +particular build by setting `RELAX_USES_LIBRARY_CHECK=true`. + +### Compatibility libraries - construct_context.py + +Compatibility libraries are libraries that didn’t exist prior to a certain SDK +version (say, `N`), but classes in them were in the bootclasspath jars, etc., +and in version `N` they have been separated into a standalone uses-library. +Compatibility libraries should only be in the CLC of an app if its +`targetSdkVersion` in the manifest is less than `N`. + +Currently compatibility libraries only affect apps (but not other libraries). + +The build system cannot see `targetSdkVersion` of an app at the time it +generates dexpreopt build rules, so it doesn't know whether to add compatibility +libaries to CLC or not. As a workaround, the build system includes all +compatibility libraries regardless of the app version, and appends some extra +logic to the dexpreopt rule that will extract `targetSdkVersion` from the +manifest and filter CLC based on that version during Ninja stage of the build, +immediately before executing the dexpreopt command (see the +soong/scripts/construct_context.py script). + +As of the time of writing (January 2022), there are the following compatibility +libraries: + +- org.apache.http.legacy (SDK 28) +- android.hidl.base-V1.0-java (SDK 29) +- android.hidl.manager-V1.0-java (SDK 29) +- android.test.base (SDK 30) +- android.test.mock (SDK 30) + +### Manifest fixer + +Sometimes uses-library tags are missing from the source manifest of a +library/app. This may happen for example if one of the transitive dependencies +of the library/app starts using another uses-library, and the library/app's +manifest isn't updated to include it. + +Soong can compute some of the missing uses-library tags for a given library/app +automatically as SDK libraries in the transitive dependency closure of the +library/app. The closure is needed because a library/app may depend on a static +library that may in turn depend on an SDK library (possibly transitively via +another library). + +Not all uses-library tags can be computed in this way, because some of the +uses-library dependencies are not SDK libraries, or they are not reachable via +transitive dependency closure. But when possible, allowing Soong to calculate +the manifest entries is less prone to errors and simplifies maintenance. For +example, consider a situation when many apps use some static library that adds a +new uses-library dependency -- all the apps will have to be updated. That is +difficult to maintain. + +There is also a manifest merger, because sometimes the final manifest of an app +is merged from a few dependency manifests, so the final manifest installed on +devices contains a superset of uses-library tags of the source manifest of the +app. + + +[make/core/dex_preopt.mk]: https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt.mk +[make/core/dex_preopt_config.mk]: https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt_config.mk +[make/core/dex_preopt_config_merger.py]: https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt_config_merger.py +[make/core/dex_preopt_odex_install.mk]: https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt_odex_install.mk +[soong/dexpreopt]: https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt +[soong/dexpreopt/class_loader_context.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt/class_loader_context.go +[soong/dexpreopt/config.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt/config.go +[soong/dexpreopt/dexpreopt.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt/dexpreopt.go +[soong/java]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java +[soong/java/app.go:deps]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/app.go?q=%22func%20\(u%20*usesLibrary\)%20deps%22 +[soong/java/app.go:verifyUsesLibraries]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/app.go?q=%22func%20\(u%20*usesLibrary\)%20verifyUsesLibraries%22 +[soong/java/bootclasspath_fragment.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/bootclasspath_fragment.go +[soong/java/dexpreopt.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/dexpreopt.go +[soong/java/dexpreopt_bootjars.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/dexpreopt_bootjars.go +[soong/java/dexpreopt_config.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/dexpreopt_config.go +[soong/java/java.go:addCLCFromDep]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/java.go?q=%22func%20addCLCfromDep%22 +[soong/java/platform_bootclasspath.go]: https://cs.android.com/android/platform/superproject/+/master:build/soong/java/platform_bootclasspath.go +[soong/scripts/construct_context.py]: https://cs.android.com/android/platform/superproject/+/master:build/soong/scripts/construct_context.py +[soong/scripts/manifest_check.py]: https://cs.android.com/android/platform/superproject/+/master:build/soong/scripts/manifest_check.py