Commit Graph

8345 Commits

Author SHA1 Message Date
Mårten Kongstad
fa23d2993b aconfig: separate flag declarations and flag values
Simplify how aconfig configurations work: remove the ability to set flag
values based on build-id.

The aconfig files now some in two flavours:

  - flag declaration files: introduce new flags; aconfig will assign the
    flags a hard-coded default value (disabled, read-write)

  - flag value files: assign flags new values

`aconfig create-cache` expects flags to be declared exactly once, and
for their values to be reassigned zero or more times.

The flag value files are identical what used to be called override
files.

Also, remove the now obsolete build-id parameter: this was used to
calculate default values before applying overrides, and is no longer
needed.

Also rename a few more structs and functions to be closer to the .proto
names. This will make it easier to use the generated proto structs
directly, and get rid of the hand-crafter wrappers.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I7bf881338b0567f932099ce419cac457abbe8df8
2023-05-12 08:50:49 +02:00
Jeongik Cha
fa9090946e parallelize add_partition_calls only if output_zip is None
if output_zip isn't None, writing into zip file in parallel is not
thread-safe.

Bug: 281960217
Test: m dist
Change-Id: I10d68a4bb779cee244f40410ec95d38ca6040306
2023-05-12 15:14:58 +09:00
Treehugger Robot
d5dc0c17fe Merge "parallelize add_partition_calls" am: 96b51b54b6
Original change: https://android-review.googlesource.com/c/platform/build/+/2585397

Change-Id: I58e1b516a5109eaed483af1c8b9979f838e0b96f
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-12 05:19:41 +00:00
Daniel Zheng
6b558b3fe4 Merge "Add fastboot-info to updatepackage" am: 717591dd4e
Original change: https://android-review.googlesource.com/c/platform/build/+/2583539

Change-Id: I92a70b845d04e2f251eed2b96fbcfe58d57ffeea
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-12 05:19:25 +00:00
Treehugger Robot
96b51b54b6 Merge "parallelize add_partition_calls" 2023-05-12 04:54:11 +00:00
Daniel Zheng
717591dd4e Merge "Add fastboot-info to updatepackage" 2023-05-12 04:31:55 +00:00
Jeongik Cha
03d014a03f parallelize add_partition_calls
it made total time equal to the longest image build

1m10s->30s in local build

Bug: 281960217
Test: m dist
Change-Id: I13d4f45d9b46b39292a014e3b4e1913365d89b7a
2023-05-12 08:43:44 +09:00
Wei Li
fd7e6517d3 Some changes to support SBOM generation for b build unbundled APEXs.
1) Use output file path of installed files in build system since there is no PRODUCT_OUT in Bazel
2) Use CONTAINS to describe the relationship between a APEX and files it contains
3) Generate SBOM of APEXs, which is similar to SBOM of products

Bug: 275472038
Test: CIs
Change-Id: I41622366e5e6ed9dc78cca7bc7bb69a1f8f9bd9f
2023-05-11 13:58:37 -07:00
Cole Faust
0a9d1d6025 Merge "Allow importing starlark code in makefiles" am: 99d89154f1
Original change: https://android-review.googlesource.com/c/platform/build/+/2583544

Change-Id: If559b5f4119df6b19c42f5bf60f738ddf4365d8f
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-11 17:14:36 +00:00
Cole Faust
99d89154f1 Merge "Allow importing starlark code in makefiles" 2023-05-11 17:04:46 +00:00
Mårten Kongstad
86d0c527c0 Merge "aconfig: Add codegen for java" am: 7b6aacb055
Original change: https://android-review.googlesource.com/c/platform/build/+/2583478

Change-Id: I8bd49c346fdd4f2e30c5033f63783e66bc7d4bb8
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-11 08:24:31 +00:00
Zhi Dou
eb74489b3a aconfig: Add codegen for java
Add codegen for java skeleton

Bug: 279485059
Test: atest aconfig.test
Change-Id: Ia0481cec9c2e137e88e9a77d1b82412529b64adc
2023-05-11 03:08:53 +00:00
Treehugger Robot
13456aa63c Merge changes from topic "aconfig-part-4" am: ef0a2c21dd
Original change: https://android-review.googlesource.com/c/platform/build/+/2584571

Change-Id: I7bb9c8cfab6084b27ce8e62afabe747bb05e43a0
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-10 19:38:09 +00:00
Treehugger Robot
ef0a2c21dd Merge changes from topic "aconfig-part-4"
* changes:
  aconfig: introduce namespace, rename proto messages
  aconfig: add dump protobuf format
2023-05-10 19:22:00 +00:00
Treehugger Robot
02591d64de Merge "Improve finalization perfomance by building only current version." am: b1a7ba8735
Original change: https://android-review.googlesource.com/c/platform/build/+/2584105

Change-Id: I2386fa0a10a112343d0ba20ac256368bec7dbcc3
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-10 16:05:07 +00:00
Treehugger Robot
b1a7ba8735 Merge "Improve finalization perfomance by building only current version." 2023-05-10 15:19:12 +00:00
Mårten Kongstad
3095078981 aconfig: introduce namespace, rename proto messages
Flags belong to a namespace. Update the proto files to reflect this.

Config files can only refer to a single namespace. Override files can
refer to multiple namespaces; an override directive for a flag in a
different namespace than the one represented by the cache will be
silently ignored.

Rename the proto messages to make it more clear what they are. Propagate
this change through the wrappers in aconfig.rs and the rest of the code.

Also, settle on Tracepoint instead of TracePoint.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I16e69dd14687bc498b2ba89d6a35879459903801
2023-05-10 17:17:20 +02:00
Kelvin Zhang
06417aaa9e Merge "Fix non-AB ota generation for directories" am: ea70695d92
Original change: https://android-review.googlesource.com/c/platform/build/+/2585207

Change-Id: I7a6472946360238393a734640e27557f7623446f
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-10 04:46:20 +00:00
Kelvin Zhang
ea70695d92 Merge "Fix non-AB ota generation for directories" 2023-05-10 04:07:09 +00:00
Kelvin Zhang
b036007592 Fix non-AB ota generation for directories
Test: ota_from_target_files -v --block yukawa-target_files-10097950.zip
test.zip

Change-Id: I4291010d66cc28bdb6b6b334cdbed06daf3de837
2023-05-09 20:30:53 -07:00
Alex Buynytskyy
aac2a187d4 Improve finalization perfomance by building only current version.
Bug: 275409981
Test: run locally
Change-Id: If531178a17792166726ff7ed230592c392eeb899
2023-05-10 03:10:32 +00:00
Cole Faust
c63ce1ab58 Allow importing starlark code in makefiles
Adds a new `$(call run-starlark,my/starlark/file.bzl)` function that
will run the starlark file and set all the variables in the
variables_to_export_to_make dictionary as make variables.

Fixes: 280685526
Test: m nothing repeatedly causes no ninja regeneration, but touching all_versions.bzl does. go test, ./out/rbcrun -mode=rbc ./build/make/tests/run.rbc
Change-Id: Ic72e18dd28dba8233ba2dfb658b5d03ccece1bfd
2023-05-09 15:12:58 -07:00
Daniel Zheng
380cb8065a Add fastboot-info to updatepackage
Modifying img_from_target_files to also add fastboot-info to
updatepackage

Test: m updatepackage
Bug: 194686221
Change-Id: I2e08c4269f0d83417b9d7079633bc28796d1cdd6
2023-05-09 13:44:22 -07:00
Mårten Kongstad
a102909e09 aconfig: add dump protobuf format
Introduce a new protobuf format to represent the all flags parsed by
aconfig. This data in this new format is similar to that of the internal
cache object, but the protobuf is a public API for other tools to
consume and any changes to the proto spec must be backwards compatible.

When aconfig has matured more, Cache can potentially be rewritten to
work with proto structs directly, removing some of the hand-written
wrapper structs and the logic to convert to and from the proto structs.
At this point, the intermediate json format can be replaced by the
protobuf dump.

Also, teach `aconfig dump` to accept an --out <file> argument (default:
stdout).

Also, teach `aconfig dump` to read more than once cache file.

Note: the new protobuf fields refer to existing fields. It would make
sense to split the .proto file in one for input and one for output
formats, and import the common messages, but the Android build system
and cargo will need different import paths. Keep the definitions in the
same file to circumvent this problem.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I55ee4a52c0fb3369d91d61406867ae03a15805c3
2023-05-09 15:27:42 +02:00
Po Hu
fc3387bb0e Support to merge PREBUILT_IMAGES
For example, pvmfw.img comes from framework target files package
and dtbo.img comes from vendor target files package.

Bug: 270241811
Test: merge_target_files
Merged-In: If2e6fac5ce3671e3387e801808a195fb7b2c55b5
Change-Id: If2e6fac5ce3671e3387e801808a195fb7b2c55b5
(cherry picked from commit 81c3f4a884)
2023-05-09 08:38:39 +00:00
Treehugger Robot
9840061379 Merge "Remove unneeded rbcrun features" am: 872023376e
Original change: https://android-review.googlesource.com/c/platform/build/+/2580473

Change-Id: Ia8b8f735b272a8a61eb989ecef819516b3a9f2b7
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-09 08:15:26 +00:00
Treehugger Robot
872023376e Merge "Remove unneeded rbcrun features" 2023-05-09 07:31:40 +00:00
Treehugger Robot
8bad8c04de Merge changes from topic "aconfig-part-3" am: d279e5d79c
Original change: https://android-review.googlesource.com/c/platform/build/+/2581670

Change-Id: I0479ee840934c8195172698248fcb06887072ebd
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-08 18:42:27 +00:00
Mårten Kongstad
2fc13df5d7 Merge changes from topic "aconfig-part-2" am: 80bff11aef
Original change: https://android-review.googlesource.com/c/platform/build/+/2578555

Change-Id: I9f41660ea3cefad70f405b93587baa005b5b4f6e
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-08 11:40:43 +00:00
Mårten Kongstad
98b0eeb2fc aconfig: simplify argument parsing in main
Improve readability by extracting how multiple input files are
transformed from a clap argument to an Input.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I299179d39eca80e107342f6978984da79ee81e34
2023-05-08 12:47:34 +02:00
Mårten Kongstad
76adff2af0 aconfig: improve flag value tracing
Improve how the flag values (state, permission) are tracked when parsing
config and override files: migrate from a raw string to a proper struct,
as this will make it easier when aconfig will output this data in some
structured format for other tools to consume.

Also, rename Cache.debug to Cache.trace.

Bug: 279485059
Test: atest aconfig.test
Change-Id: Idad57c969515f697e9065429d8a44c38d8a512d2
2023-05-08 12:47:34 +02:00
Mårten Kongstad
80bff11aef Merge changes from topic "aconfig-part-2"
* changes:
  aconfig: change flag values to enabled/disabled enum
  aconfig: add read/write permission
  aconfig: Source: remove unnecessary #[derive(Serialize, Deserialize)]
2023-05-08 10:46:26 +00:00
Treehugger Robot
9cc5216ed0 Merge "Script fixes and improvements." am: be8badd74e
Original change: https://android-review.googlesource.com/c/platform/build/+/2581131

Change-Id: Icf9617f5f6ddf548ce5994f2c8945bcd9c06829c
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-07 01:43:18 +00:00
Alex Buynytskyy
00a88ec3ec Script fixes and improvements.
Fix to mark all versions REL.

More robust patch application. Keep getting these build breakages, not sure why. Looks like git is
confusing one folder for the other?
https://android-build.googleplex.com/builds/submitted/10065142/test_suites_arm64/latest/logs/build_error.log

Bug: 275409981
Test: run locally
Change-Id: I6173b9ecae5e9ab12d9abf2eee3f2136dc353210
2023-05-06 20:55:10 +00:00
Cole Faust
a874f88cee Remove unneeded rbcrun features
- rblf_cli and rblf_env
- -c and -f

This is in preparation for making rbcrun able to function as a more
general purpose starlark interpreter.

Bug: 280685526
Test: go test, ./out/rbc ./build/make/tests/run.rbc, ./build/bazel/ci/rbc_dashboard.py --quick aosp_arm64
Change-Id: Ifff9ce7b4369422f39c5003bb85a168c78bde7cf
2023-05-05 11:52:02 -07:00
Mårten Kongstad
c68c4eabea aconfig: change flag values to enabled/disabled enum
Change the underlying type of a flag's value from bool to an explicit
enum (Disabled, Enabled): this will hopefully reduce future confusion on
how flags are intended to be used.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I9535f9b23baf93ad5916ca06fb7d21277b4573eb
2023-05-05 16:20:09 +02:00
Mårten Kongstad
416330b060 aconfig: add read/write permission
Introduce the concept of flag read/write permissions: a read-only flag
can only have its value set during the build; a writable flag can by
updated in runtime.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I3ec5c9571faa54de5666120ccd60090d3db9e331
2023-05-05 13:45:28 +02:00
Mårten Kongstad
2937566c55 aconfig: Source: remove unnecessary #[derive(Serialize, Deserialize)]
The Source struct is no longer serialized/deserialized. Remove the
unused derives.

Bug: 279485059
Test: atest aconfig.test
Change-Id: Ifd78988ed8134ab43013314b4437e428a8927981
2023-05-05 10:57:19 +02:00
Mårten Kongstad
787a0d1ffd Merge changes from topic "aconfig-part-1" am: 0b6b92e5a5
Original change: https://android-review.googlesource.com/c/platform/build/+/2578416

Change-Id: Icdeb54a877d6e0e7df55321e4ed61d04a9011e91
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-04 21:27:00 +00:00
Mårten Kongstad
0b6b92e5a5 Merge changes from topic "aconfig-part-1"
* changes:
  aconfig: add support for changing flag value based on build
  aconfig: introduce cache
  aconfig: define Aconfig proto
  aconfig: add support for cargo
2023-05-04 20:36:22 +00:00
Mårten Kongstad
09c28d1689 aconfig: add support for changing flag value based on build
Teach aconfig about build IDs (continuously increasing integers). Extend
the aconfig file format to allow flags to say "by default, my value is
X, but starting from build ID A, it's Y, and from build ID B, it's Z".

Bug: 279485059
Test: atest aconfig.test
Change-Id: Idde03dee06f6cb9041c0dd4ca917c8b2f2faafdd
2023-05-04 14:34:55 +02:00
Mårten Kongstad
4d2b4b047b aconfig: introduce cache
Introduce the Cache struct to represent parsed and verified aconfig and
override content. Most commands in aconfig will work of an existing
cache file, eliminating the need to re-read the input every time.

Restructure main.rs to use clap to create a proper command line
interface with support for sub-commands. main.rs is responsible for
parsing the command line, performing disk I/O and calling the correct
subcommand implementation (in commands.rs).

To simplify unit tests, subcommands never perform explicit I/O; instead
they only work with Read and Write traits.

Also add dependencies on clap, serde and serde_json.

Bug: 279485059
Test: atest aconfig.test
Change-Id: Ib6abf2eabd264009804f253874b6fba924fc391b
2023-05-04 10:40:26 +02:00
Mårten Kongstad
bb520729e8 aconfig: define Aconfig proto
Fill in aconfig.proto. Define Aconfig definitions (for introducing flags
and setting their values) and Overrides (for overriding flags regardless
of what their definitions say). More changes to the proto schema are
expected when more of the aconfig project is outlined.

Use proto2 instead of proto3: this will cause the protobuf text parser
to error on missing fields instead of returning a default value which is
ambiguous, especially for booleans. (Also, the text protobuf parser
doesn't provide good error messages: if the input is missing a field,
the error is always "1:1: Message not initialized").

Unfortunately the generated Rust wrappers around the proto structs land
in an external crate, which prevents aconfig from adding new impl
blocks. Circumvent this by converting the data read from proto into
structs defined in the aconfig crate.

Change main.rs to parse (static) text proto.

Also add dependency on anyhow.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I512e3b61ef20e2f55b7699a178d466d2a9a89eac
2023-05-04 10:35:25 +02:00
Treehugger Robot
5e4f59f9ca Merge "Support generating partial OTAs from extracted target_files" am: c3216a321b
Original change: https://android-review.googlesource.com/c/platform/build/+/2577071

Change-Id: I79fa834772dc12fe26678238bcf80ecffd2a34ce
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-04 00:54:58 +00:00
Treehugger Robot
c3216a321b Merge "Support generating partial OTAs from extracted target_files" 2023-05-04 00:05:29 +00:00
Treehugger Robot
acbf5e5c45 Merge "Update sdk finalizer for change to api_levels_released_versions" am: 626b012556
Original change: https://android-review.googlesource.com/c/platform/build/+/2573034

Change-Id: Id8fb00b47f16bcd17f0503e5f4fb733d3c5fc03a
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-03 18:01:54 +00:00
Treehugger Robot
626b012556 Merge "Update sdk finalizer for change to api_levels_released_versions" 2023-05-03 17:24:31 +00:00
Mårten Kongstad
fe753f5365 aconfig: add support for cargo
Officially, aconfig is build using the Android tool-chain. However, to
speed up the local development cycle, add support for building with
cargo.

While it is possible to tell cargo to place the build artifacts outside
the source tree, there is no way to tell it to not generate the cargo
lock file in the same directory as Cargo.toml. Add a .gitignore to
ignore Cargo.lock and the target directory.

The way the Android build system and cargo generates code from the
protobuf files is slightly different. Tell cargo to enable the "cargo"
feature and introduce src/protos.rs to hide this difference from the
rest of the aconfig source.

Bug: 279485059
Test: m aconfig && aconfig
Test: atest aconfig.test
Test: cargo build
Test: cargo test
Change-Id: I85741f58cadae353ed95c124f566e4f4a7484186
2023-05-03 09:41:11 +02:00
Mårten Kongstad
7b9a0b649c Merge "aconfig: add project scaffolding" am: e7a850e942
Original change: https://android-review.googlesource.com/c/platform/build/+/2563154

Change-Id: I88c6b05de525f296d879805e099f020f7c5e4fb4
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-05-03 07:07:49 +00:00
Mårten Kongstad
e7a850e942 Merge "aconfig: add project scaffolding" 2023-05-03 06:18:02 +00:00