So that the build can't access extra information unintentionally.
Particuarly ANDROID_BUILD_TOP is dangerous.
In the future PATH should be locked down as well.
Bug: 307824623
Test: Added a all_genrules target and built that
Change-Id: I88bb0efb0a82529a1c85875a53cf20c8384d07fe
Currently, OUT_DIR is inherited from the parent process, leading to
scripts being able to find the output directory when the enviornment
variable is set to an absolute path. When sandboxing a command,
also rewrite the OUT_DIR environment variable to the sandboxed one,
so that scripts can't find the real out dir.
Bug: 307824623
Test: Presubmits
Change-Id: I325071121a60bddc4105df680fbdfe3d11dc94e2
Genrules that set uses_order_only_build_number_file: true will now
have the ability to reference a $(build_number_file) label that
will point to the build number file. It will also caused the build
number file to be added as an order-only dependency, which will make
it show up in the genrule sandbox.
This is needed for converting make code that references the build
number to soong, and for sandboxing the remaining unsandboxed genrules
that reference the build number.
Bug: 341873065
Test: m nothing --no-skip-soong-tests
Change-Id: I9092cbb0eb39c5449a79f0ee40a4202262cef206
This allows a build rule to set the phony_output argument to ninja.
It's used to get ninja to ignore output files similar to .PHONY for
make.
It will cause the build rule to be run every time regardless of if or
when the output file was created.
Test: m android.hardware.bluetooth.audio-update-api &&
git restore bluetooth/audio/aidl/aidl_api/* &&
m android.hardware.bluetooth.audio-update-api
Bug: 353287999
Change-Id: I47b0b9e3ce2d8cb637b0102031193c81de24a680
xsdc commands end up using the same output directory inside sbox
for reading and writing the output archive, resulting in including
a partial copy of the archive in itself.
This change allows that rule to specify a sub-directory for the inputs
leaving the output archive outside of it.
Used by https://r.android.com/2955427
Bug: b/322788229
Test: Ran a couple of builds and confirmed no cache misses from xsdc.
Also manually checked an archive and confirmed
abcence of partial self-copies.
Change-Id: If709b111458a83d58906d919c3d2a7787bcbe4ee
When debugging problems with, or just modifying a ninja rule that uses
the SBox it is often necessary to read the manifest file. This change
makes them easier to read by pretty printing them onto multiple lines
with a small but clear indent.
This increased the size of the following file by abot 6% from 66K to
70K bytes:
out/soong/.intermediates/packages/modules/Wifi/framework/framework-wifi.stubs.source.system/android_common/metalava.sbox.textproto
Test: m out/soong/.intermediates/packages/modules/Wifi/framework/framework-wifi.stubs.source.system/android_common/metalava.sbox.textproto
less out/soong/.intermediates/packages/modules/Wifi/framework/framework-wifi.stubs.source.system/android_common/metalava.sbox.textproto
Change-Id: Icfadeaad1c6334bd0010128f4437669f35822850
symlink_outputs was added so bazel could run ninja files, but we
abanoned that approach in roboleaf, and then roboleaf was cancelled
entirely. Remove this feature so we're more compatible with upstream
ninja / n2.
Bug: 160568334
Test: Presubmits
Change-Id: Ic368c48dd01b68e51c471c3fe90d0c02c55956e9
Previously, the path to built files in the sandbox would be
out/.intermediates/... instead of out/soong/.intermediates/....
After this cl, it will be out/soong/.intermediates/.... This makes
it more consistent with the non-sandboxed paths, which is easier for
developers.
CtsApkVerityTestDebugFiles is a genrule that's used to find the paths
to other modules. Developers were expected to build it, then copy all
the files listed in its output file to a temporary directory. Those
paths would be wrong before this change.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py CtsApkVerityTestDebugFiles
Change-Id: Iadf9e3240a2c828567e46b6b02cc14004f30e8bf
The next CL will need a TestContext parameter in
ContentFromFileRuleForTests in order to retrieve the file rule contents
from the Config. Add it and update all the tests that use it in order
to simply review of the next CL.
Bug: 306029038
Test: go test ./...
Change-Id: Ia4b4c9854017ea3472fa2f8ba42cf7f72720496e
For rules which use RuleBuilder with Ninja variables (using the
BuildWithUnescapedNinjaVars function) but are not sandboxed, the command
output by RuleBuilder must not be escaped, or else Ninja variables will
not be evaluated until they are run on the command line.
Test: m libnum_traits liboid_registry crosvm droid
Change-Id: I81611368ab1975220f7f517df2f57646deac2af8
The RuleBuilder API would not expand Ninja variables because the
variables would be written verbatim to the sandbox manifest file. This
commit allows a rule to specify that the manifest file should be written
in an un-escaped format so that Ninja variables are expanded before
writing the manifest file.
Bug: 286077158
Test: rust sandboxing topic + go test
Change-Id: I1915431f6e24d04d343dacc213c9079674ec8251
The use case for this is for building rules_go's root builder which runs
into issues when built in a directory that contains a symlink to
prebuilts/go
The implementation will involve two changes of working dir
- `sbox` to change the working directory to
__SBOX_SANDBOX_DIR__
- the generated manifest will change the working
directory to mixed build execution root relative to that
Implemenation details
1. Create a unique intermediate path by hashing the outputs of a buildAction.
"out/bazel/output/execroot/__main__/" was deliberately not chosen as
the outpuDir for the sandbox because ruleBuilder would wipe it.
`sbox` will generate the files in __SBOX_SANDBOX_DIR__ and then place
the files in this intermediate directory.
2. After the files have been generated in (1), copy them to
out/bazel/output/execroot/__main__/...
3. For bazel depsets that are inputs of an action, copy the direct
artifacts into the sandbox instead of the phony target
4. Make sandboxing an opt-in. Currently we will only use it for
`GoToolchainBinaryBuild`
In the current implementation, (3) will increase the size of the ninja
file. With sboxing turned on for only GoToolchainBinaryBuild, this will
increase the size of the ninja file by around 1.3% on aosp's cf
Test: m com.android.neuralnetworks (will build soong_zip from source
using rules_go)
Test: OUT_DIR=out.other m com.android.neuralnetworks
Bug: 289102849
Change-Id: I7addda9af583ba0ff306e50c1dfa16ed16c29799
Add support to sbox for only writing output files if they are changed
to support enabling restat. Move the clearing of the output directory
into sbox, and when write if changed is enabled delete obsolete output
files without deleting output files that will be replaced.
Enable restat for local metalava rules. Restat for metalava rules
run remotely is still blocked on b/183630617.
Bug: 222095735
Test: TestRuleBuilder_Build
Test: m SystemUI
Change-Id: If8fc47e711f4a6f43947ab69f17cccc8ba6ae61a
Assume calls to RuleBuilderCommand.BuiltTool may refer to a tool
that was built against musl libc, and add it as a dependency
so that it is copied into the sandbox. This emulates the
behavior of compiling against glibc, which is available from the
host sysroot when running in the sandbox.
Bug: 190084016
Test: m USE_HOST_MUSL=true sdk-repo-build-tools
Change-Id: Ieafdcceb818f9c31595487aab3ffbafba1412b3a
Move cc module installation rules into Soong by overriding
InstallBypassMake. Update the locations that find host tools
to look in the Make installation directory instead of the Soong
installation directory, which will no longer be used.
The methods that find host tools are also used on go binaries,
so update the config methods that tell Blueprint where to install
go binaries to the Make installation directory too.
Bug: 204136549
Test: m checkbuild
Change-Id: Id172592c195e506102982a4af0084f6d9c68a896
Use pathForInstall instead of PathForOutput for HostToolPath,
HostJNIToolPath and sboxPathForToolRel so that they internally produce
an InstallPath that can later support being converted to Make install
path.
Bug: 204136549
Test: m checkbuild
Change-Id: Ie16a62641d113873daeec4d1dd4261251bc0d0eb
Previously, the RuleBuilder would add all supplied Paths directly to
one of its lists of Paths without checking to make sure it was not nil
only to panic somewhere in its Build() method when it attempted to
convert it to a string. Deferring the failure made it harder to debug.
This change fails fast by checking every Path when it is passed to the
RuleBuilder.
Test: m nothing
Change-Id: I06b35da02101f6dfab80d2daaf3d8e44ded8b6be
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
Add RuleBuilderCommand.Validation and RuleBuilderCommand.Validations
that propagate to BuildParams.Validations.
Test: TestRuleBuilder_build, TestRuleBuilder
Change-Id: I9a08bbc073a432fd0b53b87d8948ac0f6afcf637
Order only dependencies that were added to a RuleBuilderCommand were
only propagated to the rule that was created for a missing dependencies
error. Set them on the main rule instead.
Test: TestRuleBuilder_build
Change-Id: I049d1bb921778d18d678439d236f69081927dc70
Use an interface instead of a string to retrieve the value for
$(location) or $(locations) expansions in genrules to allow
delaying the evaluation until the RuleBuilderCommand is available.
This allows using helpers like RuleBuilderCommand.PathForInputs
to properly rewrite the values for sandboxing.
Also remove the standalone SboxPathFor* methods that don't operate
on a specific RuleBuilderCommand that are now unnecessary.
Test: genrule_test.go
Change-Id: I8bb2647332ef118204a216cead23d062517e2b8c
The lint rule is manually creating a second rsp file because Ninja
only supports on per rule. Move the support into RuleBuilder so
that it can apply the same rewrites that it does to the primary
one.
Test: TestRuleBuilder_Build
Change-Id: Iec250a2d60e74ccf1b4ad085a960fec6867ea559
The current implementation causes inputs listed in an rsp file used with
sbox to be duplicated 3 times in the build.ninja file; once as a
dependency of the rule, once in the rspfile_content field of the rule
with the paths rewritten to be relative to the sandbox, and once in the
rule to write the sbox manifest. When RBE is enabled it also gets a
fourth copy in the list of files to be treated as inputs by rewrapper.
Reduce this to a single copy by using "$in" for the rspfile_content so
that the files only have to be listed in the input dependencies of the
rule, and then add support to sbox to rewrite the rsp file while copying
it into the sandbox, and pass it to rewrapper as well.
Test: m lint-check
Change-Id: I3f46f61119508d39a8bb231c99fc130153fb6f04
rewrapper supports a comma separate list of rsp files, replace
REParams.RSPFile with REParmas.RSPFiles.
Test: remoteexec_test.go
Change-Id: I7850c071c23d368d6fad4480dd527d146c13c6d3
Allow passing a remoteexec.REParams to RuleBuilder to configure it to
run the rule remotely through RBE. Requires the rule to use
SandboxInputs, which ensures that RuleBuilder is aware of all of the
inputs and outputs of the rule.
Running sbox in RBE initially seems unnecessary, as RBE is already a
good sandbox, but reproxy can execute RBE actions locally when configured
for local execution, local fallback or racing. Using sbox in RBE ensures
that these local actions are also sandboxed, giving consistent results
between directly executed actions, local RBE actions, and remote RBE
actions.
Bug: 182612695
Test: manual
Change-Id: Icf2f24dde8dee833eb680ba22566a8e1c0143b15
When RuleBuilder.SandboxInputs() is called configure sbox to copy
all the input files into the sandbox directory and then change the
working directory there when running the command.
Copying input files into the sandbox directory gets tricky when
the input file is the output file from another rule, and could
be at an arbitrary, possibly absolute path based on the value
of OUT_DIR. They will need to be copied to a directory in the
sandbox using the path relative to OUT_DIR.
RSP files need special handling, they need to both be copied into
the sandbox as an input, rewritten to contain paths as seen in the
sandbox, and references to them on the command line need to use
sandbox paths.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: Ic0db961961b186e4ed9b76246881e3f04971825c
Output files in sandboxed RuleBuilder rules must be under the output
directory, but output paths that were marked as temporaries were
not error checked.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I09616402ef1637c2c455ec7d345a296711582e8f
WriteFileRule shouldn't force the caller to ninja escape the input,
and should shell escape spaces.
Bug: 182612695
Test: manual
Change-Id: Ide2f1ed92783eef7883279238de209d992d8f735
Now that rsp files use an explicit path instead of $out.rsp the
unescaped spans and NinjaEscapedCommands() support can be removed
from RuleBuilder.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I6705151ff0528c59aa1af56ee52c294da15a7a66
Using $out.rsp as the rsp file adds extra complexity around keeping
the $ unescaped. Make callers to FlagWithRspFileInputList provide
an explicit path for the rsp file instead.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I3f531d80c1efa8a9d09aac0a63790c5b11a9f0c6
The CommandDeps and Rspfile properties can contain arbitrary paths
that may need ninja escaping.
Bug: 182612695
Test: genrule_test.go
Change-Id: I1426e9dfeab3bde9f65b61d0c47840f97c1c7526
This relands I38393900677c5dbe8e72fe06a7dd7d71f3c09f82 after
I8038ed5c6f05c989ac21ec06c4552fb3136b9a7a, which makes the ASAN
variants of libraries have PackagingSpecs so they can be copied
into sandboxed genrules.
Copy tools used by genrules into the sandbox directory. This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.
Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I57c0d5fc8bba216fac4deb972d0d2098593e8963
Copy tools used by genrules into the sandbox directory. This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.
Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I38393900677c5dbe8e72fe06a7dd7d71f3c09f82
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
In preparation for more complicated sandboxing support in sbox, use
a single implementation of the sbox sandboxing by moving genrule to
use RuleBuilder's sbox support instead of creating an sbox rule
directly.
Also move genrule's input list hash support into RuleBuilder.
Test: genrule_test.go
Test: rule_builder_test.go
Change-Id: I292184d02743c7e6887ebbcd232ba565db2ab0cc
