Commit Graph

131 Commits

Author SHA1 Message Date
Mohd Faraz
d1d72351b2 extract_utils: Added Suffix uses COMMON_SUFFIX
* COMMON_JAVA_PACKAGE_SUFFIX for jar

 * COMMON_ANDROID_PACKAGE_SUFFIX for apk

Change-Id: I812405dac12ef7183985c66a6e43b0ea5f85989c
Signed-off-by: Mohd Faraz <mohd.faraz.abc@gmail.com>
2019-10-27 11:24:28 +00:00
Rashed Abdel-Tawab
34b5cdc1e3 extract_files: Implement blueprint file generation
Switch to blueprint on:
 - shared objects
 - $partiton/etc/ files
 - JARs
 - executable binaries and scripts
 - APKs

Only /sbin binaries are still in Android.mk because blueprint
doesn't handle sbin installation yet

Change-Id: I1dfd7e8bb575367b2a7fa9e333c4c6fa3aa68180
2019-10-27 11:24:28 +00:00
Rashed Abdel-Tawab
42752d4d4e extract_files: Add initial blueprint creation
Start the copyrights for these at 2019 because the blueprints won't
have existed before then.

Change-Id: I5300109949785dd19555e35d17a1e80daef498b0
2019-10-27 11:24:28 +00:00
Rashed Abdel-Tawab
8aae50df3c extract_files: Match non-treble path prefixes
Some devices put stuff on /system, /system/vendor or even
/system/vendor/odm. Search for these paths too when generating
TARGET_COPY_OUT_$partition variables.

Change-Id: Ie2c087e57aaca02d5ea93f290d5fc50d1315a600
2019-10-27 11:24:28 +00:00
Rashed Abdel-Tawab
8de76b204b extract_files: Support system/ prefixes
With support for 4 independent partitions now, we seriously
need to start putting /system blobs in their own directory.
Add support for file lists with system/ prefixes while
maintaining support for old file lists without it.

Also, TARGET_COPY_OUT_SYSTEM is a thing now, and all devices,
regardless of treble or not, set TARGET_COPY_OUT_$partition
so let's get rid of the treble compat option and default it
to true.

Change-Id: I5b798d293768d7c1e16db3ba01e2de3e083088d7
2019-10-27 11:24:06 +00:00
PIPIPIG233666
17d7dcba43 extract_utils: Fix typo
* ./../../xiaomi/sm6150-common/../../../vendor/lineage/build/tools/extract_utils.sh: line 1: /#!/bin/bash: No such file or directory.

Signed-off-by: PIPIPIG233666 <2212848813@qq.com>
Change-Id: I178f745d4ecb818c38706ff100611df19221065d
2019-09-29 12:49:44 -04:00
Bruno Martins
8194b8e583 extract_utils: Source the prebuilts from the new repo
The prebuilt tools now live in prebuilts/tools-lineage.

Change-Id: I4022210c52e1002aae0c6e622a480e5c7e10c366
2019-09-29 03:06:49 +02:00
Bruno Martins
3dff7da473 build/tools: Move prebuilts to the new repository
Going to be imported into prebuilts/tools-lineage.

Change-Id: I81dd06249a8de8f4970be30d75b396c85d4c4fb7
2019-09-29 03:06:49 +02:00
Rashed Abdel-Tawab
08e3a27135 extract_files: Add ODM partition blob support
Change-Id: Id94bbef879b6f4bcc20645c259981f9d163649c9
2019-09-27 06:46:44 -07:00
LuK1337
c62a9fbc31 repopick: Print change subject as unicode
* This prevents from seeing stuff like
  "b'frameworks: Add unlinked ringtone and notification volumes'"
  when using python3 as default.

Change-Id: Ie1fa85681b648edcee65680d784da4dff1779616
2019-09-21 11:48:49 +02:00
razorloves
b5c2c96a56 extract_utils: Support extracting from product partition
Change-Id: I2d0d01253fdf93d63a26556208d45cd69e76329f
(cherry picked from commit ea15be6e92e3be0c103948d220de64fd2e1d781f)
2019-09-10 11:28:05 +02:00
Rashed Abdel-Tawab
19c36cd7a6 extract_utils: Support multidex
Co-authored-by: Erfan Abdi <erfangplus@gmail.com>
Change-Id: I4881658d303b6c5c7f0b141beb50c1e9100af611
2019-07-18 07:31:36 +02:00
Rashed Abdel-Tawab
1c29c373dd extract_utils: Add functions to extract vendor blobs from vendor.img
Change-Id: I6f37adc955856fa78e719eb30249800abe80020c
2019-06-26 04:37:44 +02:00
Luca Weiss
5ee35ea356 repopick: cmp() is not available in Python 3, define it manually
Change-Id: I16a89aa555e62f105d888e87cb43afd26606076b
2019-06-08 00:01:05 +02:00
Vladimir Oltean
4818c230b3 extract_utils: introduce support for executing blob fixups
* Traditionally, the task of hex-editing blobs has been approached in 2 ways:
  (1) Do it out-of-band, commit the modified blob, and record its edited
      sha1sum in proprietary-files.txt (aka pin it).
  (2) Do it in-band, by adding code to the device-level extract-files.sh
      (usually this performs patchelf or sed). This code runs after the
      extract_utils functions were invoked.
* Problems of approach (1):
    - It relies on verbal (basically commit message) documentation of
      the hex-editing that was done. Makes it more difficult to reproduce.
    - Each time blobs are updated, pinning needs to be temporarily removed,
      hex-editing done again manually and new hash put back.
* Problems of approach (2):
    - It is incompatible with the concept of pinning, which is useful
      for kanging blobs from another device. A pinned blob would either:
        - Match the hash, get hex-edited, then it won't match the hash
          next time around.
        - Not match the hash (because of, say, hex-editing), then the
          extraction script would use an unwanted blob version instead of the
          pinned one (either that, or say "!! file not found in source").

* In summary, this patch adds system-wide support for approach (2) in order
  to address the aforementioned shortcomings.

* At device level, users of extract_utils who wish to perform blob
  fixups can override a blob_fixup() Bash function in their
  extract-files.sh immediately after running "source ${HELPER}". The
  blob_fixup() function will be called by the common extract() function
  after extracting every individual blob, giving the user the
  opportunity to hook custom code after this operation takes place.

* In proprietary-files.txt, the line corresponding to this blob which
  needs fixups can look in one of 2 ways:

(a) vendor/lib64/vendor.qti.gnss@1.0_vendor.so

    Do this if you are taking the blob from the stock ROM. The fixup
    script will always run after the blob is extracted.

(b) vendor/lib64/vendor.qti.gnss@1.0_vendor.so|249c76153f8de014bf2dd2ab623ee3d87741fbc8|f7e9ee8e3804887a2f3939128e860767e6f27258

    Do this if you are kanging the blob from somebody else. The pinning
    logic now applies for both the pre- and the post-fixup hashes. The
    fixup script will only run if the blob doesn't match the hex-edited blob,
    although the fixup script should really be idempotent.

Change-Id: Ifdd73c885d995c645f6210597537693d1a2f903f
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2019-03-03 16:44:53 +02:00
Vladimir Oltean
5238ba8fe5 extract_utils: make --section match words anywhere in the line
Change-Id: Ia3fe2160f0f4bc7da1b5bd31a21080af51b41fab
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2019-03-03 16:42:24 +02:00
Vladimir Oltean
bec92ebaec extract_utils: introduce -k (kang mode) option
* The use case is easier updating of pinned blobs. When --kang is set,
  pinning is automatically ignored, and the script will print lines at
  its output that can be directly copied back into the
  proprietary-files.txt.
* Best served together with --section ${SECTION}, and proper grouping
  of the proprietary-files.txt.

Change-Id: I648fbcbd4580a4a002b00828bcfee18d1e265d7b
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2019-03-03 16:42:24 +02:00
Vladimir Oltean
c503446ec3 extract_utils: cleanup variable names and arguments passed to extract() function
* This also makes the --section argument non-positional, since otherwise
  it is not possible to easily support more than one optional positional
  argument. This is in preparation of one more optional argument to come
  in a follow-up patch: --kang.

Change-Id: Ieb142e0854319defb9a278ab68cd4aeefd0fbdd5
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2019-03-03 16:42:24 +02:00
Vladimir Oltean
78d690d12c extract_utils: be compatible with system-as-root layouts
* The use case is that if you have the following layout:
  $TOP --- system.img
       |
       +-- vendor.img
  you should be able (from $TOP) to:
    mkdir system; mount -o ro,loop system.img system
    mkdir vendor; mount -o ro,loop vendor.img vendor
  and then (from device tree)
    ./extract-files.sh $TOP

  But this doesn't work if system.img is SAR and contains another
  "system" dir inside. This patch makes sure it searches for a "system"
  dir in the provided path as well, if it couldn't find the blob
  anywhere else.

Change-Id: Ib49cd5b587b3a57478a66ff69cf840270c2b1403
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2019-03-03 16:42:24 +02:00
Vladimir Oltean
4fe79b11c1 extract_utils: drop the "/system/" prefix from prints of blob names
* This makes the printed output closer to the proprietary-files.txt syntax

Change-Id: I81b844bb6bb1d1a2f91a39151a892fbfc0bed20b
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2019-03-03 16:42:24 +02:00
Vladimir Oltean
b2c382193d extract_utils: fix mismatch with non-lowercase pinned sha1sum
Change-Id: I86421ca69007ad1992aa67deedf7d2e32bbe4dba
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2019-01-17 11:26:34 +01:00
Vladimir Oltean
b8084ec8a5 extract_utils: avoid doing stuff on files that don't exist
* If an apk/jar doesn't exist, the script would still try to deodex it.
* If an xml doesn't exist, the script would still try to "fix" it.
* Take it easier, man, it's not your fault.

Change-Id: I3061fb48b403da5121e3c17dd9ecdb6cd148bf97
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-12-24 01:32:53 +02:00
Luca Stefani
99a66bffdc extract_utils: Redirect vdexextractor log to /dev/null
Change-Id: I55028997c0396d57c024e6c42a53907f6d1510c8
2018-11-13 22:06:17 +01:00
Tim Schumacher
49d26ba39a repopick: Try to fast-forward first
Change-Id: I4890ce8075556eb3f57d89c283543f620f39fa02
2018-10-27 21:57:56 +02:00
Luca Stefani
f6096e989a Add CDEX support for boot JAR files
Change-Id: I0039256c883a58c5a5125fd9bff6e6ec3b98688d
2018-10-21 10:28:37 +02:00
TheStrix
5aad07422e extract_utils: Fix output dex file name generated by vdexExtractor
- Reflect the changes required by:
   ff58255706
 - Also fix JARVDEX file path for deodexing jars

Change-Id: Ief355b0bd9e69a67b552d0bd2060f43adb39fb5f
2018-10-07 12:17:35 +02:00
codeworkx
1c29bf66dd extract_utils: add CDEX support via compact_dex_converter
Source: https://github.com/anestisb/vdexExtractor#compact-dex-converter

Change-Id: I63664cae02e51a05e5a62c0c5c0595eead6c6513
2018-10-05 18:25:46 +02:00
codeworkx
485f65f633 tools: update smali/baksmali to 2.2.5
Change-Id: I0d9b2f8acde17b20576d66d87ab9a8c51865e4dc
Source: https://bitbucket.org/JesusFreke/smali/downloads/
2018-09-24 16:22:10 +02:00
codeworkx
fcad7cfc40 tools: update vdexExtractor from 0.4.1 to 0.5.2 for P compat
Source: https://github.com/anestisb/vdexExtractor

Linux
Compiled on  : Arch Linux, 4.18.8-arch1-1-ARCH
Compiled with: Clang 6.0.1 (tags/RELEASE_601/final)
Target: x86_64-pc-linux-gnu

Darwin
Compiled on  : MacOSX 10.13
Compiled with: Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.4.0

Change-Id: I4f1ac8a9aa01887edf54a4f59e4a99f34e870606
2018-09-24 16:22:09 +02:00
Adrian DC
0f8230b528 repopick: Warn about empty commits instead of failing
* If a cherry-pick results in an empty commit
    (hints about git commit --allow-empty),
    we should warn instead of failing all repopicks

Change-Id: I8410d7d02c4118c8072de609cf3c291e2d8523e6
2018-09-05 22:41:06 +02:00
Bruno Martins
92746e08f5 tools: Bump baksmali/smali to v2.2.4
Change-Id: Ida7b7a1045080b905a9e93f15d9380df21d3799b
2018-07-26 10:27:17 +02:00
Luca Stefani
a9a9cc730c extract_utils: Newline love for adb extraction
Change-Id: Ibbf7a03ed6920d556ed7658c277f37b02d8876b8
2018-07-12 21:28:21 +02:00
Vladimir Oltean
d577325ae2 extract_utils: make get_file() able to search paths with and w/o /system prefix
Change-Id: I75eba627646caa8ceb8a30575124c57e5ee9445c
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
5f15e3e63c extract_utils: extract(): fix src not being searched in "src:dst;args" spec
* Root cause of the issue is improper naming of variables. Turns out,
  there was no variable to even denote where the blob should have been
  searched for, at "src".
* Previously there was one such variable, suggestively called "TARGET", that
  was desperately trying to serve as both, depending on who +2d hacks harder.
* One such example is "c982836 extract_utils: Fix makefile generation issues".
  That patch deliberately trimmed the "src:" from a spec (therefore
  obviously breaking the search at src) but enabling the searching at
  dst, via the good-for-all TARGET variable.
* This patch introduces the following variables:
  - SRC_FILE: absolute path corresponding to SPEC_SRC_FILE in the
    Android filesystem.
  - DST_FILE: absolute path corresponding to SPEC_DST_FILE in the
    Android filesystem. Somewhat analogous to the old TARGET variable,
    but actually contains the leading / as well (/system/bin/adsprpcd
    instead of system/bin/adsprpcd).
* Use existing common get_file() function (which previously was
  impossible to use, because it was impossible to determine calling
  arguments properly) to reduce complexity of handling adb and disk
  image as blob sources.
* Via the new SRC_FILE and DST_FILE variables, search for a blob in all
  possible locations (including paths stripped of "/system" which
  transforms an absolute path in the Android filesystem into a proper
  relative path to that file in a disk image).

Change-Id: Ic40fb4dc93541d8b3f33fde586b773199cf4ded2
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
d674771760 extract_utils: extract(): rename DEST variable to VENDOR_REPO_FILE
* This denotes the path of the file that results from the extraction
  process, relative to the "proprietary" directory.
* This is a cleanup patch.

Change-Id: I38e759bb6ed697f0a31ca35a7aa9b9b92f8b6793
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
d652a0675c extract_utils: extract(): rename ARGS variable to SPEC_ARGS
* This is a cleanup patch.

Change-Id: I2a4aac840c06c4f9b94b77b32ead33352ce395ff
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
da3b64413d extract_utils: extract(): rename FILE variable to SPEC_SRC_FILE
* This is a cleanup patch.

Change-Id: I971b2b9eba5c1f303c0d5f902feaaad25b3b26ed
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
411e0695f2 extract_utils: extract(): rename FROM variable to SPEC_DST_FILE
* This is a cleanup patch.

Change-Id: I308b0478484ccf9a594078825731606fce5f5507
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
9564328417 extract_utils: create src_file() function to mirror behavior of target_file()
* This is a cleanup-only patch.

Change-Id: I144252de765686ac3469652994a04820ecc96d4d
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
2654eaa7ef extract_utils: prefix_match(): do not strip target_args from its output
* The write_product_copy_files() and write_product_packages() functions
  rely on its undocumented behavior of keeping target_args in the
  returned list, because they are users of target_args (such as
  ";PRESIGNED" etc).
* Make the behavior documented.

Change-Id: If71595dca32abd40039706d4fed2d7f12e005365
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
6a7946baeb extract_utils: target_file() and target_args() cleanup
* Strip target_args from target_file at callee instead of at caller
* WARNING! Changes (improperly documented) behavior of prefix_match()
  function, which is expected to not strip target_args(), and the root
  cause why stripping target_args was currently done at caller. Will be
  addressed in next patch.

Change-Id: I820d2350aa64ff41374809fcb22f812257132652
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-07-08 20:14:12 +02:00
Vladimir Oltean
59074e6832 Revert "extract_files: Add support for paths without system/"
This reverts commit 8b87b09ef7.
This will be addressed cleaner in the cleanup commit to follow.

Change-Id: Iec013bfafdbb334ab2657b10c433158002809b62
2018-07-08 20:14:12 +02:00
Vladimir Oltean
a8b4f887ea Revert "extract_utils: Fix makefile generation issues"
This reverts commit c982836ca6.
This breaks extracting from src in "src:dst", such as from a disk image.
Will be addressed in the cleanup commit that follows.

Change-Id: Iff84a926f0c3bf908320b43ba40235e0a89db644
2018-07-08 20:14:12 +02:00
Gabriele M
0fcc122018 repopick: Update SSH queries result to match HTTP queries
Needed after the recent repopick changes.

Change-Id: I14f57afefeb9579d1d9896d8dc4512645b5b68e9
2018-06-11 22:52:51 +02:00
Vladimir Oltean
a48b9fe9b6 write_makefiles: fix paths in BUILD_PREBUILT with destination
* proprietary-files.txt entries such as
   "-app/TimeService/TimeService.apk:priv-app/TimeService/TimeService.apk" should
   generate a "LOCAL_SRC_FILES := proprietary/priv-app/TimeService/TimeService.apk"
   in the Makefile definition.
 * However, currently, the prefix_match function is being called on the whole
   PRODUCT_PACKAGES_LIST entry (whole line, including ":"), and therefore,
   TimeService.apk would be included in the APPS list instead of PRIV_APPS.
 * Furthermore, because of the incorrect prefix_match, the generated
   LOCAL_SRC_FILES is "proprietary/app/priv-app/TimeService/TimeService.apk",
   which breaks the build because there is no file at that wrong path.
 * The fix is to make the extract function match up with write_product_packages
   by applying the target_file function on all BUILD_PREBUILT source files.

Change-Id: Ib4859b8854db0f2142bb3f28cce8dd25f7141732
Signed-off-by: Vladimir Oltean <olteanv@gmail.com>
2018-04-09 12:55:31 -07:00
Gabriele M
af970b695c repopick: Allow to pick changes by URL
Change-Id: I460c987cc03c4f6ff03ecab1bc39e52cec413c13
2018-04-05 17:34:16 +02:00
Gabriele M
88c0e5d1af repopick: Actually print the patchset number
We are printing the change number twice...

Change-Id: I4f63766d4162f5d016badad662842badd6da3588
2018-04-05 17:34:15 +02:00
Gabriele M
de9e0ae0a5 repopick: Ignore the trailing slash in change numbers
Trailing slashes cause 'patchset' to be an empty string, which
cannot be converted to an int. If we have a trailing slash, ignore
it and pick the current patchset.

Change-Id: Ifad60b7f6ff3dcfd9dd80b2e93b1f909aa360666
2018-04-05 17:34:15 +02:00
Gabriele M
5b610aede3 repopick: Try to keep the changes sorted
Sort the changes according to their parent first and then according
to their number.

Change-Id: Iebdb8789728b2ccd528e19437e162129eb27973c
2018-04-05 17:34:14 +02:00
Joe Maples
9be579f8d7 extract_utils: Use vdexExtractor for extracting oreo classes.dex
vdexExtractor is a tool made by anestisb that is written in C++
with code taken directly from art. However, anestisb has also added
a quicken decompiler to oatdump that was merged upstream, so we only
need vdexExtractor for 8.1 and 8.0.

Change-Id: Ic2cf2dc627a1ad2fa4d500d02d9eac8b8a9577b5
Signed-off-by: Joe Maples <joe@frap129.org>
2018-03-30 20:46:07 +00:00