Fixes and extra info + rootless SSL unpinning

[web-hugo.git] / content / posts / reverse-engineering-a-mobile-app-protobuf-api.md

diff --git a/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md b/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md
index c28e6e4e7de1aad1e3ac193feaaa42cb669a4a56..5c94bd1a47a4f9225a4d291d13727ad795502393 100644 (file)
--- a/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md
+++ b/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md
@@ -16,12 +16,17 @@ is often left behind in source format within client applications.
 In this series of blogposts, I will be using a mobile game "Egg, Inc." as the target for
 demonstration. It's a simple time killer app that got me through boring long waits when I was still at school.
 
 In this series of blogposts, I will be using a mobile game "Egg, Inc." as the target for
 demonstration. It's a simple time killer app that got me through boring long waits when I was still at school.
 

-I won't bore you with details, but in essence it's an incremental game with partial online features
-such as cloud save, co-op contracts and server scheduled boosts.
+Egg, Inc. is a basic incremental "idler" game where your goal is to take over the world food supply with ever-increasing supply of eggs,
+if you have ever played Cookie Clicker, you know the premise of something like that. You have to unlock denser and denser eggs - the game
+is also designed around the fact that you can do certain online-tied activites such as Contracts to unlock more Soul Eggs (prestige boost) and
+"Eggs of Prophecy" which increase potency of your Soul Eggs.
+
+It's rather simple game with a very minimal API, making it perfect for learning. You may not like the game, but that's beside the point.
+The simplicity of our target matters here.

 
 ## The existing works
 
 ## The existing works

-In some cases, you will find previous works on the target you pick, in my case, some clever people have created
-[scripts to extract .proto file out of app.](https://github.com/DavidArthurCole/EggIncProtoExtractor)
+In some cases, you will find previous works on the target you pick. In my case, some clever people have created
+[scripts to extract .proto file out of the app.](https://github.com/DavidArthurCole/EggIncProtoExtractor)

 I advise you to check it out if you wish to get a better understanding of how you would go about retrieving the
 API spec .proto file for your target.
 
 I advise you to check it out if you wish to get a better understanding of how you would go about retrieving the
 API spec .proto file for your target.
 


@@ -30,6 +35,13 @@ Further there are a few dedicated individuals in the game's community who have c
 For this blog purposes, we will assume the game server is shut down (as in we cannot query from the live API) and our goal is
 to make a semi-functional selfhosted gameserver for our own needs, assuming we are the only one on said server.
 
 For this blog purposes, we will assume the game server is shut down (as in we cannot query from the live API) and our goal is
 to make a semi-functional selfhosted gameserver for our own needs, assuming we are the only one on said server.
 

+## How to source builds of a game
+There are two methods of sourcing the apk file here - one method is if you already have the app installed, install something like ZArchiver
+and extract it from /data/app/ - identifying the app by its icon. From there you will find `base.apk` which is enough for most apps.
+
+Alternatively, if the app is still available on Google Play, you can use an app like Aurora Store to go to the store detail page, select
+"Manual Download" and enter a known Build ID.
+

 ## Getting Started
 Thanks to the previously mentioned script, it's easy to get started - find the APK, extract protobuf spec file, convert it with
 protoc and we're done there. One small problem - due to cheaters, latest version of the game includes "AuthenticatedMessage" structure,
 ## Getting Started
 Thanks to the previously mentioned script, it's easy to get started - find the APK, extract protobuf spec file, convert it with
 protoc and we're done there. One small problem - due to cheaters, latest version of the game includes "AuthenticatedMessage" structure,


@@ -39,7 +51,7 @@ At this point, after a bit of internal dilemma, I decided to not further the pro
 more morally sound decision of picking a version prior to these integrity checks. We can crack that another day as all the needed information
 is retained in the app itself.
 
 more morally sound decision of picking a version prior to these integrity checks. We can crack that another day as all the needed information
 is retained in the app itself.
 

-Going forward with this, we are targetting game version 1.12.13 (Build ID 111121).
+Going forward with this, we are targetting game version 1.12.13 (Build ID 111121 - use that in Aurora Store).

 
 With all that out of the way, lets get into actual commands used here:
 ```
 
 With all that out of the way, lets get into actual commands used here:
 ```


@@ -85,7 +97,7 @@ certain global values.
 
 So we have a potential API endpoint, let's put it to the test. We're not going to recompile anything yet or do any byte-patching,
 let's try a quick smoke-test. Ensure your phone is rooted and you have a variant of Xposed Framework installed (I used LSPosed).
 
 So we have a potential API endpoint, let's put it to the test. We're not going to recompile anything yet or do any byte-patching,
 let's try a quick smoke-test. Ensure your phone is rooted and you have a variant of Xposed Framework installed (I used LSPosed).

-We will need to untrip the SSL pinning present in most apps, including this one, I used [io.github.tehcneko.sslunpinning](https://github.com/Xposed-Modules-Repo/io.github.tehcneko.sslunpinning) module.
+We will need to unarm the SSL pinning present in most apps, including this one, I used [io.github.tehcneko.sslunpinning](https://github.com/Xposed-Modules-Repo/io.github.tehcneko.sslunpinning) module.

 (Note: I know it is possible to repackage the app to do SSL unpinning in most cases, but in many cases, you won't know if it's worth the effort yet)
 
 Next, install [AdAway app from F-Droid](https://f-droid.org/packages/org.adaway/) so we can setup a redirection on any network we are on.
 (Note: I know it is possible to repackage the app to do SSL unpinning in most cases, but in many cases, you won't know if it's worth the effort yet)
 
 Next, install [AdAway app from F-Droid](https://f-droid.org/packages/org.adaway/) so we can setup a redirection on any network we are on.


@@ -380,7 +392,7 @@ GENEROUS DRONES
 ```
 This looks promising, right off the bat, first strings I'd check here are `r_icon_drone_rewards`, `b_icon_drone_boost`, `drone-boost` and `GENEROUS DRONES`.
 
 ```
 This looks promising, right off the bat, first strings I'd check here are `r_icon_drone_rewards`, `b_icon_drone_boost`, `drone-boost` and `GENEROUS DRONES`.
 

-I inspected all 4 of them, and when I got to final 2, I found the enum string translations used for event IDs - here they are extracted for game version 1.12.13
+I inspected all 4 of them, and when I got to the final 2, I found the enum string translations used for event IDs - here they are extracted for game version 1.12.13

 ```
 piggy-boost (Rate piggy fills is increased.)
 piggy-cap-boost (UNLIMITED PIGGY;Gains are retained when event ends.)
 ```
 piggy-boost (Rate piggy fills is increased.)
 piggy-cap-boost (UNLIMITED PIGGY;Gains are retained when event ends.)


@@ -428,7 +440,7 @@ At this point, we can start dog-fooding the project. Lets start with whatever ba
 
 ### Contracts
 As we progress the game and start performing prestiges, we unlock a feature called "Contracts" - but disaster strikes as we don't have any contracts we could
 
 ### Contracts
 As we progress the game and start performing prestiges, we unlock a feature called "Contracts" - but disaster strikes as we don't have any contracts we could

-accept. So far we stil see our good friends `/ei/get_periodicals` and `/ei/save_backup` hammering the server at regular intervals.
+accept. So far we still see our good friends `/ei/get_periodicals` and `/ei/save_backup` hammering the server at regular intervals.

 
 When we created the periodicals response payload, you might have noticed in the protobuf message an optional field called `ContractsResponse contracts`. Lets see
 what this ContractsResponse message contains.
 
 When we created the periodicals response payload, you might have noticed in the protobuf message an optional field called `ContractsResponse contracts`. Lets see
 what this ContractsResponse message contains.


@@ -487,7 +499,7 @@ beginners will get easier contract goals while more advanced players get harder
 We can put two-and-two together here and infer that `repeated Goal goals` is the legacy contract system - where everyone was on equal footing
 and `repeated GoalSet goal_sets` is the *new* goal system that is split into Standard and Elite.
 
 We can put two-and-two together here and infer that `repeated Goal goals` is the legacy contract system - where everyone was on equal footing
 and `repeated GoalSet goal_sets` is the *new* goal system that is split into Standard and Elite.
 

-We also learn that in future game version, they completely reworked how contracts work *yet* again into a grading "bracket" system. Fortunately,
+We also learn that in future game versions, they completely reworked how contracts work *yet* again into a grading "bracket" system. Fortunately,

 we do not have to worry about that in our current target revision.
 
 Now to get the ball rolling, there is conveniently a starting point set ahead for us already. The developer of game intended to ease new players into
 we do not have to worry about that in our current target revision.
 
 Now to get the ball rolling, there is conveniently a starting point set ahead for us already. The developer of game intended to ease new players into


@@ -541,7 +553,7 @@ This page tells us all the information we need to compose our first Contract, so
                return base64.b64encode(CurrentPeriodicals.SerializeToString())
 ```
 
                return base64.b64encode(CurrentPeriodicals.SerializeToString())
 ```
 

-Lets try that out in-game now - after waiting for a minute, we see our contract prop up, but I immediately noticed one thing amiss.
+Lets try that out in-game now - after waiting for a minute, we see our contract pop up, but I immediately noticed one thing amiss.

 The contract goals are swapped! I am getting Elite contract rewards for a Standard contract.
 
 This piece of information now tells us that the first entry in GoalSets refers to Elite rewards and the second entry in GoalSets to Standard rewards.
 The contract goals are swapped! I am getting Elite contract rewards for a Standard contract.
 
 This piece of information now tells us that the first entry in GoalSets refers to Elite rewards and the second entry in GoalSets to Standard rewards.


@@ -553,6 +565,68 @@ Now, the above code could be a lot neater. For your homework, if you're not skip
 a contract database and try scheduling them like the game originally did - a "Leggacy" contract every Friday and regular contracts showing up every 1-2 weeks
 for roughly 2 weeks.
 
 a contract database and try scheduling them like the game originally did - a "Leggacy" contract every Friday and regular contracts showing up every 1-2 weeks
 for roughly 2 weeks.
 

+## Rootless SSL Unpinning + Endpoint URL patching
+Let's make the app not require a VPN or root privileges - let's make user CAs work and the endpoint URL something we control on the public net.
+Start off by pulling the following repository
+```
+git clone https://github.com/ilya-kozyr/android-ssl-pinning-bypass.git
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+cp /path/to/your/apk .
+python3 apk-rebuild.py egginc.apk --pause
+```
+
+Open a new terminal window, the script will wait for us to perform modifications, enter the created folder `egginc.apk-decompiled` and `lib`.
+
+We have two folders here now, `arm64-v8a` and `armeabi-v7a`, just as we saw when we pulled the .so file out of the apk earlier. Let's tackle
+the 64-bit build first.
+
+For arm64 build it was really simple to perform bytepatch on the said endpoint. We already know it's supposed to look as `G?www.auxbrain.com` - let's probe the .so library a bit.
+```
+$> hexdump -C libegginc.so | grep "ww.auxbrain.co" -A2 -B2
+00b02b40  cd cc 4c 3f 00 00 00 00  00 00 00 00 00 00 80 3f  |..L?...........?|
+00b02b50  00 00 00 00 00 00 00 00  00 00 00 00 14 ae 47 3f  |..............G?|
+00b02b60  77 77 77 2e 61 75 78 62  72 61 69 6e 2e 63 6f 6d  |www.auxbrain.com|
+00b02b70  00 48 54 54 50 20 52 45  51 3a 20 25 64 00 64 61  |.HTTP REQ: %d.da|
+00b02b80  74 61 3d 00 65 69 2f 66  69 72 73 74 5f 63 6f 6e  |ta=.ei/first_con
+```
+
+We seem to have nothing blocking our way, let's create hex representations of `G?www.auxbrain.com` and a target domain of equal length, for example `G?eggs.based.quest`.
+
+(Note: You can choose a shorter name as well, if you null-terminate the extra bytes as padding)
+```
+$> echo "G?www.auxbrain.com" | hexdump -ve '1/1 "%.2X"'
+473F7777772E617578627261696E2E636F6D0A
+$> echo "G?eggs.based.quest" | hexdump -ve '1/1 "%.2X"'
+473F656767732E62617365642E71756573740A
+```
+
+Remove the trailing `0A` from end of both hex strings and now proceed as follows:
+```
+# Place the source in first bracket of sed and the new URL at second bracket.
+hexdump -ve '1/1 "%.2X"' libegginc.so | sed "s/473F7777772E617578627261696E2E636F6D/473F656767732E62617365642E7175657374/g" | xxd -r -p > patched.so
+```
+
+Huzzah! We now have a patched linked-library for the arm64 build. Let's also patch the 32-bit version.
+```
+$> hexdump -C libegginc.so | grep "ww.auxbrain.co" -A2 -B2
+0087b770  69 67 68 5f 74 6f 6f 5f  6d 61 6e 79 5f 70 78 00  |igh_too_many_px.|
+0087b780  74 61 62 6c 65 74 5f 68  64 70 69 00 00 00 00 00  |tablet_hdpi.....|
+0087b790  77 77 77 2e 61 75 78 62  72 61 69 6e 2e 63 6f 6d  |www.auxbrain.com|
+0087b7a0  00 00 00 00 00 00 00 00  65 69 2f 66 69 72 73 74  |........ei/first|
+0087b7b0  5f 63 6f 6e 74 61 63 74  00 00 00 00 00 00 00 00  |_contact........|
+```
+This one lacks the `G?` prefix on API endpoint, but we still have null terminators we can rely on. Let's replace the `473F` from our previous strings with `0000`.
+```
+# Place the source in first bracket of sed and the new URL at second bracket.
+hexdump -ve '1/1 "%.2X"' libegginc.so | sed "s/00007777772E617578627261696E2E636F6D/0000656767732E62617365642E7175657374/g" | xxd -r -p > patched.so
+```
+
+Replace both of the libegginc.so files with the patched.so files. Move back to main terminal window and press ENTER.
+
+We now have a patched and debug signed apk for the game that isn't SSL pinned and contains a custom API endpoint we control without a VPN.
+

 ## Conclusion so far
 We have created a (rather ugly looking) server emulator for the game. It functions, but it needs a lot of work still before we can call it ready.
 If you have followed this far, give yourself pat on the back - if you actually tried to run this code, give yourself an extra pat on the back.
 ## Conclusion so far
 We have created a (rather ugly looking) server emulator for the game. It functions, but it needs a lot of work still before we can call it ready.
 If you have followed this far, give yourself pat on the back - if you actually tried to run this code, give yourself an extra pat on the back.