Truncate reverse engineering post, move some stuff for part two

[web-hugo.git] / content / posts / reverse-engineering-a-mobile-app-protobuf-api.md

diff --git a/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md b/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md
index c28e6e4e7de1aad1e3ac193feaaa42cb669a4a56..2f412e48f0fd18529711ba77e62b79f789ec0976 100644 (file)
--- a/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md
+++ b/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md
@@ -16,12 +16,17 @@ is often left behind in source format within client applications.
 In this series of blogposts, I will be using a mobile game "Egg, Inc." as the target for
 demonstration. It's a simple time killer app that got me through boring long waits when I was still at school.
 
 In this series of blogposts, I will be using a mobile game "Egg, Inc." as the target for
 demonstration. It's a simple time killer app that got me through boring long waits when I was still at school.
 

-I won't bore you with details, but in essence it's an incremental game with partial online features
-such as cloud save, co-op contracts and server scheduled boosts.
+Egg, Inc. is a basic incremental "idler" game where your goal is to take over the world food supply with ever-increasing supply of eggs,
+if you have ever played Cookie Clicker, you know the premise of something like that. You have to unlock denser and denser eggs - the game
+is also designed around the fact that you can do certain online-tied activites such as Contracts to unlock more Soul Eggs (prestige boost) and
+"Eggs of Prophecy" which increase potency of your Soul Eggs.
+
+It's rather simple game with a very minimal API, making it perfect for learning. You may not like the game, but that's beside the point.
+The simplicity of our target matters here.

 
 ## The existing works
 
 ## The existing works

-In some cases, you will find previous works on the target you pick, in my case, some clever people have created
-[scripts to extract .proto file out of app.](https://github.com/DavidArthurCole/EggIncProtoExtractor)
+In some cases, you will find previous works on the target you pick. In my case, some clever people have created
+[scripts to extract .proto file out of the app.](https://github.com/DavidArthurCole/EggIncProtoExtractor)

 I advise you to check it out if you wish to get a better understanding of how you would go about retrieving the
 API spec .proto file for your target.
 
 I advise you to check it out if you wish to get a better understanding of how you would go about retrieving the
 API spec .proto file for your target.
 


@@ -30,6 +35,13 @@ Further there are a few dedicated individuals in the game's community who have c
 For this blog purposes, we will assume the game server is shut down (as in we cannot query from the live API) and our goal is
 to make a semi-functional selfhosted gameserver for our own needs, assuming we are the only one on said server.
 
 For this blog purposes, we will assume the game server is shut down (as in we cannot query from the live API) and our goal is
 to make a semi-functional selfhosted gameserver for our own needs, assuming we are the only one on said server.
 

+## How to source builds of a game
+There are two methods of sourcing the apk file here - one method is if you already have the app installed, install something like ZArchiver
+and extract it from /data/app/ - identifying the app by its icon. From there you will find `base.apk` which is enough for most apps.
+
+Alternatively, if the app is still available on Google Play, you can use an app like Aurora Store to go to the store detail page, select
+"Manual Download" and enter a known Build ID.
+

 ## Getting Started
 Thanks to the previously mentioned script, it's easy to get started - find the APK, extract protobuf spec file, convert it with
 protoc and we're done there. One small problem - due to cheaters, latest version of the game includes "AuthenticatedMessage" structure,
 ## Getting Started
 Thanks to the previously mentioned script, it's easy to get started - find the APK, extract protobuf spec file, convert it with
 protoc and we're done there. One small problem - due to cheaters, latest version of the game includes "AuthenticatedMessage" structure,


@@ -39,7 +51,7 @@ At this point, after a bit of internal dilemma, I decided to not further the pro
 more morally sound decision of picking a version prior to these integrity checks. We can crack that another day as all the needed information
 is retained in the app itself.
 
 more morally sound decision of picking a version prior to these integrity checks. We can crack that another day as all the needed information
 is retained in the app itself.
 

-Going forward with this, we are targetting game version 1.12.13 (Build ID 111121).
+Going forward with this, we are targetting game version 1.12.13 (Build ID 111121 - use that in Aurora Store).

 
 With all that out of the way, lets get into actual commands used here:
 ```
 
 With all that out of the way, lets get into actual commands used here:
 ```


@@ -83,17 +95,16 @@ When I jumped to it, I saw an exactly adjacent string to it which could give mor
 Interesting, `www.auxbrain.com`. If we jump to its XREF, we get a garbled function, but what it seems to be doing is setting up
 certain global values.
 
 Interesting, `www.auxbrain.com`. If we jump to its XREF, we get a garbled function, but what it seems to be doing is setting up
 certain global values.
 

-So we have a potential API endpoint, let's put it to the test. We're not going to recompile anything yet or do any byte-patching,
-let's try a quick smoke-test. Ensure your phone is rooted and you have a variant of Xposed Framework installed (I used LSPosed).
-We will need to untrip the SSL pinning present in most apps, including this one, I used [io.github.tehcneko.sslunpinning](https://github.com/Xposed-Modules-Repo/io.github.tehcneko.sslunpinning) module.
-(Note: I know it is possible to repackage the app to do SSL unpinning in most cases, but in many cases, you won't know if it's worth the effort yet)
+## The smoke-test

 
 

-Next, install [AdAway app from F-Droid](https://f-droid.org/packages/org.adaway/) so we can setup a redirection on any network we are on.
+So we have a potential API endpoint, let's put it to the test. We can do a quick smoke test by setting up a webserver.
+
+Install [AdAway app from F-Droid](https://f-droid.org/packages/org.adaway/) so we can setup a redirection on any network we are on.

 Inside AdAway, add a redirection rule for the address we just found and point it to an IP address in your LAN that will run the API server.
 
 Inside AdAway, add a redirection rule for the address we just found and point it to an IP address in your LAN that will run the API server.
 

-Generate a self-signed certificate authority and a certificate signed by it and run a webserver with both HTTP and HTTPS on the API server machine.
-Import the self-signed CA to your phone's truststore. Once all of that is done, run the app for first time.
+(NOTE: AdAway doesn't detect any subdomains nor can it do wildcard, you will need to include the FQDN of the API endpoint `www.auxbrain.com`)

 
 

+Once you're done setting up the redirection, run any webserver such as nginx for a quick and dirty test.

 ```
 192.168.1.212 - - [...] "POST /ei/first_contact HTTP/1.1" 404 0 "-"
 ```
 ```
 192.168.1.212 - - [...] "POST /ei/first_contact HTTP/1.1" 404 0 "-"
 ```


@@ -380,7 +391,7 @@ GENEROUS DRONES
 ```
 This looks promising, right off the bat, first strings I'd check here are `r_icon_drone_rewards`, `b_icon_drone_boost`, `drone-boost` and `GENEROUS DRONES`.
 
 ```
 This looks promising, right off the bat, first strings I'd check here are `r_icon_drone_rewards`, `b_icon_drone_boost`, `drone-boost` and `GENEROUS DRONES`.
 

-I inspected all 4 of them, and when I got to final 2, I found the enum string translations used for event IDs - here they are extracted for game version 1.12.13
+I inspected all 4 of them, and when I got to the final 2, I found the enum string translations used for event IDs - here they are extracted for game version 1.12.13

 ```
 piggy-boost (Rate piggy fills is increased.)
 piggy-cap-boost (UNLIMITED PIGGY;Gains are retained when event ends.)
 ```
 piggy-boost (Rate piggy fills is increased.)
 piggy-cap-boost (UNLIMITED PIGGY;Gains are retained when event ends.)


@@ -428,7 +439,7 @@ At this point, we can start dog-fooding the project. Lets start with whatever ba
 
 ### Contracts
 As we progress the game and start performing prestiges, we unlock a feature called "Contracts" - but disaster strikes as we don't have any contracts we could
 
 ### Contracts
 As we progress the game and start performing prestiges, we unlock a feature called "Contracts" - but disaster strikes as we don't have any contracts we could

-accept. So far we stil see our good friends `/ei/get_periodicals` and `/ei/save_backup` hammering the server at regular intervals.
+accept. So far we still see our good friends `/ei/get_periodicals` and `/ei/save_backup` hammering the server at regular intervals.

 
 When we created the periodicals response payload, you might have noticed in the protobuf message an optional field called `ContractsResponse contracts`. Lets see
 what this ContractsResponse message contains.
 
 When we created the periodicals response payload, you might have noticed in the protobuf message an optional field called `ContractsResponse contracts`. Lets see
 what this ContractsResponse message contains.


@@ -487,7 +498,7 @@ beginners will get easier contract goals while more advanced players get harder
 We can put two-and-two together here and infer that `repeated Goal goals` is the legacy contract system - where everyone was on equal footing
 and `repeated GoalSet goal_sets` is the *new* goal system that is split into Standard and Elite.
 
 We can put two-and-two together here and infer that `repeated Goal goals` is the legacy contract system - where everyone was on equal footing
 and `repeated GoalSet goal_sets` is the *new* goal system that is split into Standard and Elite.
 

-We also learn that in future game version, they completely reworked how contracts work *yet* again into a grading "bracket" system. Fortunately,
+We also learn that in future game versions, they completely reworked how contracts work *yet* again into a grading "bracket" system. Fortunately,

 we do not have to worry about that in our current target revision.
 
 Now to get the ball rolling, there is conveniently a starting point set ahead for us already. The developer of game intended to ease new players into
 we do not have to worry about that in our current target revision.
 
 Now to get the ball rolling, there is conveniently a starting point set ahead for us already. The developer of game intended to ease new players into


@@ -541,7 +552,7 @@ This page tells us all the information we need to compose our first Contract, so
                return base64.b64encode(CurrentPeriodicals.SerializeToString())
 ```
 
                return base64.b64encode(CurrentPeriodicals.SerializeToString())
 ```
 

-Lets try that out in-game now - after waiting for a minute, we see our contract prop up, but I immediately noticed one thing amiss.
+Lets try that out in-game now - after waiting for a minute, we see our contract pop up, but I immediately noticed one thing amiss.

 The contract goals are swapped! I am getting Elite contract rewards for a Standard contract.
 
 This piece of information now tells us that the first entry in GoalSets refers to Elite rewards and the second entry in GoalSets to Standard rewards.
 The contract goals are swapped! I am getting Elite contract rewards for a Standard contract.
 
 This piece of information now tells us that the first entry in GoalSets refers to Elite rewards and the second entry in GoalSets to Standard rewards.


@@ -568,5 +579,7 @@ about the project you are currently doing - refactoring becomes an essential par
 I won't give any promises for a part 2 any time soon, but I will be trying to make this feature complete, so without further ado, here are the git repository links:
 [github.com](https://github.com/cernodile/reEgg), [git.based.quest](https://git.based.quest/?p=reEgg.git;a=tree;h=refs/heads/master;hb=refs/heads/master).
 
 I won't give any promises for a part 2 any time soon, but I will be trying to make this feature complete, so without further ado, here are the git repository links:
 [github.com](https://github.com/cernodile/reEgg), [git.based.quest](https://git.based.quest/?p=reEgg.git;a=tree;h=refs/heads/master;hb=refs/heads/master).
 

+Next time we will dive into apps that use SSL/TLS and making onboarding for your friends easier.
+

 Thank you for reading and making it all the way to the end,
 - Cernodile
 Thank you for reading and making it all the way to the end,
 - Cernodile