Commit | Line | Data |
---|---|---|
2a5bb5a5 JM |
1 | --- |
2 | title: "Reverse Engineering a Mobile App Protobuf API" | |
3 | date: 2024-05-11T12:00:00+03:00 | |
4 | description: "In recent times more than ever, live service and overuse of APIs soon-to-be-stale is only increasing. This causes a lot of once-written software to become either unusable or handicapped in many regards. One way to fight this is to learn reverse engineering for sake of digital preservation. In this blog, I take you through a journey of reverse engineering a simple mobile game's protobuf API." | |
5 | tags: ['tutorial', 'reverse-engineering', 'opensource'] | |
6 | type: blog | |
7 | draft: false | |
8 | --- | |
9 | ||
10 | # Why | |
11 | Why not? Digital preservation is important, even if you don't care for a specific program. | |
12 | This is also a good way to get started with protocol reverse engineering due to way protobuf | |
13 | is often left behind in source format within client applications. | |
14 | ||
15 | ## The target | |
16 | In this series of blogposts, I will be using a mobile game "Egg, Inc." as the target for | |
17 | demonstration. It's a simple time killer app that got me through boring long waits when I was still at school. | |
18 | ||
1e75f355 JM |
19 | Egg, Inc. is a basic incremental "idler" game where your goal is to take over the world food supply with ever-increasing supply of eggs, |
20 | if you have ever played Cookie Clicker, you know the premise of something like that. You have to unlock denser and denser eggs - the game | |
21 | is also designed around the fact that you can do certain online-tied activites such as Contracts to unlock more Soul Eggs (prestige boost) and | |
22 | "Eggs of Prophecy" which increase potency of your Soul Eggs. | |
23 | ||
24 | It's rather simple game with a very minimal API, making it perfect for learning. You may not like the game, but that's beside the point. | |
25 | The simplicity of our target matters here. | |
2a5bb5a5 JM |
26 | |
27 | ## The existing works | |
1e75f355 JM |
28 | In some cases, you will find previous works on the target you pick. In my case, some clever people have created |
29 | [scripts to extract .proto file out of the app.](https://github.com/DavidArthurCole/EggIncProtoExtractor) | |
2a5bb5a5 JM |
30 | I advise you to check it out if you wish to get a better understanding of how you would go about retrieving the |
31 | API spec .proto file for your target. | |
32 | ||
33 | Further there are a few dedicated individuals in the game's community who have created numerous tools and historical databases. | |
34 | ||
35 | For this blog purposes, we will assume the game server is shut down (as in we cannot query from the live API) and our goal is | |
36 | to make a semi-functional selfhosted gameserver for our own needs, assuming we are the only one on said server. | |
37 | ||
1e75f355 JM |
38 | ## How to source builds of a game |
39 | There are two methods of sourcing the apk file here - one method is if you already have the app installed, install something like ZArchiver | |
40 | and extract it from /data/app/ - identifying the app by its icon. From there you will find `base.apk` which is enough for most apps. | |
41 | ||
42 | Alternatively, if the app is still available on Google Play, you can use an app like Aurora Store to go to the store detail page, select | |
43 | "Manual Download" and enter a known Build ID. | |
44 | ||
2a5bb5a5 JM |
45 | ## Getting Started |
46 | Thanks to the previously mentioned script, it's easy to get started - find the APK, extract protobuf spec file, convert it with | |
47 | protoc and we're done there. One small problem - due to cheaters, latest version of the game includes "AuthenticatedMessage" structure, | |
48 | which contains a salted sha256sum of the payload message. | |
49 | ||
50 | At this point, after a bit of internal dilemma, I decided to not further the problem while service is still live for people playing and did the | |
51 | more morally sound decision of picking a version prior to these integrity checks. We can crack that another day as all the needed information | |
52 | is retained in the app itself. | |
53 | ||
1e75f355 | 54 | Going forward with this, we are targetting game version 1.12.13 (Build ID 111121 - use that in Aurora Store). |
2a5bb5a5 JM |
55 | |
56 | With all that out of the way, lets get into actual commands used here: | |
57 | ``` | |
58 | git clone https://github.com/DavidArthurCole/EggIncProtoExtractor.git | |
59 | cd EggIncProtoExtractor | |
60 | ./apkextract.sh com.auxbrain.egginc_1.12.13.apk | |
61 | # We should have a new folder "protos" now with resulting files | |
62 | cd protos | |
63 | # There should be a file called ei.proto - that's our protobuf spec file | |
64 | # At this point, we can use the protoc utility which can convert the specfile | |
65 | # to interfaces in C++, C#, Java, Kotlin, Objective-C, PHP, Python and Ruby with | |
66 | # additional plugin support for Dart and Go. | |
67 | # To make this easier to understand, we will use Python in this demonstration | |
68 | protoc -I=. --python_out=. ./ei.proto | |
69 | # Success! We now have a "ei_pb2.py" file which can be directly imported to Python programs | |
70 | ``` | |
71 | ||
72 | With the protobuf interface in Python created, we can now proceed with creating the API emulator - but there's a slight problem. | |
73 | What URL? What endpoints? How do we find this out? Simple answer, disassembling the game. Get your RE tool of choice, I will be | |
74 | using [Ghidra](https://ghidra-sre.org/) myself. | |
75 | ||
76 | (Note: You can also just try to find this out using tools such as WireShark) | |
77 | ||
78 | The game contains a linked-library written in C++, which you can find inside the .apk `lib` folder, named as `libegginc.so`. | |
79 | This is perfect for our use-case, Ghidra is going to slice through this like butter. Import the file to your RE tool of choice | |
80 | and let it perform some analysis on it, have a cup of tea or coffee as this is going to take a hot minute. | |
81 | ||
82 | Once that's done, we are going to start by looking at the defined strings - try our luck there. Search for any debug prints left | |
83 | behind or maybe some clues. I started by searching for `http`, which lead me to following string `"HTTP REQ: %d"`, seems promising. | |
84 | When I jumped to it, I saw an exactly adjacent string to it which could give more clues: | |
85 | ``` | |
86 | s_www.auxbrain.com_00c02b60 XREF[0,1]: FUN_00518ab8:00518b38(R) | |
87 | 00c02b5e 47 3f 77 ds "G?www.auxbrain.com" | |
88 | 77 77 2e | |
89 | 61 75 78 | |
90 | s_HTTP_REQ:_%d_00c02b71 XREF[1]: makeRequestInternal:0067bbd4(*) | |
91 | 00c02b71 48 54 54 ds "HTTP REQ: %d" | |
92 | 50 20 52 | |
93 | 45 51 3a | |
94 | ``` | |
95 | Interesting, `www.auxbrain.com`. If we jump to its XREF, we get a garbled function, but what it seems to be doing is setting up | |
96 | certain global values. | |
97 | ||
be8f6a91 JM |
98 | ## The smoke-test |
99 | ||
2a5bb5a5 JM |
100 | So we have a potential API endpoint, let's put it to the test. We're not going to recompile anything yet or do any byte-patching, |
101 | let's try a quick smoke-test. Ensure your phone is rooted and you have a variant of Xposed Framework installed (I used LSPosed). | |
1e75f355 | 102 | We will need to unarm the SSL pinning present in most apps, including this one, I used [io.github.tehcneko.sslunpinning](https://github.com/Xposed-Modules-Repo/io.github.tehcneko.sslunpinning) module. |
9656149b | 103 | (NOTE: Users without root might want to skip to end of article where I showcase unpinning the app manually) |
2a5bb5a5 JM |
104 | |
105 | Next, install [AdAway app from F-Droid](https://f-droid.org/packages/org.adaway/) so we can setup a redirection on any network we are on. | |
106 | Inside AdAway, add a redirection rule for the address we just found and point it to an IP address in your LAN that will run the API server. | |
107 | ||
108 | Generate a self-signed certificate authority and a certificate signed by it and run a webserver with both HTTP and HTTPS on the API server machine. | |
be8f6a91 JM |
109 | ``` |
110 | # Create an ext file containing the Subject Alternative Name (SAN) | |
9656149b JM |
111 | # DNS.1 should correspond to the API endpoint of the app. |
112 | # NOTE! If you are changing the API endpoint to a public domain, you can just use a public cert, no need for any of this. | |
be8f6a91 JM |
113 | cat > auxbrain.ext << EOF |
114 | authorityKeyIdentifier=keyid,issuer | |
115 | basicConstraints=CA:FALSE | |
116 | keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment | |
117 | subjectAltName = @alt_names | |
118 | ||
119 | [alt_names] | |
120 | DNS.1 = www.auxbrain.com | |
121 | EOF | |
122 | ||
123 | # Create your own Certificate Authority | |
124 | openssl genrsa -des3 -out myCA.key 2048 | |
125 | openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.pem | |
126 | # Create a CSR and lets have the new CA sign it | |
3bcb1df5 | 127 | openssl genrsa -out auxbrain.key 2048 |
be8f6a91 | 128 | openssl req -new -key auxbrain.key -out auxbrain.csr -nodes |
596663d8 JM |
129 | openssl x509 -req -in auxbrain.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out auxbrain.pem -days 825 -sha256 -extfile auxbrain.ext |
130 | # You now have: | |
131 | # myCA.pem - the public certificate of your root CA | |
132 | # auxbrain.key - the private key for your webserver | |
133 | # auxbrain.pem - the public cert for your webserver. | |
be8f6a91 JM |
134 | ``` |
135 | ||
136 | Use the generated `auxbrain.pem` and `auxbrain.key` files for your webserver SSL/TLS configuration. For nginx, append following values to your server directive: | |
137 | ``` | |
138 | listen 443 ssl; | |
139 | ssl_certificate /path/to/auxbrain.pem; | |
140 | ssl_certificate_key /path/to/auxbrain.key; | |
141 | ssl_session_cache shared:SSL:1m; | |
142 | ssl_session_timeout 5m; | |
143 | ssl_ciphers HIGH:!aNULL:!MD5; | |
144 | ssl_prefer_server_ciphers on; | |
145 | ``` | |
146 | ||
147 | Import the self-signed CA (myCA.pem) to your phone's truststore (Check under your phone's Security/Encryption settings). Once all of that is done, run the app for first time. | |
2a5bb5a5 JM |
148 | |
149 | ``` | |
150 | 192.168.1.212 - - [...] "POST /ei/first_contact HTTP/1.1" 404 0 "-" | |
151 | ``` | |
152 | ||
153 | Bingo. We have contact and we have an API endpoint. Searching for "ei/" in the strings reveals a extensive list of API endpoints, we now have something | |
154 | to go off from. We have everything we need to start creating the server. | |
155 | ||
156 | ## Implementing the Server - Getting first contact | |
157 | Next up, we create a new project - as we generated the protobuf definitions for Python, we will proceed accordingly. | |
158 | If you are following along, get respective packages for your operating system to create python venvs. | |
159 | As the protobufs are being sent over HTTP, we will be serving our application over flask which is being reverse proxied by nginx. | |
160 | ||
161 | ``` | |
162 | # Lets stage the environment | |
163 | mkdir apiserver | |
164 | cd apiserver | |
165 | python -m venv .venv | |
166 | source .venv/bin/activate | |
167 | touch app.py | |
168 | cp ~/EggIncProtoExtractor/protos/ei.proto . | |
169 | ||
170 | # Get some dependeices | |
171 | pip install protobuf | |
172 | pip install flask | |
173 | ``` | |
174 | ||
175 | We now have the project set up for reading protobuf definitions and a framework to listen for HTTP and routes sent to it. | |
176 | Let's create an actual listener application, open app.py with your favourite IDE or text editor. | |
177 | ||
178 | ``` | |
179 | import ei_pb2 as EIProto | |
180 | from flask import Flask | |
181 | from flask import request | |
182 | ||
183 | @app.route("/ei/<path:subpath>", methods=["POST"]) | |
184 | def ei_routes(subpath): | |
185 | print("HTTP POST /ei/" + subpath) | |
186 | print(request.headers) | |
187 | return "" | |
188 | ``` | |
189 | ||
190 | This should get the ball rolling, we will see whatever call comes in and we can see what the payload of each request contains. | |
191 | At this point you should setup the reverse proxy, override your nginx / directive with: | |
192 | ``` | |
193 | location / { | |
194 | proxy_pass http://127.0.0.1:5000; | |
195 | } | |
196 | ``` | |
197 | ||
198 | Reload your nginx and start the flask application you just created with `flask run`. | |
199 | ||
200 | Run the app again and have it phone home and see what it contains. | |
201 | ``` | |
202 | HTTP POST /ei/first_contact | |
203 | Host: 127.0.0.1:5000 | |
204 | Connection: close | |
205 | Content-Length: 37 | |
206 | Content-Type: application/x-www-form-urlencoded | |
207 | User-Agent: Dalvik/2.1.0 (Linux; U; Android 13; M2012K11AG Build/TQ3A.230901.001) | |
208 | Accept-Encoding: gzip | |
209 | ``` | |
210 | ||
211 | We can see there's a form payload attached to this request, let's modify our app route a bit: | |
212 | ``` | |
213 | @app.route("/ei/<path:subpath>", methods=["POST"]) | |
214 | def ei_routes(subpath): | |
215 | print("HTTP POST /ei/" + subpath) | |
216 | print(request.form) | |
217 | return "" | |
218 | ``` | |
219 | ||
220 | Now if we run the modified flask application again, we see following output on the first_contact endpoint. | |
221 | ``` | |
222 | HTTP POST /ei/first_contact | |
223 | ImmutableMultiDict([('data', 'ChAzNTVlNDZlOTA4OWQxZTRjEAAYAg==')]) | |
224 | ``` | |
225 | ||
226 | We have a base64-encoded protobuf binary data - which isn't terribly useful for reading plain-text, since protobuf *is* a binary | |
227 | format, so we will need to figure out what protobuf message this payload belongs to. | |
228 | ||
229 | Remember that ei.proto file alongside the ei_pb2.py we got earlier? Lets go back there and inspect it a bit. | |
230 | We know we just contacted something called "first_contact", maybe there is something in that file that could help us? | |
231 | ||
232 | ``` | |
233 | message EggIncFirstContactRequest { | |
234 | optional string user_id = 1; | |
235 | optional uint32 client_version = 2; | |
236 | optional Platform platform = 3; | |
237 | } | |
238 | ||
239 | message EggIncFirstContactResponse { | |
240 | optional Backup backup = 1; | |
241 | } | |
242 | ``` | |
243 | ||
244 | Seems like the application is using message names in almost similar fashion to API endpoint names themselves. This will prove | |
245 | to be useful knowledge. We now know what the payload should be, lets put this to the test. | |
246 | ||
247 | Edit your app routine again | |
248 | ``` | |
249 | # add "import base64" to top of the file | |
250 | @app.route("/ei/<path:subpath>", methods=["POST"]) | |
251 | def ei_routes(subpath): | |
252 | print("HTTP POST /ei/" + subpath) | |
253 | if subpath == "first_contact": | |
254 | # Create the protobuf object so we can load data from the b64 payload | |
255 | FirstContact = EIProto.EggIncFirstContactRequest() | |
256 | FirstContact.ParseFromString(base64.b64decode(form["data"])) | |
257 | print(FirstContact) | |
258 | else: | |
259 | print(request.form) | |
260 | return "" | |
261 | ``` | |
262 | ||
263 | We should now be able to see deserialized output when we run the flask application and the mobile app, let's try it out: | |
264 | ||
265 | ``` | |
266 | HTTP POST /ei/first_contact | |
267 | user_id: "355e46e9089d1e4c" | |
268 | client_version: 0 | |
269 | platform: DROID | |
270 | ``` | |
271 | ||
272 | Nice! We now know how to identify which protobuf object corresponds to which API endpoint. We can now make an educated guess | |
273 | on what would come next. | |
274 | ||
275 | Seeing how we got `EggIncFirstContactRequest` and saw an adjacent `EggIncFirstContactResponse` message in the proto file, we | |
276 | can safely assume that this is what the game is expecting from us in return. | |
277 | ||
278 | Lets modify the server a bit to account for that. | |
279 | ||
280 | ``` | |
281 | @app.route("/ei/<path:subpath>", methods=["POST"]) | |
282 | def ei_routes(subpath): | |
283 | print("HTTP POST /ei/" + subpath) | |
284 | if subpath == "first_contact": | |
285 | # Create the protobuf object so we can load data from the b64 payload | |
286 | FirstContact = EIProto.EggIncFirstContactRequest() | |
287 | FirstContact.ParseFromString(base64.b64decode(form["data"])) | |
288 | print("We got a first contact hello from user " + FirstContact.user_id) | |
289 | # Lets respond with a FirstContactResponse | |
290 | FirstContactResp = EIProto.EggIncFirstContactResponse() | |
291 | # This takes only one optional argument - a Backup object - but we have no account | |
292 | # system yet, so we will opt out of sending that for now. | |
293 | # --- | |
294 | # We send the payload back as a base64 string - the same way we retrieved it. | |
295 | return base64.b64encode(FirstContactResp.SerializeToString()) | |
296 | else: | |
297 | print(request.form) | |
298 | return "" | |
299 | ``` | |
300 | ||
301 | Now when we run the app again, we notice that we no longer get spammed this endpoint, but instead in its place we see a few new friends | |
302 | ||
303 | ## Implementing the Server - New Friends | |
304 | Say hello to `/ei/save_backup` and `/ei/get_periodicals`. We can infer from the name, that save_backup would involve a Backup message | |
305 | and get_periodicals would involve a GetPeriodicalsRequest, both of which are defined fully in the proto spec file. | |
306 | ||
307 | Both of these are clogging up the flask application log periodically, we should check out what they are so we can have a sane log again. | |
308 | ||
309 | ``` | |
310 | @app.route("/ei/<path:subpath>", methods=["POST"]) | |
311 | def ei_routes(subpath): | |
312 | print("HTTP POST /ei/" + subpath) | |
313 | if subpath == "first_contact": | |
314 | # Create the protobuf object so we can load data from the b64 payload | |
315 | FirstContact = EIProto.EggIncFirstContactRequest() | |
316 | FirstContact.ParseFromString(base64.b64decode(form["data"])) | |
317 | print("We got a first contact hello from user " + FirstContact.user_id) | |
318 | # Lets respond with a FirstContactResponse | |
319 | FirstContactResp = EIProto.EggIncFirstContactResponse() | |
320 | # This takes only one optional argument - a Backup object - but we have no account | |
321 | # system yet, so we will opt out of sending that for now. | |
322 | # --- | |
323 | # We send the payload back as a base64 string - the same way we retrieved it. | |
324 | return base64.b64encode(FirstContactResp.SerializeToString()) | |
325 | elif subpath == "save_backup": | |
326 | # NOTE: This took me way longer to realize than it should have, but the base64 | |
327 | # payload you receive from client is broken due to some Android bug, where it | |
328 | # substitutes "+" symbols with a " " whitespace. | |
329 | # I don't want you to waste half hour to hours figuring out why you're getting | |
330 | # corrupted data, so you're welcome. | |
331 | Backup = EIProto.Backup() | |
332 | Backup.ParseFromString(base64.b64decode(form["data"].replace(" ", "+")) | |
333 | print(Backup) | |
334 | elif subpath == "get_periodicals": | |
335 | Periodicals = EIProto.GetPeriodicalsRequest() | |
336 | Periodicals.ParseFromString(base64.b64decode(form["data"]) | |
337 | print(Periodicals) | |
338 | else: | |
339 | print(request.form) | |
340 | return "" | |
341 | ``` | |
342 | ||
343 | We should now see what these payloads actually contain when deserialized (for your reading experience, I advise you to rather | |
344 | try this out yourself - the Backup structure is VERY large). | |
345 | ||
346 | Upon these payloads reaching the server, we see that a very much populated Backup message makes its way through and a relatively | |
347 | thin payload of PeriodicalsRequest comes through, which isn't very useful by itself. | |
348 | ||
349 | Now, this is why the game developer ended up creating forms of anticheat in future versions of this app - the Backup message contains | |
350 | your entire game state, which is often sent as a way to save your progress to cloud, but there is no actual sanity checking in place | |
351 | to ensure you're not progressing way too fast. Personally, I am of the mind that anticheat should be done on the server-side, not | |
352 | on the client-side, but I digress. We can use this to prove a very obvious vulnerability when using trust-client-always architecture. | |
353 | ||
354 | The game has an In App Purchase for "Pro Permit", which allows you to build more Silos, which in turn allow you to get offline | |
355 | rewards for a longer period of time. If we look at protobuf definition file, you can see under Backup.game, a field called "permit_level", | |
356 | which by default is zero. Lets try to change that and present a modified Backup the next time user opens the game. | |
357 | ||
358 | ||
359 | ``` | |
360 | cache = {} | |
361 | ||
362 | @app.route("/ei/<path:subpath>", methods=["POST"]) | |
363 | def ei_routes(subpath): | |
364 | print("HTTP POST /ei/" + subpath) | |
365 | if subpath == "first_contact": | |
366 | # Create the protobuf object so we can load data from the b64 payload | |
367 | FirstContact = EIProto.EggIncFirstContactRequest() | |
368 | FirstContact.ParseFromString(base64.b64decode(form["data"])) | |
369 | print("We got a first contact hello from user " + FirstContact.user_id) | |
370 | # Lets respond with a FirstContactResponse | |
371 | FirstContactResp = EIProto.EggIncFirstContactResponse() | |
372 | if FirstContact.user_id in cache: | |
373 | FirstContactResp.backup.CopyFrom(cache[FirstContact.user_id]) | |
374 | del cache[FirstContact.user_id] | |
375 | return base64.b64encode(FirstContactResp.SerializeToString()) | |
376 | elif subpath == "save_backup": | |
377 | # NOTE: This took me way longer to realize than it should have, but the base64 | |
378 | # payload you receive from client is broken due to some Android bug, where it | |
379 | # substitutes "+" symbols with a " " whitespace. | |
380 | # I don't want you to waste half hour to hours figuring out why you're getting | |
381 | # corrupted data, so you're welcome. | |
382 | Backup = EIProto.Backup() | |
383 | Backup.ParseFromString(base64.b64decode(form["data"].replace(" ", "+")) | |
384 | if Backup.game.permit_level == 0: | |
385 | print("Saved a modified Backup for next game load") | |
386 | # Modify the permit level, force offer the backup | |
387 | Backup.game.permit_level = 1 | |
388 | Backup.force_offer_backup = True | |
389 | Backup.force_backup = True | |
390 | cache[Backup.user_id] = Backup | |
391 | elif subpath == "get_periodicals": | |
392 | Periodicals = EIProto.GetPeriodicalsRequest() | |
393 | Periodicals.ParseFromString(base64.b64decode(form["data"]) | |
394 | print(Periodicals) | |
395 | else: | |
396 | print(request.form) | |
397 | return "" | |
398 | ``` | |
399 | ||
400 | Lets load up the game. Nothing interesting seems to be happening yet - lets wait until we see the "Saved a modified Backup for next game load" message | |
401 | show up in the server console. Once this shows up, restart the game - you are presented with a popup that you are offered to load a | |
402 | Backup from server. Let's accept that. | |
403 | ||
404 | Now click on your silos, you have the Pro Permit for free. | |
405 | ||
406 | Now, it goes without saying, I do not condone piracy - the future versions of this game are very much guarded against this, rightfully so. | |
407 | If you attempt this in actual game servers, this is considered fraud and IS detectable by the developer (every IAP has a receipt, logically!). | |
408 | ||
409 | This version of the game is defunct as the protocol has changed quite a bit in the years since this version and additional anticheat | |
410 | measures have been added since. You cannot transfer this status (or even purchase what you just did) from this game version to the next. | |
411 | ||
412 | ### Onto the PeriodicalsRequest | |
413 | This one is a bit more fun to delve into blindly - the proto spec wont help you much here. We will need to use our trusty RE tools again and delve into | |
414 | the game disassembly again. | |
415 | ||
416 | By public knowledge, we know there are server events for "Epic Research Sale", "Research Sale", "Drone Bonus" and "Prestige Boost". We can use this information to try | |
417 | and look at any potential leads in strings. Drone seems like a good canditate, lets look into that. | |
418 | ``` | |
419 | drone_fans2 | |
420 | drone_crash | |
421 | drone_enemy | |
422 | drone_hunter | |
423 | r_icon_drone_rewards | |
424 | b_icon_drone_boost | |
425 | drone_touch | |
426 | ei_drone_lights_green | |
427 | ei_drone_lights_red | |
428 | ei_drone_package | |
429 | ei_drone_propeller | |
430 | drone-boost | |
431 | GENEROUS DRONES | |
432 | ``` | |
433 | This looks promising, right off the bat, first strings I'd check here are `r_icon_drone_rewards`, `b_icon_drone_boost`, `drone-boost` and `GENEROUS DRONES`. | |
434 | ||
1e75f355 | 435 | I inspected all 4 of them, and when I got to the final 2, I found the enum string translations used for event IDs - here they are extracted for game version 1.12.13 |
2a5bb5a5 JM |
436 | ``` |
437 | piggy-boost (Rate piggy fills is increased.) | |
438 | piggy-cap-boost (UNLIMITED PIGGY;Gains are retained when event ends.) | |
439 | prestige-boost (PRESTIGE BOOST;Collect more soul eggs on prestige, you must prestige to take advantage of this event.) | |
440 | earnings-boost (CASH BOOST;Regular earnings are increased.) | |
441 | gift-boost (GENEROUS GIFTS;Boost applies to random gifts and video gifts.) | |
442 | drone-boost (GENEROUS DRONES;Drones will produce larger rewards.) | |
443 | epic-research-sale (EPIC RESEARCH SALE;Only applies to Epic Research.) | |
444 | vehicle-sale (VEHICLE SALE;Applies to all vehicles.) | |
445 | boost-sale (BOOST SALE;Applies to the gold price of boosts.) | |
446 | boost-duration (BOOST TIME+;Boosts last longer, you must start a boost during the event.) | |
447 | ``` | |
448 | I recall there being a few more boosts, but this is useful for getting started with compositing PeriodicalsResponse with an active running event. | |
449 | ||
450 | ### Putting together the response | |
451 | We have the enum, we have the names, descriptions, lets try to create a sample server event when the client enqueries about current server periodical events. | |
452 | ``` | |
453 | elif subpath == "get_periodicals": | |
454 | # We don't actually need the information client sends us, | |
455 | # we aren't verifying any stats about client in our server. | |
456 | CurrentPeriodicals = EIProto.PeriodicalsResponse() | |
457 | # In order to add items to a repeatable field in protobuf structure, | |
458 | # we need to call .add() method on it | |
459 | event = CurrentPeriodicals.events.events.add() | |
460 | # Refer to ei.proto - we are filling fields for EggIncEvent structure here. | |
461 | event.type = "drone-boost" | |
462 | event.multiplier = 5.00 | |
463 | event.subtitle = "Drones will produce larger rewards." | |
464 | event.identifier = "GENEROUS DRONES" | |
465 | event.seconds_remaining = 300.0 | |
466 | # Lets make it respond with a 5 minute event (this will re-arm itself when client calls | |
467 | # for get_periodicals again every 6 minutes) | |
468 | return base64.b64encode(CurrentPeriodicals.SerializeToString()) | |
469 | ``` | |
470 | ||
471 | Launch the server and observe as the client periodically calls this endpoint again, it will now receive a 5 minute 5x Drone Rewards boost on the game. | |
472 | ||
473 | ## Created the Server - What now? | |
474 | We have now created a very basic server, which appropriately responds to a first contact, misuses the game backup feature to prove a point about weaknesses of | |
475 | trusting client in server. We also created a very basic server event, which always rearms itself to never expire. | |
476 | ||
477 | What do we do next? | |
478 | ||
479 | At this point, we can start dog-fooding the project. Lets start with whatever ball game throws at us as we progress. | |
480 | ||
481 | ### Contracts | |
482 | As we progress the game and start performing prestiges, we unlock a feature called "Contracts" - but disaster strikes as we don't have any contracts we could | |
1e75f355 | 483 | accept. So far we still see our good friends `/ei/get_periodicals` and `/ei/save_backup` hammering the server at regular intervals. |
2a5bb5a5 JM |
484 | |
485 | When we created the periodicals response payload, you might have noticed in the protobuf message an optional field called `ContractsResponse contracts`. Lets see | |
486 | what this ContractsResponse message contains. | |
487 | ||
488 | ``` | |
489 | message ContractsResponse { | |
490 | repeated Contract contracts = 1; | |
491 | optional string warning_message = 4; | |
492 | optional double server_time = 2; | |
493 | optional uint32 max_eop = 3 [default = 1000]; | |
494 | } | |
495 | ``` | |
496 | ||
497 | Notice there being an array of Contract messages right off the bat - lets find its message structure next: | |
498 | ||
499 | ``` | |
500 | message Contract { | |
501 | optional string identifier = 1; | |
502 | optional string name = 9; | |
503 | optional string description = 10; | |
504 | optional Egg egg = 2; | |
505 | ||
506 | repeated Goal goals = 3; | |
507 | message Goal { | |
508 | optional GoalType type = 1; | |
509 | optional double target_amount = 2; | |
510 | optional RewardType reward_type = 3; | |
511 | optional string reward_sub_type = 4; | |
512 | optional double reward_amount = 5; | |
513 | optional double target_soul_eggs = 6; | |
514 | } | |
515 | ||
516 | repeated GoalSet goal_sets = 16; | |
517 | message GoalSet { | |
518 | repeated Goal goals = 1; | |
519 | } | |
520 | ||
521 | optional bool coop_allowed = 4; | |
522 | optional uint32 max_coop_size = 5; | |
523 | optional uint32 max_boosts = 12; | |
524 | optional double minutes_per_token = 15 [default = 60]; | |
525 | optional double expiration_time = 6; | |
526 | optional double length_seconds = 7; | |
527 | optional double max_soul_eggs = 13; | |
528 | optional uint32 min_client_version = 14; | |
529 | optional bool debug = 11; | |
530 | } | |
531 | ``` | |
532 | ||
533 | We will need to do a bit of reading. Fortunately, the game has a community wiki, lets look into how contracts should work. I took an | |
534 | [older revision of Contracts wiki page from 2021](https://egg-inc.fandom.com/wiki/Contracts?oldid=13015) and did some slight research. | |
535 | ||
536 | From what I gather, at one point, there was only one set of contract rewards, shared between everyone - then they created a system where | |
537 | beginners will get easier contract goals while more advanced players get harder contract goals. | |
538 | ||
539 | We can put two-and-two together here and infer that `repeated Goal goals` is the legacy contract system - where everyone was on equal footing | |
540 | and `repeated GoalSet goal_sets` is the *new* goal system that is split into Standard and Elite. | |
541 | ||
1e75f355 | 542 | We also learn that in future game versions, they completely reworked how contracts work *yet* again into a grading "bracket" system. Fortunately, |
2a5bb5a5 JM |
543 | we do not have to worry about that in our current target revision. |
544 | ||
545 | Now to get the ball rolling, there is conveniently a starting point set ahead for us already. The developer of game intended to ease new players into | |
546 | contracts by creating a simple & easy contract called [Your First Contract](https://egg-inc.fandom.com/wiki/Contracts/Your_First_Contract?oldid=13547). | |
547 | ||
548 | This page tells us all the information we need to compose our first Contract, so lets try to make one. | |
549 | ||
550 | ``` | |
551 | elif subpath == "get_periodicals": | |
552 | # We don't actually need the information client sends us, | |
553 | # we aren't verifying any stats about client in our server. | |
554 | CurrentPeriodicals = EIProto.PeriodicalsResponse() | |
555 | # [...] | |
556 | Contract = CurrentPeriodicals.contracts.contracts.add() | |
557 | Contract.identifier = "first-contract" | |
558 | Contract.name = "Your First Contract" | |
559 | Contract.description = "We heard you are open to contract work! Help fill this order from the local pharmacy!" | |
560 | Contract.egg = EIProto.Egg.MEDICAL | |
561 | Contract.coop_allowed = False | |
562 | Contract.minutes_per_token = 5 | |
563 | # Lets set expiry time to always be 3 days into future | |
564 | Contract.expiration_time = time.time() + (3600.0 * 72.0) | |
565 | Contract.length_seconds = 3600.0 * 4.0 | |
566 | # The wiki mentions that you cannot get this contract after you reach 5000 Soul Eggs | |
567 | Contract.max_soul_eggs = 5000.0 | |
568 | # We should have the basic metadata set now, lets create the goalsets. | |
569 | FirstSet = Contract.goal_sets.add() | |
570 | Goal = FirstSet.goals.add() | |
571 | # There is only one type of goal in this verison | |
572 | Goal.type = EIProto.GoalType.EGGS_LAID | |
573 | Goal.target_amount = 100000.0 | |
574 | Goal.reward_type = EIProto.RewardType.GOLD | |
575 | Goal.reward_amount = 192 | |
576 | Goal = FirstSet.goals.add() | |
577 | Goal.type = EIProto.GoalType.EGGS_LAID | |
578 | Goal.target_amount = 500000000.0 | |
579 | Goal.reward_type = EIProto.RewardType.PIGGY_FILL | |
580 | Goal.reward_amount = 10000 | |
581 | # Lets now add the Elite table, we can pretty much copy-paste the above here. | |
582 | SecondSet = Contract.goal_sets.add() | |
583 | Goal = SecondSet.goals.add() | |
584 | Goal.type = EIProto.GoalType.EGGS_LAID | |
585 | Goal.target_amount = 100000.0 | |
586 | Goal.reward_type = EIProto.RewardType.GOLD | |
587 | Goal.reward_amount = 500 | |
588 | Goal = SecondSet.goals.add() | |
589 | Goal.type = EIProto.GoalType.EGGS_LAID | |
590 | Goal.target_amount = 500000000.0 | |
591 | Goal.reward_type = EIProto.RewardType.PIGGY_FILL | |
592 | Goal.reward_amount = 10000 | |
593 | return base64.b64encode(CurrentPeriodicals.SerializeToString()) | |
594 | ``` | |
595 | ||
1e75f355 | 596 | Lets try that out in-game now - after waiting for a minute, we see our contract pop up, but I immediately noticed one thing amiss. |
2a5bb5a5 JM |
597 | The contract goals are swapped! I am getting Elite contract rewards for a Standard contract. |
598 | ||
599 | This piece of information now tells us that the first entry in GoalSets refers to Elite rewards and the second entry in GoalSets to Standard rewards. | |
600 | After swapping the sets around, we now see a contract with the corrected rewards. | |
601 | ||
602 | I playtested it a bit and the contract worked as expected. | |
603 | ||
604 | Now, the above code could be a lot neater. For your homework, if you're not skipping to the public source release in the end, you should try to create | |
605 | a contract database and try scheduling them like the game originally did - a "Leggacy" contract every Friday and regular contracts showing up every 1-2 weeks | |
606 | for roughly 2 weeks. | |
607 | ||
1e75f355 JM |
608 | ## Rootless SSL Unpinning + Endpoint URL patching |
609 | Let's make the app not require a VPN or root privileges - let's make user CAs work and the endpoint URL something we control on the public net. | |
610 | Start off by pulling the following repository | |
611 | ``` | |
612 | git clone https://github.com/ilya-kozyr/android-ssl-pinning-bypass.git | |
613 | python3 -m venv .venv | |
614 | source .venv/bin/activate | |
615 | pip install -r requirements.txt | |
616 | cp /path/to/your/apk . | |
617 | python3 apk-rebuild.py egginc.apk --pause | |
618 | ``` | |
619 | ||
9656149b JM |
620 | **NOTE!** IF you do not intend to patch the API endpoint and just want to proceed with AdAway redirecting traffic, you can stop here and press ENTER! |
621 | Proceed only if you own a domain in your control (that is equal or less in length to www.auxbrain.com) and want to use the app without a VPN/redirection. | |
622 | ||
1e75f355 JM |
623 | Open a new terminal window, the script will wait for us to perform modifications, enter the created folder `egginc.apk-decompiled` and `lib`. |
624 | ||
625 | We have two folders here now, `arm64-v8a` and `armeabi-v7a`, just as we saw when we pulled the .so file out of the apk earlier. Let's tackle | |
626 | the 64-bit build first. | |
627 | ||
628 | For arm64 build it was really simple to perform bytepatch on the said endpoint. We already know it's supposed to look as `G?www.auxbrain.com` - let's probe the .so library a bit. | |
629 | ``` | |
630 | $> hexdump -C libegginc.so | grep "ww.auxbrain.co" -A2 -B2 | |
631 | 00b02b40 cd cc 4c 3f 00 00 00 00 00 00 00 00 00 00 80 3f |..L?...........?| | |
632 | 00b02b50 00 00 00 00 00 00 00 00 00 00 00 00 14 ae 47 3f |..............G?| | |
633 | 00b02b60 77 77 77 2e 61 75 78 62 72 61 69 6e 2e 63 6f 6d |www.auxbrain.com| | |
634 | 00b02b70 00 48 54 54 50 20 52 45 51 3a 20 25 64 00 64 61 |.HTTP REQ: %d.da| | |
635 | 00b02b80 74 61 3d 00 65 69 2f 66 69 72 73 74 5f 63 6f 6e |ta=.ei/first_con | |
636 | ``` | |
637 | ||
638 | We seem to have nothing blocking our way, let's create hex representations of `G?www.auxbrain.com` and a target domain of equal length, for example `G?eggs.based.quest`. | |
639 | ||
640 | (Note: You can choose a shorter name as well, if you null-terminate the extra bytes as padding) | |
641 | ``` | |
642 | $> echo "G?www.auxbrain.com" | hexdump -ve '1/1 "%.2X"' | |
643 | 473F7777772E617578627261696E2E636F6D0A | |
644 | $> echo "G?eggs.based.quest" | hexdump -ve '1/1 "%.2X"' | |
645 | 473F656767732E62617365642E71756573740A | |
646 | ``` | |
647 | ||
648 | Remove the trailing `0A` from end of both hex strings and now proceed as follows: | |
649 | ``` | |
650 | # Place the source in first bracket of sed and the new URL at second bracket. | |
651 | hexdump -ve '1/1 "%.2X"' libegginc.so | sed "s/473F7777772E617578627261696E2E636F6D/473F656767732E62617365642E7175657374/g" | xxd -r -p > patched.so | |
652 | ``` | |
653 | ||
654 | Huzzah! We now have a patched linked-library for the arm64 build. Let's also patch the 32-bit version. | |
655 | ``` | |
656 | $> hexdump -C libegginc.so | grep "ww.auxbrain.co" -A2 -B2 | |
657 | 0087b770 69 67 68 5f 74 6f 6f 5f 6d 61 6e 79 5f 70 78 00 |igh_too_many_px.| | |
658 | 0087b780 74 61 62 6c 65 74 5f 68 64 70 69 00 00 00 00 00 |tablet_hdpi.....| | |
659 | 0087b790 77 77 77 2e 61 75 78 62 72 61 69 6e 2e 63 6f 6d |www.auxbrain.com| | |
660 | 0087b7a0 00 00 00 00 00 00 00 00 65 69 2f 66 69 72 73 74 |........ei/first| | |
661 | 0087b7b0 5f 63 6f 6e 74 61 63 74 00 00 00 00 00 00 00 00 |_contact........| | |
662 | ``` | |
663 | This one lacks the `G?` prefix on API endpoint, but we still have null terminators we can rely on. Let's replace the `473F` from our previous strings with `0000`. | |
664 | ``` | |
665 | # Place the source in first bracket of sed and the new URL at second bracket. | |
666 | hexdump -ve '1/1 "%.2X"' libegginc.so | sed "s/00007777772E617578627261696E2E636F6D/0000656767732E62617365642E7175657374/g" | xxd -r -p > patched.so | |
667 | ``` | |
668 | ||
669 | Replace both of the libegginc.so files with the patched.so files. Move back to main terminal window and press ENTER. | |
670 | ||
671 | We now have a patched and debug signed apk for the game that isn't SSL pinned and contains a custom API endpoint we control without a VPN. | |
672 | ||
2a5bb5a5 JM |
673 | ## Conclusion so far |
674 | We have created a (rather ugly looking) server emulator for the game. It functions, but it needs a lot of work still before we can call it ready. | |
675 | If you have followed this far, give yourself pat on the back - if you actually tried to run this code, give yourself an extra pat on the back. | |
676 | ||
677 | Before I give you the public source to the project, you might want to try your hand at creating a few more things. | |
678 | - "Cloud" save, present a Backup to any new device that just started playing. | |
679 | - Contracts Database and scheduler | |
680 | - Server Event scheduler | |
681 | ||
682 | I apologize if my method of documenting this has been messy, but that's also part of the chaos of reverse engineering, you are constantly learning new things | |
683 | about the project you are currently doing - refactoring becomes an essential part once you have documented the protocol to a comfortable degree. | |
684 | ||
685 | I won't give any promises for a part 2 any time soon, but I will be trying to make this feature complete, so without further ado, here are the git repository links: | |
686 | [github.com](https://github.com/cernodile/reEgg), [git.based.quest](https://git.based.quest/?p=reEgg.git;a=tree;h=refs/heads/master;hb=refs/heads/master). | |
687 | ||
688 | Thank you for reading and making it all the way to the end, | |
689 | - Cernodile |