From: Joann Mõndresku Date: Wed, 15 May 2024 20:26:28 +0000 (+0300) Subject: Truncate reverse engineering post, move some stuff for part two X-Git-Url: https://git.based.quest/?p=web-hugo.git;a=commitdiff_plain;h=HEAD;hp=d5f70bedf4ca5e06af580bf2c2b910c53bdfbe96 Truncate reverse engineering post, move some stuff for part two --- diff --git a/content/posts/moving-from-nginx-to-caddy.md b/content/posts/moving-from-nginx-to-caddy.md new file mode 100644 index 0000000..898c691 --- /dev/null +++ b/content/posts/moving-from-nginx-to-caddy.md @@ -0,0 +1,137 @@ +--- +title: "Moving from nginx to Caddy" +date: 2023-09-03T13:00:00+03:00 +description: "I describe my journey into using Caddy and the great config truncation from 2000+ lines by 20x" +tags: [site, news] +type: blog +draft: false +--- + +# Backstory +I've heard of [Caddy](https://caddyserver.com/) for a long while, but have always been dismissive about it - ignoring it as "yet another webserver software" when I had my +perfectly good trusty nginx to rely on for many years already. Over the years, my nginx config has become somewhat bloated and even take a while +to do restarts on due to the complexity of said config. I did attempt to resolve it by reducing amount of expensive routines - which did help to +an extent - I was complacent with my setup. Until I took a deeper dive into Caddy. + +One Saturday night (...yesterday as of writing this post), BieHDC brought to my attention that Caddy has a nginx configuration adapter. +I thought to myself why not try it out if it's really plug-and-play and try to see what all the fuss is about. What followed this was all worth it. + +## The nginx-adapter +I started off with building the [nginx adapter for Caddy](https://github.com/caddyserver/nginx-adapter) which took a while. +Very quickly after building it, I ran into issues - no matter what I did, I kept getting null routes on the generated output with no idea where it's coming from. +You would think I would have stopped here, but I foolishly tried to make it work for something that was never meant to be, you see, I forgot to do +one key thing before even starting this whole thing - RTFM (Read The Fucking Manual). The README had a list of directives it supported and I had a ton of which +were not supported, but very essential to the work of my services. + +I shelved the nginx-adpater for good at this point, after wasting an hour with it. + +## The story doesn't end +While I may have been burnt by the mischiefs of nginx-adapter, I wasn't ready to give up - by this point I had already familiarised myself with Caddy's +documentation and reference syntax. I decided to give it a try for based.quest at very least to have a fair and honest impression of Caddy as a software +so I wouldn't start to bash on it going forward from a bad experience on a beta-phase config adapter. + +In literally a few minutes, I had made it, the Caddyfile for based.quest - I present it to you in all its glory in this blogpost: +``` +http://based.quest, based.quest { + root * /var/www/based.quest/html + file_server +} + +breezewiki.based.quest { + reverse_proxy localhost:10416 +} + +rimgo.based.quest { + reverse_proxy localhost:3000 +} + +proxitok.based.quest { + reverse_proxy localhost:8999 +} + +quetre.based.quest { + reverse_proxy localhost:3333 +} +``` + +Yes, you are seeing this right - **this little configuration only needed**. I was shocked. All this time I had been writing more for no reason, because I +turned a blind eye to the alternative that was right in front of me for many years - I've seen the name, I've seen it used in several applications that +I host under Docker, but never gave it a shot until it was pointed out to me that an adapter exist to load nginx configs (which obviously didn't work out, but +that's beside the point). + +## The main server +I wasn't going to stop with only based.quest server - after seeing the potential, I took on the seemingly herculean task of porting my main server's 2000+ lines +of nginx config to a Caddyfile. I delved into documentation for over an hour, searching for answers for any immediate question I had. + +You may find what I wrote useful for your own configs, so here are (not fully mine) templates you can use for building your own Caddy virtualhosts: +``` +(errors) { + handle_errors { + @custom_err file /err-{err.status_code}.html /err.html + handle @custom_err { + rewrite * {file_match.relative} + file_server + } + respond "{err.status_code} {err.status_text}" + } +} +(php-fpm) { + encode gzip + php_fastcgi unix//run/php/php7.4-fpm.sock { + try_files {path} {path}/index.php =404 + } + file_server +} +(cache-static) { + @static { + file + path *.ico *.css *.js *.gif *.webp *.avif *.jpg *.jpeg *.png *.svg *.woff *.woff2 + } + header @static Cache-Control max-age=1209600 +} +``` +You can find most of this on the official documentation, but thought it may be useful to consolidate it here for your convenience. + +Migrating Matrix is super easy, but be attentive to details, they provide the Caddyfile samples on their own repository already - here is my final variant of it: +``` +http://cernodile.com, cernodile.com, cernodile.com:8448 { + header /.well-known/matrix/* Content-Type application/json + header /.well-known/matrix/* Access-Control-Allow-Origin * + respond /.well-known/matrix/server `{"m.server": "cernodile.com"}` + respond /.well-known/matrix/client `{"m.homeserver":{"base_url":"https://cernodile.com"}}` + reverse_proxy /_matrix/* localhost:8008 + reverse_proxy /_synapse/* localhost:8008 + # [...] +} +``` + +The reason I ask you to be attentive to details is because I broke my federation with it. Notice the /matrix/server field. I had copied the sample from Synapse +repository and left it as `"cernodile.com:443"` whereas my **original** well-known record was just `"cernodile.com"`. + +There were 2 blockers when migrating my services to Caddy. PeerTube and Gitweb. Unfortunately for PeerTube, since [PeerTube does not support anything else than nginx](https://docs.joinpeertube.org/install/any-os#webserver), +I had to keep one nginx virtualhost up and use reverse_proxy directive against it. As for Gitweb which doesn't supply any other configuration than +for Apache, I did manage to port it over to Caddy by adapting generic fastcgi adapter in a way that works for Gitweb, feel free to use it as well: +``` +git.based.quest { + root * /usr/share/gitweb + try_files {uri} index.cgi + handle /static/* { + file_server + } + reverse_proxy unix//var/run/fcgiwrap.socket { + transport fastcgi { + env GITWEB_CONFIG /etc/gitweb.conf + split .cgi + } + } +} +``` + +## Closing thoughts +After all this effort, I am down to 108 lines on my Caddyfile from 2000+ lines on nginx. The performance is as great as ever with no hickups noticed. +I want to thank BieHDC for bringing this to my attention, I wouldn't have probably gone down this rabbit hole if it weren't for the nginx adapter. +This experience was all the worth it and I suggest you give Caddy a try if you are currently running an nginx site - you won't regret having to write +20x less configuration for your webserver. + +Thank you for reading +- Cernodile diff --git a/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md b/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md new file mode 100644 index 0000000..2f412e4 --- /dev/null +++ b/content/posts/reverse-engineering-a-mobile-app-protobuf-api.md @@ -0,0 +1,585 @@ +--- +title: "Reverse Engineering a Mobile App Protobuf API" +date: 2024-05-11T12:00:00+03:00 +description: "In recent times more than ever, live service and overuse of APIs soon-to-be-stale is only increasing. This causes a lot of once-written software to become either unusable or handicapped in many regards. One way to fight this is to learn reverse engineering for sake of digital preservation. In this blog, I take you through a journey of reverse engineering a simple mobile game's protobuf API." +tags: ['tutorial', 'reverse-engineering', 'opensource'] +type: blog +draft: false +--- + +# Why +Why not? Digital preservation is important, even if you don't care for a specific program. +This is also a good way to get started with protocol reverse engineering due to way protobuf +is often left behind in source format within client applications. + +## The target +In this series of blogposts, I will be using a mobile game "Egg, Inc." as the target for +demonstration. It's a simple time killer app that got me through boring long waits when I was still at school. + +Egg, Inc. is a basic incremental "idler" game where your goal is to take over the world food supply with ever-increasing supply of eggs, +if you have ever played Cookie Clicker, you know the premise of something like that. You have to unlock denser and denser eggs - the game +is also designed around the fact that you can do certain online-tied activites such as Contracts to unlock more Soul Eggs (prestige boost) and +"Eggs of Prophecy" which increase potency of your Soul Eggs. + +It's rather simple game with a very minimal API, making it perfect for learning. You may not like the game, but that's beside the point. +The simplicity of our target matters here. + +## The existing works +In some cases, you will find previous works on the target you pick. In my case, some clever people have created +[scripts to extract .proto file out of the app.](https://github.com/DavidArthurCole/EggIncProtoExtractor) +I advise you to check it out if you wish to get a better understanding of how you would go about retrieving the +API spec .proto file for your target. + +Further there are a few dedicated individuals in the game's community who have created numerous tools and historical databases. + +For this blog purposes, we will assume the game server is shut down (as in we cannot query from the live API) and our goal is +to make a semi-functional selfhosted gameserver for our own needs, assuming we are the only one on said server. + +## How to source builds of a game +There are two methods of sourcing the apk file here - one method is if you already have the app installed, install something like ZArchiver +and extract it from /data/app/ - identifying the app by its icon. From there you will find `base.apk` which is enough for most apps. + +Alternatively, if the app is still available on Google Play, you can use an app like Aurora Store to go to the store detail page, select +"Manual Download" and enter a known Build ID. + +## Getting Started +Thanks to the previously mentioned script, it's easy to get started - find the APK, extract protobuf spec file, convert it with +protoc and we're done there. One small problem - due to cheaters, latest version of the game includes "AuthenticatedMessage" structure, +which contains a salted sha256sum of the payload message. + +At this point, after a bit of internal dilemma, I decided to not further the problem while service is still live for people playing and did the +more morally sound decision of picking a version prior to these integrity checks. We can crack that another day as all the needed information +is retained in the app itself. + +Going forward with this, we are targetting game version 1.12.13 (Build ID 111121 - use that in Aurora Store). + +With all that out of the way, lets get into actual commands used here: +``` +git clone https://github.com/DavidArthurCole/EggIncProtoExtractor.git +cd EggIncProtoExtractor +./apkextract.sh com.auxbrain.egginc_1.12.13.apk +# We should have a new folder "protos" now with resulting files +cd protos +# There should be a file called ei.proto - that's our protobuf spec file +# At this point, we can use the protoc utility which can convert the specfile +# to interfaces in C++, C#, Java, Kotlin, Objective-C, PHP, Python and Ruby with +# additional plugin support for Dart and Go. +# To make this easier to understand, we will use Python in this demonstration +protoc -I=. --python_out=. ./ei.proto +# Success! We now have a "ei_pb2.py" file which can be directly imported to Python programs +``` + +With the protobuf interface in Python created, we can now proceed with creating the API emulator - but there's a slight problem. +What URL? What endpoints? How do we find this out? Simple answer, disassembling the game. Get your RE tool of choice, I will be +using [Ghidra](https://ghidra-sre.org/) myself. + +(Note: You can also just try to find this out using tools such as WireShark) + +The game contains a linked-library written in C++, which you can find inside the .apk `lib` folder, named as `libegginc.so`. +This is perfect for our use-case, Ghidra is going to slice through this like butter. Import the file to your RE tool of choice +and let it perform some analysis on it, have a cup of tea or coffee as this is going to take a hot minute. + +Once that's done, we are going to start by looking at the defined strings - try our luck there. Search for any debug prints left +behind or maybe some clues. I started by searching for `http`, which lead me to following string `"HTTP REQ: %d"`, seems promising. +When I jumped to it, I saw an exactly adjacent string to it which could give more clues: +``` + s_www.auxbrain.com_00c02b60 XREF[0,1]: FUN_00518ab8:00518b38(R) + 00c02b5e 47 3f 77 ds "G?www.auxbrain.com" + 77 77 2e + 61 75 78 + s_HTTP_REQ:_%d_00c02b71 XREF[1]: makeRequestInternal:0067bbd4(*) + 00c02b71 48 54 54 ds "HTTP REQ: %d" + 50 20 52 + 45 51 3a +``` +Interesting, `www.auxbrain.com`. If we jump to its XREF, we get a garbled function, but what it seems to be doing is setting up +certain global values. + +## The smoke-test + +So we have a potential API endpoint, let's put it to the test. We can do a quick smoke test by setting up a webserver. + +Install [AdAway app from F-Droid](https://f-droid.org/packages/org.adaway/) so we can setup a redirection on any network we are on. +Inside AdAway, add a redirection rule for the address we just found and point it to an IP address in your LAN that will run the API server. + +(NOTE: AdAway doesn't detect any subdomains nor can it do wildcard, you will need to include the FQDN of the API endpoint `www.auxbrain.com`) + +Once you're done setting up the redirection, run any webserver such as nginx for a quick and dirty test. +``` +192.168.1.212 - - [...] "POST /ei/first_contact HTTP/1.1" 404 0 "-" +``` + +Bingo. We have contact and we have an API endpoint. Searching for "ei/" in the strings reveals a extensive list of API endpoints, we now have something +to go off from. We have everything we need to start creating the server. + +## Implementing the Server - Getting first contact +Next up, we create a new project - as we generated the protobuf definitions for Python, we will proceed accordingly. +If you are following along, get respective packages for your operating system to create python venvs. +As the protobufs are being sent over HTTP, we will be serving our application over flask which is being reverse proxied by nginx. + +``` +# Lets stage the environment +mkdir apiserver +cd apiserver +python -m venv .venv +source .venv/bin/activate +touch app.py +cp ~/EggIncProtoExtractor/protos/ei.proto . + +# Get some dependeices +pip install protobuf +pip install flask +``` + +We now have the project set up for reading protobuf definitions and a framework to listen for HTTP and routes sent to it. +Let's create an actual listener application, open app.py with your favourite IDE or text editor. + +``` +import ei_pb2 as EIProto +from flask import Flask +from flask import request + +@app.route("/ei/", methods=["POST"]) +def ei_routes(subpath): + print("HTTP POST /ei/" + subpath) + print(request.headers) + return "" +``` + +This should get the ball rolling, we will see whatever call comes in and we can see what the payload of each request contains. +At this point you should setup the reverse proxy, override your nginx / directive with: +``` +location / { + proxy_pass http://127.0.0.1:5000; +} +``` + +Reload your nginx and start the flask application you just created with `flask run`. + +Run the app again and have it phone home and see what it contains. +``` +HTTP POST /ei/first_contact +Host: 127.0.0.1:5000 +Connection: close +Content-Length: 37 +Content-Type: application/x-www-form-urlencoded +User-Agent: Dalvik/2.1.0 (Linux; U; Android 13; M2012K11AG Build/TQ3A.230901.001) +Accept-Encoding: gzip +``` + +We can see there's a form payload attached to this request, let's modify our app route a bit: +``` +@app.route("/ei/", methods=["POST"]) +def ei_routes(subpath): + print("HTTP POST /ei/" + subpath) + print(request.form) + return "" +``` + +Now if we run the modified flask application again, we see following output on the first_contact endpoint. +``` +HTTP POST /ei/first_contact +ImmutableMultiDict([('data', 'ChAzNTVlNDZlOTA4OWQxZTRjEAAYAg==')]) +``` + +We have a base64-encoded protobuf binary data - which isn't terribly useful for reading plain-text, since protobuf *is* a binary +format, so we will need to figure out what protobuf message this payload belongs to. + +Remember that ei.proto file alongside the ei_pb2.py we got earlier? Lets go back there and inspect it a bit. +We know we just contacted something called "first_contact", maybe there is something in that file that could help us? + +``` +message EggIncFirstContactRequest { + optional string user_id = 1; + optional uint32 client_version = 2; + optional Platform platform = 3; +} + +message EggIncFirstContactResponse { + optional Backup backup = 1; +} +``` + +Seems like the application is using message names in almost similar fashion to API endpoint names themselves. This will prove +to be useful knowledge. We now know what the payload should be, lets put this to the test. + +Edit your app routine again +``` +# add "import base64" to top of the file +@app.route("/ei/", methods=["POST"]) +def ei_routes(subpath): + print("HTTP POST /ei/" + subpath) + if subpath == "first_contact": + # Create the protobuf object so we can load data from the b64 payload + FirstContact = EIProto.EggIncFirstContactRequest() + FirstContact.ParseFromString(base64.b64decode(form["data"])) + print(FirstContact) + else: + print(request.form) + return "" +``` + +We should now be able to see deserialized output when we run the flask application and the mobile app, let's try it out: + +``` +HTTP POST /ei/first_contact +user_id: "355e46e9089d1e4c" +client_version: 0 +platform: DROID +``` + +Nice! We now know how to identify which protobuf object corresponds to which API endpoint. We can now make an educated guess +on what would come next. + +Seeing how we got `EggIncFirstContactRequest` and saw an adjacent `EggIncFirstContactResponse` message in the proto file, we +can safely assume that this is what the game is expecting from us in return. + +Lets modify the server a bit to account for that. + +``` +@app.route("/ei/", methods=["POST"]) +def ei_routes(subpath): + print("HTTP POST /ei/" + subpath) + if subpath == "first_contact": + # Create the protobuf object so we can load data from the b64 payload + FirstContact = EIProto.EggIncFirstContactRequest() + FirstContact.ParseFromString(base64.b64decode(form["data"])) + print("We got a first contact hello from user " + FirstContact.user_id) + # Lets respond with a FirstContactResponse + FirstContactResp = EIProto.EggIncFirstContactResponse() + # This takes only one optional argument - a Backup object - but we have no account + # system yet, so we will opt out of sending that for now. + # --- + # We send the payload back as a base64 string - the same way we retrieved it. + return base64.b64encode(FirstContactResp.SerializeToString()) + else: + print(request.form) + return "" +``` + +Now when we run the app again, we notice that we no longer get spammed this endpoint, but instead in its place we see a few new friends + +## Implementing the Server - New Friends +Say hello to `/ei/save_backup` and `/ei/get_periodicals`. We can infer from the name, that save_backup would involve a Backup message +and get_periodicals would involve a GetPeriodicalsRequest, both of which are defined fully in the proto spec file. + +Both of these are clogging up the flask application log periodically, we should check out what they are so we can have a sane log again. + +``` +@app.route("/ei/", methods=["POST"]) +def ei_routes(subpath): + print("HTTP POST /ei/" + subpath) + if subpath == "first_contact": + # Create the protobuf object so we can load data from the b64 payload + FirstContact = EIProto.EggIncFirstContactRequest() + FirstContact.ParseFromString(base64.b64decode(form["data"])) + print("We got a first contact hello from user " + FirstContact.user_id) + # Lets respond with a FirstContactResponse + FirstContactResp = EIProto.EggIncFirstContactResponse() + # This takes only one optional argument - a Backup object - but we have no account + # system yet, so we will opt out of sending that for now. + # --- + # We send the payload back as a base64 string - the same way we retrieved it. + return base64.b64encode(FirstContactResp.SerializeToString()) + elif subpath == "save_backup": + # NOTE: This took me way longer to realize than it should have, but the base64 + # payload you receive from client is broken due to some Android bug, where it + # substitutes "+" symbols with a " " whitespace. + # I don't want you to waste half hour to hours figuring out why you're getting + # corrupted data, so you're welcome. + Backup = EIProto.Backup() + Backup.ParseFromString(base64.b64decode(form["data"].replace(" ", "+")) + print(Backup) + elif subpath == "get_periodicals": + Periodicals = EIProto.GetPeriodicalsRequest() + Periodicals.ParseFromString(base64.b64decode(form["data"]) + print(Periodicals) + else: + print(request.form) + return "" +``` + +We should now see what these payloads actually contain when deserialized (for your reading experience, I advise you to rather +try this out yourself - the Backup structure is VERY large). + +Upon these payloads reaching the server, we see that a very much populated Backup message makes its way through and a relatively +thin payload of PeriodicalsRequest comes through, which isn't very useful by itself. + +Now, this is why the game developer ended up creating forms of anticheat in future versions of this app - the Backup message contains +your entire game state, which is often sent as a way to save your progress to cloud, but there is no actual sanity checking in place +to ensure you're not progressing way too fast. Personally, I am of the mind that anticheat should be done on the server-side, not +on the client-side, but I digress. We can use this to prove a very obvious vulnerability when using trust-client-always architecture. + +The game has an In App Purchase for "Pro Permit", which allows you to build more Silos, which in turn allow you to get offline +rewards for a longer period of time. If we look at protobuf definition file, you can see under Backup.game, a field called "permit_level", +which by default is zero. Lets try to change that and present a modified Backup the next time user opens the game. + + +``` +cache = {} + +@app.route("/ei/", methods=["POST"]) +def ei_routes(subpath): + print("HTTP POST /ei/" + subpath) + if subpath == "first_contact": + # Create the protobuf object so we can load data from the b64 payload + FirstContact = EIProto.EggIncFirstContactRequest() + FirstContact.ParseFromString(base64.b64decode(form["data"])) + print("We got a first contact hello from user " + FirstContact.user_id) + # Lets respond with a FirstContactResponse + FirstContactResp = EIProto.EggIncFirstContactResponse() + if FirstContact.user_id in cache: + FirstContactResp.backup.CopyFrom(cache[FirstContact.user_id]) + del cache[FirstContact.user_id] + return base64.b64encode(FirstContactResp.SerializeToString()) + elif subpath == "save_backup": + # NOTE: This took me way longer to realize than it should have, but the base64 + # payload you receive from client is broken due to some Android bug, where it + # substitutes "+" symbols with a " " whitespace. + # I don't want you to waste half hour to hours figuring out why you're getting + # corrupted data, so you're welcome. + Backup = EIProto.Backup() + Backup.ParseFromString(base64.b64decode(form["data"].replace(" ", "+")) + if Backup.game.permit_level == 0: + print("Saved a modified Backup for next game load") + # Modify the permit level, force offer the backup + Backup.game.permit_level = 1 + Backup.force_offer_backup = True + Backup.force_backup = True + cache[Backup.user_id] = Backup + elif subpath == "get_periodicals": + Periodicals = EIProto.GetPeriodicalsRequest() + Periodicals.ParseFromString(base64.b64decode(form["data"]) + print(Periodicals) + else: + print(request.form) + return "" +``` + +Lets load up the game. Nothing interesting seems to be happening yet - lets wait until we see the "Saved a modified Backup for next game load" message +show up in the server console. Once this shows up, restart the game - you are presented with a popup that you are offered to load a +Backup from server. Let's accept that. + +Now click on your silos, you have the Pro Permit for free. + +Now, it goes without saying, I do not condone piracy - the future versions of this game are very much guarded against this, rightfully so. +If you attempt this in actual game servers, this is considered fraud and IS detectable by the developer (every IAP has a receipt, logically!). + +This version of the game is defunct as the protocol has changed quite a bit in the years since this version and additional anticheat +measures have been added since. You cannot transfer this status (or even purchase what you just did) from this game version to the next. + +### Onto the PeriodicalsRequest +This one is a bit more fun to delve into blindly - the proto spec wont help you much here. We will need to use our trusty RE tools again and delve into +the game disassembly again. + +By public knowledge, we know there are server events for "Epic Research Sale", "Research Sale", "Drone Bonus" and "Prestige Boost". We can use this information to try +and look at any potential leads in strings. Drone seems like a good canditate, lets look into that. +``` +drone_fans2 +drone_crash +drone_enemy +drone_hunter +r_icon_drone_rewards +b_icon_drone_boost +drone_touch +ei_drone_lights_green +ei_drone_lights_red +ei_drone_package +ei_drone_propeller +drone-boost +GENEROUS DRONES +``` +This looks promising, right off the bat, first strings I'd check here are `r_icon_drone_rewards`, `b_icon_drone_boost`, `drone-boost` and `GENEROUS DRONES`. + +I inspected all 4 of them, and when I got to the final 2, I found the enum string translations used for event IDs - here they are extracted for game version 1.12.13 +``` +piggy-boost (Rate piggy fills is increased.) +piggy-cap-boost (UNLIMITED PIGGY;Gains are retained when event ends.) +prestige-boost (PRESTIGE BOOST;Collect more soul eggs on prestige, you must prestige to take advantage of this event.) +earnings-boost (CASH BOOST;Regular earnings are increased.) +gift-boost (GENEROUS GIFTS;Boost applies to random gifts and video gifts.) +drone-boost (GENEROUS DRONES;Drones will produce larger rewards.) +epic-research-sale (EPIC RESEARCH SALE;Only applies to Epic Research.) +vehicle-sale (VEHICLE SALE;Applies to all vehicles.) +boost-sale (BOOST SALE;Applies to the gold price of boosts.) +boost-duration (BOOST TIME+;Boosts last longer, you must start a boost during the event.) +``` +I recall there being a few more boosts, but this is useful for getting started with compositing PeriodicalsResponse with an active running event. + +### Putting together the response +We have the enum, we have the names, descriptions, lets try to create a sample server event when the client enqueries about current server periodical events. +``` + elif subpath == "get_periodicals": + # We don't actually need the information client sends us, + # we aren't verifying any stats about client in our server. + CurrentPeriodicals = EIProto.PeriodicalsResponse() + # In order to add items to a repeatable field in protobuf structure, + # we need to call .add() method on it + event = CurrentPeriodicals.events.events.add() + # Refer to ei.proto - we are filling fields for EggIncEvent structure here. + event.type = "drone-boost" + event.multiplier = 5.00 + event.subtitle = "Drones will produce larger rewards." + event.identifier = "GENEROUS DRONES" + event.seconds_remaining = 300.0 + # Lets make it respond with a 5 minute event (this will re-arm itself when client calls + # for get_periodicals again every 6 minutes) + return base64.b64encode(CurrentPeriodicals.SerializeToString()) +``` + +Launch the server and observe as the client periodically calls this endpoint again, it will now receive a 5 minute 5x Drone Rewards boost on the game. + +## Created the Server - What now? +We have now created a very basic server, which appropriately responds to a first contact, misuses the game backup feature to prove a point about weaknesses of +trusting client in server. We also created a very basic server event, which always rearms itself to never expire. + +What do we do next? + +At this point, we can start dog-fooding the project. Lets start with whatever ball game throws at us as we progress. + +### Contracts +As we progress the game and start performing prestiges, we unlock a feature called "Contracts" - but disaster strikes as we don't have any contracts we could +accept. So far we still see our good friends `/ei/get_periodicals` and `/ei/save_backup` hammering the server at regular intervals. + +When we created the periodicals response payload, you might have noticed in the protobuf message an optional field called `ContractsResponse contracts`. Lets see +what this ContractsResponse message contains. + +``` +message ContractsResponse { + repeated Contract contracts = 1; + optional string warning_message = 4; + optional double server_time = 2; + optional uint32 max_eop = 3 [default = 1000]; +} +``` + +Notice there being an array of Contract messages right off the bat - lets find its message structure next: + +``` +message Contract { + optional string identifier = 1; + optional string name = 9; + optional string description = 10; + optional Egg egg = 2; + + repeated Goal goals = 3; + message Goal { + optional GoalType type = 1; + optional double target_amount = 2; + optional RewardType reward_type = 3; + optional string reward_sub_type = 4; + optional double reward_amount = 5; + optional double target_soul_eggs = 6; + } + + repeated GoalSet goal_sets = 16; + message GoalSet { + repeated Goal goals = 1; + } + + optional bool coop_allowed = 4; + optional uint32 max_coop_size = 5; + optional uint32 max_boosts = 12; + optional double minutes_per_token = 15 [default = 60]; + optional double expiration_time = 6; + optional double length_seconds = 7; + optional double max_soul_eggs = 13; + optional uint32 min_client_version = 14; + optional bool debug = 11; +} +``` + +We will need to do a bit of reading. Fortunately, the game has a community wiki, lets look into how contracts should work. I took an +[older revision of Contracts wiki page from 2021](https://egg-inc.fandom.com/wiki/Contracts?oldid=13015) and did some slight research. + +From what I gather, at one point, there was only one set of contract rewards, shared between everyone - then they created a system where +beginners will get easier contract goals while more advanced players get harder contract goals. + +We can put two-and-two together here and infer that `repeated Goal goals` is the legacy contract system - where everyone was on equal footing +and `repeated GoalSet goal_sets` is the *new* goal system that is split into Standard and Elite. + +We also learn that in future game versions, they completely reworked how contracts work *yet* again into a grading "bracket" system. Fortunately, +we do not have to worry about that in our current target revision. + +Now to get the ball rolling, there is conveniently a starting point set ahead for us already. The developer of game intended to ease new players into +contracts by creating a simple & easy contract called [Your First Contract](https://egg-inc.fandom.com/wiki/Contracts/Your_First_Contract?oldid=13547). + +This page tells us all the information we need to compose our first Contract, so lets try to make one. + +``` + elif subpath == "get_periodicals": + # We don't actually need the information client sends us, + # we aren't verifying any stats about client in our server. + CurrentPeriodicals = EIProto.PeriodicalsResponse() + # [...] + Contract = CurrentPeriodicals.contracts.contracts.add() + Contract.identifier = "first-contract" + Contract.name = "Your First Contract" + Contract.description = "We heard you are open to contract work! Help fill this order from the local pharmacy!" + Contract.egg = EIProto.Egg.MEDICAL + Contract.coop_allowed = False + Contract.minutes_per_token = 5 + # Lets set expiry time to always be 3 days into future + Contract.expiration_time = time.time() + (3600.0 * 72.0) + Contract.length_seconds = 3600.0 * 4.0 + # The wiki mentions that you cannot get this contract after you reach 5000 Soul Eggs + Contract.max_soul_eggs = 5000.0 + # We should have the basic metadata set now, lets create the goalsets. + FirstSet = Contract.goal_sets.add() + Goal = FirstSet.goals.add() + # There is only one type of goal in this verison + Goal.type = EIProto.GoalType.EGGS_LAID + Goal.target_amount = 100000.0 + Goal.reward_type = EIProto.RewardType.GOLD + Goal.reward_amount = 192 + Goal = FirstSet.goals.add() + Goal.type = EIProto.GoalType.EGGS_LAID + Goal.target_amount = 500000000.0 + Goal.reward_type = EIProto.RewardType.PIGGY_FILL + Goal.reward_amount = 10000 + # Lets now add the Elite table, we can pretty much copy-paste the above here. + SecondSet = Contract.goal_sets.add() + Goal = SecondSet.goals.add() + Goal.type = EIProto.GoalType.EGGS_LAID + Goal.target_amount = 100000.0 + Goal.reward_type = EIProto.RewardType.GOLD + Goal.reward_amount = 500 + Goal = SecondSet.goals.add() + Goal.type = EIProto.GoalType.EGGS_LAID + Goal.target_amount = 500000000.0 + Goal.reward_type = EIProto.RewardType.PIGGY_FILL + Goal.reward_amount = 10000 + return base64.b64encode(CurrentPeriodicals.SerializeToString()) +``` + +Lets try that out in-game now - after waiting for a minute, we see our contract pop up, but I immediately noticed one thing amiss. +The contract goals are swapped! I am getting Elite contract rewards for a Standard contract. + +This piece of information now tells us that the first entry in GoalSets refers to Elite rewards and the second entry in GoalSets to Standard rewards. +After swapping the sets around, we now see a contract with the corrected rewards. + +I playtested it a bit and the contract worked as expected. + +Now, the above code could be a lot neater. For your homework, if you're not skipping to the public source release in the end, you should try to create +a contract database and try scheduling them like the game originally did - a "Leggacy" contract every Friday and regular contracts showing up every 1-2 weeks +for roughly 2 weeks. + +## Conclusion so far +We have created a (rather ugly looking) server emulator for the game. It functions, but it needs a lot of work still before we can call it ready. +If you have followed this far, give yourself pat on the back - if you actually tried to run this code, give yourself an extra pat on the back. + +Before I give you the public source to the project, you might want to try your hand at creating a few more things. +- "Cloud" save, present a Backup to any new device that just started playing. +- Contracts Database and scheduler +- Server Event scheduler + +I apologize if my method of documenting this has been messy, but that's also part of the chaos of reverse engineering, you are constantly learning new things +about the project you are currently doing - refactoring becomes an essential part once you have documented the protocol to a comfortable degree. + +I won't give any promises for a part 2 any time soon, but I will be trying to make this feature complete, so without further ado, here are the git repository links: +[github.com](https://github.com/cernodile/reEgg), [git.based.quest](https://git.based.quest/?p=reEgg.git;a=tree;h=refs/heads/master;hb=refs/heads/master). + +Next time we will dive into apps that use SSL/TLS and making onboarding for your friends easier. + +Thank you for reading and making it all the way to the end, +- Cernodile diff --git a/content/posts/reverse-engineering-mobile-apps-part-two.md b/content/posts/reverse-engineering-mobile-apps-part-two.md new file mode 100644 index 0000000..ecdf100 --- /dev/null +++ b/content/posts/reverse-engineering-mobile-apps-part-two.md @@ -0,0 +1,134 @@ +--- +title: "Reverse Engineering Mobile Apps Part Two" +date: 2024-05-15T23:21:00+03:00 +description: "" +tags: [] +type: blog +draft: true +--- + +## Note to the git snooper - I moved these here due to revalations made later, I will deep-dive these topics at a later date. + +## The smoke-test + +So we have a potential API endpoint, let's put it to the test. We're not going to recompile anything yet or do any byte-patching, +let's try a quick smoke-test. + +**UPDATE 15/05/2024: It turns out this old app version uses HTTP only, instead of HTTPS. You will only need to perform the AdAway instruction here. +As this information is still vastly useful for reverse engineering most apps, I will be leaving this section intact.** + +Ensure your phone is rooted and you have a variant of Xposed Framework installed (I used LSPosed). +We will need to unarm the SSL pinning present in most apps, including this one, I used [io.github.tehcneko.sslunpinning](https://github.com/Xposed-Modules-Repo/io.github.tehcneko.sslunpinning) module. +(NOTE: Users without root might want to skip to end of article where I showcase unpinning the app manually) + +Next, install [AdAway app from F-Droid](https://f-droid.org/packages/org.adaway/) so we can setup a redirection on any network we are on. +Inside AdAway, add a redirection rule for the address we just found and point it to an IP address in your LAN that will run the API server. + +Generate a self-signed certificate authority and a certificate signed by it and run a webserver with both HTTP and HTTPS on the API server machine. +``` +# Create an ext file containing the Subject Alternative Name (SAN) +# DNS.1 should correspond to the API endpoint of the app. +# NOTE! If you are changing the API endpoint to a public domain, you can just use a public cert, no need for any of this. +cat > auxbrain.ext << EOF +authorityKeyIdentifier=keyid,issuer +basicConstraints=CA:FALSE +keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment +subjectAltName = @alt_names + +[alt_names] +DNS.1 = www.auxbrain.com +EOF + +# Create your own Certificate Authority +openssl genrsa -des3 -out myCA.key 2048 +openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.pem +# Create a CSR and lets have the new CA sign it +openssl genrsa -out auxbrain.key 2048 +openssl req -new -key auxbrain.key -out auxbrain.csr -nodes +openssl x509 -req -in auxbrain.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out auxbrain.crt -days 825 -sha256 -extfile auxbrain.ext +cat auxbrain.crt myCA.pem > auxbrain.pem +# You now have: +# myCA.pem - the public certificate of your root CA +# auxbrain.key - the private key for your webserver +# auxbrain.pem - the public cert for your webserver. +``` + +Use the generated `auxbrain.pem` and `auxbrain.key` files for your webserver SSL/TLS configuration. For nginx, append following values to your server directive: +``` +listen 443 ssl; +ssl_certificate /path/to/auxbrain.pem; +ssl_certificate_key /path/to/auxbrain.key; +ssl_session_cache shared:SSL:1m; +ssl_session_timeout 5m; +ssl_ciphers HIGH:!aNULL:!MD5; +ssl_prefer_server_ciphers on; +``` + +Import the self-signed CA (myCA.pem) to your phone's truststore (Check under your phone's Security/Encryption settings). Once all of that is done, run the app for first time. + + +## Rootless SSL Unpinning + Endpoint URL patching +Let's make the app not require a VPN or root privileges - let's make user CAs work and the endpoint URL something we control on the public net. +Start off by pulling the following repository +``` +git clone https://github.com/ilya-kozyr/android-ssl-pinning-bypass.git +python3 -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +cp /path/to/your/apk . +python3 apk-rebuild.py egginc.apk --pause +``` + +**NOTE!** IF you do not intend to patch the API endpoint and just want to proceed with AdAway redirecting traffic, you can stop here and press ENTER! +Proceed only if you own a domain in your control (that is equal or less in length to www.auxbrain.com) and want to use the app without a VPN/redirection. + +Open a new terminal window, the script will wait for us to perform modifications, enter the created folder `egginc.apk-decompiled` and `lib`. + +We have two folders here now, `arm64-v8a` and `armeabi-v7a`, just as we saw when we pulled the .so file out of the apk earlier. Let's tackle +the 64-bit build first. + +For arm64 build it was really simple to perform bytepatch on the said endpoint. We already know it's supposed to look as `G?www.auxbrain.com` - let's probe the .so library a bit. +``` +$> hexdump -C libegginc.so | grep "ww.auxbrain.co" -A2 -B2 +00b02b40 cd cc 4c 3f 00 00 00 00 00 00 00 00 00 00 80 3f |..L?...........?| +00b02b50 00 00 00 00 00 00 00 00 00 00 00 00 14 ae 47 3f |..............G?| +00b02b60 77 77 77 2e 61 75 78 62 72 61 69 6e 2e 63 6f 6d |www.auxbrain.com| +00b02b70 00 48 54 54 50 20 52 45 51 3a 20 25 64 00 64 61 |.HTTP REQ: %d.da| +00b02b80 74 61 3d 00 65 69 2f 66 69 72 73 74 5f 63 6f 6e |ta=.ei/first_con +``` + +We seem to have nothing blocking our way, let's create hex representations of `G?www.auxbrain.com` and a target domain of equal length, for example `G?eggs.based.quest`. + +(Note: You can choose a shorter name as well, if you null-terminate the extra bytes as padding) +``` +$> echo "G?www.auxbrain.com" | hexdump -ve '1/1 "%.2X"' +473F7777772E617578627261696E2E636F6D0A +$> echo "G?eggs.based.quest" | hexdump -ve '1/1 "%.2X"' +473F656767732E62617365642E71756573740A +``` + +Remove the trailing `0A` from end of both hex strings and now proceed as follows: +``` +# Place the source in first bracket of sed and the new URL at second bracket. +hexdump -ve '1/1 "%.2X"' libegginc.so | sed "s/473F7777772E617578627261696E2E636F6D/473F656767732E62617365642E7175657374/g" | xxd -r -p > patched.so +``` + +Huzzah! We now have a patched linked-library for the arm64 build. Let's also patch the 32-bit version. +``` +$> hexdump -C libegginc.so | grep "ww.auxbrain.co" -A2 -B2 +0087b770 69 67 68 5f 74 6f 6f 5f 6d 61 6e 79 5f 70 78 00 |igh_too_many_px.| +0087b780 74 61 62 6c 65 74 5f 68 64 70 69 00 00 00 00 00 |tablet_hdpi.....| +0087b790 77 77 77 2e 61 75 78 62 72 61 69 6e 2e 63 6f 6d |www.auxbrain.com| +0087b7a0 00 00 00 00 00 00 00 00 65 69 2f 66 69 72 73 74 |........ei/first| +0087b7b0 5f 63 6f 6e 74 61 63 74 00 00 00 00 00 00 00 00 |_contact........| +``` +This one lacks the `G?` prefix on API endpoint, but we still have null terminators we can rely on. Let's replace the `473F` from our previous strings with `0000`. +``` +# Place the source in first bracket of sed and the new URL at second bracket. +hexdump -ve '1/1 "%.2X"' libegginc.so | sed "s/00007777772E617578627261696E2E636F6D/0000656767732E62617365642E7175657374/g" | xxd -r -p > patched.so +``` + +Replace both of the libegginc.so files with the patched.so files. Move back to main terminal window and press ENTER. + +We now have a patched and debug signed apk for the game that isn't SSL pinned and contains a custom API endpoint we control without a VPN. + diff --git a/layouts/partials/links.html b/layouts/partials/links.html index 7609e46..67cfd73 100644 --- a/layouts/partials/links.html +++ b/layouts/partials/links.html @@ -1,3 +1,3 @@ -

Quick-links to services I host: PeerTube, Searx, Nitter, Piped, Teddit.

+

Quick-links to services I host: PeerTube, Searx, Nitter, Proxitok, BreezeWiki, Quetre, Teddit.

Hall of Based: ReactOS, Matrix, PostmarketOS, Pine64, Landchad.net, based.cooking, borgbackup.

-

Fellow landchads: okass.net, ghativega.in.

+

Fellow landchads: okass.net, ghativega.in, dujemihanovic.xyz.