Keeping the Secret Safe

Joel, my best bud, and I are working on an Android application for IsThereAnyDeal, a service that helps users track the price of PC games across a number of digital distribution stores.

We’re both big believers in open source and so right at the start we made the decision to open up development and host our source code on Github, a site which lets users share, distribute, and collaborate on code.

To connect user’s IsThereAnyDeal accounts with our application we utilize their OAuth2 API which works on the basis of a client ID and client secret, the combination of which are unique to our application. Our client ID can be shared publicly but our client secret, as the name might suggest, is meant to be known only to us.

Unfortunately, during one of our coding sessions both our client ID and client secret got stored in our source code, eventually committed to our GitHub repository, and consequently made public. Oops.

Tomáš, the author and maintainer of IsThereAnyDeal, was quick to write a feature for us to obtain a new client secret, but before he did, I took a few steps to limit the damage.

Using BFG

To manage our project we’re using a version control system (a tool for storing our code as a series of version to allow us to rollback to a known working version of code if we break) called Git. Git stores our progress as a series of changes meaning that simply saving a new version of our code without the client secret wouldn’t be enough to stop people seeing our client secret. We literally needed to rewrite history.

Luckily there’s a tool out there that does exactly that, the BFG file cleaner. Below I’ll outline the steps I took to totally scrub our client id and client secret from our repository.

First, I grabbed a copy of BFG.

Then I had to make a new commit. BFG assumes the latest commit in your repository is a good copy and will only change history before that. I changed the strings that used our client secret and ID to empty strings, committed, and pushed up a new copy to GitHub.

Then I cloned down a new copy of the repository using the mirror flag.

git clone --mirror https://github.com/arranf/IsThereAnyDeal.git

The mirror flag creates a bare copy of your repository. This blog post does a great job of explaining what a bare repository is but in short a it’s a copy of your repository that doesn’t contain a working copy/checked out copy of your files. It’s a version of your repository designed primarily for being shared, like on Github.

Next I made a .txt document containing all the strings we wanted to be removed from the repository. For us, it was a document called keys.txt containing the client ID and client secret we wanted to be removed. Something like this:

133622423420992
6b0880726a21a89dfcb07c19c7807817

Then I copied the bfg.jar file and keys.txt to the same directory that contained our bare repository and ran the following commands.

java -jar bfg.jar --replace-text keys.txt IsThereAnyDeal.git

This cleans all the copies of the strings in keys.txt with **REMOVED** but doesn’t remove the previous history. To do that you’ll need to use git gc.

cd IsThereAnyDeal.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive
git push

This removes all the surplus information, finalizing your new history, and pushes it to the remote replacing the old history.

You’ll want to clone down a new version of your history to prevent conflicts and if anyone else is sharing your repository you’ll want to let them know too.

Look out for a part two where I’ll share how we’ve started storing our client secret to keep it a secret.