Building a secure API Token mechanism

I am building a programmatic access to my website. Or maybe that’s just a fancy way of saying that I’m designing an API. Either way, one of the core elements of the project is the authentication method. I need a way of knowing who is the sender of any given message (API request).

Since this is more of an experiment rather than a serious feature in a mature project, some corner-cutting was in order. In the initial draft, I just derived a single, static token based on the user ID. This would look something like this pseudocode (of course the secret would be generated once and stored):

secret := random_bytes 32
hash := hash_hmac secret id
token := join "_" id hash

Which then can be used in the following way:

id, hash := split "_" token
verify = hash_hmac secret id
match verify == hash:
  true → id
  false → null

The primary upside of that solution was that it worked, and it took about 5 minutes to implement. The obvious downside: it was only slightly more secure than attaching a „bro, trust me” to the payload.

The characteristics of a static API token

So, a lot of the web apps do this actually. Or at list a similar implementation, but for all intents and purposes, the results are the same. You navigate to the settings page, where you can find a single token you can use for authorization.

Let’s examine its characteristics, and I’ll try to be generous:

• ✅ It does not try to roll out a fancy pseud-crypto algorithm to derive the value, it just calculates a message digest. That in itself allows us to avoid some common pitfalls, like a padding attack for example. And although, in all fairness, I don’t believe that using a padding attack would be feasible in this scenario, using some strong foundations is a good start

I think that is the only positive of this solution. I won’t even count implementation time, since I can’t imagine a world in which a few hours spared while implementing would outweigh the risks that it brings:

• ⛔ There is no way of revoking a token. Once it’s compromised, it’s game over. And I don’t even mean that the customer does not have that option in the UI. Even I, as the system author, cannot do anything about that.
• ⛔ In a similar vein, there is no expiration attached to a token. This means that once someone saves it somewhere, it has the potential to be valid 10 years from now, assuming the site is up and we didn’t migrate away from this authorization method.

I think that with little effort I can do significantly better.

Opaque tokens

My first improvement would be replacing a single static token with a random, opaque secret stored in the database. Now, a user needs to request a new token, and the backend gladly generates one for them (and stores it in the database for later use).

When attached to an API request, the backend would lookup the token in the database, verify that it exists and is valid, and use the stored userId value.

How does this fix the issues I mentioned a couple of paragraphs prior?

• ✅ Now the customer can simply have a section in their panel to manage the list of tokens. Each of them can simply be revoked at their request, and its database record removed. If a token gets compromised, it can get revoked (individually) and that random string or characters would lose any of its meaning.
• ✅ Each token gets an expiration date. This might be 2 weeks, 3 months or a year, depending on your situation, but at least there is one, which means that by default, these tokens will not be valid indefinitely.

Great, what other upsides did we get out of the box?

• ✅ There can be multiple tokens at once. The customer can name them, give them different expiration dates, and possibly — even if that’s not the case for my app right now — attach specific permissions to each of them.
• ✅ Each token will have a date of last request attached, so the user can determine if the token is in use or not

This is already much better, and the effort to implement a simple system like that isn’t large. So far it seems like it pays off, but there is one glaring flaw.

Storing raw tokens

While storing raw passwords in a database wouldn’t even cross my mind, when I first implemented the above mechanism, the fact that the raw token secrets were just sitting there didn’t make me flinch.

I must admit, that storing raw passwords has some key differences:

• Users remember their passwords, so changing them is a chore. On the other hand, people store the API tokens, so if we were to invalidate one, changing it would be a one-time activity without a lasting impact
• Users reuse passwords, so a potential breach could impact much more than just our system
• And finally, API tokens would probably authorize you to access just a subset of areas in comparison to the account password, which would most likely grant access to every app’s feature

Either way, exposing raw API tokens in a breach would mean service disruption for your customers, as they would need to be revoked immediately. And depending on the circumstance, some of the tokens might have been already used, causing a wider breach. The optics of the situation is also not good: a leak might happen, but a good system architecture would put more safeguards in place, so the impact is minimised.

Hashing token secrets

We could apply rules similar to storing passwords: just hash the value, and when a request comes in, hash that too and compare the hashes. There are some key differences that complicate things:

• Passwords are usually attached to an account identifier (like a username or an email), which allows us to look up the correct hash to run password_verify input hash on. Tokens, on the other hand, lack that and are mostly meant to work on their own. And since secure password hashing algorithms such as Argon2 generate salt in their output, we cannot simply hash the input and look it up, since each invocation of the hash function produces different outputs.
• The app wouldn’t know the secret value. You probably know all this prompts: copy the secret now, as this is the last time the app can show it to you. This is a key principle of this security architecture, and at the same time a huge UX downside.

Let’s pause to think before we make our next step

Gathering requirements

Next steps wouldn’t be as simple as our first improvement. Basically now the paths split, and whichever one we take, there would be a compromise that we need to make. To help us with the decision, let’s decide on the requirements we have for our system.

In my case, that would be:

• Since the system is not processing any sensitive data, I can sacrifice some of the security guarantees for user convenience
• I would like to be able to present a token secret to the user at any point. If that’s not obvious, I would like to do that without regenerating its value (and invalidating the previously used one in the process)
• At the same time, I want to avoid exposing raw values in case of a database leak

This means that my next step will not be to hash the tokens. At the same time I need to point out that it applies specifically to my current use case, and shouldn’t be treated as a silver bullet for any situation.

Adding a runtime secret

Going forward, the security guarantees of our tokens will be based on the fact that there are two distinct sources of secrets in our app:

• Each token’s random secret value
• And some kind of global application secret

That application secret could be different per user or account, but the most important part is that it wouldn’t be stored in the same place as the tokens, so they would never leak if the database gets compromised. Ideally, it would only come from a secrets vault during the app’s runtime.

The easiest way would be just to use the app secret as a symetric key and store the tokens in an encrypted form.

• Gaining access to the token database wouldn’t bring the attacked any immediate gains, as the token secrets would be encrypted.
• Similarly just having the encryption key is of no use unless you get a leaked token

Having a database breach would still mean that the tokens are tainted and require rotation, but they wouldn’t be immediately usable by an attacker, and if the key is strong enough, it wouldn’t even be worth decrypting them using brute force.

On the other hand, leaking the app secret would give us time to re-encrypt using a new key, without any service disruption.

Summary

Architecting a security model of your app is always a game of multiple conflicting requirements. In the end it’s always a matter of which sacrifices are you willing to make, what risks are you accepting to have, and how your system holds as a whole, instead of just its individual parts.

While I might know a thing or two, I am not a security professional. I might be the best person to secure the API of my product, and the end product could be good enough to be accepted at my company, but it’s obvious that your requirements and risk tolerance are likely completely different. So while I hope sharing my thoughts might be useful to some, always stay frosty when listening to security advice from strangers on the internet.