Understanding the guts of Twitter's OAuth for client apps

January 18, 2010

In November 2009 at Web 2.0 Expo in New York, someone from Twitter said that they will be phasing out Basic Authentication for their API in favor of OAuth some time in 2010. (This may or may not be 100% correct; I remember reading/seeing this but could not find the reference.) But regardless of what exactly they said, it is clear that they are moving towards OAuth, as are many other providers.

So, I decided the time has come for me to learn more about OAuth, and specifically how to work with Twitter API using it, instead of HTTP Basic Authentication. I found some shortcomings in existing docs and thought to write a post.

This post is for you if you:

know how HTTP works on a general level: e.g you know what’s a request header.
kinda sorta know what OAuth is and does even if you haven’t developed for it before; if not, read e.g Wikipedia or Beginner’s Guide to OAuth.
want to understand how the protocol works for client apps on the barebones/wire level, instead of just using a client library.

Starting with existing docs

Let’s see what resources there are today to learn about OAuth:

Some of these were somewhat helpful, but there wasn’t one single guide that satisfied me. The RFC is probably the most accurate as reference material, but it is very dense and hard to parse/follow if you are starting out. Terminology has also changed over time, and the RFC points out the mapping between “old” and “new” terms.

What I find most helpful when working with a new HTTP-based protocol is to examine what “goes on the wire” and what comes back. There are some examples in the above, but they are not one coherent story. The examples are also generic, and not specific to Twitter, which is what I was interested in. So here we go… an introductory OAuth session in three (+1) steps.

0. Register your app with Twitter

Go to Twitter OAuth clients and register your app. Select “Client” as the app type. You will get two very important pieces of information: “consumer key” and “consumer secret.” You will use them below. RFC calls these “client credentials.” (Confused about terminology yet?)

1. Request temporary credentials from Twitter

You will now issue a HTTP GET request to the following URL:

https://twitter.com/oauth/request_token

(Twitter actually uses HTTP throughout the examples on their site, but HTTPS is better for security and works fine.)

It is a simple GET request, but as part of it, you include the following HTTP header:

Authorization: OAuth realm="", oauth_nonce="92673243246",
oauth_timestamp="12642432725", oauth_consumer_key="9874239869",
oauth_signature_method="HMAC-SHA1", oauth_version="1.0",
oauth_signature="l%2FXBqib2y423432LCYwby3kCk%3D"

Whoa. What?

Yes, that’s basically how OAuth works. It is otherwise like simple HTTP, but you just include this special header in all your requests. (It should all be one line, it has linebreaks above just for readability.) So. Let’s talk about this header. (There are other ways to do this besides the header, but RFC section 3.5 says header is preferred anyway, so we will just use the header.)

You see it has a bunch of fields. Here’s what they should be:

realm: “” is fine for Twitter

oauth_nonce: a unique value. I myself use sha1(timestamp

str(random)), i.e current timestamp concatenated with a random string. Anything should be fine as long as it is unique string. Keep it alphanumeric or hex for simplicity.

oauth_timestamp: current UNIX timestamp, seconds since epoch (Jan 1 1970).
oauth_consumer_key: what you got from Twitter.
oauth_signature_method: just use HMAC-SHA1. There are other methods, but I won’t talk about them.
oauth_version: just use “1.0.”
oauth_signature: this is a signature calculated across a set of fields in a very specific way, so let’s talk about that.

OAuth signature and the signature base string

OAuth headers need to include a signature. It is calculated thus:

signature = base64(hmac-sha1(signature_base_string, signature_key))

So, there is some hmac-sha1 function that takes two things as input: a base string and a key. It signs the base string with the key and returns a signature. The output value must be base64-encoded. Treat these functions as black boxes, just find a hmac-sha1 and base64 implementation and use those, they are standard.

Let’s talk about the key first. The key is defined thus (“

” means concatenation):

signature_key = consumer_secret || "&" || token_secret

“But I don’t have a token_secret by this point?” Correct, you don’t. So if the consumer secret was “abcd”, your signature key in this stage is “abcd&”.

Now, let’s talk about the base string. Here is an example base string.

GET&https%3A%2F%2Ftwitter.com%2Foauth%2Frequest_token&oauth_consumer_key
%3D1rB94WF54ayRvryLQIZ01A%26oauth_nonce%3D92676946%26
oauth_signature_method%3DHMAC-SHA1%26
oauth_timestamp%3D1263777725%26oauth_version%3D1.0

Again, it should all be one long string without linebreaks.

The algorithm to calculate the signature base string is:

signature_base_string = HTTP_METHOD || "&" || urlencode(normalized_url)
    || "&" || urlencode(parameters_string)
parameters_string = join(sorted_urlencoded_parameter_keypairs, '&')

In English:

Take the request parameter keypairs. These are your GET or POST key/value pairs. We don’t have any in this first example. In addition, you have a set of core fields that are included, which you can see above.
Sort the keypairs alphabetically by key, merge them into a big string “key1=value1&key2=value2”, where the keys and values are URL-encoded. (RFC specifies a specific method of URL-encoding; see section 3.6.)
Construct the base string by merging the HTTP method, normalized URL, and URL-encoded whole parameter string into the base string. This means that for some parameter keys/values, you will see double URL-encoding: first the individual value was encoded, and then the result got encoded again due to the whole parameter string being encoded.

OK… so, let’s go back to where we were. We have managed to construct a signature base string, and have signed it. We have put together the OAuth Authorization header. We fired off the GET request. At this point, one of two things will happen.

First, you may get a “401 Not Authorized” response from Twitter. It is helpful to examine the response body for more detail. It will say something like “Invalid signature” or “Invalid / expired nonce.” Use this knowledge to fix what you were doing. I had lots of trouble to get the base string and signature calculation right. Don’t forget that the signature key always contains “&”. And Twitter really does make sure that nonces are not reused: if you send the same correctly constructed request twice with the same nonce, then the first one will be successful, but the second one will fail.

Let’s say you did everything correctly. In this case, you will get a 200 OK, and the response body will be:

oauth_token=lLUrf57ghMfIt24173AgJiSUHjTbT
EdNRqOqQFz5Pk&oauth_token_secret=5K60EOjryJVtiC
NkwoEteTQkGhMkrwFsEMs&oauth_callback_confirmed=true

Aha! Now we are getting somewhere. Capture and save the oauth_token and oauth_token_secret values; you’ll need them soon. Disregard oauth_callback_confirmed; since you are a client app, there are no callbacks.

1.5 Redirecting user to obtain the PIN

Next, you need to redirect the user to this URL in their browsers:

https://twitter.com/oauth/authorize?oauth_token=Bo2bmtiydaKScpMqzIsEWuZuG2x8

Where the token value is, of course, the one that you got from Twitter just a second ago.

The users will then go through these screens:

Picture 3.png

Picture 4 twtr.png

They will come back to your app with the PIN, and will expect to input or copypaste it somewhere. Let’s see what happens after they do that.

2. Obtaining an access token from Twitter

You now have your consumer key and secret, token and token secret (which are temporary by this point), and the PIN that the user input (in the spec terms this is “verifier”). You now make the following HTTP GET request:

https://twitter.com/oauth/access_token

The header should be something like:

Authorization: OAuth realm="", oauth_nonce="926534246",
oauth_timestamp="1890725", oauth_consumer_key="9874239869",
oauth_signature_method="HMAC-SHA1", oauth_version="1.0",
oauth_signature="l%2FXBqib2y422LCYwby3kCk%3D",
oauth_token="31242343232", oauth_verifier="8689612"

The signature base string is something like:

GET&https%3A%2F%2Ftwitter.com%2Foauth%2Faccess_token&
oauth_consumer_key%3D1rB94WF54ayRvryLQIZ01A%26oauth_nonce%3D
86345093%26oauth_signature_method%3DHMAC-SHA1%26
oauth_timestamp%3D1263780501%26oauth_token%3D
Bo2d2oZewAyOHstF5bmtiydaKScpMqzIsEWuZuG2x8%26
oauth_verifier%3D2383255%26oauth_version%3D1.0

This is actually very similar to what we did above. Only three changes:

The URL is https://twitter.com/oauth/access_token.
The header and base string contain two new parameters, oauth_verifier (user-entered PIN) and oauth_token (what you got from Twitter as response to previous request).

Remember how signature key is calculated: signature_key = consumer_secret

“&”

token_secret. You had both of these by this point, so the key was something like “abc123&456cde.”

(The Twitter API wiki says that the method for this call should be POST. I am not sure why, but as empirically shown, GET works just fine.)

The outcome is one of two things. Either “401 Not Authorized,” in which case it is again time to debug. But if all is well, Twitter responds with 200 OK and the following body:

oauth_token=103393708-XKheKolp4P775W8Ejr1DQ
Z82AciwFOicaRg6hw1Y&oauth_token_secret=vRaADq6
v7vUZ90fisaIF2jqJZW29pKVS8vLKMUb
f3k&user_id=103393708&screen_name=exampleuser

Cool! If you get by this point, you have done the bulk of the work and are pretty much done. As you see, we got four things back from Twitter:

user_id and screen_name. These are the identifiers for the user who signed in. Note that you haven’t actually asked the user for anything in your app; you just get the userid and screenname like this back from Twitter. You can now save and use them.
oauth_token and oauth_token_secret. Here is the confusing part: these parameters are called the same as they were before, but they mean an entirely different thing. These are now AUTHORIZED, which means that you will now use these to make further requests. Throw away the previous oauth_token and secret, and keep these new ones. In RFC terms, these are “access token and secret.”

3. Making further requests

You would do the above steps only once per user in your app. They would authenticate once, and then forget about it. Now you will continue to make Twitter API requests. Let’s look at a simple example of updating status. Say you POST to this URL:

https://twitter.com/statuses/update.json

And your post body (parameters encoded in a standard way) is:

status=hello+world

Your header would be:

Authorization: OAuth realm="", oauth_nonce="926534246",
oauth_timestamp="1890345", oauth_consumer_key="9874239869",
oauth_signature_method="HMAC-SHA1", oauth_version="1.0",
oauth_signature="l%2FXBqib242y422LCYwby3kCk%3D",
oauth_token="31242343232", status="hello%20world"

And the signature base string would be:

POST&https%3A%2F%2Ftwitter.com%2F1%2Fstatuses
%2Fupdate.json&oauth_consumer_key%3D1rB7654ayRvr
yLQIZ01A%26oauth_nonce%3D45586507%26oauth_
signature_method%3DHMAC-SHA1%26oauth_
timestamp%3D1263781497%26oauth_token%3D1033
93708-XKheKolp4P775W8Ejr4234246hw1Y%26o
auth_version%3D1.0%26status%3Dhello%2520world

Note the new “status” field, and how it may be differently encoded in the POST body and in the signature base string. It is in the end since the keys were alphabetically sorted.

Phew… that’s it.

A note on debugging

The above may be difficult to debug in your client app. One strategy that I found extremely helpful was to use curl. Here is how you would do the curl request for the above status update example:

curl -v -X POST -H 'Authorization: OAuth realm=""
oauth_nonce=... etc' -d "status=hello world"
https://twitter.com/statuses/update.json

So, in your client app, you would generate this header, but not actually run the request. Remember that nonces need to be unique; you can only run the request once with the same nonce value; changing it arbitrarily would invalidate the signature. So, just generate and print the header from your client app, and try to curl it, and see what you get back. The -v flag ensures you see plenty of debug info.

Acknowledgements

Tweepy is a great Twitter API implementation in Python. I used it to debug the protocol, just inserting print statements in relevant places, letting me trace what went on the wire and what came back.