REST in Practice: HTTP Security Essentials

This excerpt from REST in Practice covers HTTP security. A coffee shop called Restbucks is used for the examples.

---

HTTP Authentication and Authorization

As we’ve often seen on the World Wide Web, HTTP natively supports authentication (to establish identity) and authorization (to help establish trust). When a consumer attempts to access a privileged resource, credentials must be provided in an Authorization header, or the consumer will be refused access.

In Restbucks, we can secure access to resources using this capability. For instance, we might allow only authorized consumers to access payment resources. If an unauthorized consumer tries to access a payment resource, it will be refused because it lacks appropriate credentials.

In Example 9-1, the consumer system sends a GET request to the Restbucks payment service. Seeing that there is no WWW-Authenticate header present, Restbucks responds with the challenge shown in Example 9-2.

Example 9-1. Accessing a payment resource

GET /payment/1234 HTTP/1.1
Host: restbucks.com

Example 9-2. Restbucks challenges the consumer

401 Unauthorized
WWW-Athenticate: Basic realm="payments@restbucks.com"

The challenge tells the consumer that Restbucks requires a Basic digest to access some resources in the realm payments@restbucks.com.^[92] If the consumer knows username and password credentials for that realm, it hashes them and embeds them in an Authorization header and retries the request, as shown in Example 9-3.



As an optimization, if a consumer already knows that a resource is protected through Basic authentication, it can provide credentials with the initial request message and so avoid the challenge-response steps.

Example 9-3. Attempted authorized access to a payment resource

GET /payment/1234 HTTP/1.1
Host: restbucks.com
Authorization: Basic Zm9vOmJhcg==

Unfortunately, HTTP Basic authentication is not secure. Although the username and password are never sent in plain text, the base64-encoded text is easily intercepted and decoded. To make HTTP authentication less susceptible to simple attacks, we need either a secure channel such as HTTPS (which we’ll see in the next section), or a mechanism that allows credentials to be passed securely through an insecure channel. Fortunately, this is precisely what HTTP Digest authentication provides.

Example 9-4 shows a Digest challenge from the payment service. Note that this challenge contains far more metadata than a Basic one.

Example 9-4. Restbucks challenges the consumer

401 Unauthorized
WWW-Athenticate: Digest realm="payments@restbucks.com",
  qop="auth",
  nonce="1e8c46a7d793433490cb8303f18a86e5",
  opaque="ff1eccda9ef442b3b38cabb2435d5967"

The Digest challenge metadata shown in Example 9-4 allows the consumer to transmit credentials safely over the Web:


qop




nonce




opaque



Based on the information presented in the challenge, the consumer retries the request with credentials attached to the Authorization header, as shown in Example 9-5.

Example 9-5. Consumer submits a digest

GET /payment/1234 HTTP/1.1
Host: restbucks.com
Authorization: Digest username="beancounter",
  realm="payments@restbucks.com",
  nonce="1e8c46a7d793433490cb8303f18a86e5",
  uri="/payment/1234"
  qop="auth",
  nc=00000001,
  cnonce="cf45f0087f33bce12332aef430945dff",
  response="ff14aa3457acd60aa2091232a98756ff",
  opaque="ff1eccda9ef442b3b38cabb2435d5967"

As you can see in Example 9-5, much of the information in the initial challenge is returned without modification in the response. However, the consumer produces additional metadata to fulfill its part of the authentication protocol:


username




uri




nc




cnonce




response



The response is calculated using a three-step process, as follows:

• The MD5 hash of the username, realm, and password is computed.^[93]
  
  
  _{[93] http://tools.ietf.org/html/rfc1321}
  
  
  
• The MD5 hash of the method and digest URI is computed.
  
  
• The MD5 hash of the value produced by step 1—the service’s nonce, request count (nc), cnonce, and qop—and the result from step 2 is calculated, yielding the consumer’s response value.

Since the consumer and service both know the shared secret (the consumer’s password) the consumer’s request can be checked by the service by repeating the calculation.

---

Note: Unlike Basic authentication, Digest authentication always requires a challenge/response, since the challenge step provides metadata to parameterize the response.

---

This scheme provides a number of benefits when compared to Basic authentication. First, because passwords are not transmitted over the network, security credentials are far less easily compromised. Second, because the Authorization header’s response entry is never repeated (as a result of using nonces) brute force replay attacks are rendered ineffective. Third, passwords no longer have to be transmitted over the network in an easily compromised form.

Despite being more secure than Basic authentication, HTTP Digest authentication can still be broken using a man-in-the-middle attack if transport-level security isn’t also used. In Figure 9-1, an interloper mimicking the behavior of the remote service intercepts the initial authentication and responds with a Basic authentication challenge to the consumer. Basic authentication is easily broken; when the consumer responds to the Basic authentication challenge, the man in the middle is able to compromise the credentials.

---

Warning: If you are going to use HTTP authentication, transport-level security should be considered mandatory.

---

Critically, the consumer may not know this has happened if the man in the middle presents a faithful reproduction of the target service! To prevent this, transport-level security is still necessary, even for Digest authentication. To provide comprehensive defense against man-in-the-middle attacks, we need transport-level mechanisms. Enter HTTPS.

---

Note: In addition to Basic and Digest authentication, a third scheme called WSSE has evolved, driven in part by the desire to secure Atom and AtomPub services. The WSSE scheme borrows an authentication approach from WS-Security’s UsernameToken profile and maps those features onto HTTP headers. WSSE is superior to Basic insofar as it doesn’t transmit passwords in clear text, requiring instead shared secrets between client and server. Like the other authentication mechanisms, WSSE is vulnerable to man-in-the-middle attacks in the absence of transport-level security.

---

Transport-Level Confidentiality and Integrity

We often take HTTPS for granted, and yet it supports vast numbers of secure interactions every day on the Web. While it might not be glamorous, humble HTTPS is mature, widely deployed, and well understood, making it ideally suited to hostile environments such as the public Internet. Of all the security protocols we could use, nothing else comes close in terms of ubiquity, maturity, and the huge volume of value transactions it supports every single minute.

All this gives service developers a degree of comfort: HTTPS is a well-understood and researched protocol that routinely provides the underlay for secure interactions at global scale. We can be sure it’s fit for purpose.

Conceptually, HTTPS is straightforward. Instead of exchanging HTTP requests and responses over TCP, we transmit those same requests and responses over Transport Layer Security, or TLS.^[95] While TLS transparently provides integrity and confidentiality for HTTP interactions, it’s important to understand how it works.

---

Note: TLS started life as Netscape’s SSL (Secure Sockets Layer). The IETF renamed the protocol to TLS as it evolved into an Internet RFC.

---

The TLS protocol has three phases, each of which must be completed before we can transfer representations via HTTPS:

• Handshake
  
  
• Secure session
  
  
• Channel setup

An HTTPS session begins with a normal TCP three-way handshake, which opens a connection to port 443 (the standard TLS port) on the remote server. Once the TCP connection has been established, the secure session can be initiated, as shown in Figure 9-2.




To initiate a secure session, the consumer begins by sending a ClientHello message to the remote service. As part of this message, the consumer includes its highest supported TLS version (or SSL version if the consumer is old), the ciphers and hashing algorithms that the consumer understands (in order of preference), and an optional session identifier.^[96] The consumer also sends some pseudorandom data for service-side key generation.

---

Note: TLS is a versatile protocol; besides security, it also supports data compression as an optimization. Compression, however, is not widely used. We favor keeping any compression at the HTTP level using the Content-Encoding header so that we can compress representations irrespective of whether we’re using TLS or TCP as the transport protocol.

---

In response, the service sends a ServerHello message back to the consumer, indicating the TLS (or SSL) version chosen, the selected cipher and hash algorithms, an optional session identifier, and its own pseudorandom data for consumer-side key generation.^[97]



Following on from the ServerHello message, the service transmits a Certificate message to the consumer. The Certificate message contains the service’s public key, and optionally the service name and certificate authority (CA). The consumer uses this information to authenticate the service; it does this by matching the name of the service and the certificate to the name (or domain) of the server. (There’s a dependency here on DNS in that we are only as secure as the DNS infrastructure we are using.) The final message of the negotiation phase is ServerHelloDone, which the service sends to the consumer to indicate that it’s finished negotiating.

---

Note: TLS also supports client-side certificates. If the service sends a ClientCertificateRequest message prior to ServerHelloDone, the consumer should immediately respond with its certificate in a ClientCertificate message.This is useful because it enables the consumer and service to establish mutual trust at the transport level. Any traffic flowing over a bilaterally certified session has a trusted source and origin, allowing service implementations to authorize or disallow interactions based on the well-known source of an HTTP request.

---

Having negotiated parameters for a secure session, the service and consumer are now ready to set up a cryptographically secure channel, as shown in Figure 9-3.




The consumer transmits a ClientKeyExchange message encrypted using the service’s public key. This message typically contains a PreMasterSecret computed from the consumer and service’s random data.^[98] If the service is able to decrypt the ClientKeyExchange message, the consumer can be sure the service has the corresponding private key, thereby proving the service’s authenticity. To defend against an attacker reverting the session to an earlier and weaker protocol version (known as a rollback attack) the service double-checks the protocol version number in the ClientKeyExchange message.

---

Note: The negotiation of a shared secret is secure and cannot be eavesdropped, even by a man-in-the-middle attack. An attacker cannot modify the messages in the handshake without being detected, making the negotiation reliable.

---

The consumer follows with a ChangeCipherSpec message, which indicates to the service that from now on the consumer will use the session key for hashing and encrypting messages. Hashing a message with the session key creates a Message Authentication Code (MAC). The MAC provides integrity and authenticity by allowing recipients who know the key to repeat the hash function in order to detect whether the message content has been changed.

As further evidence that nothing has gone awry, the consumer securely transmits a ClientFinished message containing a hash of the entire conversation up to this point. The ClientFinished message is the first message encrypted and hashed using the shared session key computed by the consumer and service.

To complete the protocol, the service responds first with a ChangeCipherSpec message, indicating that it, too, will now use a secure channel using the negotiated keys. The service then sends a ServerFinished message that verifies its common understanding of the parameters for the session. The ServerFinished message contains a hash of the conversation up to this point for the consumer to verify.

From this point onward, the consumer and service can exchange secure messages using computationally inexpensive shared key methods. That is, we can now transmit HTTP requests and responses whose privacy and integrity are assured.

Network and Performance Considerations

Because HTTPS is so common, we might be tempted to use it liberally in an attempt to provide blanket security. If all a service’s resources are secured with HTTPS (particularly with bilateral certificates), its attack surface is significantly reduced. In practice, however, there are drawbacks to such an approach. In particular, forcing every interaction over a secure channel may limit scalability.

The use of a secure channel not only incurs cryptography costs, it also prevents caching in the network; because intermediaries cannot see the representations being exchanged, they cannot understand their caching metadata. Losing the important visibility aspect of HTTP means we only have consumer-side caching (which does not permit cached representations to be shared among many consumers) to help reduce load on servers.

The questions we have to ask ourselves when considering securing resources are “What is the value of the resource?” and “Who should be able to access it?” Only by answering these questions can we determine whether interactions with a resource should be secured, and plan for any performance challenges that arise.

---

Note: Establishing HTTPS requires several network interactions and cryptographic operations before a TLS channel can be used to transfer representations. This overhead makes a single HTTPS call several times more expensive than a single HTTP call. However, if the TLS channel is used for many exchanges, the setup cost and cost of ongoing cryptographic operations becomes less significant and the cost of missed intermediary caching becomes the primary consideration.

---

To gain the dual benefits of caching and secure interactions, we must turn aside from using encrypted channels to enforce security considerations, and consider instead patterns that allows us to encrypt content on unencrypted (and therefore cacheable) channels. As Figure 9-4 shows, it is possible to publish data on the public Web and still maintain confidentiality and integrity.




The pattern in Figure 9-4 shows a service publishing an Atom feed onto the Web. The feed and its entries are cacheable, and therefore scale well at modest latencies—as we have come to expect from the Web. However, the content of each atom:entry can be encrypted so that it is meaningless for any consumers that don’t have the key to decrypt it.

In Figure 9-4, the top two consumers both access the same (shared) key, while the third accesses a different shared key. This allows the service to broadcast information in the knowledge that only certain recipients will understand it (assuming that any shared keys have already been transmitted and stored securely). Because the content is useless without the decryption keys, the pattern enables the Atom service to publish a highly cacheable feed without worrying about attackers getting hold of the content.

---

Warning: This technique is not without its drawbacks. Having to share keys increases coupling, and in practice can be tricky to implement and scale.

---

In the general case, services will expose both secure and insecure resources. It is our duty as service developers to understand the trade-off between security and scalability, and to decide which resources to secure based on their risk/performance profiles.