Walt Stoneburner's Ramblings

Currating Chaos

Resolving a Redirect Error

This morning I was greeted by a Google Search Console email stating there was a Coverage Issue on one of my domains. Specifically, there was a Redirect Error.

The problem, in slightly more detail, is that if one goes to http://mydomain.com/ it should redirect to https://mydomain.com/ and then serve the pages securely. The problem was that the site was getting into a redirection loop and not serving at all.

The domain happens to be hosted on DreamHost with its DNS and caching handled by Cloudflare, two services I can't say enough good things about.

This was particularly odd, as the domain had been working for quite a while, and according to the files, provider settings, and domain management tools, nothing had been significantly altered in about two years.

So, either I changed something one day and thought it'd have little consequence, or something at the provider's end changed, or there was an error that would either need waiting out or raising to support's attention. As I had other domains hosted the same way with the same providers, I went on the assumption it was a problem within my control.

When this kind of problem crops up, it's because DreamHost wants to manage the domains's DNS (since it knows what the machines' IP addresses really are) and Cloudflare wants to manage the domain's DNS (since it is providing high-speed responses for users at the edge). As long as both systems are in agreement about the contents of the DNS, all is fine. But, if DreamHost moves machines (very rare, but it happens), then the two fall out of sync and the DNS records must be manually synchronized again.

The easiest way to check is using dig.

dig @ns1.dreamhost.com mydomain.com

This asks DreamHost's DNS, to which I can compare the entries on Cloudflare's DNS. In this case, they were all the same.

Logging on to DreamHost's server, which I first expected would use its own DNS overriding Cloudflare but that seemed not to be the case, which also makes sense because of what the domain name's registry entries say is authoritative, I got this:

$ lynx http://mydomains.com/

Looking up mydomain.com
Making HTTP connection to mydomain.com
Sending HTTP request.
HTTP request sent; waiting for response.
HTTP/1.1 301 Moved Permanently
'A'lways allowing from domain '.mydomain.com'.
Data transfer complete
HTTP/1.1 301 Moved Permanently
Using https://www.mydomain.com/
Looking up www.mydomain.com
Making HTTPS connection to www.mydomain.com
Verified connection to www.mydomain.com (subj=*.mydomain.com)
Certificate issued by: /C=US/O=Cloudflare\, Inc./CN=Cloudflare Inc ECC CA-3
Secure 256-bit TLS1.2 (ECDHE_ECDSA_CHACHA20_POLY1305) HTTP connection
Sending HTTP request.
HTTP request sent; waiting for response.
HTTP/1.1 301 Moved Permanently
Data transfer complete
HTTP/1.1 301 Moved Permanently

lynx: Start file could not be found or is not text/html or text/plain
      Exiting...

Sure enough, we see the redirection loop happening. The HTTP request lands at a HTTP 301 that says to use HTTPS. That gets a secured connection, which ...makes an HTTP request? ...that results in the 301, and we're in cycles, so it quits.

Just to make sure, I saw there was nothing in .htaccess and that the index.html file was present. File dates, again, showed it had been about two years since anything had changed, so this very much was some configuration issue.

It turns out if you go to Cloudflare's Speed Test tab, it will tell you if something is wrong. You can then go to the help center, check your specific domain, and see what it says.

I found this page to be a little tricky to get to, it's title is "Are you experiencing a problem with Cloudflare" and the button you want is "Error Analytics".

  • https://dash.cloudflare.com/{some-hex-key-here}/support
  • https://dash.cloudflare.com/{some-hex-key-here}/support/error-analytics

Cloudflare also has a Cloudflare system status page, if you want to check their services overall. DreamHost also has a DreamHost system status as well.

In my case, it confirmed I had a redirect loop and pointed me to Cloudflare's Troubleshooting redirect loop errors page.

Here's where things appear to get sticky. When one utilizes Lets Encrypt with DreamHost, DreamHost as the site provider will automatically redirect any HTTP traffic to HTTPS, which is actually what I want.

Meanwhile, Cloudflare in their SSL/TLS Overview table has a Flexible SSL encryption mode that encrypts traffic between the browser and Cloudflare even if the request is HTTP. Clever. It does this by sending requests to the origin web server unencrypted over HTTP, but your connection to Cloudflare is encrypted. However, that means redirect loops can happen if the origin server is redirecting HTTP to HTTPS, perhaps never answering the original HTTP request.

What also threw me was that direct HTTPS requests were not being served either, although this could merely have been a cached response.

Just to incrementally check things, on Cloudflare I put the server into Development Mode and cleared the cache; I didn't expect it to help, but sometimes glitches happen.

While this wasn't going to be the final solution, I went to disable the automatic redirect to HTTPS on DreamHost. Again, this didn't help and I turned it back on.

The Solution

Assuming that DreamHost was doing the Rewrite rules on the domain automatically, this left making changes on CloudFlare.

Comparing to another site I was working, I saw the working site's Cloudflare SSL option (in the SSL/TLS Overview tag) was set to Full.

One should use Full if there is a SSL certificate configured at the origin web server, which there was, the Let's Encrypt certificate.

However my setting was currently Flexible.

Because this was an "older" website, I suspect the following happened:

  • I set up the site originally as HTTP long ago
  • Let's Encrypt came onto the scene and I added it
  • When I moved to CloudFlare, it had an automatic HTTP to HTTPS redirection that I turned on at some point
  • DreamHost in improving its service also opted to turn on HTTP to HTTPS redirection at their end as well
  • Where ClouldFlare was doing the redirect itself before, now both systems were in conflict

The solution was to change the setting to Full, letting the origin server's certificate do all the HTTPS magic, and therefore let the origin server's HTTP redirect where it wanted. Meanwhile, Cloudflare could still intercept and translate any originating HTTP requests to HTTPS as well.