Story time: When email link behaviour changed

Last year, a client started getting reports from users that links in their emails weren’t working. They would click them and get an invalid certificate error. It was time to investigate!

The first step was getting a sample email and link to try out. Right away, I was able to reproduce the issue. I wasn’t aware of any recent relevant changes. It was time to dig in.

The email service they used provides simple click tracking by wrapping links in emails. The user would hit a service-hosted URL, which would immediately redirect to the destination one. This was set up, but with no SSL certificates. It used to work fine, and I could find no recent configuration changes. Nothing in the email service changelogs indicating relevant changes in their system.

The immediate fix was to configure SSL correctly, which I did, and everything worked fine. But why the change? I noticed that the links in the email were http and not https. Previously the browser would happily follow the http, but something was triggering a redirect to https. I tested in a separate browser and noticed something very strange…it worked there! No auto-upgrade to https. What gives? I verified it was also working in private-browsing mode in my normal browser.

Something about the browser state was causing the auto-upgrade. Looking at the browser logs, they noted that the reason for upgrading was HSTS. I did some research and learned there is a way to specify a “force SSL” state on all subdomains. I checked the root domain, and there it was! Coincidentally, the marketing website (at the root domain) had recently been relaunched and included the HSTS subdomain header configuration.

Mystery solved. A marketing website update caused product email links to be auto-upgraded to https.

Lessons learned: Root domain settings can affect behaviour of any subdomains via the HSTS header. Always have secure certificates set up.

Do you have any debugging stories? I’m very interested in hearing them!