Let's talk
engineering

Emails not delivered to Gmail? Why confirmations bounced

Every booking confirmation to a Gmail address was refused from one afternoon. About 37 went unconfirmed and no code had changed. How to find the cause.

· · updated

Diagram: Every Gmail confirmation refused from noon. Delivery clean until 12:00 on 5 September; Records set to reject anything not signed by the domain itself; Sender signed under its own name; record named only Google; First estimate 692 emails, real figure roughly 37; No code was at fault

If customers say your emails never arrive and Gmail is refusing them, the usual cause is not the software but a change to the domain’s email records that made the policy stricter than the service sending the mail could satisfy. The first check is to find a bounce message, read its reason, and ask who changed the domain’s records and when.

Here is how that looked at one car rental operator, and what fixing it took.

A customer books a car, pays, and hears nothing. They email you to ask whether the booking went through. Then a second customer does the same. Then you get a bounce message in your own inbox saying a message from your business could not be delivered, in language that looks like it was written for somebody else, and you forward it to us with the question every owner asks: what did you change?

The answer, for this self-drive car rental operator in Goa (the system behind GOI Car & Taxi), was that nobody had changed the software. From noon on 5 September, every confirmation sent to a Gmail address was refused. The first estimate of the damage, worked out from the domain’s settings, was every customer email since 22 August, 692 of them. Checked against the actual delivery records, that was wrong: delivery was clean until 12:00 on 5 September and broke from that minute, which puts the real figure at roughly 37 bounced confirmations. The wrong number was corrected before it reached the owner, but it is worth recording, because the tempting figure was twenty times the true one.

What was actually going on

Email has a set of public records, held with the domain name, that tell receiving mail systems which senders are allowed to use that name. On the afternoon of 5 September, the records for this operator’s domain were set to the strictest setting: reject anything that is not signed by, and sent from, the domain itself. That is a reasonable setting for a business that spoofs easily. But the service that actually sends the portal’s email was signing messages under its own name, not the operator’s, and the record listing approved senders named only Google. So nothing lined up, and Gmail did what the record told it to: refused every message.

The sending service, on inspection, had never been set up to sign as the operator’s domain at all. It held the sending address as a bare address with signing switched off, and no domain identity existed. Half of a separate bounce-handling arrangement had been published in the domain’s records but never told to the sending service, and would not have helped anyway under a strict setting.

So the bounce was not a code fault. It was a domain configuration that had been made stricter than the sending arrangement could satisfy.

What we changed

The operator’s domain was registered with the sending service as a proper identity, signing was switched on, and the three records that publishes were added to the domain. Then, rather than assume, we sent a test message through the exact production sender to a Gmail inbox and read the result out of the received message’s own headers: signed by the operator’s domain, passed the domain’s policy, no adverse disposition.

While there, a file of live credentials was found sitting in the repository directory one command away from being committed. It was excluded from version control and reorganised into a properly named configuration file.

What it did not fix

The roughly 37 confirmations that were refused during the window were refused. A domain record cannot resend an email, and the log for this work records no resend.

Three things were flagged for the owner and left. The sending service’s access key and secret are hard-coded as fallbacks in two source files and therefore live in the repository’s history, which no later edit removes. A second sending identity, for the portal’s own hostname, is in a failed state. And the administrator credentials supplied for this work should be deleted and rotated now that it is done.

How to check your own email delivery

  1. Get hold of one bounce message and read the refusal reason, even if the language looks written for somebody else.
  2. Ask who changed the domain’s email records recently, and compare the date with the first customer complaint.
  3. Send a test from the exact production sender to a Gmail inbox and read the received message’s headers: it should be signed by your domain and pass the domain’s policy.
  4. Work out the damage from delivery records, not from the domain settings. Our first estimate was twenty times too high.
  5. Decide whether the refused messages will be resent. A domain record cannot resend an email.

The mechanism, briefly

_dmarc on the domain published p=reject with adkim=s and aspf=s; the SPF record listed only Google’s sender, and SES was signing with amazonses.com, so neither DKIM nor SPF aligned to the domain and Gmail rejected with 550-5.7.26. Fixed by creating the domain identity in SES with Easy DKIM, publishing the three CNAMEs DNS-only at the DNS host, and verifying dkim=pass header.i=@ the domain, dmarc=pass and dis=NONE in the raw headers of a message sent through the production Source.

Where this ends up

A confirmation that is refused at the door is indistinguishable, to the customer, from one that was never sent, which is why Sazinga Rentals treats the domain’s mail authentication as part of the booking flow and checks delivery from the received headers, not from the send succeeding.

This came out of building Sazinga Rentals

Bookings, availability and the fleet standing behind them. The problem above is one we met while building it, and what we did about it is in the product.

If you run something like this, there is one thing you can do without a call: send one week's booking sheet.