evmar
> A portion of this code implemented a SMTP client.

If I wanted to root cause this, the real problem is right there. Implementing protocols correctly is hard and bugs like in the post are common. A properly implemented SMTP client library, like one you would pull off the shelf, would accept text and encode it properly per the SMTP protocol, regardless of where the periods were in the input. The templating layer shouldn't be worrying about SMTP.

aftoprokrustes
I will not comment on the technical part, as others already did it better than I could, but it just reminded me of an anecdote that reminds of the importance of such trivial things as a period at the end of a sentence:

In Germany, where I work, it is usual at the end of employement to ask for a letter of recommendation ("Zeugnis") that lists the tasks performed, and how good the employee was. It is an important document, as it will typcally be required when applying for jobs. Obviously, no employee would accept a document explicitly stating "this guy is a lazy bastard, do not hire him", so there is a "Zeugnissprache", a "secret code" to disguise this information as praise. One part of this code is that a missing period in the last sentence means "please ignore everything said here, this guy is horrible".

How do I know? I let a lawyer check my Zeugnis after my last employment, and (I assume out of lack of care, as all my performance reviews were positive) the last sentence was missing the period.

kazinator
Why would a cron job that sends e-mails need to implement its own SMTP client???

You just use the mail program from mailutils or whatever.

Just from a point of view of deliverability, developing bare bones SMTP interaction over a socket is a nonstarter. You can't just connect to random mail exchange hosts directly and send mail these days. A solution has to be capable of connecting to a specific SMTP forwarding host (e.g. provided by your ISP). For that, you need to implement connections over TLS, with authentication and all.

Also, a slightly ironic thing is that cron already knows how to send mail. The output of a cron job is mailed to the owner. Some crons let that mail address be overriden with a MAILTO variable in the crontab or some such thing.

teddyh
I see two huge bad habits here. The first is the obvious one, as pointed out by many commenters here: Don’t implement standards haphazardly, if you even should do so yourself. Either give the implementations the necessary care and attention, or use a pre-made library.

But the other thing is: Don’t vendor your dependencies. Those libraries you use need to be updated regularly and timely, and absolutely not “only as necessary”. If updates lag behind or are avoided entirely, bugs like this can be huge problems even when the upstream code has been fixed, for people who thought that they should update only when they, themselves, see a problem or need.

userbinator
I suspect a lot of people are no longer being taught these fundamental protocols by manual interaction with a terminal, since that's what SMTP seems to have been originally intended for; and as someone who actually made use of that for a nontrivial amount of time, the "single dot on a line" to end a message has been permanently etched into my memory.

Relatedly, escaping somehow seems to be a foreign concept for a lot of programmers, who wouldn't ever see the above situation and ask themselves "but what if I want to send an email with a line containing a single dot?" yet another large group of them finds it perfectly logical and easy to understand.

talkingtab
This reminds me of an experience debugging a network protocol implementation - specifically AppleTalk NBP for other ancient people. I had coded everything but my packets were rejected (aka silently dropped) when real (aka Apple implementation) packets were not. I had a copy of the good and bad packets on the screen of my computer and had gone over them byte by byte to find the problem. And there was none. From start to finish they were exactly the same, with correct check sums etc. It was time to go home and I decided just to print the stupid things to look at later.

As soon as I printed them, the error was clear. My version ran to two pages and the good implementation one page. I had not been careful to clear the buffer before sending the data (mbufs don't you know).

This still cracks me up.

kazinator
But the line "We are happy to welcome you to our family." is not anywhere near the line limit. There is something else going on here, like perhaps the whole thing actually being an HTML MIME attachment, perhaps? IN which it is like

  ... lots of text ... <br>We are happy to welcome you to our family.<br>
or whatever. But if you blindly split HTML into lines, it will break tags.
jeffbee
If a person has never heard of dot stuffing they're never going to believe what other horrors lie within the email space. Header folding, quoting in the local part, ipv6 literals, etc.
irrational
> Seeing that the SMTP client code was borrowed from a previous project we thought it good to let our other teams know about this bug in case they needed to patch it as well. They thanked us and we called it a day.

As soon as I read the above, I knew the below would be the result.

> It seems one of our other teams haven't gotten around to patching this bug in their code.

carimura
Good story. Reading this reminds me of all the "curious cases of....." that I've solved (or in some cases not solved) over my career and how that feeling of triumph is so deeply tied to why I got into computers in the first place. The pure joy of unraveling the mysteries of engineering... like forgetting a semi-colon somewhere in a 50k LOC Perl backend.
eschneider
I guessed the cause of the problem (well, "leading period") from the description, but that's because I've experienced a lot of pain in my life...
linsomniac
Show of hands: Who knew exactly where this was going as soon as "SMTP" was mentioned?
jcpham2
As soon as I started reading one of the first things that came to mind was the termination period in the smtp server spec. If you’ve ever had to troubleshoot an SMTP transport issue typing SMTP con Ds by hand is a familiar thing. Cool read.
jayceedenton
> A portion of this code implemented a SMTP client.

What the...

davidmurdoch
Loved reading this. This is now one of my new favorite bug hunt stories!
tuck1s
The need for dot-stuffing is a side-effect of email's use of the period as part of the termination sequence CR LF . CR LF. Mishandling of that sequence has led to other recently discussed bugs. https://smtpsmuggling.com/
layer8
SMTP = Sporadically Missing Trailing Period

;-)

readyman
The title sounds like a pregnancy scare
blueflow
You can avoid this mess altogether by using the quoted-printable content encoding when generating emails.
camel_gopher
I have two kids. The case is not curious at all.
pyuser583
For female readers, the title might have a very different meaning.
bhaney
This made me realize I have the opposite problem. Now I have to go update the toy SMTP server that I ended up implementing in a perl script so it handles SMTP clients double-dotting a line.
256_
Sometimes I feel like I'm wasting my time by obsessively reading RFCs and specifications for the things I use. This made me feel better about that. And also much more smug.

Other than the obvious moral that protocols should be implemented properly, the moral of the story is that all abstractions are leaky, and it will always be useful to understand the lower levels.

renewiltord
Suspected it was the 990 char limit per line but this is another one. I assume this is an old system? Despite many claims about “best practices”, there were definitely past platforms where these didn’t exist and a minimal implementation was safer just because you knew the scope of error.

Of course if it was modern, different question.

sethammons
I recognized the problem on sight; we solved this exact issue but since it was MTA software we knew about periods being special. Unfortunately, people are often solving problems that many others have solved. Maybe AI will allow the lines to be connected or solve for known edge cases like a dot in the smtp data
croes
>the first character is a period and there are other characters on the line, the first character is deleted.

Why is it implemented that way?

If a single period means end of mail then more than a period means it's mail data.

Why deleting the period in the first place? Couldn't they store one byte to check the next?

jiveturkey
> This meant some customers received emails informing them their new premium was now $2700 instead of $27.00.

there's a secondary issue here, why in the world would you auto split a monetary value across a numeric decimal indicator? why would you split lines at all for this use case?

867-5309
>Every time an employee of our client needed to send out a document via email or needed to print a document that needed to be sent out by the postal services to the customer the employee would have to replace all the placeholders within the document

ironically titled..

readthenotes1
All this about a missing period and nothing about the consistently missing commas. In fact I would say leaving off punctuation on one line paragraphs would be more consistent.
davidwritesbugs
As I'm implementing an NNTP server based on RFC specs I knew instantly what was happening here without RFA. Dot stuffing, yea 80s protocols baby.
marcosdumay
So... SMTP client misses basic SMTP functionality that fits in the 20 lines summary.

Good luck having it handle any of the SMTP craziness that isn't on the short introduction to the protocol.

m3kw9
For a second I thought this was the health forum
ijuelz
The night is dark and full of errors.
EVa5I7bHFq9mnYK
Given that the words Night and Club appear in the picture, the title looked mildly intriguing ..
bruce343434
As someone who configures email servers for a living (among other things): email needs to be replaced, frankly. This is a stupid protocol with even stupider file formats. What is the reason for such a hard coded line limit? It's just a stream of bytes...

Not to mention all the weird bandaids on top of bandaids to try to get sender verification and tamper proof emails working. That alongside the complete lack of end to end encryption.

It's just an incredibly unpleasant tech stack from top to bottom, through and through. The amount of moving parts/pieces of running software needing to cooperate just right to even function as a simple outgoing-only mail server is too damn high.

Learner100
- The SMTP client they implemented could insert a newline such that a line was comprised of only a single period.

- The SMTP client spec says that an additional period would be added here.

- The SMTP server spec says that it would remove this additional period, bringing us back to one period.

I don’t get how this led to there being no period at all. Am I missing something?

edweis
I like the simplicity of the email membership price increase.
bufordtwain
What about the missing comma at the end of the first line? :)
cat_plus_plus
Congratulation on upcoming new addition to your family!
xmjw
Holy shit. This is one of the 2 bugs in my career I never solved. (That I knew about, and lost sleep over, etc…)
BrandonMarc
This reads like a story from the daily wtf
nytesky
Wasn’t this an episode of Silicon Valley?
billy99k
[flagged]