Google has a vested interest in reading your email. Google lets users think that setting up a mail server is difficult, that email is a dead end communication medium compared to walled gardens, and that phishing is difficult to combat effectively.
FWIW though, setting up a mail server is difficult. Setting up a mail client with reasonable spam filtering to make email usable is also difficult. Compare the list of steps on either of those tasks to the list for getting email via Gmail: "1) Sign up to a Google account, 2) visit gmail.com"; it's obvious almost all users are going to do the latter (or its equivalent at a Google competitor).
Setting up a mail server isn't difficult. It's about as easy as installing and configuring a web server, and just about as technical.
Email has been around for decades. The fact that Gmail, Yahoo, Hotmail, and many paid services have had one-click email addresses for twenty-plus years should demonstrate that it's not difficult. Businesses just don't want to make it easy for lay people to do it.
Unless email servers have gotten a lot more turnkey in the last few years, I'm gonna disagree with this, sorry. :) I can get nginx running on most systems with literally a couple commands ("package-manager-thing install nginx" and "service-manager-thing start nginx"). Getting a mail server running -- at least in any form that qualifies as "Gmail replacement" -- seems to involve "install postfix, immediately edit the configuration because it has no out-of-the-box defaults, create multiple configuration files beyond that because that's how postfix rolls, install dovecot, immediately edit its configuration because it has no out-of-the-box defaults either, create some directories and user accounts and more configuration files because that's how dovecot rolls, then test that everything's working." And I'm leaving out the "install MySQL" part that many guides you'll find online will also have you do. Also, I haven't talked about running a spam filter and YOUR HARD DRIVE IS ALREADY FULL OF NIGERIAN PRINCES TRYING TO SELL YOU PENIS-ENLARGING BITCOIN I'M SO SORRY.
I would love it if "package-manager-thing install mail-server-thing" and "service-manager-thing start mail-server-thing" could get things going with sensible defaults like they do with nginx, even if you'd probably need to tweak things after they were going just like you're almost certainly going to do with a web server. But AFAIK, that just ain't the case.
I've run a mail server, and still help out with one, and getting consistent deliverability is hard. We've wanted to move to another hosting service for years, but the only reason we're doing ok right now is that our dedicated IP has a solid history and we just can't give that up in a migration.
The services you mention all have entire large teams devoted to tackling spam, reputation, uptime, etc. That it's easy and cheap on a marginal basis for them to add one more inbox to their millions doesn't mean the overall infrastructure is easy to keep running.
> The services you mention all have entire large teams devoted to tackling spam, reputation, uptime, etc.
Can you provide specific numbers to the count of people devoted to tackling spam, reputation, uptime, etc? I suspect the number's a lot smaller than you think. I also suspect the peoples' duties are less devoted than you think.
If Google, for example, truly worked to tackle spam then it wouldn't have a spam problem today. If Yahoo truly worked to tackle reputation, people wouldn't have trouble sending email using an email client instead of a browser.
I'll grant you that uptime does have dedicated people. But it's not to tackle spam or reputation. It's because offline services don't make a profit.
There are teams that actively (and solely) work on spam and abuse detection. They're larger than you seem to believe, though I won't give exact numbers. There's also obviously sre teams that maintain uptime. (Note I said teams, and there's a public approximate minimum size for an are team at Google of 8-12 people)
The problem is that spam can't be "solved". Reputation is easy to solve: only accept email from a known list of good senders. Gmail, Yahoo, MailChimp (or not), etc. But that makes people on HN complain. So your have to try and infer reputation of mailservers on shared hosts. And spammers are always trying to beat you, and there are thousands, maybe tens of thousands of spam outfits. Af they're sneaky. They try to use awa or gcp to send email, or even send spam from Gmail, prevent trickier things. So you're left to defend from a spam campaign from Yahoo while also trying to not block everyone at Yahoo, and detect the spammers who are using Gmail to spam Yahoo too.
And the spammers are always innovating, so you have to as well.
My personal belief is that Google likely considers spam detection to be an area of competitive advantage so investments are warranted.
> The problem is that spam can't be "solved". Reputation is easy to solve: only accept email from a known list of good senders. Gmail, Yahoo, MailChimp (or not), etc. But that makes people on HN complain. So your have to try and infer reputation of mailservers on shared hosts. And spammers are always trying to beat you, and there are thousands, maybe tens of thousands of spam outfits. Af they're sneaky. They try to use awa or gcp to send email, or even send spam from Gmail, prevent trickier things. So you're left to defend from a spam campaign from Yahoo while also trying to not block everyone at Yahoo, and detect the spammers who are using Gmail to spam Yahoo too.
Spam absolutely can be solved.
1) enforce identity. If the sender isn't authentic, then the sender is spam.
2) enforce reportability. If the user reports the sender as spam, then don't permit the sender to send more messages to the person who complained. if a lot of people report the problem, then block the sender.
3) enforce liability. if an ISP hosts spammers, then block the ISP.
If someone complains then walk them through the process. Just like people shouldn't drive vehicles without understanding that vehicles are dangerous, the same should be done with computers.
What is identity? A name? An SSN? How do you verify that for people in all the countries of the world?
How do you do that at scale? With hundreds of millions of users, you can't exactly call them up.
How many users are you going to have after you start adding measures to verify their identities at signup? How will the board of directors feel about that? And feel free to run your own company into the ground doing the right thing, but there are other email providers in the world who will happily accept the users you drive away.
What happens when people have their accounts taken over and start spamming? Were the accounts ever "real"? How can you even know?
What happens when the reports themselves are spam? Spammers will report other spammers to remove the competition. Or they'll just overwhelm it with useless fake reports to DOS your human reviewers.
You have to realize that every input to your system is a potential avenue for abuse. There are people sitting there all day thinking about how to prevent you from achieving your goals. Humanity went to the moon, we're problem solvers. If there's a way to manipulate and undermine your spam defenses it will be found.
> If the sender isn't authentic, then the sender is spam.
What's your definition of authentic? Is a self-hosted email server authentic? How do you decide?
> If the user reports the sender as spam, then don't permit the sender to send more messages to the person who complained.
If 10000 yahoo accounts are sending spam emails to other websites, what do you do? Block all yahoo senders? Try to block the yahoo accounts as they appear?
> 3) enforce liability. if an ISP hosts spammers, then block the ISP.
All major ISPs host spammers. Often they don't know that they do. Is it worth cutting off all comcast users nationwide from being able to use email? If anything, this would further centralize on one or two trustworthy email hosts, because those providers are essentially their own ISPs.
> What's your definition of authentic? Is a self-hosted email server authentic? How do you decide?
Authentic in terms of DNS. That means using and enforcing DKIM at the minimum.
Also in terms of from: and reply-to: addresses matching each other.
> If 10000 yahoo accounts are sending spam emails to other websites, what do you do? Block all yahoo senders? Try to block the yahoo accounts as they appear?
If 10000 yahoo accounts are sending spam emails, then that's a Yahoo problem. Yes, I would refuse to accept incoming mail from @yahoo.com until they've fixed their complicity.
> All major ISPs host spammers. Often they don't know that they do.
I disagree about not knowing that they do. ISPs must respond to fraud and abuse reports or they would lose the ability to do business. ISPs not responding to spam reports are offloading the cost of policing their users onto you.
> Authentic in terms of DNS. That means using and enforcing DKIM at the minimum.
Sure, these are basic things that are generally used as strong signals, but all this does is filter out the incompetent spammers. If you're sending from yahoo or from gmail, you've already solved the reputation problem. And there are other ways of doing the same.
> If 10000 yahoo accounts are sending spam emails, then that's a Yahoo problem. Yes, I would refuse to accept incoming mail from @yahoo.com until they've fixed their complicity.
I'd expect that this is approximately the baseline number of yahoo accounts sending spam when they aren't being actively targeted. Its less than 1% of 1% of the active monthly accounts on yahoo. So you'd like to just block yahoo constantly?
> I disagree about not knowing that they do.
Sure they know, in the sense that I also know that there are always people spamming from every major ISP. That doesn't mean that they can immediately address things. And while you're busy blocking all comcast users from sending your users email, your users are busy moving to a different email provider that identifies individual spam senders so that they can still receive legitimate email.
In closing, a simple question: if solving spam is this straightforward, why hasn't an upstart competitor (yahoo, protonmail, etc.) taken advantage of this strategy to fix the spam problem? It appears you're presuming a centralized system, which defeats the point of email and significantly simplifies the problem.
Internet systems are one part technology and one part social.
If my mail server is banning mail from Yahoo, I can't communicate with my grandparents and I stop using that mail server. Enough people do that and the mail server has no users.
inetknght, I get the sense that you run a mail server of your own. Have you taken your own advice here and blocked @yahoo.com incoming? Is it inconvenient? Is it more inconvenient than the two-step process of setting up a Gmail account?
> If my mail server is banning mail from Yahoo, I can't communicate with my grandparents and I stop using that mail server. Enough people do that and the mail server has no users.
Why are your grandparents using Yahoo instead of your mail server?
> Have you taken your own advice here and blocked @yahoo.com incoming? Is it inconvenient? Is it more inconvenient than the two-step process of setting up a Gmail account?
I haven't had any correspondence from anyone who uses @yahoo.com. Or, if I have, they haven't complained about me not receiving their email. Or, if they have, their complaint was also not received in which case it doesn't exactly matter. If it did matter then I would address it then. And, importantly, it also means there's another (less noisy) communication medium available already.
> Why are your grandparents using Yahoo instead of your mail server?
Because internet systems are one part technology and one part social. My grandparents already have Yahoo accounts and are unwilling to change that.
And if your solution to interoperating with Yahoo servers is "I don't have anyone to talk to using Yahoo servers," then I'm afraid it sounds like you're trying to solve a problem other than the one email is designed to solve.
The simplest web server I'm aware of is `python http.server 8080`
... and even for that, I had to disable / re-enable the firewall and fish around in the docs just last night to get the dang thing to accept connections from a source other than localhost (PROTIP: when running on a pretty-out-of-the-box Windows 10 config, the default options bind listeners to IPv6, not IPv4). And I definitely wouldn't recommend that configuration for production; you'll open yourself up to a universe of pain.
But as an analogy for setting up an email server vs. just subscribing to Google or whoever, I accept it. ;)
Phishing is difficult to combat effectively. Part of the problem is technological in that we never adopted a way of verifying senders are who they claim to be, but a large part of it is that humans are really good at fooling other humans, at least enough of the time to cause trouble.
Ironically, it is the technical problem of verification that makes it difficult to run (not setup) a mailserver: no one wants to trust it.
I don't think verification of sender is the problem (see PGP), it's that email was built such that anyone can contact anyone without setup ahead of time. I can give my address to some guy I meet at a conference and he can email me later with no issues, we don't need to agree ahead of time that we know who each other are. If he uses PGP, I need to accept the first communication is him, which means anyone can send me anything claiming to be anyone. Unfortunately, as soon as you solve this part of the issue with technology, I think you've just reinvented social media.
I wasn't trying to discuss whether or not PGP is a good or widely used technology. My point was that even if something like it were widely adopted and made seamless, it would not be able to solve the problem of bootstrapping the verification connections without eventually become social media itself.
> Part of the problem is technological in that we never adopted a way of verifying senders are who they claim to be, but a large part of it is that humans are really good at fooling other humans, at least enough of the time to cause trouble.
Tell me more about how DKIM and SPF and TLS with client certificates don't verify senders.
I didn't say they didn't, I said we haven't adopted a way of verifying senders. If DKIM, SPF, and TLS were universal, it would be a solved issue, but they aren't for a whole lot of reasons.
P.S.: However, come to think of it, they don't. You can receive a perfectly legit and verified email from scammer@consrus.ru with the name displayed as "Mr. Your CEO".
