Spam isn't killing email, automatic spam filter is killing it. WHY isn't nobody fixing this?

( I know, the correct one is "Spam" but I like "SPAM" :] )

I don't think SPAM alone is killing email, well not killing you know what I mean, took the joy out of it... I think automatic SPAM-filters also are killing email because they have made email a really unreliable way of communication.

I use Google Apps for some of my email hosting and occasionally, emails from known people, like my mom get cought in the SPAM-filter. Since I use Outlook, I don't check the SPAM-folder often. It has happened that I have missed emails this way.

It has become so bad that many ISPs in Europe and almost ALL ISPs in my country turned SMTP port off and forcing people to use ISP's own SMTP to send email. Which kind'a makes the Internet, slowly become CableTV, managed at top level. What is next, port 80? then the Internet will REALLY become CableTV. ...am I sensing conspiracy here?

There is something very wrong with email, WHY isn't anybody fixing it? Email is such an essential part of the Internet, specially shopping! Imagine a webshop without the email function, yes I thought so too, it wouldn't really work. ...Seriously, people talked about this for like 10 years now, no? WHEN is it time to roll up your sleeves, sit down and fix email, once and for all?

( ...for the info, I use white-listing aliases normally. I create <contact>@domain.com and give that one out to people so in case it starts to collect SPAM, I can shut it down + I know where SPAM comes from, so it gives me zero SPAM not a good solution for "normal" people, however. )

Well one problem is that we need a new email system and no one wants to make the move.

i was on a volunteer internet working group on how to deal with spam back in the late end of the 1990's

about 1998 i think.

when i pointed out that smtp had some very basic issues that should be changed the majority of the folks on the group said that was not whyat they would support, that the anti spam extesions had to work within smtp as it was.

well after that i dropped off the list cause they were not going to fix the real problem.

the issues are not trivial.

think of how many smtp servers there are in the world right now.... a lot....

as for blocking the smtp port... actually i have no problem as long as the isp has a policy of allowing you to request that your connection be open.

see one of the problems is that *if* an isp lets all traffice thru then pc's on that network can all become mail servers ...

if for example a trojan / bot control program gets even a few hundred pc's sending spam from that network then it becomes a real problem for the isp and the global email community.

while i want the internet to be very open i also think that if you are on a network you have to play within some limits.

the network operator should be able to limit the number of mail servers - not to keep you from sending email but to keep bot nets from flooding the network with spam email.

figuerres said:

Well one problem is that we need a new email system and no one wants to make the move.

 

i was on a volunteer internet working group on how to deal with spam back in the late end of the 1990's

about 1998 i think.

when i pointed out that smtp had some very basic issues that should be changed the majority of the folks on the group said that was not whyat they would support, that the anti spam extesions had to work within smtp as it was.

 

well after that i dropped off the list cause they were not going to fix the real problem.

 

the issues are not trivial.

think of how many smtp servers there are in the world right now.... a lot....

 

as for blocking the smtp port... actually i have no problem as long as the isp has a policy of allowing you to request that your connection be open.

 

see one of the problems is that *if* an isp lets all traffice thru then pc's on that network can all become mail servers ...

if for example a trojan / bot control program gets even a few hundred pc's sending spam from that network then it becomes a real problem for the isp and the global email community.

 

while i want the internet to be very open i also think that if you are on a network you have to play within some limits.

the network operator should be able to limit the number of mail servers - not to keep you from sending email but to keep bot nets from flooding the network with spam email.

Our company has created a system that could replace e-mail, that is secure by default, that uses a white-list system to eliminate spam, but apparently people are not interested in such a system.

This is primarily a political, not technical issue. Addressing it would require virtually every service provider to agree on a set of changes and implement them at the same time and the level of coordination (and authority) just doesn't exist. A central issue in spam is spoofing, and even addressing this one aspect through SPF is unevenly and inconsistently applied. If we can't get mail admins to uniformly use this standard, then there's not much hope of doing anything more stringent and broadly based. Today, people have pretty much just resigned themselves to the fact that spam is something they have to cope with (via filters, whitelists, graylisting and so on), rather than viewing it as an important issue that requires fundamental change.

Or to put it another way, we're so focused on addressing the symptoms of spam, there's no will to actually take the measures necessary to treat the underlying disease (i.e.: addressing inherent flaws in the protocol).

SPAM isn't the problem.  Spammers are the problem.  No matter what process or mechanism is put in place the infrastructure can't combat what is essentially subjective.  What is the expression ... one man's garbage is another man's gold.

The best way to combat a spammer is to DOS them (maybe with loads of their own spam), Wild Wild West style.  And just like in the wild west you need to watch out for miss kitty and treat her nice, her computer might be a zombie for the spammer.

Suggestions for Outlook.  Turn off spam checking, turn off the reading pane, if something comes in your inbox that is unexpected or looks suspicious delete it, and empty your deleted items folder.

mstefan said:

This is primarily a political, not technical issue. Addressing it would require virtually every service provider to agree on a set of changes and implement them at the same time and the level of coordination (and authority) just doesn't exist. A central issue in spam is spoofing, and even addressing this one aspect through SPF is unevenly and inconsistently applied. If we can't get mail admins to uniformly use this standard, then there's not much hope of doing anything more stringent and broadly based. Today, people have pretty much just resigned themselves to the fact that spam is something they have to cope with (via filters, whitelists, graylisting and so on), rather than viewing it as an important issue that requires fundamental change.

 

Or to put it another way, we're so focused on addressing the symptoms of spam, there's no will to actually take the measures necessary to treat the underlying disease (i.e.: addressing inherent flaws in the protocol).

 

mstefan: exactly right, it's the willing ness or lack of to take the steps needed and to pony up to the cost also.

Dave: spammers are the problem in large part, the REAL way to combat the issue is not in filtering and testing.

the first real step to halt it is to make the replacement of SMTP be much more strict in the format of the email message - the headers esp.

and to setup the system to make it possiblt to track a message back to the sending server that put in on the network.

if you can see that users get traffic from a given host and that 99% of that traffic gets rejected by the users and the traffic is massivly inbound to your server then you can throttle them or block them with better tracking of why.

*IF* that model was used on the majority of mailservers then the bulk senders would find they would become less and less effective and would have to "Play fair"

while a good IT team can find the sources of the bulk inbound junk the current smtp syste make that to hard to really do and still be cost effective. so the alternative is the spam rating and blocking we see today.

it's cost effective and scales well.

but very far from perfect.

I would start with a simple upgrade to SMTP that would allow servers to "opt in" on a new message format based on a strict XML dialect.

that and some logic that says in short:

if a message comes in that you can not verify that it came from the server and identity that it claims to be from then you block it and report it as suspect.

just like we cut back on open relay servers a few years back... if a server in say north korea is sending messages that say they came from a hotmail account STOP right there and do not accept or deliver that message. also report that server to a server abuse network.

as more reports come in more servers can block that sender.

yes some of this is the same as the RBL / DNS blacklist systems but i start with the idea of halting the bogus message headers.

sending server and sending domain and sending account all need to be evaluated to halt the masking of the bulk sender.

expose them as what they are and make them take complaints for real.

and look at ways to account for the cost to the reciving server ....

i bet that at least 25% of the global cost of the internet is due to bulk / junk email

think what we could all do with 25% cash back on that scale....

figuerres said:

mstefan said:

*snip*

mstefan: exactly right, it's the willing ness or lack of to take the steps needed and to pony up to the cost also.

 

Dave: spammers are the problem in large part, the REAL way to combat the issue is not in filtering and testing.

the first real step to halt it is to make the replacement of SMTP be much more strict in the format of the email message - the headers esp.

and to setup the system to make it possiblt to track a message back to the sending server that put in on the network.

if you can see that users get traffic from a given host and that 99% of that traffic gets rejected by the users and the traffic is massivly inbound to your server then you can throttle them or block them with better tracking of why.

*IF* that model was used on the majority of mailservers then the bulk senders would find they would become less and less effective and would have to "Play fair"

 

while a good IT team can find the sources of the bulk inbound junk the current smtp syste make that to hard to really do and still be cost effective. so the alternative is the spam rating and blocking we see today.

it's cost effective and scales well.

but very far from perfect.

 

I would start with a simple upgrade to SMTP that would allow servers to "opt in" on a new message format based on a strict XML dialect.

that and some logic that says in short:

if a message comes in that you can not verify that it came from the server and identity that it claims to be from then you block it and report it as suspect.

 

just like we cut back on open relay servers a few years back... if a server in say north korea is sending messages that say they came from a hotmail account STOP right there and do not accept or deliver that message. also report that server to a server abuse network.

as more reports come in more servers can block that sender.

 

yes some of this is the same as the RBL / DNS blacklist systems but i start with the idea of halting the bogus message headers.

 

sending server and sending domain and sending account all need to be evaluated to halt the masking of the bulk sender.

expose them as what they are and make them take complaints for real.

 

and look at ways to account for the cost to the reciving server ....

i bet that at least 25% of the global cost of the internet is due to bulk / junk email

think what we could all do with 25% cash back on that scale....

What I'd like to see is really a more stringent form of SPF, but rather than relying on DNS records, to use the mail servers themselves to validate the message. Basically, the recipient would contact the mail server who supposedly originated the message and ask "I received a message that was addressed from user-at-domain, did this message really originate from you? If so, then send me back a hash of the sender's email address and the message ID." Of course, that would impose some additional workload on the servers, but it would put a serious crimp in spoofing and the net reduction in spam would more than offset the increased traffic from the verification process.

mstefan said:

figuerres said:

*snip*

What I'd like to see is really a more stringent form of SPF, but rather than relying on DNS records, to use the mail servers themselves to validate the message. Basically, the recipient would contact the mail server who supposedly originated the message and ask "I received a message that was addressed from user-at-domain, did this message really originate from you? If so, then send me back a hash of the sender's email address and the message ID." Of course, that would impose some additional workload on the servers, but it would put a serious crimp in spoofing and the net reduction in spam would more than offset the increased traffic from the verification process.

 

Yes I think i see a good idea there.

it does go along with what i said that we need to test that the claim matches the truth.

also I would like to see the actual sender have to recive the traffic from delivery failure and from users telling them to stop filling thier inbox with stuff they do not want.

make the bulk sender pay more operating costs automaticly w/o the need for lawyers and courts.

the more it costs them to send the messages the less they will want to send mass emails w/o good demographics and sale per x messages sent rates.

it's one ting to send 100,000 messages and get 3 sales if the send only costs say 1K and you get paid for sending them not for the sales.

say you get paid 0.05 per delivery attempt then you make $4,000

but if the cost is say .25 then you have to raise the price or target the email so that you get a better return rate.

that should change the market to make junk mail less attractive to the companies that generate it.

figuerres said:

mstefan said:

*snip*

Yes I think i see a good idea there.

it does go along with what i said that we need to test that the claim matches the truth.

also I would like to see the actual sender have to recive the traffic from delivery failure and from users telling them to stop filling thier inbox with stuff they do not want.

 

make the bulk sender pay more operating costs automaticly w/o the need for lawyers and courts.

the more it costs them to send the messages the less they will want to send mass emails w/o good demographics and sale per x messages sent rates.

 

it's one ting to send 100,000 messages and get 3 sales if the send only costs say 1K and you get paid for sending them not for the sales.

say you get paid 0.05 per delivery attempt then you make $4,000

but if the cost is say .25 then you have to raise the price or target the email so that you get a better return rate.

 

that should change the market to make junk mail less attractive to the companies that generate it.

To increase the "cost" of sending messages for spammers, one approach is to simply use time as a cost factor. For example, using the system I described above, if the originating mail server "plays nice" with others and validates the messages that it sends, then the recipient's server will immediately deliver valid messages. However, if the originating server refuses to validate the sender (claiming not to support the protocol extension, etc.) then the recipient server quarantines the message for a configurable period of time; say, 3 days. When the quarantine time period is up, the recipient server then performs a secondary check of the originating server against known DNSRBLs before completing the delivery.

This does two things. First, it makes spam more expensive to send in terms of time. Fewer people are going to be interested in spamming if they know that the recpients aren't going to get their emails for days. The second thing is that the delay itself provides a larger window in which a spammer's website can be shutdown and their mail server added to blacklists. Their spamming activities will also be noticed faster because honeypots can be setup to look for servers that don't validate.

Another thing that can be done to increase the time cost for spammers is make it so recipient servers can choose to do things like intentionally increase the transaction time for the message submission if it's coming from a server that has never validated a message for them before; for individual messages, an increase in submission time by a few seconds means nothing. To a spammer, it could cumulatively mean that what they used to be able to do in hours would now take days because every server they're talking to is making them wait a few seconds after every command they issue.

Simply put, make the act of spamming so time-intensive that it loses its economic value.

Clint said:

TommyCarlier said:

*snip*

whitelist systems have been done and in my opinion are extremely annoying.  They force an extra step that semi defeats the ease of email.

White-listing can work, if you do it properly. An example: with our system, you can define rules on the level of the server that (for example) allow all messages from a certain domain to all (or a subset of) accounts on your server. BTW: white-listing is being used more and more. Facebook is a great example of a white-listing system. As is Skype or Live Messenger.

TommyCarlier said:

Clint said:

*snip*

White-listing can work, if you do it properly. An example: with our system, you can define rules on the level of the server that (for example) allow all messages from a certain domain to all (or a subset of) accounts on your server. BTW: white-listing is being used more and more. Facebook is a great example of a white-listing system. As is Skype or Live Messenger.

But white-listing can only work properly if you can't spoof a from-address. And with e-mail, that's incredibly trivial to do (unless you system requires someting like SPF).

To add a pointless anecdote, I always have to tell Outlook to white-list my supervising professor's e-mail address. The man has a tendency to write these really short messages in poor English that nearly always get classified as spam. Half the time the entire message is in the subject field (usually just one sentence + his name) with the message body empty.

mstefan said:

figuerres said:

*snip*

What I'd like to see is really a more stringent form of SPF, but rather than relying on DNS records, to use the mail servers themselves to validate the message. Basically, the recipient would contact the mail server who supposedly originated the message and ask "I received a message that was addressed from user-at-domain, did this message really originate from you? If so, then send me back a hash of the sender's email address and the message ID." Of course, that would impose some additional workload on the servers, but it would put a serious crimp in spoofing and the net reduction in spam would more than offset the increased traffic from the verification process.

 

Our system kind of works like this. All the communication is encrypted. Every account has an RSA key pair, every file or message is digitally signed with the private key of the sender (to verify the authenticity and to tamper-proof) and encrypted with the public key of the receiver (to make sure only the receiver can decrypt it).

Sven Groot said:

TommyCarlier said:

*snip*

But white-listing can only work properly if you can't spoof a from-address. And with e-mail, that's incredibly trivial to do (unless you system requires someting like SPF).

 

To add a pointless anecdote, I always have to tell Outlook to white-list my supervising professor's e-mail address. The man has a tendency to write these really short messages in poor English that nearly always get classified as spam. Half the time the entire message is in the subject field (usually just one sentence + his name) with the message body empty.

I'm not saying white-listing works with SMTP. But it can work if the whole system is designed from the start with security in mind. The biggest problem is still the users. The more secure a system is, the more difficult it is to use (at least with current systems). Making it easier to use usually means lowering the security.

TommyCarlier said:

mstefan said:

*snip*

Our system kind of works like this. All the communication is encrypted. Every account has an RSA key pair, every file or message is digitally signed with the private key of the sender (to verify the authenticity and to tamper-proof) and encrypted with the public key of the receiver (to make sure only the receiver can decrypt it).

In a more general sense, simply forcing all SMTP sessions to use TLS, with both the sender and recipient MTAs requiring a valid certificate would also be another thorn in the hide of spammers. The sender connects securely to the recipient's server SMTP and the recipient then establishes a second TLS connection back to the sender; if either certificate chain can't be validated, then the session is aborted. Spammers would find their certificates revoked, and not only would everyone be rejecting their mail, they'd be out the fee charged by the CA (on the other end, CAs couldn't hand out "test" certificates because those would definitely be abused by spammers).

Personally, I think it's advantageous to keep all of this on the MTA side of the fence so that existing mail clients don't need to be modified; there's a lot more of those out there than there are servers, and requiring a simultaneous change in both servers and clients at the same time would be a non-starter. It'd be hard enough to adopt these kinds of changes on just the server end; requiring it on the client end as well (such as requiring messages to be encrypted) is just too much of a heavy lift.

Edit: Made a slight change there to make it clear that I was talking about the sender and recipient MTAs requiring TLS, not requiring secure connections from MUAs. 

mstefan said:

TommyCarlier said:

*snip*

In a more general sense, simply forcing all SMTP sessions to use TLS, with both the sender and recipient MTAs requiring a valid certificate would also be another thorn in the hide of spammers. The sender connects securely to the recipient's server SMTP and the recipient then establishes a second TLS connection back to the sender; if either certificate chain can't be validated, then the session is aborted. Spammers would find their certificates revoked, and not only would everyone be rejecting their mail, they'd be out the fee charged by the CA (on the other end, CAs couldn't hand out "test" certificates because those would definitely be abused by spammers).

 

Personally, I think it's advantageous to keep all of this on the MTA side of the fence so that existing mail clients don't need to be modified; there's a lot more of those out there than there are servers, and requiring a simultaneous change in both servers and clients at the same time would be a non-starter. It'd be hard enough to adopt these kinds of changes on just the server end; requiring it on the client end as well (such as requiring messages to be encrypted) is just too much of a heavy lift.

 

Edit: Made a slight change there to make it clear that I was talking about the sender and recipient MTAs requiring TLS, not requiring secure connections from MUAs. 

Don't tell me. Convincing people to install a new piece of client software is hard. They're used to their Outlook and often don't want to run separate software with a separate inbox.

TommyCarlier said:

mstefan said:

*snip*

Don't tell me. Convincing people to install a new piece of client software is hard. They're used to their Outlook and often don't want to run separate software with a separate inbox.

Not to mention the huge number of legacy MUAs that are still out there which only implement RFC 821, and not all of those are clients running on a desktop PC. For example, a lot of devices have options to send email notifications, and unlike software on desktops, those devices aren't usually replaced and/or updated frequently (presuming you can even get a firmware update). So requiring any kind of substantive change on the mail client side would just break a lot of code and tick off a lot of users and their admins.

For anything to be really successfully adopted, I think it has to be solely on the server end of things, and it needs to be implemented in a way that penalizes servers that don't comply with the "new rules", but doesn't actually drop the message (at least during a transition period). Instead, the postmaster for the domain gets hate mail from the other servers saying "Hey, update your mail server software or we're going to stop delivering to you eventually".