Reverse MX

Reverse MX (RMX) is an email authentication method developed by Hadmut Danisch. It became a basis for the two most commonly used methods, Sender Policy Framework and Sender ID.

Development of RMX

Background and motivation

Between 1990 and 1998, the author of RMX, Hadmut Danisch, was working as a security researcher and system administrator at the European Institute for System Security (E.I.S.S.) at the University of Karlsruhe, Germany. Subject of research at the E.I.S.S. were both cryptographical (e.g. RFC 1824) and non-cryptographical methods (such as early firewall technology), with special focus on authentication and communication security. As a demonstration of a so called 'organizational security measure', Danisch had developed a scheme to prevent forgery of SMTP e-mail sender addresses, implemented as a complex and recursive rule set for sendmail. The basic idea was to perform a recursive sequence of database lookups with both the sender's IP address and given e-mail address after the 'MAIL FROM' command in the SMTP protocol. The first matching database lookup would tell whether to accept or to decline the message for delivery. At that time, the system was working well under lab conditions and in experimental implementations, but did not yet have a particular purpose, except for the demonstration of security technologies.

Around 1996/1997 the system became practically useful when suddenly an increasing number of spam messages began to fill and jam mail systems and mailboxes. While spam messages had formerly been seen merely on Usenet, spammers now began to systematically collect e-mail addresses and send spam by e-mail. At first, spammers used their own sender addresses, which were blacklisted soon. Later they used random sender addresses, but again this could easily be filtered by querying whether the sender's domain has a valid MX record, i.e. to perform a simple check whether mail could be sent back to the (alleged) sender. Then spammers started to use real domains or e-mail addresses (randomly picked from their lists of collected addresses) to forge sender addresses, thus bringing the first spam storms that could not be systematically detected and blocked by SMTP-based mailsystems. Mailboxes started to contain more spam than regular mail. Since the number of registered domains and personal e-mail contacts was rather small and surveyable at that time, the experimental sendmail ruleset drastically reduced the amount of spam coming into the institute's mailboxes, once the most important sender domains and their legitimate sender machines had been put in the local database (<and optionally been whitelisted).

In 1998, Danisch left the E.I.S.S. and became a security consultant at the first German internet provider, soon being confronted with the increasing number of harsh complaints of commercial internet customers, who's leased lines had been jammed by spammers, and who had, on top of that, been billed for the spam traffic. At that time, internet traffic was expensive and billed on a volume base, and spam could easily increase the costs by ten times or even more, so a technical solution was urgently needed. Although - and because - that internet provider had busted with the dot-com bubble in spring 2002, Danisch was still busy with finding a technical solution.

Beyond that, there was an ongoing intense technical and scientifical dispute between Danisch and the University of Karlsruhe about the nature, the basics, and the principles of IT security in common, and the asserted necessity of cryptography in particular. Therefore, a hard and large scale technical proof of concept in the real world outside the university labs was needed to prove that real, robust, and easy to use security can be achieved by organizational methods without cryptography. RMX was also meant as such a proof of concept, which should work in reality and on a world wide scale, in contrast to the science methods usual at the German university, where security science was seen as strictly limited to paperwork or maybe a lab.

Design criteria

The development of RMX was based on the following design criteria:

• The main strategy to fight spam was to not try to detect spam by content, but to prevent forgery of sender addresses in order to enable domain based black- and whitelisting, reputation databases, easier tracking of spammers.
• There should be no governmental or centralized measures. Measures should be implemented and controlled by sender and receiver of a message without interaction with any third party.
• The mechanism should conform to German and European data protection regulations, and not violate the secrecy of telecommunications.
• The mechanism should be smart, simple, and robust, easy to understand, easy to test and debug, easy to repair. Common knowledge typical for mail and domain administrators should be sufficient to work with the system.
• The system should be cheap and durable, no costs, no fees, no costly subscriptions and updates.
• The system should be implemented on the mail relays (MTA) without the need to interfere with the end user mail programs (MUA).
• The system should avoid cryptography for several reasons:
  • The system should in principle be resistant against bots, hackers, and malware. Since this obviously cannot be achieved in common, because an authorized and infected system can by design send spam, the transport of spam should be immediately and reliably be terminated once the system is cleaned or taken down after notification (in contrast to, e.g., cryptographic systems, where secret keys could have been leaked and fast methods to revoke and replace keys where required).
  • Cryptography is secure on small scale use and within closed user groups only. There are about a billion of internet users world wide and millions of domains. Even if only 0.1% of the secret keys of these domains were compromised (which is an extremely low estimation), it would still mean thousands of domains that can still be used by spammers.
  • Cryptography requires the key holders to cooperate and to behave well. There is no way for a receiver to detect whether the sender intentionally revealed his key to spammers.
  • It must be possible to easily and immediately verify the integrity of the system (in contrast to, e.g., cryptographic systems where you cannot verify that no unauthorized party has knowledge of a secret key).
  • It must be easy to implement by programmers, no skills required beyond regular internet programming.
  • It should be available all over the world, even in those countries that ban the use of cryptography or the possession of secret keys. The system should not use cryptography or provide any infrastructure that could be used for secret communication, that could prevent the world wide use and legality of the system. No country should be excluded.
  • It was meant as a proof of concept for non-cryptographical security methods.
  • Cryptography is error prone and complex. Keep in mind that even today, more than 30 years after the invention of public key cryptography, there is still no common use of cryptography, and the use and configuration of those services, that are in use such as HTTPS, IPSec, PGP, X.509, S/MIME, are complicated to use and require experts or specially trained users. Any cryptography based system would necessarily mean lots of misconfigurations and lots of lost legitimate mail. Even if it worked, the overhead of using cryptography makes it expensive.

The concept of reverse MX records

The old sendmail based system suffered from a major shortcoming. It was the receiver's task to continuously keep his database of the systems authorized to send mail from a given domain up to date. This is unfeasible, since the receiver cannot know which system the owners of all domains consider as authorized to send mail from their domains, and it would require continuous maintenance.

The basic idea was that the owner of a domain publishes a list of all systems authorized to send e-mail from that domain. The receiver would query this list when receiving a message or on a regular base and verify, whether the sending machine is authorized to do so. This is a typical task for a distributed directory service, and the only world wide established domain-based distributed directory service is currently DNS.

DNS is already used for mail delivery, it lists the authorized receivers in the MX record for a given domain. A first approach would be to use the MX records for senders as well, but at a first glance it can be seen that sending and receiving machines are not identical, and MX records cannot be used for senders at the same time. Furthermore, sender lists are far more complex than receiver lists and can include large networks, e.g. a full /8 address range (or even more with IPv6). While a list of receiving machines can be just a list of some machines, the list of sending machines can contain network address ranges.

Another problem is the problem of delegation to remote networks, such as e-mail service providers. In the case of MX records, delegation is achieved by using a symbolic name of a domain under control of the delegate, who can then assign arbitrary IP addresses. In case of the sender, the delegation of complete lists of different network ranges would be required. So none of the existing record types would have these properties, and a new record type was designed to list the authorized senders of a domain. Since it does the reverse task of an MX record, it was called Reverse MX Record (RMX).

Publication at the IRTF and IETF

By incidence and due to the dramatical increase of spam sendings and the lack of solutions, the IRTF had opened the Anti Spam Research Group (ASRG) and a mailing list, which remained completely silent. In fact, the announcement of the first RMX draft was the very first message on that mailing list, and initiated an extreme intensive and longlasting debate and discussions on the mailing list. Thus RMX was the starter of the IRTF and IETF research on spam fighting and the first approach to establish a world wide spam and fraud protection system. Until then, only commercial solutions limited to some customers, basically subscriptions of lists of spam patterns, existed, comparable to virus filters. (See the bibliography for the RMX drafts submitted to the IETF.)

Technical description

Overall principle of RMX

The idea of RMX is to detect forging of sender addresses used in e-mails and their transfer. While RMX can be used for both the sender's address given in the e-mail header (RFC 2822) and the envelope address (RFC 2821), RMX was intended primarily for and would be most efficient with the envelope sender address. After transmission of the MAIL FROM SMTP command, the receiving mail relay would perform a DNS lookup for the RMX record(s) for the DNS name that was used as the domain part (what's on the right side of the @). These RMX records would describe a list of machines authorized to send e-mail with that given sender domain. In other words, the owner of a domain can publish a statement about where e-mail from his domain can legitimately come from. If the sending machine is listed in the RMX record, the mail is accepted. Otherwise, depending on the receiver's policy, the mail could be rejected right after the MAIL FROM command, or tagged as spam. In simple words, MX records tell, where mail for a given domain should go to, and RMX records tell, where mail from a given domain should come from.

To be precise in terms of security science, RMX is an authorization scheme, and DNS is it's database. The identity of an entity is it's IP address, and the underlying (hidden) authentication method is the TCP handshake needed to establish a TCP connection for SMTP. Due to the TCP sequence number used in that handshake, it is difficult to forge the sender's IP address. While this is not a strong security mechanism, it is cheap, robust, and appropriate to reduce spam.

Implementation as binary DNS records

When using DNS as a directory service, it is important to keep DNS records as small as possible, since there is a (historical) limit of 512 bytes (not counting the IP or UDP headers; see section 4.2.1 in RFC 1035). Although DNS allows to use TCP for DNS queries and replies as well, TCP considerably slows down DNS, and many firewall rulesets would not let DNS TCP traffic through. Therefore, there are two methods to keep DNS records as small as possible: The compact binary encoding of the records, where the encoding scheme is determined by the resource record type and type number, and a compression scheme for DNS names. In contrast to its successors like [SPF], which used TXT records and plain text, RMX was originally designed to use both methods, the compact and DNS-like binary encoding, and the DNS compression scheme for DNS names. Due to technical flaws in DNS explained below, the use of compression had to be omitted.

Structure of RMX records

An RMX record is basically a simple list of entries (a sequence). Since DNS allows a domain name to have more than one record of the same type, it could have more than one RMX record. This is interpreted as a concatenation of these records into one large RMX record. However, DNS does not maintain the order of records, and therefore multiple RMX records must be avoided unless order does not matter. As usual with common security rulesets, these entries are processed in sequence, and the first matching entry determines whether the sender, identified by its IP address, is authorized or not. To do so, every entry can be negated. Positive entries declare permission, negated entries declare non-permissions. These entry types were supported:

unused

This domain will not be used for sending e-mails, no sender can be authorized.

ipv4 and ipv6

The entry contains an IP address and a CIDR mask length, to authorize a single IP address or address range.

DNS hostnames

The entry contains the DNS name, which has to be resolved to its A or AAAA record in a subsequent DNS query. This allows to delegate authorization to someone else and to authorize machines with dynamically assigned IP addresses (with DynDNS).

APL reference

RFC 3123 introduces DNS APL records, which are basically lists of IP address ranges. This also allows to delegate authorization to someone else (e.g. an e-mail service provider or a company branch).

Domain member

This entry type does not take parameters. It means, that the reverse (and verified) DNS name of the sender's IP address belongs to the domain, thus effectively authorizing all machines that have a domain name of the same domain.

Full Mail Address Lookup

This entry type does not take parameters. It means, that the RMX lookup is to be done with the full sender's email address instead of the domain part only to allow per-address granularity.

MX Reference

This entry type does not take parameters. It means, that all MX hosts of the domain are also authorized to send mail.

In addition, RMX had some experimental entry types:

TLS fingerprint

This authorizes a sending machine that initiated SMTP with TLS encryption, identifying itself with a client certificate matching the fingerprint.

TLS and LDAP

Verify the sender's certificate with an LDAP lookup.

SASL

The sender has to authenticate through SMTP SASL mechanisms.

PGP or S/MIME

Accept messages with a content signed by the given key (which differs from all other entry types in that it authorizes the content, not the sender).

Limitations of the Reverse MX approach

Reverse MX is actually not a method to detect spam. It does, by design, not (except for the experimental entry types based on signatures) take any part of the e-mail message itself (except for - optionally - the From: header entry) into consideration. RMX is solely based on the IP address of the sending relay and the sender's envelope e-mail address as given in the SMTP MAIL FROM command. That way, RMX can deny a message before transmission. In contrast to many other anti-spam proposals like greylisting, content-based heuristics, or external blacklists, RMX is fast, precise, predictable, and easy to debug. On the other hand, RMX has serious limitations:

• RMX provides protection against forgery only. Unforged messages will never be blocked or marked. Spammers who legally use their own domains are not affected by RMX.
• Since RMX is an authorization scheme only and is based on the (weak and cheap) TCP handshake authentication of the sender's IP address, it does not protect against the forgery of the IP address itself or other ways to take it over. Installing malware like bots or TCP relays on an authorized machine will still allow to send messages on behalf of the alleged sender.
• Since RMX relies on DNS, it is vulnerable against DNS attacks like cache poisoning.
• With the RMX records, RMX reveals the IP addresses of the sending machines to the public. If these addresses should be kept secret, either a central relay should be used or other measures should be negotiated with those authorized to know the sending SMTP addresses. Alternatively, symbolic DNS names can be used that can be resolved over the local hosts files only.
• Because of this sort of unforged spam, RMX requires (and enables) additional black- or whitelisting. Black- and whitelisting require protection against forgery for being effective and reliable.
• Automatic mail forwarding has become very common in the internet. The recipient's relay then rewrites the envelope recipient address and forwards the message as if it were a regular part of the delivery chain. Since this way of forwarding is undistinguishable from forgery of the sender address, RMX breaks this simple forwarding, but still allows forwarding in principle. When forwarding, the forwarding relay must verify the RMX entry of the message's origin. The final recipient's relay must have the forwarding relay whitelisted as trusted in common, rely on it having RMX previously verified, and bypass its own RMX routines.
• RMX can be effective even when only a very limited number of domains support it. E.g. a mailbox can whitelist all messages from a given domain, if its RMX records prevent forgery. But to become effective against spam in common, RMX entries would have to be used on a world wide base, or alternatively for all hosts of a given top level domain (like .com). An important prerequisite would be that mail relays could deny mails from domains without an RMX entry or with unplausible wide RMX entries (like allowing mail from 0.0.0.0/0).
• RMX does not really support domains where a large number of users sends e-mail from their personal computers outside a central area, and thus from remote and changing IP addresses. Such a domain would basically authorize anyone to send e-mails and thus defeat the idea of sender verification based on IP address and make spamming easy. RMX protects the traffic between domains, but relies on the fact that the domain authority itself does the job of authentication and protecting against abusive mails. Such domains should therefore maintain central mail relays where users or machines deliver their outgoing messages to and authenticate with any method chosen by the domain administration, like SMTP password authentication.

Reasons for failure

While RMX technically worked exactly as expected in the lab, it never succeeded in reality for various reasons. More can be learned from why RMX failed and security mechanisms in general fail than from the way it actually worked.

Shortcomings of the DNS design

DNS was designed between 1983 and 1987 (RFC 882 and RFC 883), and it's design matched the state of the art of programming of that time. Although DNS was principally extensible with new record types, DNS lacks the ability to transport and process new data types without being explicitly extended for these data types (in contrast to e.g. LDAP, which can be extended to new objectClasses by configuration and handles all data encoded as ASN.1). As a consequence, four major design and implementation details prevent DNS from being easily extended:

• Authoritative DNS servers need to be explicitly extended to be able to handle new DNS resource record types. Without a software update, a DNS server will not be able to serve new record types. Upgrading the internet's DNS infrastructure to new software within limited time is virtually impossible.
• DNS cache and resolver implementations originally did not forward unknown record types, and would require software update for the same reason. Only newer implementations are able to forward unknown record types.
• DNS UDP messages are by design limited to 512 byte. Since RMX records would tend to be rather complex and to refer to other record types such as A, MX, or APL records, which could be transported in the same message as additional answers. It is therefore essential to keep messages short by making use of the DNS name compression scheme. Unfortunately, compressed records require decompression and recompression for caching and forwarding, and thus binary de- and reencoding. Even if a DNS server would forward unknown new record types untouched, it would break the compression and make the record unusable.
• Client side DNS implementations do not support querying arbitrary record types through the standard API, and tie each known record type to a particular API function such as gethostbyname(). Implementing the client side for new record type would require either an extension of the C/Unix or POSIX runtime environment, or move the implementation details into the application program (usually any mail transfer agent) and cause incompatibilities and use of undocumented DNS functions.

While the correct and intended way to transport new types of data in DNS would be to define new record types, these characteristics of the DNS make it more than difficult, to deploy on a world wide scale. Even for experimental purposes it would require all DNS servers and libraries involved in the tests to be patched and recompiled every time the record type definitions change.

To overcome this problem and to be able to at least perform experiments easily, the IRTF working group proposed to use (existing and implemented) TXT resource records, and to use plaintext instead of binary encoding, allowing to use any DNS servers and resolvers without update. This was named SPF (see below). However, this method significantly increased the length of DNS answers and could easily exceed the size limit for DNS UDP packets, forcing DNS into the much slower TCP mode (which is not even allowed by many firewall configurations).

Problems of the SMTP e-mail protocol and its implementations

During the experiments made for the Anti Spam Research Group of the IRTF and IETF, further complications showed up. Although the protocol definitions in RFC 821 and RFC 2821 seem to be rather precise, they were not precise enough, and too many SMTP implementations did not conform exactly to the specs. Especially the different ways to forward e-mail from the first recipient's mailbox to other mailboxes, a common practise in the internet, caused problems. Usually, mail forwarders simply replace the recipient's envelope address (and sometimes the address given in the header's To: or Cc: field), but not the sender's address. That way they act on behalf of the original sender without the sender's knowledge and approval, and without taking any care about how the final recipient could verify the origin or trust the forwarder. Technically, there is no difference at all between a forwarder and a spammer forging the sender address, acting the very same way as a forwarder. The process of forwarding of e-mail had never been precisely defined.

Another problem was that the RMX principle primarily focusses on and verifies the envelope sender address. While most traditional mail transport and user agents in the Unix environment preserve and display the envelope sender address (particularly in the first "From ..." line in the mbox format), some commercial e-mail systems turned out to not be completely compatible with the SMTP world. Some of them internally used proprietary message and storage formats, that did not support the SMTP idea of having and storing separate header and envelope addresses, making it impossible to verify an envelope sender address that is not preserved.

Further discussion showed that due to lack of a sufficient precise and enforced definition of the e-mail protocols, the worldwide e-mail infrastructure had grown uncontrolled and developed a huge variety of flavors, dialects, and versions of SMTP, its extensions, and background mechanisms. Maybe this can be compared to the unmanageable difference in HTML formats and interpretations we had for several years with the browser types (until stricter definitions had been published).

Spam and forgery could be seen as trivial exploits of fundamental security flaws of SMTP. IT seems as if the deployed worldwide e-mail infrastructure can neither easily be fixed nor replaced with or updated to better and more secure protocols. Most probably, the worldwide internet infrastructure has grown that far out of any central control, that the problem of spam and forgery cannot be fixed in reasonable time and on a worldwide scale anymore. Maybe the next chance to fix these problems will be when SMTP in common or even the Internet protocol itself will be replaced by more modern and secure protocols.

The asymmetry of the POP and IMAP protocols

Originally and by SMTP design, mail routing was basically static and symmetric. Sending email from A to B would usually take always the same way through the same (group of) relays, and sending a message back from B to A would in most cases take the same way backwards. E-mail users were usually reading e-mails directly on those relay machines.

The invention and spreading of personal computers, that are not always powered on and online, and the availability of home internet access based on dynamically assigned IP addresses, changed this dramatically (and at least temporarily). Two new protocols for pulling e-mails from the last regular relay in the e-mail routing path to the end user's machine were invented, POP and IMAP, and became standard for e-mail retrieval. Unfortunately, both protocols cover downloading of incoming messages, but not uploading of outgoing messages, which should be seen as a severe design flaw. Sending messages would still require use of SMTP.

At the time when RMX was proposed, SMTP authentication was not yet common and available in software, although SASL authentication had been introduced with RFC 2554 in 1999. (See RFC 4409 as well.) Although some mail providers started to use SMTP after POP as an authentication method, many organizations did not really care about how their domain users were delivering e-mail and authentication. They offered POP or IMAP for receiving messages, but left it completely to the end user with dynamically assigned IP addresses, to deliver outgoing messages directly to the world wide internet over SMTP and without any authentication. Obviously, this defeats any attempts to limit the use of the organization's domain, as it by design leaves sending open to the world.

Soon after the proposal of RMX, several universities complained that RMX would break their infrastructure and cut them from the e-mail system, since thousands of their members would send e-mail from home over their private internet access lines, often tens or hundreds of miles away from their university, or even from any point on the world when traveling, involving an unsurveyable number of internet providers and IP address ranges. The typical e-mail system of the typical university would be a one-way system for the incoming mail only, and leave it to every user to deliver by SMTP on her own. Therefore it would be necessary to allow the use of the organizations domain from any IP address. Which was actually not an infrastructure but an absence of.

Although RMX in principle supported such schemes by simply having a 0.0.0.0/0 entry and leaving it on the recipient's decision whether to accept messages from domains open to the world for abuse, those universities (i.e. their admins) strictly declined RMX. It is important to understand, that RMX-like schemes force the network administrators to organize their e-mail systems and to give a precise and public statement about where e-mail from their domain comes from. Not all administrators or domain owners were able and willing to do so, for various reasons. Apart from RMX, these structures and administrators, and intentionally leaving domains and networks open for abuse, were a major problem of fighting spam and abuse in common. At least, RMX would allow to detect such domains by their too wide open RMX record, and allow the recipient to decide whether to accept e-mail from such poorly maintained domains. That's a major reason why RMX-like schemes were declined.

Interestingly, today most of these e-mail structures are gone. Spam traffic from dynamically assigned IP addresses was growing fast, forcing mail administrators to block mails from dynamically assigned numbers, and forcing organizations to provide local relays for their members. After a phase of experiments like SMTP after POP, SMTP authentication with SASL or TLS became common and supported by most mail user programs. Today it is state of the art to provide e-mail end users not only with POP or IMAP, but with an SMTP access point with password authentication as well.

Cultural reasons and perceptions of identity

Communication security is usually seen as a technical discipline of computer science and engineering. The RMX concept was, however, facing harsh resentments with a rather cultural background. The influence of culture and mentality on IT engineering and security is significantly stronger than expected.

The RMX principle was blamed to violate the freedom of speech, as it could restrict the sending of e-mail messages to mailboxes or allow systematic filtering (which was, after all, the idea of RMX in particular and anti-spam measures in common). This criticism was perplexing, since RMX had explicitly been designed to not touch or take the message content into account in any way, specifically to avoid filtering of particular contents, and to avoid censorship. RMX was designed to be usable and deployed on a world wide scale, and therefore to not use any sort of cryptography, that could violate local laws and prevent the acceptance, and to not provide an infrastructure for censorship (like most spam filtering methods do). However, an astonishing large number of people claimed that the freedom of speech included the right to forge anyone else's e-mail sender address and to speak on behalf of someone else without his consent. This point of view directly contradicts any method of sender authentication or spam fighting. If every aspect of spam is seen as covered by the freedom of speech, then any method of reducing spam would violate. This raised intense discussions, and indeed many people took up the position that the freedom of speech guarantees the right to write anything in anyone's mailbox, and that it is solely the reader's task and problem to cope with that. It was claimed that anti spam measures must be restricted to methods to support the reader while reading the messages, as the only way to not violate the freedom of speech. In contrast, RMX was based on the (European) point of view, that the freedom of speech does not allow to force people to listen, to provide communication bandwidth, and to pay the costs.

As a consequence, some of the critics demanded to invent a central reputation database. The idea was, that every mail of whatever content should reach the recipient's mailbox. While reading the messages, the reader should be supported by the reputation database, telling whether a sender was worth of reading his messages. While this scheme may match the American culture and mentality better, the advocators of that principle could, ironically, never explain how it could work without reliable sender authentication. If it is possible to forge other people's sender address, it would be possible to send spam on behalf of someone with a good reputation, or to systematically drive someone's reputation down by sending spam, thus inventing censorship. Beyond those technical problems, such reputation databases would infringe European data protection law. It would not be allowed to transmit reputation scores outside Europe.

Another cultural clash was the perception of identity. In Europe, people are usually much closer tied to their name and public identity, than in other countries, especially the United States. Although the RMX scheme was not based on individual authentication, but on verification of the domain part of an e-mail address only, several people felt attacked and affronted by a European bringing an authentication scheme to American users, and unobjective and abusive analogies to the German history had been voiced. It should be noticed that these objections had not been raised when the very same proposal was later on made again from American authors under the name SPF.

These discussions showed, that security design must, if it should be international, concern large parts of the population, and involve their every-day habits, always respect cultural and legal characteristics for being accepted. As a result, it could be impossible to find a single anti-spam mechanisms applicable to and effective for the world.

Economical reasons

From the very beginning, RMX was heavily antagonized by industrial lobbyists, which turned out to have extreme influence at the IRTF and IETF. On one hand, some people heavily agitating against anti-spam measures could later be linked to commercial spammers, trying to sabotage any attempt to impede spam. On the other hand, many opponents were involved in the business of subscriptions of commercial spam filters that have to be updated daily like virus filters. These subscriptions are a billion dollar market that would have been dried out by every simple and free anti-spam measure. There was an intense lobbyism at the IRTF and IETF to bring down efforts against spam. One provider of commercial spam filters even put the author and other disputants on his filtering lists to keep the discussion about his business model away from his customers, who could then see only parts of the discussions on the mailing lists.

SPF and IRTF/IETF-specific reasons

Perhaps the most important reason for failure were the IRTF and IETF themselves. Although RMX initialized the ASRG's work and provided the main principle the ASRG was working on for years, the ASRG, and later on the IETF made clear to be unwilling to accept an anti spam mechanism coming from outside of North America.

A first scandal, causing intense arguments and complaints in the ASRG, was the ASRG meeting on the 56th IETF conference in San Francisco in 2003. At that time, spam was a major subject of the public discussion, and a solution was badly needed. The meeting was - very unusual for an IETF session - heavily crowded with accredited journalists, radio and TV reporters. The IETF chair however abused his role to show himself and friends up in front of the cameras with pointless talks, while not mentioning the ASRG's several month's work including RMX at all.

Soon after the first discussion about RMX, the ASRG mailing list was flooded with similar proposals. One of these proposal was SPF, which was at that time explicitly introduced as an almost direct copy of RMX. The major difference was that it did not use binary encoding and put the sender statement in DNS TXT resource records. At that time, SPF was named Sender Permitted From, which even indicated the use of the RMX principle. While the ASRG had asked the author of RMX, to further support the ASRG on a closed mailing with all the authors of these slightly differing proposals, the chair of the ASRG gave an interview to the Washington Post incorrectly telling that the ASRG's anti spam method was SPF and had been invented by the SPF author, which soon spread to journals and newspapers around the world. When the author of RMX complained, the ASRG chair denied to have given an interview and to be reliable for the Washington Post's article, but the Washington Post proved that the interview had been given. The ASRG, that formerly had declined all complaints as unsubstantial, then silently replaced the chair, but did not react or comment otherwise or corrected its public statements.

Since then, the method was intensively published and advertised as SPF and as made by an American company. Taking over that intellectual property was justified with the claim that the (European) author of RMX had not been present enough on US conferences to advertise RMX, which was considered as giving up the property and leaving it to be picked up by others.

While the protagonists of SPF were good advertisers, they could not agree on a common strategy about SPF's future. SPF diverged into several versions with uncontrolled growth, gathering more and increasingly farfetched functions and extensions, causing SPF to lose its shape. People finally began to deny SPF due to its confusing and sprawling structure, and SPF then fell back to the RMX-like state, then called 'SPF classic', thus questioning itself.

Later on, after the ASRG had been transformed into the IETF MARID working group and dramatically failed, the IETF published all proposals derived from RMX as RFCs for historical reasons, except RMX itself. Opinions were voiced that the ASRG had never been anything but a vehicle for advertising people and commercial interests.

Microsoft's role

Another reason for failure was Microsoft's role. Around the time when RMX was introduced, Microsoft had publicly announced to solve the spam problem, but did not have a proper method to do so. After the RMX proposal was published, Microsoft adopted the principle, but had to modify it because of compatibility problems between Microsoft's mail software and SMTP, and published it as 'Caller-ID' as an analogon to the caller's phone number displayed when a phone rings, allowing the callee to decide whether to accept the call, but still having sever shortcomings. Microsoft then merged SPF and Caller-ID into a new proposal named Sender-ID, and applied for a patent. Based on that patent claim, Microsoft announced to license the method for free to open source mail relay software such as sendmail and postfix, but to insist on every admin using that software on applying by fax for a license.

Microsoft's claim provoked heavy reactions in the open source scene, and many authors of common software publicly denied to support any anti-spam mechanism under these circumstances. Since it was unclear, on what exactly Microsoft had claimed a patent for, authors denied not just Sender-ID, but any method that could be under the risk of being a target of patent claims. That conflict culminated in the meeting in the 60th IETF meeting in San Jose in 2004 with a full-day controversial dispute between Microsoft and open source authors, and ended with the final rejection of Sender-ID and all related methods, including SPF and RMX. This conference was the point were the RMX proposal and all its successors had finally failed to be accepted.

Personal and European reasons

As said above, the author of RMX had been blamed for not being sufficiently present on conferences to publish RMX. RMX had never been funded in any way and was a completely private project of its author, and therefore limited by affordability. Typically, at the time when RMX was proposed, there was no support at all from the German government, German universities, or the European Union. While spam fighting was a matter of extreme public interested in the united states and attracted scientists, industry, journalists, and politics, it was seen as irrelevant by European officials. Funding was not available at all, and German scientists considered the method as not scientific and therefore as unremarkable. In fact, RMX had been build as a proof of concept to show, that robust, easy to use, and world-wide deployable security mechanisms can be built without cryptography and complicated maths, which had been denied by German universities.

Ironically, as a side effect of this dispute between the RMX author and a German university and on the RMX author's demand, a German upper court found that it is a crime under § 206 of the German criminal code to suppress e-mail in a university network. Although this applied to regular e-mails blocked by university officials only, and not to spam, it provoked international newspaper articles about Germany would prohibit filtering spam.

When spam became a public annoyance, the German federal parliament discussed how to get rid of spam, and whether filtering spam could be in conflict with the court findings. The parliament, however, did not find a way to reduce spam, and still today there is no common spam protection effective in Germany. Although meanwhile the German government began to improve network security, it recently stated that from the government's point of view IT security is when security business and revenues are growing up. In contrast to filtering subscriptions or mail service providers, RMX does not raise revenues and therefore would not be considered as desirable security in Germany, although developed in Germany.

References

(See the bibliography as well.)