According to recent statistics, around 90-95% of email traffic around the world is spam. The disruptive impact of spam and other email abuse has resulted in two types of approaches to deal with the menace.
- The first approach focuses on detecting and filtering problem messages. This is done by techniques that attempt to identify bad mails using pattern detection and based on other facts about the contents in the email.
- A complementary, but quite different approach, seeks a basis for trusting a message rather than for mistrusting it. This is done by profiling and identifying the sender and validating the sender’s authenticity.
Rediffmail uses a combination of both the techniques to filter out unwanted mails and keeps the users mailbox free from spam. The Rediffmail spam control system is like a funnel, which filters out unwanted messages at various stages and delivers only the good messages to the user’s ,mailbox. Here is a list of the different modules in the Rediffmail spam control system:
IP based filtering
This is the first level of filtering. At this stage, the reputation of the sender’s IP is established, based on which messages are accepted, rejected or deferred. Generally messages are rejected which originate from IPs that are known to send spam, malware or belong to machines that have been compromised or are open relays. Message are deferred which originate from IPs that tend to send Bulk mail in large numbers or from IPs that send unsolicited mails that have not been requested by the recipients. Rediffmail uses a combination of internal as well as external sources to generate the IP reputation database.
Domain Signatures / RDNS and Fake NDR filters.
At this stage, the spam filter tries to establish the reputation of the sender by validating the sender’s SPF, DK, DKIM and RDNS records. Mails from senders with incorrect signatures or with non-existent RDNS records are filtered at this stage. Spammers who send mails by faking sender ids result in lot of Non Delivery Reports (NDRs). Though the NDRs are not really spam messages, but they can be a cause of nuisance for users who receive a lot of such messages. Rediffmail uses a home grown technique to identify and block such fake NDRs and not let them reach the users mailbox.
Content based filtering
At this stage, the spam filter uses techniques which try to finger print messages and try to classify messages based on the patterns in the content of the email. Finger printing of messages is done by examining patterns in messages, rather than just the content and comparing them against the corpus of bad and good patterns that have been built over a period of years of learning. The filter at this stage also considers recurrence of patterns over short time spans to identify unsolicited bulk messages. One of the techniques used to filter out bad messages is to examine the nature of URLs in the content. Rediffmail uses a combination of internal as well as external sources to build a database of bad URLs. The corpus of bad patterns and URLs is also generated based on feedback from other users who report messages as spam. At the end of this stage, messages are classified as containing malware, phishing mails, other scams like lottery scams, etc.
Reputation based filtering
This is the final stage of filtering in the spam control system. At this stage, the reputation of the sender is established with respect to the recipient, based on the actions taken by uses on the emails that they send and receive. Messages from senders having a bad are filtered out and messages from senders having a good reputation are accepted and delivered to the user’s mailbox. This technique to do so has been built and grown in house. Apart from this Rediffmail has a dedicated team which works on daily Spam control issues, such as detecting False Positives, monitoring the classification of messages, monitoring the reputation of IPs, etc. Over and above the above spam filtering funnel, the administrator of domains can control the SPAM filter or individuals as well as for the entire domain. Administrators can enable/ disable spam filters as per the requirements and also enforce restrictions on incoming & outgoing mails with respect to senders, recipients, attachments, etc.
So do you have all these in your built-in Anti-Spam & Anti-virus system in your current email service? If not you may be in for some serious trouble.
Rediffmail Enterprise is a CIO choice award-winning business email service for small, medium & large enterprises with spam & virus protection of global standards. A representative of Messaging anti abuse working group (MAAWG, an international organization responsible for addressing messaging abuse holistically by knowledge sharing, industry collaboration and public policy) who has contributed to address issues relating to SPAM originating from India. Rediffmail Enterprise’s multi-layered SPAM filtering mechanisms help filter out up to 99.9% of SPAM messages and known viruses trying to reach user’s inbox.
