Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Topics - Netino

Pages: [1]
1
How to / [Howto] Install Rspamd
« on: January 07, 2019, 12:08:43 AM »
Howto Instal Rspamd Antispam

This howto describes the main steps to get and start working with Rspamd, in particular with the following setup:
- CentOS 7
- Postfix MTA
- Redis cache
- Dovecot with Sieve plugin to sort mail and learn by moving messages to Junk folder

Attention: Use this how to at your own risk and make a backup before trying it. This howto should only be used for advanced users.

Rspamd is a fast, astonishing fast, and light open source spam filtering system which utilizes multiple techniques to prevent spam from reaching your mailbox.
I simply couldn't install stably spamassassin in the server. Looking for alternatives, I found it, and it seemed much better in almost every aspects.

Rspamd is actively developed and appears to be a much more modern project. Written in C, it is quite fast and integrates directly into postfix as a milter. It uses bayesian filtering and machine learning to learn what users consider to be spam and ham, global, by domain and by user. However, it also incorporates much many other anti-spam measures: RBL checks, SPF/DKIM/DMARC/ARC validation, DCC bulk mail checks, antivirus checks, and greylisting, to name a few. In addition, it has some other useful features like DKIM signing of outgoing mail, automatic whitelisting when you reply to someone, and a web interface where you can see spam checking results in real time.

Rspamd is an advanced spam filtering system supporting a variety of filtering mechanisms including regular expressions, statistical analysis and custom services such as URL black lists. Each message is analysed by rspamd and given a spam score. According to this spam score and the userís settings rspamd recommends an action for the MTA to apply to the message- for example to pass, reject or add a header. Rspamd is designed to process hundreds of messages per second simultaneously.

It replaces amavisd-new, spamassassin, opendmarc, Policyd-SPF in just one software.
I could save about 1.0 Gb of memory with just this replacement, and Rspamd now takes up only 98Mb of memory, ie 10 times less !.
Each E-mail was analyzed in one to two minutes, and is now analyzed in 2 to 3 seconds!
I've been using it for over a year now.

What you need to know:
-General: all information below are important but need to be checked.
-General: this howto was made using CentOS7 server x64, so there could be differences to other distributions, and must be adapted some paths!
-Rspamd: Do not edit any default configuration file in /etc/rspamd/. Rspamd is designed to load configuration files from the /etc/rspamd/local.d/ folder, where is store customizations. Take a look into the documentation for available configs and parameters.
-Spamassassin: It is needed to disable spamassassin, but you can import almost all your spamassassin rules, include custom rules, and use it in Rspamd. To migrate, more information at <https://rspamd.com/doc/tutorials/migrate_sa.html>.
-Redis: This installation is really is not needed, but is strongly recommended, is a *astonishingly* fast database. If for some reason you won't install, supress all steps 'for redis use' in configuration. The steps 'with redis use' are affected by redis use, and must be adapted before you use it. The other steps are not using redis, or are unaffected by redis.
-Opendkim: Despite this configuration no longer use opendkim, as CWP uses it as the "default" configuration for automatic domain creation with DKIM support, the proposed configuration here makes use of the DKIM keys installed by opendkim, so I tried to maintain the structure of the DKIM key files for you can use it normally, preserving the autocreation of DKIM keys at CWP, so you do not need to make any changes to the existing structure, just by including the user '_rspamd' in the opendkim group.

Installing and Configuring Rspamd
=================================

Access ssh as root in your server.

- Install Rspamd, as: https://rspamd.com/downloads.html, executing these commands as 'root':
Code: [Select]
# curl http://rspamd.com/rpm-stable/centos-7/rspamd.repo > /etc/yum.repos.d/rspamd.repo
# rpm --import http://rspamd.com/rpm-stable/gpg.key
# yum update
# yum install rspamd
(don't start it yet)

- Install Redis
Code: [Select]
# yum install redis
# systemctl start redis
# systemctl enable redis

- Test if is Redis is working. Execute:
Code: [Select]
# redis-cli ping
the result must be 'PONG'.

- Change Redis configuration in file /etc/redis.conf:
Code: [Select]
maxmemory 500mb
maxmemory-policy volatile-lru

- and after change the system configuration: (for redis use)
Code: [Select]
# echo 1 > /proc/sys/vm/overcommit_memory

- add at file /etc/sysctl.conf (for redis use)
Code: [Select]
vm.overcommit_memory = 1

- and update the system configuration:
Code: [Select]
# sysctl -p

- ... and restart redis:
Code: [Select]
# systemctl restart redis

- Create the file /etc/rspamd/local.d/redis.conf with the following content: (for redis use)
Code: [Select]
servers = "127.0.0.1";

- file /etc/rspamd/local.d/options.inc
Code: [Select]
dns {
  enable_dnssec = true;
  timeout = 4s;
  retransmits = 5;
}

- file /etc/rspamd/local.d/worker-normal.inc
Code: [Select]
bind_socket = "127.0.0.1:11333";

- file /etc/rspamd/local.d/worker-proxy.inc
Code: [Select]
bind_socket = "127.0.0.1:11332";
milter = yes;
timeout = 120s;
upstream "local" {
 default = yes;
 self_scan = yes;
}
[code]

- file /etc/rspamd/local.d/logging.inc
[code]
type = "file";
filename = "/var/log/rspamd/rspamd.log";
level = "error";
debug_modules = [];

- Create the user password (use your own password instead 'P4ssvv0rD')
Code: [Select]
# rspamadm pw --encrypt -p P4ssvv0rD
$2$htwknhydfj45j58nuej1kffpegykzmer$i9pup6hpz3izzz3iqi99kohokmtfbnoh1k1oz3ph33xio6sgr41b
... and use it in the below file.

# /etc/rspamd/local.d/worker-controller.inc
Code: [Select]
bind_socket = "127.0.0.1:11334";
# password for normal user
password = "$2$htwknhydfj45j58nuej1kffpegykzmer$i9pup6hpz3izzz3iqi99kohokmtfbnoh1k1oz3ph33xio6sgr41b";
# password for 'admin'
# create this admin password with the rspamd utilities
enable_password = ""
secure_ip = "127.0.0.1";

- file /etc/rspamd/local.d/classifier-bayes.conf (for redis)
Code: [Select]
servers = "127.0.0.1";
backend = "redis";
autolearn = true;
new_schema = true;
expire = 8640000;

- file /etc/rspamd/local.d/milter_headers.conf
Code: [Select]
use = ["x-spamd-bar", "x-spam-level", "authentication-results"];
authenticated_headers = ["authentication-results"];
extended_spam_headers = true;

routines {
  spam-header {
    header = "X-Spam-Flag";
    value = "YES";
    remove = 1;
  }
  authentication-results {
    header = "Authentication-Results";
    remove = 1;
    # SPF/DKIM/DMARC symbols in case these are redefined
    spf_symbols {
      pass = "R_SPF_ALLOW";
      fail = "R_SPF_FAIL";
      softfail = "R_SPF_SOFTFAIL";
      neutral = "R_SPF_NEUTRAL";
      temperror = "R_SPF_DNSFAIL";
      none = "R_SPF_NA";
      permerror = "R_SPF_PERMFAIL";
    }
    dkim_symbols {
      pass = "R_DKIM_ALLOW";
      fail = "R_DKIM_REJECT";
      temperror = "R_DKIM_TEMPFAIL";
      none = "R_DKIM_NA";
      permerror = "R_DKIM_PERMFAIL";
    }
    dmarc_symbols {
      pass = "DMARC_POLICY_ALLOW";
      permerror = "DMARC_BAD_POLICY";
      temperror = "DMARC_DNSFAIL";
      none = "DMARC_NA";
      reject = "DMARC_POLICY_REJECT";
      softfail = "DMARC_POLICY_SOFTFAIL";
      quarantine = "DMARC_POLICY_QUARANTINE";
    }
  }
}

- Include '_rspamd' user at group 'opendkim':
Code: [Select]
# usermod -a -G opendkim _rspamd

- Create the file '/etc/rspamd/local.d/dkim_signing.conf' with the following content:
(you can suppress the lines starting with '#', but I do not recommend)
Code: [Select]
# If false, messages with empty envelope from are not signed
allow_envfrom_empty = true;
# If true, envelope/header domain mismatch is ignored
allow_hdrfrom_mismatch = false;
# If true, multiple from headers are allowed (but only first is used)
allow_hdrfrom_multiple = false;
# If true, username does not need to contain matching domain
allow_username_mismatch = true;
# If false, messages from authenticated users are not selected for signing
auth_only = true;
# Default path to key, can include '$domain' and '$selector' variables
path = "/etc/opendkim/userkeys/$domain/$selector.private";
# Default selector to use
selector = "default";
# If false, messages from local networks are not selected for signing
sign_local = true;
# Map file of IP addresses/subnets to consider for signing
# sign_networks = "/some/file"; # or url
# Symbol to add when message is signed
symbol = "DKIM_SIGNED";
# Whether to fallback to global config
try_fallback = true;
# Domain to use for DKIM signing: can be "header" (MIME From), "envelope" (SMTP From) or "auth" (SMTP username)
use_domain = "header";
# Domain to use for DKIM signing when sender is in sign_networks ("header"/"envelope"/"auth")
use_domain_sign_networks = "header";
# Domain to use for DKIM signing when sender is a local IP ("header"/"envelope"/"auth")
use_domain_sign_local = "header";
# Whether to normalise domains to eSLD
use_esld = true;
# Whether to get keys from Redis
# Not using redis, keys coming from files in /etc/opendkim
use_redis = false;
# Hash for DKIM keys in Redis
key_prefix = "DKIM_KEYS";

Create a soft link (the files normally are identical)
Code: [Select]
# cd /etc/rspamd/local.d/; ln -s dkim_signing.conf arc.conf

- Create the file '/etc/rspamd/local.d/mx_check.conf' with the following content:
Code: [Select]
# Set this to enable the module
enabled = true;
# connection timeout in seconds
timeout = 30.0;
# symbol yielded if no MX is connectable
symbol_bad_mx = "MX_INVALID";
# symbol yielded if no MX is found
symbol_no_mx = "MX_MISSING";
# symbol yielded if MX is connectable
symbol_good_mx = "MX_GOOD";
# lifetime of redis cache - 1 day by default
expire = 86400;
# lifetime of redis cache for no valid mxes - 2 hours by default
expire_novalid = 7200;
# greylist first message with invalid MX (require greylist plugin)
greylist_invalid = false;
# prefix used for redis key
key_prefix = "rmx";
# module-specific redis-server configuration
servers = "127.0.0.1";
# a map of specific domains that should be excluded from MX check
# exclude_domains = "/etc/rspamd/local.d/local_wl_domains.map.inc";

- file /etc/rspamd/local.d/dmarc.conf
Code: [Select]
servers = "127.0.0.1";
# Enables storing reporting information to redis
#reporting = true;
# If Redis server is not configured below, settings from redis {} will be used
#servers = "127.0.0.1:6379"; # Servers to use for reads and writes (can be a list)
# Alternatively set read_servers / write_servers to split reads and writes
# To set custom prefix for redis keys:
#key_prefix = "dmarc_";
# Actions to enforce based on DMARC disposition (empty by default)
actions = {
quarantine = "add_header";
reject = "reject";
}
# Ignore "pct" setting for some domains
# no_sampling_domains = "/etc/rspamd/dmarc_no_sampling.domains";

- file /etc/rspamd/local.d/force_actions.conf
Code: [Select]
# Rules are defined in the rules {} block
rules {
VIRUS_DETECTED {
action = "reject";
expression = "CLAM_VIRUS";
# message setting sets SMTP message returned by mailer
message = "Rejected due to suspicion of virus";
honor_action = ["reject"];
}
}

- file /etc/rspamd/local.d/fuzzy_check.conf
Code: [Select]
timeout = 4s;
retransmits = 3;

- file /etc/rspamd/local.d/neural.conf (with redis use)
Code: [Select]
servers = "127.0.0.1";
enabled = true;
dbname = "2"; # Redis setup

# local.d/neural_group.conf
symbols = {
  "NEURAL_SPAM" {
    weight = 3.0; # sample weight
    description = "Neural network spam";
  }
  "NEURAL_HAM" {
    weight = -3.0; # sample weight
    description = "Neural network ham";
  }
}

- file /etc/rspamd/local.d/phishing.conf
Code: [Select]
phishtank_enabled = true;
phishtank_map = "https://rspamd.com/phishtank/online-valid.json.zst";
# Enable openphish support (default disabled)
openphish_enabled = true;
# URL of feed, default is public url:
openphish_map = "https://www.openphish.com/feed.txt";
openphish_premium = false;
# For premium feed, change that to your personal URL, e.g.
# openphish_map = "https://openphish.com/samples/premium_feed.json";

- Change this to true in that file, if premium feed is enabled (paid service)
Code: [Select]
openphish_premium = true;

- file /etc/rspamd/local.d/replies.conf (with redis use)
Code: [Select]
# This setting is non-default & may be desirable
action = "no action";
# These are default settings you may want to change
expire = 86400;
key_prefix = "rr";
message = "Message is reply to one we originated";
symbol = "REPLY";
# Module specific redis configuration
servers = "127.0.0.1";

# /etc/rspamd/local.d/surbl.conf
Code: [Select]
# List of domains that are not checked by surbl
whitelist = "file://$CONFDIR/surbl-whitelist.inc";
# Additional exceptions for TLD rules
exceptions = "file://$CONFDIR/2tld.inc";
redirector_hosts_map = "/etc/rspamd/redirectors.inc";

rules {
"SURBL_MULTI" {
# DNS suffix for this rule
suffix = "multi.surbl.org";
bits {
# List of bits ORed when reply is given
JP_SURBL_MULTI = 64;
AB_SURBL_MULTI = 32;
MW_SURBL_MULTI = 16;
PH_SURBL_MULTI = 8;
WS_SURBL_MULTI = 4;
SC_SURBL_MULTI = 2;
}
}
"URIBL_MULTI" {
suffix = "multi.uribl.com";
bits {
URIBL_BLACK = 2;
URIBL_GREY = 4;
URIBL_RED = 8;
}
}
"RAMBLER_URIBL" {
suffix = "uribl.rambler.ru";
# Also check images
images = true;
}
"DBL" {
suffix = "dbl.spamhaus.org";
# Do not check numeric URL's
noip = true;
}
"SPFBL_URIBL" {
suffix = "uribl.spfbl.net";
resolve_ip = false;
ips {
URIBL_SPFBL = "127.0.0.2";
}
}
"SEM_URIBL_UNKNOWN" {
suffix = "uribl.spameatingmonkey.net";
bits {
SEM_URIBL = 2;
}
noip = true;
}
"SEM_URIBL_FRESH15_UNKNOWN" {
suffix = "fresh15.spameatingmonkey.net";
bits {
SEM_URIBL_FRESH15 = 2;
}
noip = true;
}
}

- file /etc/rspamd/local.d/url_redirector.conf (with redis use)
Code: [Select]
# How long to cache dereferenced links in Redis (default 1 day)
expire = 1d;
# Timeout for HTTP requests (10 seconds by default)
timeout = 10; # 10 seconds by default
# How many nested redirects to follow (default 1)
nested_limit = 1;
# Prefix for keys in redis (default "rdr:")
key_prefix = "rdr:";
# Check SSL certificates (default false)
check_ssl = false;
max_size = 10k; # maximum body to process

- file /etc/rspamd/local.d/url_reputation.conf (with redis use)
Code: [Select]
# Enable
enabled = true;
# Key prefix for redis - default "Ur."
key_prefix = "Ur.";
# Symbols to insert - defaults as shown
symbols {
  white = "URL_REPUTATION_WHITE";
  black = "URL_REPUTATION_BLACK";
  grey = "URL_REPUTATION_GREY";
  neutral = "URL_REPUTATION_NEUTRAL";
}
# DKIM/DMARC/SPF allow symbols - defaults as shown
foreign_symbols {
  dmarc = "DMARC_POLICY_ALLOW";
  dkim = "R_DKIM_ALLOW";
  spf = "R_SPF_ALLOW";
}
# SURBL metatags to ignore - default as shown
ignore_surbl = ["URIBL_BLOCKED", "DBL_PROHIBIT", "SURBL_BLOCKED"];
# Amount of samples required for scoring - default 5
threshold = 5;
# Maximum number of TLDs to update reputation on (default 1)
update_limit = 1;
# Maximum number of TLDs to query reputation on (default 100)
query_limit = 100;
# If true, try to find most 'relevant' URL (default true)
relevance = true;

- file /etc/rspamd/local.d/url_tags.conf  (with redis use)
Code: [Select]
# cache some URL tags in redis
enabled = true;

- file /etc/rspamd/override.d/antivirus.conf
Code: [Select]
# multiple scanners could be checked, for each we create a configuration block with an arbitrary name
clamav {
  # If set force this action if any virus is found (default unset: no action is forced)
  action = "reject";
  # if `true` only messages with non-image attachments will be checked (default true)
  # attachments_only = false;
  scan_mime_parts = false;
  # If `max_size` is set, messages > n bytes in size are not scanned
  #max_size = 20000000;
  # symbol to add (add it to metric if you want non-zero weight)
  symbol = "CLAM_VIRUS";
  # type of scanner: "clamav", "fprot", "sophos" or "savapi"
  type = "clamav";
  # If set true, log message is emitted for clean messages
  log_clean = true;
  # For "savapi" you must also specify the following variable
  #product_id = 12345;
  # For "savapi" you can enable logging for clean messages
  #log_clean = true;
  # servers to query (if port is unspecified, scanner-specific default is used)
  # can be specified multiple times to pool servers
  # can be set to a path to a unix socket
  servers = "127.0.0.1:3310";
  # `whitelist` points to a map of IP addresses. Mail from these addresses is not scanned.
  whitelist = "/etc/rspamd/antivirus.wl";
}

- You can download maps and fuzzy keys from 'maps.rspamd.com'.
Currently, it has two ip addresses: 88.99.142.95 and 212.24.145.107.
So, you need to release it at firewall, to the port 11335 outbound to maps.rspamd.com.
- Change the file /etc/csf/csf.allow, including at final:
Code: [Select]
udp|out|d=11335|d=88.99.142.95
tcp|out|d=11335|d=212.24.145.107
udp|out|d=11335|d=212.24.145.107

- Start rspamd
Code: [Select]
# systemctl start rspamd

Check with the folloing command:
Code: [Select]
# ps auxww | grep rspamd

At this point, if all it's ok and there is no typing wrong, Rspamd must be working, it will result in:
Code: [Select]
_rspamd   5166  0.0  0.0 304072  2864 ?        SNs  Dez06   0:00 rspamd: main process
_rspamd   5171  0.0  0.8 474132 31944 ?        SNL  Dez06   0:26 rspamd: rspamd_proxy process (127.0.0.1:11332)
_rspamd   5172  0.0  0.9 362124 35768 ?        SN   Dez06   0:28 rspamd: controller process (127.0.0.1:11334)
_rspamd   5173  0.0  0.6 358404 25972 ?        SN   Dez06   0:07 rspamd: normal process (127.0.0.1:11333)
_rspamd   5174  0.0  0.1 304072  6580 ?        SN   Dez06   0:12 rspamd: hs_helper process

It is working, but no functional yet.
Now, we must to configure Postfix.

(...FOLLOWS AT PART 2, BELOW)

2
New Modules / Where post translation..?!
« on: December 23, 2017, 09:55:45 PM »
Where post translations..??!!

I have translated the user panel to the brazilian portuguese, but seems attachments are forbidden on this forum.

Regards,
Netino

Pages: [1]