| | |

Building a Modern Mail Server on Debian 13: Complete Series

ByPieter Bakker

Why Self-Host Your Mail Server?

Email remains one of the most critical communication channels for businesses, yet many organizations have moved away from hosting their own mail servers in favor of cloud-based solutions. While cloud email services offer convenience, they come at the cost of control over your business-critical data and privacy.

I’ve been hosting and managing my own mail servers since the early 2000s, building extensive expertise in mail filtering, spam prevention, and defending against cyberattacks. This experience has taught me one fundamental truth: maintaining control over your email infrastructure is worth the investment, despite its complexity.

However, email technology is constantly evolving, standards keep changing, security requirements are becoming stricter, and challenges continue to increase. While ensuring your outgoing messages reach their destination (even to the big players like Google and Microsoft) is important, the real challenge has become filtering out the bad messages containing spam, phishing attempts, and viruses. Modern attacks are increasingly sophisticated, making it harder than ever to maintain effective filtering without risking a high false positive rate that blocks legitimate business communications.

This is why I’m building a new mail server stack from the ground up, using rock-solid technologies that comply with all current guidelines and best practices.

Why Not Use All-in-One Solutions?

Several packaged mail server solutions exist: Mailcow, Modoboa, Stalwart Labs, iRedMail, Mail-in-a-Box, Zimbra, and others. While these can get you started quickly, they come with significant drawbacks:

Loss of Control

  • Forced to accept their proposed configurations
  • Limited ability to customize for your specific needs
  • The black-box approach makes it harder to understand what’s actually happening

Harder Debugging

  • When problems occur, you’re troubleshooting someone else’s abstraction layer
  • Error messages may be hidden behind management interfaces
  • Difficult to trace issues through multiple layers of automation

Update Risks

  • Updates may introduce breaking changes without your knowledge
  • Your customizations can be overwritten during upgrades
  • Dependency on maintainers’ release schedules and priorities
  • Example: Zimbra stopped providing open source binaries after version 8.8.15 (now EOL), forcing users to either build from source, use third-party community builds, or migrate to commercial licensing

Technology Lock-in

  • Docker/microservices architecture (not always suitable for all environments)
  • Specific software versions and combinations you may not want
  • Difficult to migrate away if you need different components

Learning Limitations

  • You don’t truly understand your mail server
  • Harder to adapt to new requirements or technologies
  • Reduced ability to troubleshoot production issues

By building from scratch, you gain:

  • Complete understanding of every component
  • Full control over configuration
  • The ability to debug at any level
  • The flexibility to adapt as requirements change
  • Deep knowledge that transfers to other projects

What This Series Covers

This comprehensive guide will walk you through creating a production-ready mail server on Debian 13 (trixie), built with modern best practices and battle-tested open-source components.

Core Technologies

Postfix – The industry-standard SMTP server for sending and receiving email. Reliable, secure, and extensively documented.

Dovecot 2.4 – Modern IMAP/POP3 server with updated configuration syntax, robust security features, and excellent performance.

PostfixAdmin – Web-based management interface for administering virtual domains, mailboxes, and aliases.

MariaDB – Reliable database backend for storing virtual domains, users, and mail server configurations.

Rspamd – Advanced spam filtering engine with machine learning capabilities, replacing outdated SpamAssassin approaches.

Roundcube – Feature-rich webmail interface providing browser-based email access for your users.

CrowdSec – Modern intrusion prevention system with crowd-sourced threat intelligence.

Unbound – Local recursive DNS resolver essential for high-volume DNSBL queries without rate limiting.

FoFoundation & Preparation

Part 1: Mail Server Preparation
Time: 30-40 minutes

Essential groundwork before installing the mail server:

  • Server naming conventions (mx1.example.com vs mail.example.com)
  • Hostname and DNS configuration
  • Configuration variables file for the entire series
  • PTR record verification
  • Timezone and time synchronization
  • Unbound DNS resolver installation
  • Why systemd-resolved must be disabled for mail servers
  • SSL certificate setup with acme.sh (SAN certificate)

Status: Published

Core Mail Stack

Part 2: Core Mail Server Setup
Time: 90-120 minutes

Complete mail server installation and configuration:

  • MariaDB database installation and setup
  • PostfixAdmin installation (creates database schema)
  • Postfix configuration for virtual domains with MariaDB
  • Dovecot 2.4 IMAP/POP3 setup with new syntax
  • Integration with MariaDB for virtual users
  • SSL/TLS configuration for both services
  • Creating domains and mailboxes
  • Testing complete mail flow (send and receive)

Status: Coming Soon

Spam Filtering & Security

Part 3: CrowdSec Intrusion Prevention
Time: 30-45 minutes

Automated security and threat protection:

  • CrowdSec installation and configuration
  • Mail server scenario integration
  • Blocking brute-force attacks on SSH, PostfixAdmin, and Postfix auth
  • Crowd-sourced threat intelligence
  • Alert configuration
  • Dashboard and monitoring

Why this comes before Rspamd: After installing the core mail server, you have exposed web interfaces and authentication endpoints. CrowdSec provides immediate protection against brute force attacks, exploits, and known malicious actors before you implement spam filtering.

Status: Coming Soon

Part 4: Rspamd Spam Filtering
Time: 60-90 minutes

Advanced spam and virus protection:

  • Rspamd installation and architecture
  • Integration with Postfix and Dovecot
  • DKIM key generation, signing, and verification
  • DNSBL configuration for optimal filtering
  • ClamAV antivirus integration
  • Greylisting setup
  • Machine learning tuning
  • False positive management

Status: Coming Soon

User Access & Maintenance

Part 5: Roundcube Webmail
Time: 30-45 minutes

Browser-based email access:

  • Roundcube installation on Debian 13
  • Integration with Dovecot
  • Theme customization
  • Plugin installation (calendar, contacts)
  • Mobile responsiveness
  • User management

Status: Coming Soon

Part 6: Monitoring, Logging, and Maintenance
Time: 45-60 minutes

Keeping your mail server healthy:

  • Log aggregation and analysis
  • Performance monitoring
  • Queue management
  • Automated backups
  • Security updates
  • Troubleshooting common issues

Status: Coming Soon

Prerequisites

Before starting this series, ensure you have:

Technical Requirements

  • Server: Debian 12 or 13 (bare metal, VPS, or Incus/LXC container)
  • RAM: Minimum 2GB (4GB recommended for Rspamd)
  • Storage: 20GB+, depending on email volume
  • Network: Static public IPv4 address (IPv6 optional but recommended)
  • Access: Root/sudo access to the server

DNS Requirements

  • Domain name with full DNS control
  • Ability to create A/AAAA records
  • Ability to create MX records
  • Ability to add TXT records (SPF, DKIM, DMARC)
  • PTR record access (usually through hosting provider)

Knowledge Requirements

  • Basic Linux command-line skills
  • Understanding of text editors (nano, vi, or vim)
  • Basic networking concepts (DNS, TCP/IP)
  • Familiarity with systemd services
  • Basic SQL knowledge (helpful but not required)

Time Commitment

  • Initial setup: 2-3 hours for core mail stack (Parts 1-2)
  • Full deployment: 5-7 hours, including security and features (Parts 1-6)
  • Ongoing maintenance: 1-2 hours per month

What Makes This Series Different

Production-Ready Focus

Every configuration is tested on real production mail servers, not just lab environments. You’ll learn what actually works in the field.

Modern Best Practices

Updated for Debian 13 with Dovecot 2.4’s new syntax, current security standards, and 2024/2025 spam filtering requirements.

Complete Understanding

No black boxes. Every configuration option is explained, so you understand not just “what” but “why.”

Real-World Troubleshooting

Common issues and their solutions based on 20+ years of mail server administration experience.

Practical Security

Security configurations that balance protection with usability, avoiding both paranoid over-engineering and dangerous shortcuts.

Ready to Begin?

Start with Part 1: Mail Server Preparation to lay the groundwork for your mail server.

Similar Posts