Zimbra EMail Server on FreeBSD as Bhyve VM using IPFW

Many server operating systems, including FreeBSD, come with sendmail but sendmail by itself lacks imap and a web interface. Various mail systems include sendmail with pine, qmail, and Maia. Maia has many nice features but installation is complex and time consuming. After trying a few newer email solutions I settled on Zimbra which does not run on FreeBSD. I first ran this as a separate cloud hosted VM but when I needed to change IP I decided to reinstall as a VM on an existing FreeBSD bare metal server and save the cloud hosting fees. The bare metal server uses ipfw so we will used ipfw to forward ports for this project.

Set Phases

Setting up the VM
Network Configuration
Installing Zimbra
Checking DNS Configuration

VM & Firewall Setup

To setup bhyve to work with ipfw follow any instructions online for how to install bhyve which roughly goes like this ...

pkg update && pkg upgrade
pkg install vm-bhyve bhyve-firmware tigervnc-viewer grub2-bhyve
zfs create zroot/bhyve
zfs set recordsize=64K zroot/bhyve
zfs create zroot/bhyve/.templates

Edit /etc/rc.conf and add
—————–
# IPFW FIREWALL ENABLE
firewall_enable=”YES” # Set to YES to enable firewall functionality
firewall_type=”open” # Firewall type (see /etc/rc.firewall)
#firewall_script=”/etc/ipfw.rules” # Add firewall rules here so not “open”
firewall_quiet=”YES” # Set to YES to suppress rule display
firewall_logging=”YES” # Set to YES to enable events logging
gateway_enable=”YES”
firewall_nat_enable=”YES”
firewall_nat_interface=”em0″ # change this to your external interface
# needed for virtualization support
vm_enable=”YES”
vm_dir=”zfs:zroot/bhyve”
# vm_list=”Ubuntu” # list of machines to autostart
vm_list=””
vm_delay=”5″

Edit /etc/sysctl.conf and add
——————————
net.inet.ip.fw.enable=1
net.inet.ip.forwarding=1
#net.inet.ip.fw.one_pass=0
net.inet.tcp.tso="0"
#net.inet.ip.fastforwarding=1

Edit /etc/devfs.rules and add
———————
[localrules=10]
add path ‘/dev/tap*’ mode 0660 group libvirt

Edit /boot/loader.conf and add
—————–
vmm_load=”YES”
nmdm_load=”YES”
if_tap_load=”YES”
if_bridge_load=”YES”
ipfw_load=”YES”
ipfw_nat_load=”YES”
net.inet.ip.fw.default_to_accept="1"

Firewall Rules

Edit /etc/ipfw.rules
—————
#!/bin/sh
fwcmd=”ipfw -q”
oif=”bge0″
net=”10.111.111.0/24″
vmip=”10.88.88.88″
natnet=”10.88.88.0/24″
#trusted1=”172.22.33.0/24″
trusted1=”172.22.33.44″
# avoid net.inet.ip.fw.one_pass=0
# try to avoid statefull

ipfw -q -f flush

${fwcmd} add allow ip from any to any via lo0
${fwcmd} add deny ip from any to 127.0.0.0/8
${fwcmd} add deny ip from 127.0.0.0/8 to any
${fwcmd} add deny ip from any to ::1
${fwcmd} add deny ip from ::1 to any
${fwcmd} add allow ipv6-icmp from :: to ff02::/16
${fwcmd} add allow ipv6-icmp from fe80::/10 to fe80::/10
${fwcmd} add allow ipv6-icmp from fe80::/10 to ff02::/16
${fwcmd} add allow ipv6-icmp from any to any icmp6types 1
${fwcmd} add allow ipv6-icmp from any to any icmp6types 2,135,136

# reassemble inbound packets
# ${fwcmd} add reass all from any to any in

ipfw -q nat 1 config if ${oif} same_ports unreg_only reset \
redirect_port tcp ${vmip}:22 22222 \
redirect_port tcp ${vmip}:81 81 \
redirect_port tcp ${vmip}:444 444 \
redirect_port tcp ${vmip}:8080 8080 \
redirect_port tcp ${vmip}:8443 8443 \
redirect_port tcp ${vmip}:25 25 \
redirect_port tcp ${vmip}:119 119 \
redirect_port tcp ${vmip}:143 143 \
redirect_port tcp ${vmip}:389 389 \
redirect_port tcp ${vmip}:465 465 \
redirect_port tcp ${vmip}:587 587 \
redirect_port tcp ${vmip}:993 993 \
redirect_port tcp ${vmip}:995 995 \
redirect_port tcp ${vmip}:7071 7071 \
redirect_port tcp ${vmip}:7073 7073 \
redirect_port tcp ${vmip}:7025 7025

# NOTE YOU CAN’T DO ACCEPT IN ON ABOVE PORTS eg 22222
# Use Apache to redirect to https on 444 / eg no http allowed
# Using Apache Proxy/Reverse to 10.88.88.88:81 works also

# NAT
${fwcmd} add nat 1 ip from ${natnet} to any out via ${oif}
${fwcmd} add nat 1 ip from any to me in via ${oif}

# Allow limited broadcast traffic from my own net.
${fwcmd} add pass all from ${net} to 255.255.255.255

# Allow any traffic to or from my own net.
${fwcmd} add pass all from me to ${net}
${fwcmd} add pass all from ${net} to me

# Allow setup of incoming https request
${fwcmd} add pass tcp from any to me 80,443 in via ${oif}

# SSH WIDE OPEN TEMP
# ${fwcmd} add pass tcp from any to me 22 in via ${oif}

# Deny ssh from NATNET / don’t trust ssh from the VM
${fwcmd} add deny tcp from ${natnet} to me 22

# TEMP allow ssh traffic to my test net
# ${fwcmd} add pass tcp from 10.111.0.0/16 to me 22 in via ${oif}
# ${fwcmd} add pass tcp from me to 10.111.0.0/16 22 in via ${oif}

# TRUSTED NET OR HOST
${fwcmd} add pass tcp from ${trusted1} to me 22 in via ${oif}
${fwcmd} add pass icmp from ${trusted1} to me
${fwcmd} add pass udp from ${trusted1} to me

# deny all other SSH requests
${fwcmd} add deny tcp from any to me 22 in via ${oif}

# Allow TCP through if setup succeeded
${fwcmd} add pass tcp from any to any established

# Allow IP fragments to pass through
${fwcmd} add pass all from any to any frag

# Allow setup of incoming email
# disable external mail, this host forwards mail to VM
# ${fwcmd} add pass tcp from any to me 25 setup

# another service to host I want to allow for example
${fwcmd} add pass tcp from any to me 23232 in via ${oif}

# NATNET
${fwcmd} add allow ip from ${natnet} to any
${fwcmd} add allow ip from any to ${natnet}

# OUT SIMPLE
${fwcmd} add allow tcp from me to any setup keep-state
${fwcmd} add allow udp from me to any keep-state
${fwcmd} add allow icmp from me to any keep-state

# Global Deny
${fwcmd} add deny ip from any to any

Reboot

Before reboot go back and comment the line that says firewall_script … #firewall_script=”/etc/ipfw.rules”

When the server restarts it won’t enable the rules but will be open. Then run sh /etc/ipfw.rules. If everything works as expected good, but if not you can reboot and the rules won’t be run on startup and you can get back in. For the same reason we have set default to open in /boot/loader.conf.

Intialize vm-bhyve

Setup the VM Network and VM

# Create the switch
# vm switch destroy natnet
vm switch create -a 10.88.88.1/24 natnet
vm switch add natnet bge0
vm switch list
NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS
natnet standard vm-natnet 10.88.88.1/24 no – – bge0
# vm switch info natnet

# Copy VM sample configs to .templates
cp /usr/local/share/examples/vm-bhyve/* /zroot/bhyve/.templates

# Create this file /zroot/bhyve/.templates/ubuntu-nvme.conf
loader="uefi"
grub_run_partition="2"
cpu=4
memory=8G
network0_type="virtio-net"
network0_switch="public"
# nvme - on zfs - raw/img is fastest / virtio-blk zvol-sparse is slower
disk0_type="nvme"
disk0_name="disk0.img"
graphics="yes"
graphics_listen="0.0.0.0"
graphics_res="1024×768"
graphics_wait="auto"
xhci_mouse="yes"
bhyve_options=""

# Some useful vm commands
vm list

# graceful shutdown
vm stop Ubuntu

# to forcefully power off
vm poweroff -f Ubuntu

# to delete th VM
vm destroy Ubuntu

Create the VM
vm create -t ubuntu-nvme -s 80G Ubuntu

# change default editor
export EDITOR=nano

# Check the vm configuration and change network to natnet
# You can optionally add another interface that is not natted for LAN access
# just make sure to create the switch first and increment the network id and
# change the mac address
vm configure Ubuntu

# Ubuntu.conf vm-bhyve configuration
loader="uefi"
grub_run_partition="2"
cpu=4
memory=8G
network0_type="virtio-net"
network0_switch="natnet"
# network1_type="virtio-net"
# network1_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
graphics="yes"
graphics_listen="0.0.0.0"
graphics_res="1024×768"
graphics_wait="auto"
xhci_mouse="yes"
uuid="f24a9af7-fa5a-11ed-8ab9-c81f66f6f69d" # do not copy use vm create
network0_mac="58:9c:fc:0c:e1:95" # do not copy vm create generates these
# network1_mac="58:9c:fc:11:22:33" # do not copy vm create generates these

# Download Ubuntu Server 20.04 and run this to install
# Use the most recent version or install may fail while
# trying to do final updates or at least take very long.

vm install Ubuntu0 ubuntu-20.04.6-live-server-amd64.iso

Connect with TigerVNC 127.0.0.1:5900 or FREEBSD_IP_ADDR:5900

As a side note debian VM's and possibly ubuntu can be strange about boot and you may need to drop to a shell after install but before reboot and ...
# mkdir /target/boot/efi/EFI/BOOT
# cp /target/boot/efi/EFI/debian/grubx64.efi /target/boot/efi/EFI/BOOT/bootx64.efi

After install the system will reboot and try to reinstall so you will need to force it to stop.

vm -f poweroff Ubuntu
vm start Ubuntu and connect to the VM login with your username and reset the root password.
su passwd root
su – root
nano /etc/ssh/sshd_config
Find AllowRootLogin and change it to yes (for the time being)
ps -aux | grep sshd
kill -1 <first sshd process id>
Go to your host and ssh to the vm..

You should now have a working up to date 20.04 Ubuntu VM.

You can always shutdown the VM and remove the graphics part of the configuration so that it’s not listening on port 5900 all the time. And, if you lost access you can force shutdown and add the graphics blurb back, start it up, and get in with TigerVNC.
vm configure Ubuntu
loader="uefi"
grub_run_partition="2"
cpu=4
memory=8G
network0_type="virtio-net"
network0_switch="natnet"
disk0_type="nvme"
disk0_name="disk0"
uuid="951084d6-fc18-11ed-b3c6-0cc47a4c2264"
network0_mac="58:9c:fc:06:d3:98"

Installing Zimbra (the easy part)

I initially thought installing Zimbra would be the easy part and it is but after migration I made a mistake with the dnsmasq.conf which allowed operation but as I got more domains and aliases added zmconfigd and proxy failed to restart. The error caused me to reinstall more than 10 times till I figured out what was causing it. So remember server= in /etc/dnsmasq.conf is your upstream DNS server not the hypervisor host IP.

This is a good resource https://inguide.in/install-zimbra-on-ubuntu-20-04-step-by-step/

hostnamectl set-hostname mx.domain.com

/etc/hosts
127.0.0.1 localhost
127.0.0.1 mx.domain.com

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback localhost
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.88.88.88 mx.domain.com
10.111.111.231 domain.com mx.domain.com mail.domain.com
10.111.111.231 domain2.com mx.domain2.com mail.domain2.com

The second entry here is in case you host another domain mail on your server.

On your name server add A Records for your mx machine(s) and add MX records.
The A Record is always the IP ADDRESS mx.domain.com. In this example you also need an A Record IP ADDRESS mail.domain.com. Note that mx and mail are expanded to mx.domain.com and mail.domain.com.
A mx IP_ADDR
A mail IP_ADDR
The MX Record should only have 1 entry unless you setup Zimbra with multiple servers (not covered here). In this example I setup a standalone so I only need 1 MX Record. You could add mail as an MX Record but that would be redundant since they both point to the same IP address. Here is the example with 2 records. Here the @ symbol is an abbreviation for domain.com and the number is priority. The lower MX priority number is higher/first priority.
MX @ mx.domain.com 10
MX @ mail.domain.com 20

You would put these entries in the DNS for each mail domain you plan to host on your server. So the MX records for domain2.com would be…
A mx IP_ADDR # expands to mx.domain2.com
A mx IP_ADDR # expands to mail.domain2.com
MX @ mx.domain.com 10
MX @ mail.domain.com 20

You will also want spf dkim and dmarc records but I’ll get to that later.

Install dnsmasq

apt install dnsmasq ifupdown net-tools

systemctl stop systemd-resolved
systemctl disable systemd-resolved

Edit the file /etc/dnsmasq.conf
server=8.8.8.8
domain=domain.com
mx-host=domain.com,mx.domain.com,10
mx-host=mx.domain.com,mx.domain.com,20
listen-address=127.0.0.1
listen-address=10.88.88.88

systemctl enable dnsmasq
systemctl restart dnsmasq

Download Zimbra
wget https://files.zimbra.com/downloads/8.8.15_GA/zcs-8.8.15_GA_4179.UBUNTU20_64.20211118033954.tgz

Extract and Install
tar xfzv zcs-8.8.15_GA_4179.UBUNTU20_64.20211118033954.tgz
cd zcs-8.8.15_GA_4179.UBUNTU20_64.20211118033954
./install.sh

Here I would follow the guide
https://inguide.in/install-zimbra-on-ubuntu-20-04-step-by-step/
do not install
-dnscache
-drive
-imapd evaluation copy
-zimbra-chat // well I’ve never tried the chat

Now the tricky part, I made changes to the proxy which I thought caused some zimbra modules to fail but I managed to trace it back to dnsmasq being misconfigured (set the upstream server). The symptoms were that nslookup was very very slow and zmcontrol start and zmcontrol status showed modules not running. A slow DNS will cause modules to fail to start or start successfully then fail as more domains are added.

Use the menu for Zimbra store and set the admin password.
Select Proxy try to change https to both, it will fail and ask you to disable secure communication but doing this seems to enable this in the menu. I could not find it without trying “both” first.
Go to Global/General and the second selection of the main menu to find and disable secure communication as instructed by the error.
Go back to Proxy menu and set https to both and change the ports
Method: both
http: 81
https: 444
– continue with install as normal

Setup Apache redirects on host, or use proxy, which I think is slower

If you setup letsencrypt on the VM you can copy the certs to the host so there is seamless proxy to the VM mail server. You can also disable the ipfw forwarding rule for 81 on the host to block unencrypted access.

I initially wrote a script to generate several certs on the host and install it on the Zimbra VM but realized installing certs for each different domain is more involved and it just overwrites the main cert unless you follow the instructions for multiple domains each with their own cert.

Running letsencrypt was more trouble that it was worth since it must stop, renew the cert, and restart and then install on the host apache too. If the cert is not installed on the host going to https://mx.hostname.com just before redirect will present a cert invalid error. Namecheap certs are very affordable so bought one and installed it on both zimbra and the host under the redirect container. If I have other aliases just put redirects for those to the main mx.mydomain.net. Users of "other domains" would then be directed to a valid login page though would have to use the full email address on login. To allow users to use short login for without the domain just add certs and directs for each domain.
 
# see files under /usr/local/etc/apache24

<VirtualHost *>
ServerName mx.mydomain.com
ServerAlias mail.mydomain.com
ServerAdmin admin@mydomain.com
RedirectMatch 301 ^(.*)$ https://mx.mydomain.net:444$1
#ProxyPreserveHost On
#ProxyPass / http://10.88.88.88:81/
#ProxyPassReverse / http://10.88.88.88:81/
</VirtualHost>
<VirtualHost *>
ServerName mail5.mydomain.com
RedirectMatch 301 ^(.*)$ https://mx.mydomain.net:444$1
ServerAdmin admin@mydomain.com
</VirtualHost>

<VirtualHost *>
ServerName mx.domain2.com
ServerAlias mail.domain2.com
RedirectMatch 301 ^(.*)$ https://mx.domain2.com:444$1
ServerAdmin admin@domain.com
</VirtualHost>

# The ssl config, I run apache ssl as a separate daemon from http for performance
# mx.mydomain.net
<VirtualHost *:443>
    ServerName mx.mydomain.net
    ServerAlias mail.mydomain.net
    
    ServerAdmin admin@mydomain.net
    # UseCanonicalName On

# We can redirect rather than proxy but STILL NEED THE CERTS!
    SSLCertificateFile "/usr/local/openssl/2024/mx_mydomain_net.crt"
    SSLCertificateKeyFile "/usr/local/openssl/2024/mx_mydomain_net.key"
    SSLCACertificatePath /usr/local/openssl/2024
    SSLCertificateChainFile "/usr/local/openssl/2024/mx_mydomain_net.ca-bundle"

# Use Redirect or Proxy not both

# Redirect / https://mx.mydomain.net:444
    RedirectMatch 301 ^(.*)$ https://mx.mydomain.net:444$1

# Apache sends http not https to VM
    #ProxyPreserveHost On
    #ProxyPass / http://10.88.88.88:81/
    #ProxyPassReverse / http://10.88.88.88:81/
    
    #SSLProxyEngine On
    #SSLProxyVerify require
    ###SSLProxyCACertificateFile /usr/local/etc/letsencrypt/live/mx.mydomain.net/cert.pem
    #SSLProxyCheckPeerCN on  # or omit, default is on
    #SSLProxyCheckPeerName on  # same
</VirtualHost>

Don’t forget to setup DKIM, DMARC, and spf records in the DNS or your email may be rejected by some mail hosts or end up in spam.
TXT @ v=spf1 a:domain.com ip4:123.456.789.111 -all

# DKIM
https://wiki.zimbra.com/wiki/Configuring_for_DKIM_Signing

# find dkim
/opt/zimbra/libexec/zmdkimkeyutil -q -d domain.com

# Create new dkim don’t forget to add DKIM to DNS
I use NameCheap and GoDaddy, make sure you remove the quotes and space inbetween and input as all one line, NameCheap and GoDaddy will correct this as needed. Just paste the whole thing, go the the end, back arrow till you see something like …dkjfdlc” “sdfdfl;dkfj… and remove the gap and quotes …dkjfdlcsdfdfl;dkfj… There are 2 gaps, one near at the beginning also.

/opt/zimbra/libexec/zmdkimkeyutil -a -d domain.com

# remove dkim signature
/opt/zimbra/libexec/zmdkimkeyutil -r -d domain.com

# update dkim, need to change dkim in DNS
/opt/zimbra/libexec/zmdkimkeyutil -u -d example.com

DMARC is a record that specifies what to do in case spf or dkim fails and who to email with the reports.

Please check online for specific references for setting these up with DNS and setting up letsencrypt.

Additional DNS Settings

Setting up dnsmasq is highly advised and there is one more caviat. Some mail programs on the host itself may need to send mail to the VM but won't work correctly unless the mx host is added with the VM's LOCAL IP. I discovered this when I couldn't get a wordpress site to send emails to the mail server.