ESXI Nexenta 4, round robin, iops=1, no Hardware Accelerated Locking

Nexenta 4 (CE) on ESXI (5/6) sort of fails when you have Hardware Accelerated Locking enabled. You will see a ton of errors in your vmkernel log about this once you activate your ISCSI.

To get it all going again here is a quick snippet.

esxcli system settings advanced set -i 0 -o /VMFS3/HardwareAcceleratedLocking

esxcfg-rescan vmhba32

for i in `esxcfg-scsidevs -c |awk '{print $1}' | grep naa.600`; do esxcli storage nmp device set -d $i --psp VMW_PSP_RR;done

for i in `esxcfg-scsidevs -c |awk '{print $1}' | grep naa.600`; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=$i; done

The first line disables the HW accelerated locking, e.g. back to basics. Then we do a rescan of vmhba32 (SW/ISCSI), then push all disks to VMW_PSP_RR and set the IOPS to 1 for optimal distribution,

C’est ca..

PostFix as a mail relay Ubuntu 14 LTS

This took some time, so i’d thought I might share it. If you have any improvements or comments, feel free to contribute.

The scenario, we have multiple post-fix mail relay servers sitting behind various ISP’s in various clouds or data-centers, they all point to a single (cluster) of mail servers (Zimbra of course). It took some time to get these MX’s to do the right thing with SMTP / TLS and SASL.

The MX sits behind a firewall on a private ip ( (VyOS) which publishes TCP 25 and 465 for the communications.

First apt-get install postfix on a clean Ubuntu 14 LTS server, set it up as a satellite when dpkg-configure asks you to.

Then fetch the TLS and SASL dependencies

apt-get install  sasl2-bin postfix-tls sasl2-bin libsasl2-modules

For some odd reason, the socket links in /run between sasl authd and postfix don’t add up. A mess of security and other oddness. So in a quick and dirty way we fix this on reboot in a script called /etc/postfix/

rm -r /var/run/saslauthd/
mkdir -p /var/spool/postfix/var/run/saslauthd
ln -s /var/spool/postfix/var/run/saslauthd /var/run
chgrp sasl /var/spool/postfix/var/run/saslauthd
service postfix restart
/etc/init.d/saslauthd start

To get Postfix to use the plain login for SASL configure the file /etc/postfix/sasl/smtpd.conf

pwcheck_method: saslauthd
mech_list: plain login

You can find some more details on the how and why here

Now for the Meat and Potatoes, the /etc/postfix/ file:

inet_interfaces = all
myhostname =
mynetworks =,,
myorigin =
mydestination =
smtpd_banner = $myhostname ESMTP
smtpd_use_tls = yes
smtp_use_tls = yes
smtp_enforce_tls = no
# TLS parameters
smtp_tls_security_level = may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_note_starttls_offer = yes
smtpd_tls_auth_only = yes
smtpd_tls_security_level = may
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_exclude_ciphers = aNULL, DES, 3DES, MD5, DES+MD5, RC4, RC4-MD5
smtpd_tls_received_header = yes
tls_random_source = dev:/dev/urandom
# Sasl
smtpd_sasl_path = smtpd
smtpd_sasl_security_options = noanonymous
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain =
broken_sasl_auth_clients = yes
# Limits
default_process_limit = 100
smtpd_client_connection_count_limit = 10
smtpd_client_connection_rate_limit = 30
queue_minfree = 20971520
header_size_limit = 51200
message_size_limit = 10485760
smtpd_recipient_limit = 100
local_recipient_maps =
local_transport = error:no local delivery
parent_domain_matches_subdomains = debug_peer_list smtpd_access_maps
relay_domains = and some more domains
smtpd_recipient_restrictions =
address_verify_negative_cache = no
unverified_recipient_reject_code = 550
maximal_queue_lifetime = 21d
relayhost = [your.mail.cluster.local]

Change the bold where appropriate of course

As you can see we use some SSL certs, these come from our own private CA, but you can use some self signed ones as well, just remember to cat the authorizing CA certificate into the CRT and have no password in your key. A lengthy explanation on the how and what can me found here

Now the /etc/postfix/, un-comment (or add) the bold lines and if all is equal, it should work…

# Postfix master process configuration file. For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line:
# Do not forget to execute "postfix reload" after editing this file.
# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
smtp inet n - - - - smtpd
#smtp inet n - - - 1 postscreen
#smtpd pass - - - - - smtpd
#dnsblog unix - - - - 0 dnsblog
#tlsproxy unix - - - - 0 tlsproxy
submission inet n - - - - smtpd
# -o syslog_name=postfix/submission
 -o smtpd_tls_security_level=encrypt
 -o smtpd_sasl_auth_enable=yes
# -o smtpd_reject_unlisted_recipient=no
# -o smtpd_client_restrictions=$mua_client_restrictions
# -o smtpd_helo_restrictions=$mua_helo_restrictions
# -o smtpd_sender_restrictions=$mua_sender_restrictions
# -o smtpd_recipient_restrictions=
 -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
 -o milter_macro_daemon_name=ORIGINATING
smtps inet n - - - - smtpd
# -o syslog_name=postfix/smtps
 -o smtpd_tls_wrappermode=yes
 -o smtpd_sasl_auth_enable=yes
# -o smtpd_reject_unlisted_recipient=no
# -o smtpd_client_restrictions=$mua_client_restrictions
# -o smtpd_helo_restrictions=$mua_helo_restrictions
# -o smtpd_sender_restrictions=$mua_sender_restrictions
# -o smtpd_recipient_restrictions=
 -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
 -o milter_macro_daemon_name=ORIGINATING
#628 inet n - - - - qmqpd
pickup unix n - - 60 1 pickup
cleanup unix n - - - 0 cleanup
qmgr unix n - n 300 1 qmgr
#qmgr unix n - n 300 1 oqmgr
tlsmgr unix - - - 1000? 1 tlsmgr
rewrite unix - - - - - trivial-rewrite
bounce unix - - - - 0 bounce
defer unix - - - - 0 bounce
trace unix - - - - 0 bounce
verify unix - - - - 1 verify
flush unix n - - 1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - - - - smtp
relay unix - - - - - smtp
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - - - - showq
error unix - - - - - error
retry unix - - - - - error
discard unix - - - - - discard
#local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - - - - lmtp
anvil unix - - - - 1 anvil
scache unix - - - - 1 scache
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
# Many of the following services use the Postfix pipe(8) delivery
# agent. See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in maildrop_destination_recipient_limit=1
maildrop unix - n n - - pipe
 flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
# ====================================================================
# Recent Cyrus versions can use the existing "lmtp" entry.
# Specify in cyrus.conf:
# lmtp cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
# Specify in one or more of the following:
# mailbox_transport = lmtp:inet:localhost
# virtual_transport = lmtp:inet:localhost
# ====================================================================
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in cyrus_destination_recipient_limit=1
#cyrus unix - n n - - pipe
# user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
# ====================================================================
# Old example of delivery via Cyrus.
#old-cyrus unix - n n - - pipe
# flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
# ====================================================================
# See the Postfix UUCP_README file for configuration details.
uucp unix - n n - - pipe
 flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
# Other external delivery methods.
ifmail unix - n n - - pipe
 flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
 flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix - n n - 2 pipe
 flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman unix - n n - - pipe
 flags=FR user=list argv=/usr/lib/mailman/bin/
 ${nexthop} ${user}

Zimbra ZCS on Ubuntu 14 Lts

Zimbra, by far the best mail solution out there, making use of postfix, clamav, openldap, amavis,nginx,memcacheed and more defacto standard open source solutions to deliver an enterprise class email solution.

Deploying this to Ubuntu 14 is quite straight forward, but most of the guides out there seem to be incomplete, so here is mine.

Zimbra is a bit resource hungry since it relies on Java, so deploy a VM with 6GB mem (4 works too) and 4 cores. We use a standard 16GB OS partition and move our mailstore to an Nexenta/NFS store at later stage.

1st things first, fetch and untar the binary from, the site is rather slow so it takes some time. The version is 8.6.0/1153 at the time of writing.

tar -zxvf zcs*

Now, edit your hosts file and hostname to have the correct referrals

#cat /etc/hosts localhost cinhk1mail01 mail

Now install the pre-requisites, for us on a image in our cloud we need to add

apt-get install libaio1 pax sysstat unzip libgmp10

Next run the ./ and select the components you want. I don’t use DNS Cache since our Internal DNS is rigid and we have MX records for our domains set up correctly, if you do not have this sorted, use dnsmasq as described here:

Install zimbra-ldap [Y]
Install zimbra-logger [Y]
Install zimbra-mta [Y]
Install zimbra-dnscache [Y] N
Install zimbra-snmp [Y]
Install zimbra-store [Y]
Install zimbra-apache [Y]
Install zimbra-spell [Y]
Install zimbra-memcached [Y]
Install zimbra-proxy [Y] N

After installing the packages, this can take some time, Zimbra will complain about the domain name, change the domain from the FQDN to the actual domain-name

DNS ERROR resolving MX for
It is suggested that the domain name have an MX record configured in DNS
Change domain name? [Yes] 
Create domain: []
 MX: (

Now walk through the setup menu and set the passwords for the Admin user (in the zimbra-store submenu) I personally change all LDAP passwords to something we have recorded in our security database to make addition of servers and separation of roles at a later stage easier. Don’t forget to set the timezone in the common configuration as well. We also enable the option “Configure for use with web proxy: ” since in our house, access to webmail is done through an NGINX reverse proxy.

Now check your installation by running zmcontrol status as the zimbra user:

su - zimbra -c "zmcontrol status"

All should be running and happy at this moment, so it is time for a reboot and see if all comes back after.

Now log in to your zimbra server admin console using your favorite browser ( mind it’s an HTTPS. If your server shows all red in the server status (in contradiction to what zmcontrol status had told you before) you might have some RSYSLOG issues. In this case, just create a syslog file (/etc/syslog) with the content:

# Zimbra logs 
local0.* -/var/log/zimbra.log 
local1.* -/var/log/zimbra-stats.log 
auth.* -/var/log/zimbra.log 
mail.* -/var/log/zimbra.log

and reconfigure zimbra syslog for good measure with /opt/zimbra/libexec/zmsyslogsetup

And restart Zimbra (or reboot) (service zimbra restart), all should be green and happy after.

Now Zimbra can read the correct log files and determine the status of the services.

Next up is the install ZeXtras, an absolutely brilliant add in for Zimbra to enable ZCS to work well with mobile connections, do some easy backups, and most important easily move your data-stores around. We use it on all our production servers and it is certainly worth the few dollars they ask for it, if you need some quote or help with ZeXtras, drop us a line at

To install, first wget and untar the add-in.

tar -zxvf zextras_suite-latest.tgz
cd zextras_suite-2.0.3 
./ all

2.0.3 is the version at hand to date, follow the install questions and after the install navigate back to the admin console, reload the page to see if the plug in is installed.

Now we have a Nexenta back end storage for NFS and ISCSI, so we use this as our data-store. We created a mount point /mailstore in which we created a folder called store01. we changed the owership to zimbra on this folder (chown zimbra:zimbra /mailstore/store01). Now navigate to the zxPowerstore section of ZexTras and add a new volume pointing to the NFS mount. We don’t need compression as the Nexenta server will take care of that for us.

Screenshot from 2015-09-17 13:47:06

It’s now time to add our wildcard certificate. If you don’t have one, just go through the setup wizard and create a self signed one or create a request for an issuer and skip this step. But in our case we have our own trusted CA from where we spawn certificates. So if you have the same, do the following:

go to the folder /opt/zimbra/ssl/zimbra/commercial, copy your CA certificate as commercial_ca.crt, your wildcard certificate as commercial.crt and the key as commercial.key. Now verify if all computes with

/opt/zimbra/openssl/bin/openssl verify -CAfile commercial_ca.crt commercial.crt

and if all is good, commit the certificates with

/opt/zimbra/bin/zmcertmgr deploycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt

Now reload your admin page and check if your certificate is used

Now domains (and users) can be added to the server, for external ldap authentication, check this post:

VyOS on a PCCW PPPoE link

Man am I annoyed!, when moving some internet connections off WharfTT to PCCW (as they promised us better site to site connectivity between Macau and Hong Kong) We got their stupendous PPPoE type connections instead of just a plain old Ethernet link.

What does that matter you might think, but with these PPPoE connections you need to screw your MTU down to 1492 to allow for the PPP overhead, and even worse, if you are using VyOS (or any other Linux Kernel based router/firewall for that matter) like we are, you would need to Clamp your MSS. And of course you will only find out how to do that when emails stop flowing in and internet connections crawl to halt.

So to save you some time,.. for VyOS (My favorite flavor of opensource routers) you do the following

Edit the file:


And add the following line

iptables -t mangle -I POSTROUTING 1 -p tcp -o pppoe0 --tcp-flags SYN SYN -j TCPMSS --set-mss 1412

You would need to hack this since VyOS (using 1.6 now) does still not come with a modify (mangle) class in the firewall configuration.

For good measure, your VyOS firewalls are of course bound to the PPPoE interface like so.

ethernet eth1 {
 duplex auto
 hw-id 00:1c:c0:f1:d2:c5
 pppoe 0 {
 default-route auto
 firewall {
 in {
 name pppoe-in
 local {
 name pppoe-local
out {
name pppoe-out

OpenLdap with multiple domains

OpenLDAP or SLAPD, the heart of Zimbra authentication and Samba/AD. Not as easy as the click-er-die-click interface of Windows, but when set up correctly, just as powerful.

My headache came however when I wanted to host multiple domains or databases on the same server. One server that is the directory for and and maybe even, you get the drift.

Lesson one is to never to modify the /usr/share/slapd/slapd.conf file, configuration of OpenLdap is done through LDIF files and ldapadd.

Now the first thing we do is actually install OpenLdap and the Tools

apt-get install slapd ldap-utils
 mkdir /ldap
 chown -R openldap:openldap /ldap

Now if you already had OpenLdap/SlapD installed, purge and reinstall it like so

apt-get purge slapd ldap-utils --YES && wait
rm /var/lib/ldap/*
rm -rf /etc/ldap
rm -rf /ldap
apt-get install slapd ldap-utils
mkdir /ldap
chown -R openldap:openldap /ldap

The (re) installation will ask you for a password (from the dpkg-configure) I have not bothered to silence that yet.

You see the script made a folder /ldap and made openldap owner of that, since I don’t like to hide my databases somewhere in /var/lib

Next is to tell the AppArmor watchdog to leave our /LDAP folder alone

nano /etc/apparmor.d/usr.sbin.slapd 

and edit the section databases and logsto look like this

 # the databases and logs
 /var/lib/ldap/ r,
 /var/lib/ldap/** rwk,
 /ldap/ r,
 /ldap/** rwk,

And restart the Apparmor:

service apparmor restart

Now, the meat and potatoes, which I called all in one shell script file of course, but cut up a bit here to explain whats what.

PASS=$(slappasswd -s Supersecretpassword)
DATABASEID==$(ar=$(ls /etc/ldap/slapd.d/cn\=config/*{?}* | cut -d "{" -f2 | cut -d "}" -f1)&&nr=$(($(echo "${ar[*]}" | sort -nr | head -n1) + 1))&&echo $nr)

The variables, yes I know , clear text passwords, next time i’ll MD5 them. The database ID is the number of the DB. Get the currently used ones from  etc/ldap/slapd.d/cn\=config/*{?}* and a little bash magic adds one to the tally.

Now add the database folder and chown it to openldap, if it already exists the domain must already be created, so better stop there..

if [ ! -d /ldap/$DOMAINNAME-$DOMAINEXT ]; then
 chown -R openldap:openldap /ldap/$DOMAINNAME-$DOMAINEXT
 echo "Domain $DOMAINNAME.$DOMAINEXT exists, aborting"
 exit 1

Next up is construct the LDIF file with a simple cat << EOF >

cat << EOF > $TEMP/database.ldif
# Create directory database
dn: olcDatabase={$DATABASEID}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {$DATABASEID}hdb
olcDbDirectory: /ldap/$DOMAINNAME-$DOMAINEXT
olcRootDN: cn=admin,dc=$DOMAINNAME,dc=$DOMAINEXT
olcRootPW: $PASS
olcAccess: to attrs=userPassword 
 by dn="cn=admin,dc=$DOMAINNAME,dc=$DOMAINEXT" write 
 by anonymous auth 
 by self write 
 by * none
olcAccess: to attrs=shadowLastChange 
 by self write 
 by * read
olcAccess: to dn.base="" 
 by * read
olcAccess: to * 
 by dn="cn=admin,dc=$DOMAINNAME,dc=$DOMAINEXT" write 
 by * read
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 2097152 0
olcDbConfig: {1}set_lk_max_objects 1500
olcDbConfig: {2}set_lk_max_locks 1500
olcDbConfig: {3}set_lk_max_lockers 1500
olcDbIndex: uid pres,eq
olcDbIndex: cn,sn,mail pres,eq,approx,sub
olcDbIndex: objectClass eq
# Modifications
dn: cn=config
changetype: modify
dn: olcDatabase={-1}frontend,cn=config
changetype: modify
delete: olcAccess
dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcRootDN
olcRootDN: cn=admin,cn=config
dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcRootPW
olcRootPW: $PASS
dn: olcDatabase={0}config,cn=config
changetype: modify
delete: olcAccess
Followed by the LDAPADD that imports the database 

ldapadd -Y EXTERNAL -H ldapi:/// -f $TEMP/database.ldif
rm $TEMP/database.ldif

Due to the unique way ldapadd reads config files, sometimes the olcAccess: class causes an ldap error 80. when this happens, make sure each line in the olcAccess class has a trailing space. 

Now it is time to fill the newly created database with a base structure. Same cat <<EOF there followed by another ldapadd

cat << EOF > $TEMP/dit.ldif
# Tree root
objectClass: dcObject
objectclass: organization
description: Tree root
# Populating
dn: cn=admin,dc=$DOMAINNAME,dc=$DOMAINEXT
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: admin
userPassword: $PASS
description: LDAP administrator
dn: ou=users,dc=$DOMAINNAME,dc=$DOMAINEXT
ou: users
objectClass: organizationalUnit
objectClass: top
dn: ou=groups,dc=$DOMAINNAME,dc=$DOMAINEXT
ou: groups
objectClass: organizationalUnit
objectClass: top
#Adding user
dn: uid=test,ou=users,dc=$DOMAINNAME,dc=$DOMAINEXT
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
uid: test
sn: test
givenName: test
cn: test test
displayName: test test
uidNumber: 1002
gidNumber: 1000
userPassword: $PASS
gecos: test test
loginShell: /bin/bash
homeDirectory: /home/test
shadowExpire: -1
shadowFlag: 0
shadowWarning: 7
shadowMin: 8
shadowMax: 999999
shadowLastChange: 10877
postalCode: 31000
l: test
mobile: +852 12345678
homePhone: +852 12345678
title: test user
initials: tst
ldapadd -x -D cn=admin,dc=$DOMAINNAME,dc=$DOMAINEXT -W -f $TEMP/dit.ldif

Finally, for good measures  I normally add some certificates for the SSL authentication, I am assuming I do not need to explain how to create some SSL certs..

dn: cn=config
changetype: modify
add: olcTLSCipherSuite
olcTLSCipherSuite: NORMAL
add: olcTLSCRLCheck
olcTLSCRLCheck: none
add: olcTLSVerifyClient
olcTLSVerifyClient: never
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/ssl/certs/ldap-ca-cert.pem
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/ssl/private/ldap-ca-key.pem

and apply with ldapadd

ldapadd -Y EXTERNAL -H ldapi:/// -f ssl.ldif

Now when all is quare and equal, you should have added a domain database. I use JXplorer to check if all is well, just connect into the cn=config for the full picture.

Next up is good old replication, but I’ll save that for another time..

Cest Ca.


VirSH with ESXi for Ubuntu 14.04 LTS (and MAAS)

Why o why is the latest release of virsh with ESX not in the APT.

Anyway, lets hack away.

First of all get your virsh source tarball, I used 1.2.18 today…

Now “apt-get install” the crapload of dependencies to compile the source, the usual Gnu/Dev,XML/dev and other stuff to make things work.

Then do the ./configure (with the –with-esx=yes), and ./make
apt-get install gcc make pkg-config libxml2-dev libgnutls-dev libdevmapper-dev libcurl4-gnutls-dev python-dev libpciaccess-dev libxen-dev libnl-dev uuid-dev xsltproc
tar -zxvf libvirt-1.2.18.tar.gz
cd libvirt-1.2.18/
./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc --with-esx=yes
make install

Now try virsh -v, should come back with 1.2.8 or whatever version you installed
and virsh -c esx://root@<your.esxi.server>?no_verify=1 should allow you to control your ESXi host, wonder :) we can stop and start ESXi VM’s now, MAAS will be so happy :)

Oh, if you want to use MAAS now to power control your VM’s, use the Virsh power type and the esx://root@<your.esxi.server>/system?no_verify=1 power control and don’t forget to make your /etc/libvirt/auth.conf file for the authentication. It looks a bit like this:


hostnameX is your ESXi host of course..

Ces’t ca

Set Round Robin and IOPS for Nexenta ISCSI Luns

Have a ton of ISCSI volumes across a ton of hosts and just not feeling like clicking all those checkboxes to change paths.. Your solution would be…

esxcli nmp device list | grep naa | grep NEXENTA | awk -F '[()]' '{print $(NF-1)}' | while read disk; do esxcli nmp device setpolicy -d $disk --psp=VMW_PSP_RR; done

Oh, and while you’re at it set the iops chunk size to 1, speeds things up a bit :)
esxcli nmp device list | grep naa | grep NEXENTA | awk -F '[()]' '{print $(NF-1)}' | while read disk; do esxcli nmp roundrobin setconfig --device $disk --iops 1 --type iops ; done