Category Archives: Expert

Squid, c-icap, ClamAV: Bug in the service. Please report to the service author!!!!

If you see this error in your c-icap server logfile, it might just be that c-icap is running out of temporary disk space and that the clamav/virus scanner configuration for c-icap is wrong:

Service antivirus_module
ServiceAlias  avscan virus_scan?allow204=on&sizelimit=off&mode=simple
virus_scan.MaxObjectSize  5M
TmpDir /tmp

The option "... sizelimit=off..." for the virus_scan service means that the configuration value for "MaxObjectSize" will be ingored. If you have too many parallel squid client connections open or large files to download, c-icap is running out of temporary disk space. It will then log the following error message without further explanation:

Bug in the service. Please report to the service author!!!!

The webbrowser download will be terminated with an error message (something like "internal server error").

To solve this problem, add more free space to the partition where TmpDIr resides, and change the virus_scan service option to "... sizelimit=on ...".

In the worst case, free disk space for the c-icap TmpDIr has to be:
MaxServers * ThreadsPerChild * virus_scan.MaxObjectSize


grub-install: error: disk '...' not found

If you get an error like the following, the reason for this might not be so obvious. In my case I got the following error message trying to run grub-install:

# grub-install /dev/mapper/vg1-lv_boot
Installing for i386-pc platform.
grub-install: error: disk `lvmid/OffQLW-SofZ-KH38-jrbl-RXyw-dmDc-VOJuPf/lbiWU0-SkvY-nDET-EGvy-A1PP-fmGb-dGv7yX' not found.

The logical volume I tried to install grub onto was ok (/dev/mapper/vg1-lv_boot). The problem was somewhere else: I previously had a disk failure in a RAID0 md raid. The faulty drive was replaced online by a hot spare drive. But there was still an encrypted swap device configured for the old drive. And that swap device was not part of the md raid, so it was not automatically transferred to the new spare drive.

Only after removing this non-existing swap partition (swapoff <device>) grub-install was working again. So if you come across any error message from grub-install like the one above, the reason for it might be a problem with ANY configured disk on your system. Check for the following errors:

# swapon -s

Are there any swap partitions in use that no longer exist physically?

# pvdisplay
/dev/mapper/cryptswap2: read failed after 0 of 4096 at 0: Input/output error 
/dev/mapper/cryptswap2: read failed after 0 of 4096 at 1998520320: Input/output error 
/dev/mapper/cryptswap2: read failed after 0 of 4096 at 1998577664: Input/output error 
/dev/mapper/cryptswap2: read failed after 0 of 4096 at 4096: Input/output error 
/dev/sdb: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb: read failed after 0 of 4096 at 1000204795904: Input/output error 
/dev/sdb: read failed after 0 of 4096 at 1000204877824: Input/output error 
/dev/sdb: read failed after 0 of 4096 at 4096: Input/output error 
/dev/sdb1: read failed after 0 of 4096 at 1998520320: Input/output error 
/dev/sdb1: read failed after 0 of 4096 at 1998577664: Input/output error 
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error 
/dev/sdb5: read failed after 0 of 4096 at 998203392000: Input/output error 
/dev/sdb5: read failed after 0 of 4096 at 998203449344: Input/output error 
/dev/sdb5: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb5: read failed after 0 of 4096 at 4096: Input/output error 
--- Physical volume ---

Are there any error messages for physical LVM2 volumes? If so, try to remove the erroneous physical volumes from your running configuration. Maybe there are still active mount points on the faulty disks (including swap partitions).

# dmsetup status

All entries in the device mapper list have to be valid. There might not be an obvious error message in the output, so you have to check each dm device manually.

Important things to note:

  • Grub2 no longer relies on the file /boot/grub/ . You can create the file with "grub-mkdevicemap", but grub-install does not use it and performs a full system scan by itself.
  • grub-install also examines swap devices, even though it obviously will not use them.

Configuring wireless networks in Linux

1. Overview

This post assumes that you are already familiar with connecting Windows or Mac OS to an existing accesspoint. It also assumes that you have a working wireless network card.  If you are looking for an inexpensive wifi card that you can attach to a USB 2.0 port, take a look at my previous post (CSL 300 Mbit/s wifi adapter with Debian 8 Jessie). You might have to install additional firmware packages.

Here is a list of supported wifi devices by the Linux kernel:

Check with iwconfig that there is a working WiFi device on your computer:

$ iwconfig

wlan0     IEEE 802.11bgn  ESSID:off/any   
          Mode:Managed  Access Point: Not-Associated   Tx-Power=15 dBm    
          Retry short limit:7   RTS thr:off   Fragment thr:off 
          Encryption key:off 
          Power Management:on

This tells us that there is a WiFi device called "wlan0" capable to connect to any 802.11b/g/n accesspoint.

There are 2 ways to configure wireless networks in Linux:

  • Using the graphical tool "NetworkManager"
    The preferred method if you are using a graphical desktop environment. Very similar to Windows or Mac OS and easy to use.
  • On the command line using "wpa_supplicant"
    Only recommended for experienced Linux users.

Both of them are included in every modern Linux distribution and have advantages and disadvantages which I will explain later in this post. You should not mix both methods, just decide for one of them and stick with it. So if you already use NetworkManager to manage ethernet connections, it is easy to add one or more WiFi connections.

Both NetworkManager and the native command line method rely on the package "wpa_supplicant" (or "wpasupplicant") to actually use a wifi network. Nevertheless I will use the term "wpa_supplicant" to refer to the command line method.

There is a plethora of additional graphical network tools in Linux, e.g. graphical front ends for wpa_supplicant. Once you know the basics of wpa_supplicant it is easy to use other tools too. Therefore in this post I will only describe how to configure wpa_supplicant on the command line.

2. Encryption Protocols

WPA2 (802.11i) is today's standard for wireless data encryption. It uses 2 different keys for encrypting traffic between accesspoint and client stations.

NameDescriptionConfiguration OptionRekeying Interval (Default Value)Notes
PTK ("Pairwise Transient Key":)- Consists of several other keys / fields used to encrypt data and distribute GTK to client stations

- Unique to every client station

- Only used for unicast traffic
"wpa_ptk_rekey" in wpa_supplicant.conf?
GTK ("Group Transient Key")- Generated by accesspoint and sent to client stations

- Shared by all client stations

- Only used for multicast, / broadcast traffic
"Group Key Interval" in accesspoint configuration

rekey interval is not configurable in NetworkManager or wpa_supplicant
30 seconds- Not configurable in NetworkManager or wpa_supplicant

- Rekeying is completely up to accesspoint, so there is no way to print the rekey interval on client station (wpa_cli or nmcli)

- wpa_supplicant generates log entries like the following:
wpa_supplicant[1652]: wlan0: WPA: Group rekeying completed with 00:2a:0e:ab:cd:ef [GTK=CCMP]

Both keys are then used to encrypt traffic between accesspoint and client stations. There are 2 protocols for symmetric data encryption:

  • TKIP (Temporal Key Integrity Protocol)
    based on RC4
    insecure and obsolete
    use only in combination with additional encryption layers like VPN or SSH tunnels
  • CCMP (CCM Mode Protocol)
    based on AES
    today's standard

3. Authentication Methods

There are 2 different authentication methods for wireless networks:

  • All users share the same single key
    Primarily used for a smaller number of client stations, e.g. in home networks or small guest networks
  • Every user has his own username / password (or unique client certificate)
    Useful for a larger number of client stations, e.g. in corporate environments or where you have a lot of guest users

WPA2 Personal / PSK (Preshared Key)

The same key (8 - 63 characters) must be configured on accesspoint and client stations. It is directly used as PMK (Pairwise Master Key) by accesspoint, and then used to calculate PTK (Pairwise Transient Key). PTK is then used to calculate GTK.

WPA2 Enterprise / 802.1x

Actual authentication is not performed by the accesspoint, but by a 3rd party server called "authentication server". This is usually a Radius server running "freeradius".

Even though authentication is performed by a separate authentication server, it only knows the MK (Master Key) and its derived PMK (Pairwise Master Key). The PMK is transferred (moved, not copied) from the authentication server to the accesspoint and used to calculate a PTK (Pairwise Transient Key). So the authentication server has no access to neither PTK nor GTK and therefore cannot decrypt traffic (unicast or multicast) between accesspoint and client stations.

  • WPA2 Enterprise usually requires a username / password combination for authentication
    (authentication methods LEAP, FAST, PEAP, and TTLS)
  • Using TLS as the authentication method the client authenticates with a client X.509 certificate.
  • The client itself may use a CA certificate to verify that it is connecting to the right accesspoint (similar to HTTPS connections in webbrowsers).

4. NetworkManager

NetworkManager is part of every modern LInux distribution. After a standard installation of Linux you will see a network icon in the system bar of desktop environment. If you click on it you will see a list of options to configure NetworkManager.

Connection settings that you make in the GUI are stored as plain text files under /etc/NetworkManager/system-connections . (Explanation of all settings: )

In addition to configure wireless networks, NetworkManager offers some other useful features:

  • You can integrate NetworkManager with desktop encryption tools like kwallet to prevent passwords from being saved in plain text to the configuration files.
  • You can integrate NetworkManager with firewalld to automatically assign WiFi networks to firewall zones.
  • You can configure NetworkManager to automatically use a VPN connection once you are connected to a specific WiFi network.

General configuration

NetworkManager screenshot: General configuration

Automatically connect to this network when it is available
In most cases leave this unchecked. Otherwise there might be occasions where you involuntary connect to the WiFi network.

All users may connect to this network
Only check this option if you want to share your wifi configuration with other Linux user accounts.

Automatically connect to VPN when using this connection
Useful when using an insecure public WiFi hotspot that you only want to use in combination with a VPN tunnel.

Firewall zone
If you are using firewalld and firewall-config, you may associate this WiFi network with a specific firewall zone. If empty the default firewall zone will be used automatically.

The dialog box layout is a little bit misleading because this field has nothing to do with the previous "Firewall zone" field. If there is more than one of the "Automatically connect to this network ..." wifi networks available, "Priority" defines the order in which those networks will be activated. The first successful connection will be used.

Command line

NetworkManager can also be controlled from the command line with "nmcli".

Display current state of NetworkManager service
$ nmcli g
connected  full          enabled  enabled  enabled  enabled

Show a list of all network connections
$ nmcli c  
mynetwork           abababab-cdcd-12cc-bbef-1212121212ab  802-11-wireless  wlan0 

Stop wifi network
$ nmcli c down id mynetwork

Start wifi network
$ nmcli c up id mynetwork


5. wpa_supplicant

wpa_supplicant runs as a service process in the background. Connections are stored by default in /etc/wpa_supplicant/wpa_supplicant.conf .

Sample configuration file with detailed explanations:

The wpa_supplicant background service can be controlled from the command line with "wpa_cli".

Display list of all command line parameters
$ wpa_cli help

Display a list of configured networks
$ wpa_cli list_networks
0       mynetwork 0a:ab:ee:ef:2a:ef       [CURRENT]

Start wifi network
$ wpa_cli enable_network 0

Stop wifi network
$ wpa_cli disable_network 0

Show current wifi connection status
$ wpa_cli status



Pinentry not working over SSH with x11forwarding (Thunderbird with Enigmail)

Using Thunderbird with Enigmail over SSH sometimes does not work because you cannot input the passphrase for your private GPG key. Starting pinentry-qt / pinentry-gnome3 / pinentry-gtk2 does not work. Here is a workaround. You can cache the passphrase with gpg-agent before starting Thunderbird. Enigmail will then use the cached passphrase because it runs only gpg2 commands in a subshell in order to encrypt or sign messages.

Connect to the server using x11forwarding:

$ ssh -Y server

Note your DISPLAY environment variable:

$ echo $DISPLAY

Unset / delete the DISPLAY environment variable:


Export GPG_TTY environment variable for gpg:

export GPG_TTY=$(tty)

Make sure that gpg-agent is running:

$ ps aux | grep gpg-agent
user 2058 0.0 0.0 168068 2228 ? Ss Nov10 0:07 gpg-agent --homedir /home/user/.gnupg --use-standard-socket --daemon

Insert the passphrase for your GPG key in gpg-agent by signing a dummy message. Make sure that you enter your passphrase in the pinentry tui not the gpg command prompt.

$ echo test | gpg2 --use-agent -s

The passphrase you are about to enter should be cached by gpg-agent. The cache lifetime is controlled by settings in ~/.gnupg/gpg-agent.conf . Now set the DISPLAY environment variable again to run Thunderbird. Use the value from previous command.

export DISPLAY=localhost:10.0

Start Thunderbird. You should now be able to sign and encrypt email messages with Enigmail without having to enter your gpg passphrase again because it is already cached by gpg-agent.

thunderbird &



PAM authentication per application (Debian 8 Jessie)

PAM is the default authentication mechanism in Linux. It is very flexible and powerful, and even allows you to configure different authentication options for each application. In this example we will use the PAM module "pam_listfile" which is already included in the standard package "libpam-modules".

The application name has to match the name of the file under /etc/pam.d . So for example for application "abc" you have the following PAM configuration file /etc/pam.d/abc :

auth required onerr=fail item=group sense=allow file=/etc/group.allow
@include common-auth
@include common-account
@include common-password
@include common-session

The first line only authenticates users that are member of any group listed in /etc/group.allow. The contents of /etc/group.allow may only contain a single line and looks like this:


This will allow only members of the group "abc_group" to login to application "abc". After adding the new configuration files, make sure to always test your PAM settings with pamtester:

# id bob
uid=1003(bob) gid=1003(bob) groups=1003(bob)
# pamtester abc bob authenticate
pamtester: Authentication failure
# usermod -aG abc_group bobpamtester abc bob authenticate
pamtester: successfully authenticated

First the user "bob" is not member of the group "abc_group". pamtester fails to authenticate the user even if you provide the right password. Then after adding "bob" to the group "abc_group" authentication succeeds.



Certificate Authorities (CA) in Google Chrome / Chromium and Firefox on Linux

Firefox ships with its own set of root CAs ("Builtin Object Token" as the Security Device in advanced preference settings). Here is the list of all root CAs included in Firefox along with their fingerprints:

Builtin root CAs are hardcoded in /usr/lib/firefox/ . You can see a list of all CAs in Firefox preferences (advanced settings).

CAs marked as "Software Security Device" are usually intermediate certificates that are downloaded from websites and stored locally. These CAs that are not builtin are either stored on a PKCS#11 compatible smartcard attached to your PC/laptop or saved to your home directory:
certutil -d $HOME/.mozilla/firefox/xxx.default -L

Chromium / Google Chrome does not ship with its own CA list but uses the CAs from the underlying operating system:

In Ubuntu 16.04 these CAs are hardcoded in /usr/lib/x86_64-linux-gnu/nss/ which is part of the package "libnss3". You should therefore update this package as soon as there is an update available to keep your builtin CA list up-to-date.

CAs that are not builtin and that you installed manually are stored in your home directory:
certutil -d sql:$HOME/.pki/nssdb -L

Important things to note:

  • The security of SSL encrypted websites (https://...) depends on the root CA which is used to sign the website certificate. These CAs are stored locally on your device in different locations based on the browser you are using.
  • There are 2 kinds of CAs:
    1. Builtin CAs that ship with your browser or linux installation. They are stored in shared object files. There is probably no easy way to edit this list unless you change the source files and recompile the package. Nevertheless in both browsers you can remove all trust from a builtin certificate which is basically the same as deleting it.
    2. Manually added CAs are stored in your home directory. You can easily edit that list within the settings of the browser or on the command line.
  • Both Firefox and Chromium / Google Chrome use NSS certificate databases to store manually added CAs that are not builtin. But they use different directories. Maybe you could use symbolic links to point both directories to the same database. That way both browsers would be using the same manual CA list.Currently Firefox uses by default the legacy dbm database version (cert8.db, key3.db) and Chromium / Google Chrome uses by default the new SQLite database version (cert9.db, key4.db). There seems to be an environment variable NSS_DEFAULT_DB_TYPE that makes Firefox use the new SQLite database version as well (s.
  • Neither Firefox nor Chromium / Google Chrome are using CAs from the package "ca-certificates".

Password encryption in OpenLDAP

Passwords in OpenLDAP are SSHA encrypted by default (salted SHA1).

Changing it to SHA512:

olcPasswordHash: {CRYPT},{SSHA}
olcPasswordCryptSaltFormat: "$6$%.16s"

This will still accept already existing passwords that are SSHA encrypted. New passwords will be SHA512 encrypted.

For this to work, the GNU C library has to support SHA512:
- /etc/login.defs: ENCRYPT_METHOD SHA512
- man pam_unix (should include sha512)

SHA512 passwords for LDAP can be generated with slappasswd:

slappasswd -c '$6$%.16s'



iSCSI connection states in Open-iSCSI

This is the iSCSI connection state if the underlying network interface changes from "UP BROADCAST RUNNING MULTICAST" to "UP BROADCAST MULTICAST".

Log entries showing that the network interface has no longer the state "RUNNING":

Jul 3 14:17:31 host kernel: [974138.571169] bnx2 0000:08:05.0 eth2: NIC Copper Link is Down
Jul 3 23:05:05 host kernel: [1005760.957474] sd 10:0:0:0: rejecting I/O to offline device
... previous message repeats many times ...

Checking iSCSI connection state:

# iscsiadm -m session -P1
iSCSI Connection State: TRANSPORT WAIT
iSCSI Session State: FREE
Internal iscsid Session State: REOPEN

Log entry once the network interface state is back to "RUNNING":

Jul 4 06:56:31 host kernel: [1034019.191222] bnx2 0000:08:05.0 eth2: NIC Copper Link is Up, 1000 Mbps full duplex

Checking iSCSI connection state again:

# iscsiadm -m session -P1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE

The output of "iscsiadm -m session -P1" can be used for monitoring the iSCSI connection e.g. in a simple Nagios or Icinga Perl script.


Disk configuration for VM guests in KVM / qemu

For KVM / qemu virtualization there are 2 settings to optimize performance for the virtual disks within the VM guest:

Cache mode
IO mode

In VMM (Virtual Machine Manager) if you create a new VM guest these are both set to "default". For newer versions of KVM / qemu this default seems to be:

Cache mode: writeback
IO mode: threads

The preferred configuration for both settings depends on the kind of storage you use for guest disk images:

Disk file (e.g. qcow2-file on an ext4 partition):
Cache mode: writeback
IO mode: threads

Block device (e.g. logical volume):
Cache mode: writethrough
IO mode: native

These settings are just a rough starting point. Because there are many layers of disk io and caching involved (guest application, guest fs / kernel, host fs / kernel, raid controller, hard drive cache, etc.) every installation is different and it is therefore almost impossible to give a general rule of thumb. You need to experiment yourself to find the best combination.


Security check for postfix (STARTTLS connection)

$ openssl s_client -tls1_2 -cipher ECDHE-RSA-AES128-GCM-SHA256 -starttls smtp -verify 3 -verify_return_error -debug -CApath /etc/ssl/certs -connect

"-tls1_2" forces the TLSv1.2 protocol. Make sure protocol and cipher list match.

"-verify 3" enables server certificate verification and sets the length of the certificate chain. In this case there are 3 certificates in the certificate chain, including the root CA. Make sure the public root CA certificate is in the "-CApath" directory. "-verify_return_error" enforces the certificate verification to succeed.

The "-cipher" option specifies the list of ciphers to be transferred to the server. The server then decides which of these ciphers to use. As we only give one cipher, we force the postfix server to only use this one. If the server does not support this cipher, openssl will return with an error.

If everything goes well, you will see a long output from the server (including the protocol and cipher from your openssl command line options) and something like "Verify return code: 0 (ok)". Quit the connection with the postfix server by typing "quit" and hit return.