Category Archives: Expert

Using the German electronic identity card (eID) in Ubuntu 20.04

The new eID functionality of the German identity card enables you to identify yourself with your real name towards government or commercial web services. It makes sure that it is really you who uses the web service, and not someone else who stole your online identity by email spoofing, SIM swapping, IMSI catcher, etc. .

In the following example, we will be using the eID to sign our PGP key. The new signature will uniquely identify the owner of the German identity card as the owner of the PGP key, which can then be used to e.g. sign and encrypt emails. That way PGP no longer relies on a web of trust, but works similar to the PKI concept of S/MIME certificates, in that the real identity of the owner of a certificate will be checked and then signed by a common public authority (CA) that everyone trusts.


  • A German identity card with eID functionality.
  • A supported RFID card reader, e.g. from REINER SCT.
  • Operating system drivers for your card reader. In Ubuntu 20.04 drivers for all REINER SCT card readers (also called "cyberJack") are included in the package libifd-cyberjack6. You can download Ubuntu drivers from their website too, but they didn't work for me.
  • On Linux, the pcscd daemon that enables access to smart card readers.
  • An application called AusweisApp2 that handles authentication (PIN entry) and authorization (who wants to access what kind of information on your eID). In Ubuntu 20.04 AusweisApp2 is already included in the standard repositories (version 1.20.0). The app is also included as a snap install (newer version 1.20.2), but that didn't work for me (for the error message see below).

First steps

  • Make sure you have the letter with the initial PIN for your eID at hand.
  • IMPORTANT: Make sure your RFID card reader is updated to the latest firmware release. With most card readers, the firmware can only be updated while you install the card reader on a Windows system.
  • IMPORTANT: Remove usbguard. Even after I permanently added the card reader to the list of allowed devices, pcscd could not find my card reader, or AusweisApp2 did not properly recognize my card reader and complained about missing drivers.
  • Install all necessary software packages and drivers for Ubuntu 20.04:
    pcscd pcsc-tools libifd-cyberjack6 libusb-1.0-0 libusb-1.0-0 libccid libpcsclite1 libpcsc-perl libpcsclite-dev

Test your card reader

Start the pcscd daemon in debug mode:

$ sudo pcscd -df
00000000 [140135772616640] pcscdaemon.c:347:main() pcscd set to foreground with debug send to stdout
00000086 [140135772616640] configfile.l:293:DBGetReaderListDir() Parsing conf directory: /etc/reader.conf.d
00000017 [140135772616640] configfile.l:329:DBGetReaderListDir() Skipping non regular file: ..
00000006 [140135772616640] configfile.l:369:DBGetReaderList() Parsing conf file: /etc/reader.conf.d/libccidtwin
00000029 [140135772616640] configfile.l:329:DBGetReaderListDir() Skipping non regular file: .
00000009 [140135772616640] pcscdaemon.c:663:main() pcsc-lite 1.8.26 daemon ready.
00003514 [140135772616640] hotplug_libudev.c:299:get_driver() Looking for a driver for VID: 0xABCD, PID: 0x1234, path: /dev/bus/usb/001/001

Plug in your card reader.

IMPORTANT: If you use a USB card reader, plug it directly into your PC or laptop. Do not use a USB hub, as the hub may not provide enough power for the USB device. Also make sure to use the USB cable that came with the card reader. Longer cables may result in unstable connections.

In the output of the pcscd daemon (after a couple of seconds, wait for it!), you will see something like this:

99999999 [140135764219648] hotplug_libudev.c:655:HPEstablishUSBNotifications() USB Device add
00000158 [140135764219648] hotplug_libudev.c:299:get_driver() Looking for a driver for VID: 0x0C4B, PID: 0x0500, path: /dev/bus/usb/002/012
00000010 [140135764219648] hotplug_libudev.c:440:HPAddDevice() Adding USB device: REINER SCT cyberJack RFID standard
00000050 [140135764219648] readerfactory.c:1074:RFInitializeReader() Attempting startup of REINER SCT cyberJack RFID standard (1234567890) 00 00 using /usr/lib/pcsc/drivers/l
00001347 [140135764219648] readerfactory.c:950:RFBindFunctions() Loading IFD Handler 3.0
00023288 [140135764219648] readerfactory.c:391:RFAddReader() Using the pcscd polling thread

Notice that the pcscd daemon uses the driver from the package libifd-cyberjack we installed earlier. You can also check the output from the pcscd client tool:

$ pcsc_scan
Using reader plug'n play mechanism
Scanning present readers...
0: REINER SCT cyberJack RFID standard (1234567890) 00 00

Thu Nov 19 13:17:31 2020
Reader 0: REINER SCT cyberJack RFID standard (1234567890) 00 00
 Event number: 0
 Card state: Card removed,

As you can see, pcscd properly detected the card reader. Now insert your identity card into the card reader while pcsc_scan is running. The output of pcsc_scan will show something like this:

Thu Nov 19 13:21:24 2020
Reader 0: REINER SCT cyberJack RFID standard (1234567890) 00 00
 Event number: 3
 Card state: Card inserted,
Possibly identified card (using /usr/share/pcsc/smartcard_list.txt):
       Personalausweis (German Identity Card) (eID)

Install and start the application AusweisApp2

Install the application AusweisApp2 from the general Ubuntu repository. Do not install the snap app! In my case, the snap version of AusweisApp2 did not work properly. I got the following error message in my system logs:

Nov 18 17:32:03 server ausweisapp2-ce.pcscd[6911]: 07606784 readerfactory.c:1105:RFInitializeReader() Open Port 0x200000 Failed (usb:0c4b/0500:libudev:0:/dev/bus/usb/002/006)
Nov 18 17:32:03 server ausweisapp2-ce.pcscd[6911]: 00000015 readerfactory.c:376:RFAddReader() REINER SCT cyberJack RFID standard (1234567890) init failed.
Nov 18 17:32:03 server ausweisapp2-ce.pcscd[6911]: 00000073 hotplug_libudev.c:526:HPAddDevice() Failed adding USB device: REINER SCT cyberJack RFID standard

After you start the application, go to Start -> Settings -> USB card reader to check if the app can communicate with your card reader.

If you haven't done so before, the app will ask you to change the initial PIN that you received by mail. You have to set your own PIN before you use any online service.

Test the authentication process

Go to Start -> Self-Authentication -> See my personal data. Here you can check the data that is stored on your eID, and also make sure that the authentication process is working properly.

Click on "Proceed to PIN entry". On your card reader, you will need to confirm the service provider who wants to access your card, and also which information is requested from your card. Of course you also need to enter your new PIN.

Sign your PGP certificate

Go to Start -> Provider -> Other services -> Schlüsselbeglaubigung. The key signing service is provided by Governikus, the company that develops AusweisApp2.

Click on "To online application". This will start your default web browser and open the URL . Of course you can also enter the URL directly in your web browser. Just make sure that AusweisApp2 is running in the background.

On the website you may upload your PGP public certificate. After successful authentication by eID, you will receive an email with your certificate signed by Governikus. The signature certifies that the PGP key really belongs to you and not someone else who is impersonating you by using your email address (email spoofing) or smartphone number (SIM card swapping, IMSI catcher).


The whole eID authentication process on a website can be described as follows:

  1. Start the pcscd daemon, either by "sudo systemctl start pcscd", or if this doesn't work by "sudo pcscd -f".
  2. Plug in your card reader. You should see a confirmation in the daemon output (or by typing "systemctl status pcscd" if you started pcscd with systemctl):
    "CYBERJACK: Started"
  3. Start the application AusweisApp2.
  4. Go to the website that requests eID authentication ("elektronischer Personalausweis"), and click on "Login".
  5. Your webbrowser automatically transfers control to AusweisApp2. There you should see who is requesting what kind of information from your eID.
  6. Insert the identity card into your card reader.
  7. In AusweisApp2, click on "Proceed to PIN entry".
  8. Control is transferred to your card reader. There you need to:
    1. Confirm the service provider.
    2. Confirm the data he wants to have access to.
    3. Enter your PIN.
  9. On the display of your card reader, you should see something like "Tunnel established". AusweisApp2 shows something like "Authentication successful". The website should automatically proceed to its regular contents, just as if you would have entered username and password.
  10. That's it. You can remove your identity card from the card reader.


  • If you see the following error message in the output of pcsc_scan, it means that pcsc_scan cannot communicate with the daemon pcscd. Make sure that the daemon is running.
SCardGetStatusChange: RPC transport error.
  • If AusweisApp2 does not recognize your card reader, or complains about missing drivers, try to start pcscd from the command line ("sudo pcscd -f"), and not as a background service ("sudo systemctl start pcscd"). Also make sure that you removed usbguard and did a reboot afterwards.
  • If the authentication process is not working, try to update the firmware of your smart card reader to the latest version. This might only work under Windows 10 during Windows driver installation for the new smart card reader device.

Don't reinvent the wheel ...

As a developer or DevOps you probably heard it a hundred times before:
" ... ah and by the way for the new project ... try to find an existing library to solve the problem. Don't reinvent the wheel, that would cost too much time."

And probably there were more than a couple of times when - after scrambling through dozens of GitHub projects - you thought: "I wish I had started to reinvent the wheel right from the beginning!"

So here are my top 5 reasons why to "reinvent the wheel":

  1. The only already existing code is a "rainy-Sunday-afternoon-good-enough-for-my-Raspberry Pi-home-project-provided-as-is-without-comments-or-error-checking-only-once-tested-on-my-12-year-old-20-minutes-startup-time-crashes-every-2-hours-Windows-Vista-laptop" GitHub project.
  2. The only GitHub project you can find is bloated with functionality you don't need at all, and it takes you 2 days to find out that the 10 lines of poorly written code that you actually need does not work anyway.
  3. You need to install 15 additional mysterious libraries that were last updated 8 years ago.
  4. The only existing project is a 8 GB of RAM sucking Java monster that takes 5 minutes for cold start.
  5. You spend the next 2 years fixing bugs in code that was never meant to run in production environment.


Are you safe from BGP hijacking?

How serious is your ISP about Internet security? There is a website now where you can check if your ISP is vulnerable to BGP hijacking:

The website also contains additional background information about BGP hijacking and how to prevent it in the first place.

ISP = Internet Service Provider, the company that hooks you up to the Internet


Let's Encrypt Certificate for SMTP with STARTTLS

TLS Encryption

Let's Encrypt provides an easy way to get free certificates not only for web servers, but also for email servers like Postfix.

The way Let's Encrypt usually works requires you to setup a web server. Let's Encrypt sends you a challenge, and you have to prove ownership of the domain by providing a response to that challenge. You do this by placing the response in a certain URL on your web server:

That way you prove that you are the owner of the domain "". But there is another even easier way to prove ownership of a domain: DNS. You place the response in a specific TXT record of your domain:

  • You can use your domain hosting service (GoDaddy, Whois, etc.) to create a new TXT record.
  • The "certbot" command line client does all the rest in just one call.
  • Under Debian 9 and 10, "certbot" is part of the official package repository.
  • You can run certbot on any Linux client. You don't have to run it on the email server.


In this example the public hostname of your mail server is Therefore you have to create a TXT record called . The value of the TXT record is in the output of certbot.

# certbot certonly --manual --preferred-challenges dns -d
Saving debug log to /var/log/letsencrypt/letsencrypt.log 
Plugins selected: Authenticator manual, Installer None 
Obtaining a new certificate 
Performing the following challenges: 
dns-01 challenge for 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
NOTE: The IP of this machine will be publicly logged as having requested this 
certificate. If you're running certbot in manual mode on a machine that is not 
your server, please ensure you're okay with that. 
Are you OK with your IP being logged? 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
(Y)es/(N)o: Y 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Please deploy a DNS TXT record under the name with the following value: 
Before continuing, verify the record is deployed. 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Press Enter to Continue 
Waiting for verification... 
Cleaning up challenges 
 - Congratulations! Your certificate and chain have been saved at: 
   Your key file has been saved at: 
   Your cert will expire on 2020-02-15. To obtain a new or tweaked 
   version of this certificate in the future, simply run certbot 
   again. To non-interactively renew *all* of your certificates, run 
   "certbot renew" 
 - If you like Certbot, please consider supporting our work by: 
   Donating to ISRG / Let's Encrypt: 
   Donating to EFF:          

C++ - The Beast is Back (Halloween Special)

C++ The Beast is Back

Do you remember the time when programming languages like Visual Basic and Java came out and flourished, because they let programmers forget about all the underlying technical details of computers, so they could focus more on things like algorithms and use cases?

"I don't wanna waste my time with solving memory management problems or all those other low level stuff. These days are finally over!" Many programmers hated C++ because they were annoyed by memory leaks, pointers, byte sizes of variable types and data structures, compiler errors, linker warnings, ... the list goes on and on.

And managers? They loved Java and Visual Basic. Less tech talk about problems nobody really understands anyway, faster time to market, happy customers, what else could you want?

It seemed like the days of C and C++ were counted. Maybe they could still be used for some low level system programming, but certainly not application programming. Instead let's move on and jump on the ponderous but convenient bytecode train. Just add some more RAM modules to the server, and triple the disk space of those cloud containers, then we're done. What a beautiful simple world it is now, the world of software programming. Right? Right?? Right???

The Return of the Beast

Well ... not so fast (pun intended). New emerging technologies like Big Data, Blockchain and AI become part of everyday application development. And what about IoT (edge cloud)? Small IoT devices don't have Terabytes of RAM and server scale CPU processors. All of these rapidly growing technologies require lean and fast code modules tailored to their specific requirements.

I recently came across a free eBook from O'Reilly: C++ Today - The Beast is Back. It is from 2015, but large parts are still valid today. I highly recommend reading it. Once you have finished, here is my very own top 6 list of reasons why "the beast is back":

  1. Coding discipline
    Source code formatting, code commenting, coding guidelines, best practices: Python brought discipline back into aspiring programmers. Coders are now less annoyed by investing time in high quality source code, because they realize it will in turn create higher quality software that is easier to maintain and safes time and money in the end. C++ development also requires a lot of discipline and attention to details, but you are rewarded with a minimum disk and memory footprint and unparalleled performance.
  2. Focus on technology
    Tech is back: Logging in with SSH to a remote git server? Using vim to fix a typo in some Python source files? No problem. Today there are more tutorials out there about vim and the Linux command line than ever before. Students again want to get in touch with the underlying technology and learn how stuff works under the hood. Knowing the memory footprint of a running program is not considered evil sorcery any more.
  3. New standards
    C++ has come a long way since the last decade:
    C++11, C++14, C++17, C++20
    New programming ideas and standards are coming up every year, and C++ is adapting fast.
  4. New technologies
    Blockchain, IoT, Big Data and Deep Learning: Exciting new technologies are all about performance, data crunching, sheer numbers. You need a lean and fast beast like C++ to tame them. For example the core of TensorFlow, today's most popular machine learning framework, is written in C++.
  5. Low competition
    In July 2019 Microsoft announced they are thinking about moving from C++ to Rust for developing internal and external software. My question: What do you do with the rest of the weekend? Seriously: It might sound like a great idea to get rid of stack overflow problems and the like, but porting tons of code from C++ to Rust will probably take decades. Furthermore, Rust is not nearly as developed and stable as C++. There sure are still heaps of banana skins hidden beneath the shiny new surface of Rust.
  6. Go green, go C++
    In 2018 the first YouTube video hit 5 billion views and burned as much energy as 40,000 US homes use in a year. This should make it very clear that every innocent clickety-click-click-barely-touching-the-shiny-polished-surface-of-your-tiny-cutesy-iphone has a huge impact on telecommunication infrastructure and cloud data centers spread around the globe. Cloud services are run by software. The more efficient the software is, the less energy these services consume (CPU, hard drive, memory, etc.). Unfortunately today's most famous programming languages JavaScript, Java and Python are rather energy inefficient. Compiled languages like C++ use less memory, produce less hard drive read/writes and consume less CPU cycles, thus making them far more energy efficient.


Android smartphone "Cubot Echo"


Cubot is a Chinese Android smartphone brand that offers a wide variety of inexpensive phone models. With the Cubot Echo (released in 2016) you get surprisingly good quality at a low price.

One of the main advantages of Cubot smartphones is their native Android version (stock Android). Many smartphone manufacturers heavily modify Android and add tons of "features" and apps that you don't really need and are more annoying than helpful. They hope to create a unique customer experience that makes users get accustomed to their brand so they choose the same brand again for their next phone. Moreover these modifications often slow down overall performance and introduce security holes.

Cubot ships all their models with an almost native Android version. No modifications (except necessary adaptions to hardware), no annoying apps or background tasks that cannot be removed, etc.

Cubot Echo

+ Very good overall hardware quality compared to cheap price (unbreakable display, strong body for outdoor use)
+ Good display, camera quality and performance compared to cheap price
+ Large 5.0 inch display
+ HDR photography
+ Up to 128 GB micro sdcard (supported, but not included)
+ Native Android user experience, no annoying modifications or add-ons
+ Removable battery
+ Cheap price

- Android security patch level only from 05.06.2017, but latest firmware update (which will be installed automatically after setup) DOES include security patch for WiFi WPA2 KRACK attack (build 08.02.2018). Android 6 Marshmallow does no longer receive security updates from Google, but you can install the unofficial Android alternative LineageOS based on Android 7 Nougat.
- No 4G / LTE support
- A bit heavy
- Released in 2016, a little bit outdated

You can get the Cubot Echo for as cheap as 60 EUR. If you can live with the security issues and the missing LTE support, that's a definitive buy. Especially considering that the upcoming Google Pixel 3 flagship for 850 EUR guarantees Android security updates for only 3 years. You could buy 14 Cubot Echos for that price. And the Google Pixel 3 does not have a removable battery, which makes it very hard to replace.

Cubot comparison chart

Cubot EchoCubot J5
Android VersionAndroid 6 Marshmallow
(no longer supported)
Unofficial support for LineageOS
based on Android 7 Nougat
Android 9
ProcessorMT6580 1.3 GHz Quad-coreMT6580 1.3 GHz Quad-core
Display5" IPS
(1300:1 contrast)
5.5" IPS
(18:9 format, 1300:1 contrast)
Brightness (cd/㎡)450450
Memory (RAM / ROM)2 GB / 16 GB2 GB / 16 GB
Max. Additional Storageup to 128 GB (not included)up to 128 GB (not included)
Camera (Back / Front)13 MP / 5 MP8 MP / 5 MP (interpolated)
Extras - Micro + standard dual SIM (no eSIM)
- Special sound chip with big speaker
- Unbreakable case
- Dual nano SIM (no eSIM)
- Curved display sides
- Gradient color case
Battery3000 mAh (removable)2800 mAh (removable)
Price~ 60 €~ 65 €
Cubot NovaCubot Magic
Android VersionAndroid 8.1 OreoAndroid 7 Nougat
ProcessorMT6739 1.5 GHz Quad-coreMT6737 1.3 GHz Quad-core
Display5.5" HD+
(18:9 format, 1300:1 contrast)
5" IPS
(1300:1 contrast)
Brightness (cd/㎡)450450
Memory (RAM / ROM)3 GB / 16 GB3 GB / 16 GB
Max. Additional Storageup to 128 GB (not included)up to 128 GB (not included)
Camera (Back / Front)13 MP / 8 MP13 MP / 5 MP
(13 MP +2 MP Dual Back Camera)
Extras - Dual 4G nano SIM (no eSIM)
- Fingerprint sensor
- Dual micro SIM and dual standby (no eSIM)
- Curved display sides
Battery2800 mAh (removable)2600 mAh (removable)
Price~ 70 €~ 70 €


Security Guidelines

Computer Security

Physical Device Security

  • Always completely switch off your computer and lock your computer safely away, even if you just visit the bathroom. Screen saver locking or putting the laptop into sleep mode is not enough (Cold Boot Attacks).
  • Don't display anything important on your computer screen (Van-Eck-Phreaking).
  • Don't type in anything important on your keyboard or touchscreen.
  • Install USBGuard to protect against unknown USB devices.
    (Note that USB IDs and serial numbers of USB devices can easily be replicated. Once an attacker knows the type of USB device you are using, and its serial number, USBGuard can easily be bypassed. That means: Never lend someone your USB stick, never accept a USB device from untrustworthy persons ... which means anyone.)
  • Ensure that air-gapped systems are not only air-gapped, but also in a completely isolated room (electromagnetic signals, monitor brightness, led lights, sound, heat, etc.)
    Academics turn RAM into Wi-Fi cards to steal data from air-gapped systems

Software Security

  • Always use fingerprints to identify TLS certificates for important web or email services. Don't rely solely on CAs.

Useful Links

  • Ubuntu Security
  • Ubuntu Security Features Matrix
  • End User Device Security Guidance for Ubuntu 18.04 LTS from the NCSC (National Cyber Security Center, part of GCHQ)

Password security - it is not about length or complexity


Passwörter sollten nach Möglichkeit nicht im Klartext am Bildschirm angezeigt werden. Neben dem offensichtlichen Shoulder Surfing ("über die Schulter schauen"), gibt es auch sog. Seitenkanalangriffe in blickgeschützten Bereichen.

Das ursprünglich für ältere Röhrenmonitore entwickelte Van-Eck-Phreaking, bei dem die elektromagnetische Strahlung über größere Distanzen aufgezeichnet wird, lässt sich offenbar auch für moderne LCD-Monitore mit HDMI-Kabel ausnutzen. Aus der empfangenen elektromagnetischen Strahlung wird dann das ursprüngliche Monitorbild rekonstruiert. Die dazu notwendige Elektronik ist mittlerweile schon für ambitionierte Hobby-Bastler erschwinglich.

Einige Quellen im Internet weisen ebenso auf relativ hohe elektromagetische Strahlungen und akustische Signale von aktuellen PC-Grafikkarten und Flachbildschirmen/Touchscreens in Kombination mit Monitor- und Stromkabeln hin, die im Prinzip wie eine Antenne funktionieren.

Um Sicherheitsproblemen in diesem Bereich von vornherein aus dem Weg zu gehen, kann man z.B. moderne Passwortmanager verwenden, die Passwörter automatisch generieren und dann über die Zwischenablage in die Anwendung kopieren, ohne das Passwort selbst im Klartext eintippen oder auf dem Bildschirm anzeigen zu müssen.


Squid, c-icap, ClamAV: Bug in the service. Please report to the service author!!!!

If you see this error in your c-icap server logfile, it might just be that c-icap is running out of temporary disk space and that the clamav/virus scanner configuration for c-icap is wrong:

Service antivirus_module
ServiceAlias  avscan virus_scan?allow204=on&sizelimit=off&mode=simple
virus_scan.MaxObjectSize  5M
TmpDir /tmp

The option "... sizelimit=off..." for the virus_scan service means that the configuration value for "MaxObjectSize" will be ingored. If you have too many parallel squid client connections open or large files to download, c-icap is running out of temporary disk space. It will then log the following error message without further explanation:

Bug in the service. Please report to the service author!!!!

The webbrowser download will be terminated with an error message (something like "internal server error").

To solve this problem, add more free space to the partition where TmpDIr resides, and change the virus_scan service option to "... sizelimit=on ...".

In the worst case, free disk space for the c-icap TmpDIr has to be:
MaxServers * ThreadsPerChild * virus_scan.MaxObjectSize


grub-install: error: disk '...' not found

If you get an error like the following, the reason for this might not be so obvious. In my case I got the following error message trying to run grub-install:

# grub-install /dev/mapper/vg1-lv_boot
Installing for i386-pc platform.
grub-install: error: disk `lvmid/OffQLW-SofZ-KH38-jrbl-RXyw-dmDc-VOJuPf/lbiWU0-SkvY-nDET-EGvy-A1PP-fmGb-dGv7yX' not found.

The logical volume I tried to install grub onto was ok (/dev/mapper/vg1-lv_boot). The problem was somewhere else: I previously had a disk failure in a RAID0 md raid. The faulty drive was replaced online by a hot spare drive. But there was still an encrypted swap device configured for the old drive. And that swap device was not part of the md raid, so it was not automatically transferred to the new spare drive.

Only after removing this non-existing swap partition (swapoff <device>) grub-install was working again. So if you come across any error message from grub-install like the one above, the reason for it might be a problem with ANY configured disk on your system. Check for the following errors:

# swapon -s

Are there any swap partitions in use that no longer exist physically?

# pvdisplay
/dev/mapper/cryptswap2: read failed after 0 of 4096 at 0: Input/output error 
/dev/mapper/cryptswap2: read failed after 0 of 4096 at 1998520320: Input/output error 
/dev/mapper/cryptswap2: read failed after 0 of 4096 at 1998577664: Input/output error 
/dev/mapper/cryptswap2: read failed after 0 of 4096 at 4096: Input/output error 
/dev/sdb: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb: read failed after 0 of 4096 at 1000204795904: Input/output error 
/dev/sdb: read failed after 0 of 4096 at 1000204877824: Input/output error 
/dev/sdb: read failed after 0 of 4096 at 4096: Input/output error 
/dev/sdb1: read failed after 0 of 4096 at 1998520320: Input/output error 
/dev/sdb1: read failed after 0 of 4096 at 1998577664: Input/output error 
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error 
/dev/sdb5: read failed after 0 of 4096 at 998203392000: Input/output error 
/dev/sdb5: read failed after 0 of 4096 at 998203449344: Input/output error 
/dev/sdb5: read failed after 0 of 4096 at 0: Input/output error 
/dev/sdb5: read failed after 0 of 4096 at 4096: Input/output error 
--- Physical volume ---

Are there any error messages for physical LVM2 volumes? If so, try to remove the erroneous physical volumes from your running configuration. Maybe there are still active mount points on the faulty disks (including swap partitions).

# dmsetup status

All entries in the device mapper list have to be valid. There might not be an obvious error message in the output, so you have to check each dm device manually.

Important things to note:

  • Grub2 no longer relies on the file /boot/grub/ . You can create the file with "grub-mkdevicemap", but grub-install does not use it and performs a full system scan by itself.
  • grub-install also examines swap devices, even though it obviously will not use them.