The root cause was out of processes, and later I apparently ran out of iptable entries.
I finally found that "cat /proc/user_beancounters" shows all the available resources and imposed limits of the Virtuozzo virtualisation. For my quite old Ubuntu 14.04 these limits were significantly lower than what is advertised on the Link above. This may be because the virtual machine was set-up before new limits have taken effect.
The limits which are most relevant to me are:
- numproc: Max. 100 processes (each thread counts, not just processes in the UNIX sense).
- numiptent: Max. 400 iptables entries. This is usually sufficient for eveb complex firewalls but becomes a problem when using fail2ban.
In the end it was sufficient to move the ssh port away from port 2222, which was attacked a lot. The attacks on the mail ports come from a smaller set of IPs. I am currently (2019-04) using 88 of 400 iptable entries.
Find all destination ports used by TCP brute-force attacks on a server: zgrep firewall /var/log/syslog*|python -c "import sys;[sys.stdout.write(word + '\n') for line in sys.stdin for word in line.split() if word.startswith('DPT=')]"|sort -n -t= -k 2|uniq -c The above assumes that you log all stray TCP packets to /var/log/syslog with the prefix 'firewall', which can be done with this iptables rule: iptables -A INPUT -m limit --limit 200/hour --limit-burst 50 -j LOG --log-level warning --log-prefix firewall:
If you find that pressing Ctrl-Q and Ctrl-S has no effect at all in a specific GNU screen terminal than you most likely have accidentally switched on XON/XOFF flow control by hitting Ctrl-A F. The symptom is that you cannot save using Ctrl-X Ctrl-S in Emacs, and you cannot use Ctrl-Q for quoted-insert in Emacs. The Ctrl-S and Ctrl-Q keys are silently ignored.
To make the Ctrl-S and Ctrl-Q keys work again you must disable flow control in screen: Press Ctrl-A F until "-flow" is displayed. Now the behavior should be back to normal: GNU screen no longer handles Ctrl-Q and Ctrl-S specially and passes these to the application.
Problem: I would like to store a backup disk in a different physical place every now and then, just to be sure in case of severe incident like a fire etc. Different physical place means to give up the control where the disk goes: It might be stolen, or it might simply be lost, or it might be sold on ebay by accident. Ok, I totally made the ebay thing up. :-)
Solution: Keep the backup disk encrypted.
Whole disk encryption is not as secure as it sounds, due to the limitations of sector-wise access, but it is _way_ better than not encrypting the disk at all, and it usually provides solid confidentiality (meaning an attacker cannot read any information from the disk).
(There are nasty attacks on whole disk encryption, but they are all around an attacker writing stuff to the disk, modifying it contents in various ways. But if the encryption is done properly all of these attacks leave random garbage in the decrypted view of the disk.)
I am using 'plain dm-crypt' instead of LUKS since I do not need any of the LUKS features and it is very easy to set up and very easy to understand what happens.
I am also using the entire disk as one big partition, without a partition table. Encrypted partition tables do not make much sense in my opinion, and I only want one partition anyway.
If there was already (plain) data on the disk (e.g. an unencrypted backup) it is wise to erase that by overwriting the disk with random garbage:
shred -n 1 /dev/sdX
Create encrypted layer of the disk:
cryptsetup --cipher=aes-cbc-essiv:sha256 --key-size 128 --key-file=key.bin open --type plain /dev/sdX enc
(I am using aes-cbc-essiv:sha256 with just a 128 bit key since it seems sufficient for my purpose. If you are concerned that somebody might modify the contents of the disk you are better off with using aes-xts-essiv:sha256 with a 512 bit key, but I was not happy with the performance penalty for very little benefit (if any).)
The file key.bin contains the 16 byte (128 bit) binary AES key. I created it using:
head -c 16 /dev/random > key.bin
I keep the key.bin file in plain on my server since I am not nervous of it getting lost. People with access to the server are unlikely to want to decrypt the backup disk of the server since they have full access to the data (even full access to the data on the encrypted disk!) anyway. Of course it is necessary to keep a backup of this key in another physical place, e.g. printed on paper etc. Otherwise the backup disk is useless.
The plain view of the encrypted disk image is now available under /dev/mapper/enc.
Creating an ext4 filesystem on the encrypted disk:
mke2fs -m 0 -t ext4 /dev/mapper/enc
Mount encrypted disk:
mount /dev/mapper/enc /backup_disk
Now you can create a backup on /backup_disk.
When done, umount the disk and stop the encryption (which might flush a few sectors to the disk):
Problem: Logging dropped packets using iptables LOG policy did not work. Well, the precise problem was, that the messages did not end up in /var/log/kern.log (nor in any other log file), but they did show in 'dmesg' and /proc/kmsg.
This one was tough. Googling showed that this problem could be solved by uncommenting the line '$ModLoad imklog' in /etc/rsyslog.conf. This in turn caused kern.log to be filled with messages like 'imklog: error reading kernel log - shutting down: Bad file descriptor'. This in turn is probably caused by some strange interaction between Ubuntu and OpenVZ/virtualization combination, but initially I did not find the precise cause of the problem, nor a solution.
This link finally gives a clue what the problem is and also presents a working workaround:
The original Brother toner cartridges for the Brother MFC-9340CDW printer runs out of toner after a programmed amount of toner usage which is of course useless. This is how to reset the toners so you can continue to print for quite some time:
- Close all warning and error message windows and return to the main menu.
- Open the Fax menu
- Mark the '*' Button with a pencil, or just remember where it is. Do not press it. It will disappear in the next step.
- Return to the main menu.
- Open the printer so you can see the toners.
- Press and hold the hidden '*' button for 7 seconds.
- A toner reset menu appears. Reset the toner in question.
Holding down shift while pressing the right mouse button in the selected area copies the text and removes all linefeeds. This is not proper cut/copy/paste of multiple lines, but works for a single long line spanning multiple terminal lines. It is better than nothing.
Goal: Let Volumio 1.55 automatically start playing music automatically after the Raspberry Pi is powered on. The RPi is used completely headless and without any buttons, and while it is fine that I can control it through my phone or a computer, I sometimes just want to start the music on random play by flicking a plain old power switch and nothing more.
After a lot of fiddling with /etc/rc.local I realized that the NAS is mounted only after my commands in rc.local were executed. A sleep may help, but rc.local is killed if you sleep to long ... argh, nasty!
Instead I decided to hardcode the autoplay into the PHP scripts directly. I don't know any PHP, but how hard can it be?
Edit file /var/www/command/player_wrk.php and search for 'WORKER MAIN LOOP'.
Add the following lines before the '// --- WORKER MAIN LOOP --- //' line:
$cmd = 'amixer cset numid=3 1';
$cmd = 'mpc repeat on';
$cmd = 'mpc random on';
$cmd = 'mpc consume off';
$cmd = 'mpc single off';
$cmd = 'mpc play';
The amixer line tries to force 3.5mm jack audio output. Not sure this works. Omit it if you use your own DAC or HDMI.
Not sure the sleep(5) is necessary. This is actually not specific for NAS and should work with any source. It should play the playlist which was last active.
I would love to have this in the GUI, but I do not know how to hack that in.
Ah yes. Completely unrelated and just so I do not forget it: To get my TL-WN725N WLAN Dongle working with volumio 1.55 I had to download the firmware for it:
My Samsung SSD 840 EVO was suffering from the apparently well known problem that 'old' files (files which have been written a long while ago and not touched since) had slow read transfer rates (as low as 5 MBytes/s in places, 29 MBytes/s for some big files I had).
I have a Zotac server without a CD drive, so I needed to run the Samsung Performance Restauration tool. I used the DOS / Mac variant which is a DOS bootable disk which contains the update tool. The easiest way to prepare the USB stick is NOT to use the USB zip provided by Samsung but to use the *.iso file (Samsung_Performance_Restoration.iso) instead:
- Mount the ISO file and get the file ISOLINUX/BTDSK.IMG from it.
- dd the BTDSK.IMG directly onto a USB stick
- boot from the USB stick and follow the instructions
On my Zotac machine when booting from this USB stick I got a couple of broken error messages about not being able to boot from device XYZ, but it booted OK from the stick after a couple of seconds.
The performance restauration procedure took 4.5h for a 1TB SSD (60% full) and about 15h for another 1TB SSD (95% full), so expect this to take some time.
My Zotac server did not reboot without a monitor attached. Booting from power-up did work ok. The graphical console of grub seemed to be the problem. Since I do not use the monitor output at all not setting any graphics mode at all and using the 80x25 default console is fine for me. This is how to disable the graphical grub console:
Uncomment this line: GRUB_TERMINAL=console
In addition I disabled setting any graphics mode on the Linux kernel and setting the timeout to 1 second.
Step 1: Make sure you can execute commands as root on the remote host after logging in, without entering your password. For this there are two alternative options:
You can allow the user to execute sudo without entring the users password. You can do this by adding the following line to the end of /etc/sudoers:
You can enter the sudo password in advance once and make sure it is cached between sessions. To make sure it is cached between sessions you must add the '!tty_tickets' option to the Defaults line in /etc/sudoers, and then you must run a dummy command as root:
ssh -p PORT -t USER@REMOTE_HOST sudo id
Step 2: Do the rsync. Override the ssh port. Specify "sudo rsync" as remote rsync command. Target the rsync at the non-root user which can now sudo:
Assuming you have a PDF which has certain restrictions on it (for example you cannot edit the PDF using Preview in the usual way), but which you can print, this is how you can strip these restrictions from the PDF:
open the Printer Queue for your Printer
stop the Queue
print the PDF
now you should have a file starting with 'd' under /var/spool/cups which is an unprotected version of your PDF
in Terminal type sudo cp /var/spool/cups/d* ~/Desktop/d.pdf
this assumes there is just one file starting with a 'd' which is usually the case. If there are multiple files pick one by one until you have your file.
in Terminal type sudo chown foo:foo ~/Desktop/d.pdf (replace foo with your actual user name)
file ~/Desktop/d.pdf does not have any restrictions
in the Printer Queue delete the print job
start the Printer Queue
Note that this procedure does not allow you to strip passwords from files which you cannot print without a password. The restricted file must at least allow you to print the file.