I have found that what kernel resource ran out on the client mail server was the number of open files.  I’ve bumped that up by 5x, hopefully that will be sufficient.

Dec 3 08:57:22 mail kernel: [371342.776244] VFS: file-max limit 194527 reached
Dec 3 09:43:33 mail kernel: [374113.872327] VFS: file-max limit 194527 reached
Dec 3 09:43:33 mail kernel: [374113.905264] VFS: file-max limit 194527 reached
Dec 3 09:45:17 mail kernel: [374217.999876] VFS: file-max limit 194527 reached
Dec 3 09:45:23 mail kernel: [374223.425007] VFS: file-max limit 194527 reached
Dec 3 09:45:24 mail kernel: [374224.590837] VFS: file-max limit 194527 reached
Dec 3 10:21:11 mail kernel: [376372.038404] VFS: file-max limit 194527 reached
Dec 3 10:21:24 mail kernel: [376385.096118] VFS: file-max limit 194527 reached

     Our client mail server got flaky between around 11AM and 1PM today.

     I am pretty sure the cause is brute force password attacks exhausting some kernel resource but I have not been able to identify the resource being exhausted.

     The reason I believe this is the cause is that in the last two days the number of IP addresses we lock out for these sorts of attacks has increased from a typical number of several hundred to over 15,000.  This is probably the result of a new Windows virus that is allowing the creation of huge botnets.  This is something we see periodically.

     I rebooted the server which restored it to normal functionality and will continue to try to determine what is being exhausted and correct it.

Last night I was up until 6AM doing reboots and backups. It is not
unusual for NFS mount points to not mount or NIS to not bind after a reboot.  Those are bugs I am used to and always check for.

But postfix not starting is unusual, I didn’t check, didn’t notice,
went to sleep and so it didn’t get fixed until someone called around 2PM.

I’ll work on some sort of automated monitoring solution.

     I will be rebooting the physical host machines Friday morning around 2AM.  This will affect most everything.  Downtime should be less than about 15 minutes if everything goes as planned.  This is to load new 5.4 kernels.

     I’ve upgraded the encryption suite on our web server to modern encryption.  An unfortunate side effect is that it will break compatibility with IE8 on WinXP and versions of Android 2.73 or earlier.

     If you are running any Debian derived operating system, Ubuntu, Debian, Mint, Zorin, Julinux, etc, you may wish to try a tickless kernel.

     What tickless kernels do for you is eliminate the CPU from having to wake-up to service clock interrupts unless there is actual work to be done.  This saves a significant amount of CPU time on a single machine these days, particularly if Intel, because the overhead of context switching has increased significantly as the result of various CPU flaws that have to be worked around.

     This can significantly improve battery life in a laptop or tablet.

     Where this kernel really shines though is on the server side when hosting multiple virtual machines.  Each virtual machine has it’s own clock ticks wasting CPU, plus the host.  This can end up eating more CPU than the actual work the machines are doing.

     I have made available two kernels both based upon the recently released 5.4 final. One is called “client” and is intended for end user systems where interactive response and low latency are important.  This kernel is entirely preemptive.

     The second is called “server” and is for server loads like physical hosts hosting virtual machines, web servers, mail servers, etc.  It is the same as the client kernel except that it is non-preemptive.

     Both of these are based upon Ubuntu 19.10 configuration except modified to be completely tickless and modified to work on the i7-6850k / Asus z-190 systems.  The stock kernels do not talk to the network on these systems.  ALL CPUs and hardware supported by the Ubuntu kernels plus the above is supported by these.

     You can download these via ftp from ftp.eskimo.com:

        Login: anonymous

        Password: You@whereever.net (your e-mail address).

     Complaints, suggestions, questions can be sent to: nanook@eskimo.com

     After logging in cd /pub/kernel/linux-5.4-tickless/[client|server]

     Then prompt off

     mget *.deb

     After you’ve retrieved the files install with:

     dpkg -i *.deb

     These kernels aren’t signed.  Sorry but I’m not willing to buy in to the Microsoft Secure Boot extortion scheme, so you will need to have secure boot turned off to use these.

      Fedora.eskimo.com is now RawHide, a rolling release, rather than 29, 30, etc.  This means it’s always on the bleeding edge.  If you need a more stable Redhat based release I suggest Centos7.eskimo.com or Centos8.yellow-snow.net.

     Fedora is in the process of an operating system upgrade.  Unlike Debian based systems, RedHat systems are brain dead and can’t be upgraded while in use.

     It should be available Friday.

     Guacamole is again operational.

     I am going to take the web server down for about a half hour shortly after midnight to image it so I do not lose the work I’ve done today.