Well about an hour and a half into it, iglulik puked out an NFS related kernel oops, four of the eight knsfd’s locked up and the machine ground to a halt.  I had to go to the co-lo to reboot and bring it back to a 5.4 kernel along with the other NFS servers.  As a client 5.8 seems to work fine so I left it on those machines which aren’t NFS servers so we still are getting a substantial performance boost.

     I will file yet another bug report with bugzilla but the kernel oops gives it a better chance of getting fixed since it tells them exactly where it was and what it was doing when it barfed, and what it barfed on is a spinlock problem, a spinlock is a kind of lock used to protect critical kernel code that is not thread-safe.

      Starting at 11pm, I will be rebooting all of our Intel based servers.  I expect this to be completed by midnight.  I am doing this to install Linux 5.8, testing has shown it to be a huge performance win.  I hope that the NFS stability issues of 5.7 are totally resolved, so far no problems have shown up in testing but it is difficult to generate the diversity of traffic present in the live servers.

     I will be taking the mail server down for imaging for approximately 20 minutes.  Imaging is making a backup of the virtual machine which must be quiescent during the operation.

     This morning’s maintenance has been completed.

     All servers (except the old eskimo.com server) will be rebooted tonight shortly after midnight.  This is necessary because security updates were installed that require a reboot to be effective.  This should be completed within about half an hour.

     An in place upgrade to Mint 20 failed, their script for setting up python had inappropriate syntax for the version of python being installed.

     A clean install of Mate 20 also failed.  Mate Desktop did not work, could not fully install because of package dependency issues, on a fresh system.

     Mint has been return to 19.3 which at least functions.

     An upgrade of the Mint shell server from 19.3 to 20.0 is in progress.

     Because this is an upgrade involving an upgrade of the underlying Ubuntu system from 18.04 to 20.04, and thus has a greater chance than normal of going south, I am performing a backup of Mint first.

     This whole process will result in the server being slower than normal while it is ongoing and there will be times when various sub-systems are temporarily dysfunctional.

     If all possible it would be better to use one of the alternate debian based systems, Ubuntu, Debian, MxLinux, Julinux, or Zorin during this period.

     I will make an additional announcement when the upgrade has completed.

     Maintenance on mail, virtual private, and shell servers has been completed.

     Tonight starting at around 11pm I will be taking these machines down for maintenance, zorin, centos8, and manjaro, then vps1, centos7, then mail and Debian. 

     Downtime should be about 1/2 hour except for Debian which usually takes about an hour.

     Tonights’ maintenance is done.  Things did not take as long as I had expected.