We are always trying to improve the speed, reliability, and functionality of our products.

     To this end we’ve made the following upgrades to our web server:

• Upgraded to Apache 2.4.41
• Added read-ahead and writeback caching to the file systems.
• Added gzip compression to our server configuration.
• Enabled transparent huge pages for both web server and our physical hosts.
• Changed the way the network interface is configured.
• Made changes to systemd configuration to insure prerequisites are up and running before Apache is started.

     The last two items are primarily aimed at improving boot time so during a server reboot downtime is reduced.

     The upgrade to Apache 2.4.41 fixes a couple of flaws in Apache that could be used in a denial of service attack against our web server.

     Gzip compression will improve page load time over slower links by reducing total page time.  Note that Pingdom tests will report that we do not have gzip enabled.  This is untrue but because we include some code that calls Twitter and Twitter does not compress, it causes this test to fail.

     If you have a WordPress plugin that does gzip compression, please disable the gzip function in this plugin.  Otherwise double gzip compression will result which browsers will not know how to handle resulting in gibberish.

     Transparent huge pages is largely a major win for MariaDB used to provide database operations for WordPress, other content management systems, and other applications on our website and on customer websites.  The combination of having this enabled on our physical host machine and web server improves the response time of MariaDB by approximately 400%.

     It appears the web server crashed at 17:09 but nobody called until 19.35.  It has been restored to service.  I am unable to determine the cause at this point.  I will be changing the kernel later tonight on the off chance it is a kernel issue.  It left the machine in a state without any errors on the console and nothing logged in the way of errors that provided any clues.

     It is restored to service.

     An update to bind 9 broke the name server by requiring a very different configuration.  I had been meaning to move it to a more modern platform anyway so this is in the works.  Until this is completed, this name server will not function.

      Maintenance was completed around 4:30 AM this morning.  Anything down is broken.  Please report anything down using either https://www.eskimo.com/ -> Support -> Tickets or e-mail support@eskimo.com or call 206-812-0051.  Thank you.

     I did not complete all the maintenance I had intended last night so manjaro and social are presently down for maintenance.  Expected down time less than one hour.

     I will be performing maintenance activity tonight after about 11PM that will involve the reboot of various servers and taking some down for backup imaging.  This requires about 15-45 minutes per machine.  Where possible I will attempt to leave functional mates up but this will not always be possible, for example when rebooting igloo or iglulik which have the mail spool and home directories on them respectively, both of these will result in unavoidable interruptions system wide.

     Things are mostly back up now.  I will be checking for proper NFS/NIS mounts over the next couple of hours but things should be basically functional.

     The machine which hosts the mail spool, client mail server, and some other hosts is currently down.  It failed to reboot because of missing software prohibiting it from accessing the Internet.  Since it can not access the Internet I can not restore the missing software that way.  I attempted to restore from backups but do not know all of the files involved in the package.  I am burning a DVD and will be heading back down to the co-lo to complete the restoration once I have that in hand.

     Rebooting all servers this morning (Sunday July 21st) to make upgrades effective.

     Borg Backup is installed on all of our shell servers at the request of one of our customers.

     We make backups automatically of all user home directories and e-mail weekly.  And both of these things are also on RAID10 partitions so data will not be lost in the event of a drive failure.

     But if you need things backup more frequently than that you can use this to backup data to your machine or your machine to your file space here.