apt upgrade brings apache2 activated on my server

I have a server using Ubuntu Server 22.04 LTS with nginx and php supported for a long time. Today, after I fire a regular update by using apt update, apt upgrade and a reboot, my web server is down.

When checking the status of nginx, it reports that port 80 is taken:

nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Unknown error)

And the lsof command

lsof -i -P -n | grep LISTEN

reports that port 80 is used by apache2. That’s really weird. This server never has apache configured or started. Who wanna run apache aside nginx in a production environment, uhh?

By checking the log of apt, I found that php8.1 is upgraded with a library named libapache2-mod-php8.1 installed automatically. I don’t find any log related to apache installation. Maybe the apache2 is preinstalled for a very long time but never initiated but the libapache2-mod-php8.1 upgrading process triggered it on.

After I purge apache2* and libapache2-mod-php8.1, the nginx can be restarted without any problem.

Disable Boot ReadOnly with OverlayFS on PiOS

PiOS shipped with Overlay FileSystem support. When enabled, the main partition (the second one) will be locked. The protection of OverlayFS doesn’t cover the boot partition (the first one). Boot partition can be set to read-only by raspi-config or editing fstab config file.

The problem is due to the read-only of the main partition, the mounting mode (readonly or read-write) cannot be changed when OverlayFS enabled. While disabling OverlayFS through raspi-config, there is no way to set the boot partition back to read-write before OverlayFS disabled, which requires a reboot. But when enabling OverlayFS, raspi-config has an option to set the boot partition to read-only at the same time before rebooting. In another word, when disabling both OverlayFS and Boot partition ReadOnly, the PiOS need to be reboot twice.

For easing the process for disabling both OverlayFS and change the boot partition back to writable, I write a script to set the boot partition writable when OverlayFS is not enabled currently. Sadly, there are two version of the script, based on the version of raspi-config.

For early releases of PiOS, raspi-config command line returns nothing but set the exit code. But it changed recently to print the result directly without setting the exit code. You can check your version by running such a command /usr/bin/raspi-config nonint get_overlay_now when OverlayFS is not enabled. For the old releases, nothing will be printed but a 1 is printed when you run echo $? after that. For new ones, you will get a 1 printed directly after the command /usr/bin/raspi-config nonint get_overlay_now processed and the exit code will always set to 0. The lucky thing is the raspi-config seems not be upgraded by apt command.

Now, let’s create the script named /usr/local/sbin/disablebootro.sh on your disk. Of cause, don’t do that with OverlayFS enabled.

Here is the version for the old silenced raspi-config:

#!/bin/sh
/usr/bin/raspi-config nonint get_overlay_now
if [ $? -eq 1 ]; then
  echo "Overlay FS is disabled."
  /usr/bin/raspi-config nonint get_bootro_conf
  if [ $? -eq 0 ]; then
    echo "Boot RO is enabled. Disabling..."
    /usr/bin/raspi-config nonint disable_bootro
    echo "Done. Rebooting..."
    reboot
  else
    echo "Boot RO is disabled."
  fi
else
  echo "Overlay FS is enabled."
fi

And this is for the new releases:

#!/bin/sh
OverlayFS=$(/usr/bin/raspi-config nonint get_overlay_now)
if [[ $OverlayFS -eq "1" ]]; then
  echo "Overlay FS is disabled."
  BootRO=$(/usr/bin/raspi-config nonint get_bootro_conf)
  if [[ $BootRO -eq "0" ]]; then
    echo "Boot RO is enabled. Disabling..."
    /usr/bin/raspi-config nonint disable_bootro
    echo "Done. Rebooting..."
    reboot
  else
    echo "Boot RO is disabled."
  fi
else
  echo "Overlay FS is enabled."
fi

Then, assign permission to this script for running by this command below.

chmod +x /usr/local/sbin/disablebootro.sh

Now, let’s create a systemd service to run this script when booting by create a file /etc/systemd/system/DisableBootRO.service with the content below.

[Unit]
Description=Disable Boot RO when Overlay Disabled
ConditionPathExists=/usr/local/sbin/disablebootro.sh

[Service]
WorkingDirectory=/usr/bin
ExecStart=/usr/local/sbin/disablebootro.sh
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
User=root
KillMode=process

[Install]
WantedBy=multi-user.target

And set the service start with system by this command:

systemctl enable DisableBootRO.service

It’s done. Every time the PiOS booting, this script will be run. When OverlayFS is disabled but the boot partition is left as read only, it will set the boot to writable and reboot. You just need to disable OverlayFS from raspi-config and reboot, leaving the boot partition things to this script and service.

Enable rc.local on Ubuntu 20.04

For compatible reason, rc.local support is still supported by new version of Ubuntu. But it is disabled by default. This will guide you to enable running rc.local while starting system.

  1. Create rc.local if it does not exist.

Run nano /etc/rc.local. If the file does not exist, place this code below as the default rc.local file.

#!/bin/bash

exit 0

Run chmod +x /etc/rc.local to give the executing permission.

2. Create systemd service file.

Run nano /etc/systemd/system/rc-local.service to create the systemd service file and paste this text below.

[Unit]
Description=/etc/rc.local Support
ConditionPathExists=/etc/rc.local

[Service]
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99

[Install]
WantedBy=multi-user.target

3. Config systemd using this bash command below.

systemctl enable rc-local

All done. The rc-local will be run while system starting.

Warning: When Upgrading to Ubuntu Server 20.04

Ubuntu, including server, 20.04 is launched several days ago. Many servers with 19.10 or earlier system are planed to upgrade soon. There are some issues should be noticed before you process.

1. If do-release-upgrade returns nothing found, you may want to use

do-release-upgrade -d

instead.

2. Some servers will be stuck with “Updating database of manual pages” process. In my tests on about 20 servers, about 20% servers will pass it with no more than 2 minutes delay. More than half servers, mainly on Azure, stuck there for 20-40 minutes. During this time, around 100 GB data are read from disk with 100% CPU using, reported by Azure portal. I don’t know what it read. All my files on the server cost no more than 20GB disk space. One of my server, hosted on Azure, costs more than half day so far (and not finished yet), more than 200GB data read. One server failed in that step, SSH drop and cannot reconnect at all. Before this problem solved, I strongly NOT recommend to upgrade your product server.

Update: The one stuck on Azure resume to idle after 14 hrs, 626.14GB reading and 27GB writing. Due to dead SSH process, I have to try to reboot the server. But the server never come back online. A kernel panic captured by Azure console while booting:

Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-1010-azure #10-Ubuntu
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
Call Trace:
  dump_stack+0x57/0x7a
  ? rest_init+0x50/0xb0
  panic+0xfb/0x2d7
  ? do_execve+0x25/0x30
  ? rest_init+0xb0/0xb0
  kernel_init+0xfb/0x100
  ret_from_fork+0x35/0x40
 Kernel Offset: 0x30000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

3. Upgrading will replace php 7.3 with 7.4. If some site is powered by php-fpm with Nginx, you may need to change your web site configuration file in Nginx available sites folder. All packages installed by pip3 will be dropped, not uninstalled properly. You may want to uninstall them if you don’t want to use it any more before upgrading system. Or, you can reinstall them after upgrading.

Deploy a dotnet core site on nginx and systemd

This article is about how to deploy an ASP.Net core 3.1 site on nginx and systemd.

Preparation:

  • Prepare a server with nginx and systemd.
  • Install dotnet core support on server. Please check Microsoft site for details.
  • Build the binary files of the site to be deployed.

Step 1: Upload files

Make a folder in the server to be used to store site files. This folder will be marked as <SITEPATH> in all files below.

Upload your site files into this folder.

Give the permission to this folder.

sudo chown -R www-data:www-data <SITEPATH>
sudo chmod -R 755 <SITEPATH>

Step 2: Create systemd service

Create a service file. I suggest to put this file in the same folder of the site, aka <SITEPATH>. Let’s name it as myapp. You could change the name.
nano <SITEPATH>/myapp.service and enter this text below:

[Unit]
Description=<A_DESCRIPTION_TEXT_HERE>

[Service]
Environment=ASPNETCORE_URLS=http://localhost:<PORT_NUMBER>
Environment=ASPNETCORE_ENVIRONMENT=Production
WorkingDirectory=<SITEPATH>
ExecStart=/usr/bin/dotnet <SITEPATH>/<ENTRY_FILE>.dll
SyslogIdentifier=<A_NAME_HERE>
Restart=always
RestartSec=10
KillSignal=SIGINT
User=www-data

[Install]
WantedBy=multi-user.target

You should specify the description, port number, site path, entry file (main file), and the name to be used in syslog. Port number need to be different than all used by other services.

Link the file to systemd folder by ln -s <SITEPATH>/myapp.service /etc/systemd/system, reload systemd by systemctl daemon-reload, then start the service by systemctl start myapp.service. If everything goes will, you can see the port is listed in lsof -i -P -n | grep LISTEN. At last, set this service to start with system by systemctl enable myapp.service.

Step 3: Create nginx site.

Create the site file in sites-available folder by nano /etc/nginx/sites-available/<YOUR_SITE_NAME>, and enter this text below:

server {
    listen 80;
    listen [::]:80;
    server_name <SERVER_DOMAIN>;
    
    location / {
        proxy_pass http://localhost:<PORT_NUMBER>;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

You should specify the server domain name and the port number which is chosen for this app.

Link the file to enabled sites by ln -s /etc/nginx/sites-available/<YOUR_SITE_NAME> /etc/nginx/sites-enabled. Test config by nginx -t. If there is nothing wrong, apply the setting by systemctl reload nginx.

Further: Certbot

If you want to use certbot to apply a free ssl certificate to this site, the nginx plugin shipped with certbot can handle that without any problem. Use certbot with the nginx parameter to finish this job: certbot --nginx.

Fix Ubuntu Server bash for internet accessing (DNS querying)

I have multiple devices installed with Ubuntu Server. Some of them have a strange problem:
They can access internet using some applications, such as squid, without any problem.
They can ping to internet IP address.
They can use host / dig to query IP by domain name.
They cannot ping to internet by using domain name directly.
They cannot download file through wget / curl using domain name, but can do with IP.
They cannot use apt, stuck at connecting.
All they use a local dnsmasq as their dns.

After a long time dig, I found the problem and here is the fix.
open the file /etc/nsswitch.conf, search the line with “hosts:” and change the setting to “files dns” (without quotes).