Ubuntu upgrading kicks the dotnet apps failed.

I manages many server using Ubuntu Server for hosting dotnet apps as service. Recently, they failed one by one. The dotnet services on them cannot start. When typing dotnet –version, it reports

A fatal error occurred. The folder [/usr/share/dotnet/host/fxr] does not exist.

The whole problem is we installed the dotnet very early. In that time, we need to install the packages-microsoft-prod due to lack of official support of dotnet from ubuntu. But now it’s changed. Recently, dotnet is listed in the Ubuntu package manager feeds. That leads into the conflict which cause the services cannot start.

The way to fix is easy:

First, lists all installed packages using

sudo apt list --installed | grep dotnet

You may see some like dotnet-host dotnet-hostfxr-6.0 dotnet-runtime-6.0.

Now, removes those package listed using

sudo apt purge dotnet-host dotnet-hostfxr-6.0 dotnet-runtime-6.0

Also, packages-microsoft-prod is need to be purged also. To be mentioned, it must be purged instead of removed, which will remove the config file which is very the reason cause the conflicts.

sudo dpkg -P packages-microsoft-prod

Now, we can install the dotnet runtime from ubuntu official feeds using some like

sudo apt update && \
  sudo apt install -y aspnetcore-runtime-6.0

Hope this can save your day.

apt upgrade brings apache2 activated on my server

I have a server using Ubuntu Server 22.04 LTS with nginx and php supported for a long time. Today, after I fire a regular update by using apt update, apt upgrade and a reboot, my web server is down.

When checking the status of nginx, it reports that port 80 is taken:

nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Unknown error)

And the lsof command

lsof -i -P -n | grep LISTEN

reports that port 80 is used by apache2. That’s really weird. This server never has apache configured or started. Who wanna run apache aside nginx in a production environment, uhh?

By checking the log of apt, I found that php8.1 is upgraded with a library named libapache2-mod-php8.1 installed automatically. I don’t find any log related to apache installation. Maybe the apache2 is preinstalled for a very long time but never initiated but the libapache2-mod-php8.1 upgrading process triggered it on.

After I purge apache2* and libapache2-mod-php8.1, the nginx can be restarted without any problem.

Enable rc.local on Ubuntu 20.04

For compatible reason, rc.local support is still supported by new version of Ubuntu. But it is disabled by default. This will guide you to enable running rc.local while starting system.

  1. Create rc.local if it does not exist.

Run nano /etc/rc.local. If the file does not exist, place this code below as the default rc.local file.

#!/bin/bash

exit 0

Run chmod +x /etc/rc.local to give the executing permission.

2. Create systemd service file.

Run nano /etc/systemd/system/rc-local.service to create the systemd service file and paste this text below.

[Unit]
Description=/etc/rc.local Support
ConditionPathExists=/etc/rc.local

[Service]
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99

[Install]
WantedBy=multi-user.target

3. Config systemd using this bash command below.

systemctl enable rc-local

All done. The rc-local will be run while system starting.

Warning: When Upgrading to Ubuntu Server 20.04

Ubuntu, including server, 20.04 is launched several days ago. Many servers with 19.10 or earlier system are planed to upgrade soon. There are some issues should be noticed before you process.

1. If do-release-upgrade returns nothing found, you may want to use

do-release-upgrade -d

instead.

2. Some servers will be stuck with “Updating database of manual pages” process. In my tests on about 20 servers, about 20% servers will pass it with no more than 2 minutes delay. More than half servers, mainly on Azure, stuck there for 20-40 minutes. During this time, around 100 GB data are read from disk with 100% CPU using, reported by Azure portal. I don’t know what it read. All my files on the server cost no more than 20GB disk space. One of my server, hosted on Azure, costs more than half day so far (and not finished yet), more than 200GB data read. One server failed in that step, SSH drop and cannot reconnect at all. Before this problem solved, I strongly NOT recommend to upgrade your product server.

Update: The one stuck on Azure resume to idle after 14 hrs, 626.14GB reading and 27GB writing. Due to dead SSH process, I have to try to reboot the server. But the server never come back online. A kernel panic captured by Azure console while booting:

Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-1010-azure #10-Ubuntu
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
Call Trace:
  dump_stack+0x57/0x7a
  ? rest_init+0x50/0xb0
  panic+0xfb/0x2d7
  ? do_execve+0x25/0x30
  ? rest_init+0xb0/0xb0
  kernel_init+0xfb/0x100
  ret_from_fork+0x35/0x40
 Kernel Offset: 0x30000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

3. Upgrading will replace php 7.3 with 7.4. If some site is powered by php-fpm with Nginx, you may need to change your web site configuration file in Nginx available sites folder. All packages installed by pip3 will be dropped, not uninstalled properly. You may want to uninstall them if you don’t want to use it any more before upgrading system. Or, you can reinstall them after upgrading.

Deploy a dotnet core site on nginx and systemd

This article is about how to deploy an ASP.Net core 3.1 site on nginx and systemd.

Preparation:

  • Prepare a server with nginx and systemd.
  • Install dotnet core support on server. Please check Microsoft site for details.
  • Build the binary files of the site to be deployed.

Step 1: Upload files

Make a folder in the server to be used to store site files. This folder will be marked as <SITEPATH> in all files below.

Upload your site files into this folder.

Give the permission to this folder.

sudo chown -R www-data:www-data <SITEPATH>
sudo chmod -R 755 <SITEPATH>

Step 2: Create systemd service

Create a service file. I suggest to put this file in the same folder of the site, aka <SITEPATH>. Let’s name it as myapp. You could change the name.
nano <SITEPATH>/myapp.service and enter this text below:

[Unit]
Description=<A_DESCRIPTION_TEXT_HERE>

[Service]
Environment=ASPNETCORE_URLS=http://localhost:<PORT_NUMBER>
Environment=ASPNETCORE_ENVIRONMENT=Production
WorkingDirectory=<SITEPATH>
ExecStart=/usr/bin/dotnet <SITEPATH>/<ENTRY_FILE>.dll
SyslogIdentifier=<A_NAME_HERE>
Restart=always
RestartSec=10
KillSignal=SIGINT
User=www-data

[Install]
WantedBy=multi-user.target

You should specify the description, port number, site path, entry file (main file), and the name to be used in syslog. Port number need to be different than all used by other services.

Link the file to systemd folder by ln -s <SITEPATH>/myapp.service /etc/systemd/system, reload systemd by systemctl daemon-reload, then start the service by systemctl start myapp.service. If everything goes will, you can see the port is listed in lsof -i -P -n | grep LISTEN. At last, set this service to start with system by systemctl enable myapp.service.

Step 3: Create nginx site.

Create the site file in sites-available folder by nano /etc/nginx/sites-available/<YOUR_SITE_NAME>, and enter this text below:

server {
    listen 80;
    listen [::]:80;
    server_name <SERVER_DOMAIN>;
    
    location / {
        proxy_pass http://localhost:<PORT_NUMBER>;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

You should specify the server domain name and the port number which is chosen for this app.

Link the file to enabled sites by ln -s /etc/nginx/sites-available/<YOUR_SITE_NAME> /etc/nginx/sites-enabled. Test config by nginx -t. If there is nothing wrong, apply the setting by systemctl reload nginx.

Further: Certbot

If you want to use certbot to apply a free ssl certificate to this site, the nginx plugin shipped with certbot can handle that without any problem. Use certbot with the nginx parameter to finish this job: certbot --nginx.

Fix Ubuntu Server bash for internet accessing (DNS querying)

I have multiple devices installed with Ubuntu Server. Some of them have a strange problem:
They can access internet using some applications, such as squid, without any problem.
They can ping to internet IP address.
They can use host / dig to query IP by domain name.
They cannot ping to internet by using domain name directly.
They cannot download file through wget / curl using domain name, but can do with IP.
They cannot use apt, stuck at connecting.
All they use a local dnsmasq as their dns.

After a long time dig, I found the problem and here is the fix.
open the file /etc/nsswitch.conf, search the line with “hosts:” and change the setting to “files dns” (without quotes).