Warning: When Upgrading to Ubuntu Server 20.04

Ubuntu, including server, 20.04 is launched several days ago. Many servers with 19.10 or earlier system are planed to upgrade soon. There are some issues should be noticed before you process.

1. If do-release-upgrade returns nothing found, you may want to use

do-release-upgrade -d

instead.

2. Some servers will be stuck with “Updating database of manual pages” process. In my tests on about 20 servers, about 20% servers will pass it with no more than 2 minutes delay. More than half servers, mainly on Azure, stuck there for 20-40 minutes. During this time, around 100 GB data are read from disk with 100% CPU using, reported by Azure portal. I don’t know what it read. All my files on the server cost no more than 20GB disk space. One of my server, hosted on Azure, costs more than half day so far (and not finished yet), more than 200GB data read. One server failed in that step, SSH drop and cannot reconnect at all. Before this problem solved, I strongly NOT recommend to upgrade your product server.

Update: The one stuck on Azure resume to idle after 14 hrs, 626.14GB reading and 27GB writing. Due to dead SSH process, I have to try to reboot the server. But the server never come back online. A kernel panic captured by Azure console while booting:

Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-1010-azure #10-Ubuntu
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
Call Trace:
  dump_stack+0x57/0x7a
  ? rest_init+0x50/0xb0
  panic+0xfb/0x2d7
  ? do_execve+0x25/0x30
  ? rest_init+0xb0/0xb0
  kernel_init+0xfb/0x100
  ret_from_fork+0x35/0x40
 Kernel Offset: 0x30000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

3. Upgrading will replace php 7.3 with 7.4. If some site is powered by php-fpm with Nginx, you may need to change your web site configuration file in Nginx available sites folder. All packages installed by pip3 will be dropped, not uninstalled properly. You may want to uninstall them if you don’t want to use it any more before upgrading system. Or, you can reinstall them after upgrading.

Deploy a dotnet core site on nginx and systemd

This article is about how to deploy an ASP.Net core 3.1 site on nginx and systemd.

Preparation:

  • Prepare a server with nginx and systemd.
  • Install dotnet core support on server. Please check Microsoft site for details.
  • Build the binary files of the site to be deployed.

Step 1: Upload files

Make a folder in the server to be used to store site files. This folder will be marked as <SITEPATH> in all files below.

Upload your site files into this folder.

Give the permission to this folder.

sudo chown -R www-data:www-data <SITEPATH>
sudo chmod -R 755 <SITEPATH>

Step 2: Create systemd service

Create a service file. I suggest to put this file in the same folder of the site, aka <SITEPATH>. Let’s name it as myapp. You could change the name.
nano <SITEPATH>/myapp.service and enter this text below:

[Unit]
Description=<A_DESCRIPTION_TEXT_HERE>

[Service]
Environment=ASPNETCORE_URLS=http://localhost:<PORT_NUMBER>
Environment=ASPNETCORE_ENVIRONMENT=Production
WorkingDirectory=<SITEPATH>
ExecStart=/usr/bin/dotnet <SITEPATH>/<ENTRY_FILE>.dll
SyslogIdentifier=<A_NAME_HERE>
Restart=always
RestartSec=10
KillSignal=SIGINT
User=www-data

[Install]
WantedBy=multi-user.target

You should specify the description, port number, site path, entry file (main file), and the name to be used in syslog. Port number need to be different than all used by other services.

Link the file to systemd folder by ln -s <SITEPATH>/myapp.service /etc/systemd/system, reload systemd by systemctl daemon-reload, then start the service by systemctl start myapp.service. If everything goes will, you can see the port is listed in lsof -i -P -n | grep LISTEN. At last, set this service to start with system by systemctl enable myapp.service.

Step 3: Create nginx site.

Create the site file in sites-available folder by nano /etc/nginx/sites-available/<YOUR_SITE_NAME>, and enter this text below:

server {
    listen 80;
    listen [::]:80;
    server_name <SERVER_DOMAIN>;
    
    location / {
        proxy_pass http://localhost:<PORT_NUMBER>;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

You should specify the server domain name and the port number which is chosen for this app.

Link the file to enabled sites by ln -s /etc/nginx/sites-available/<YOUR_SITE_NAME> /etc/nginx/sites-enabled. Test config by nginx -t. If there is nothing wrong, apply the setting by systemctl reload nginx.

Further: Certbot

If you want to use certbot to apply a free ssl certificate to this site, the nginx plugin shipped with certbot can handle that without any problem. Use certbot with the nginx parameter to finish this job: certbot --nginx.

Fix Ubuntu Server bash for internet accessing (DNS querying)

I have multiple devices installed with Ubuntu Server. Some of them have a strange problem:
They can access internet using some applications, such as squid, without any problem.
They can ping to internet IP address.
They can use host / dig to query IP by domain name.
They cannot ping to internet by using domain name directly.
They cannot download file through wget / curl using domain name, but can do with IP.
They cannot use apt, stuck at connecting.
All they use a local dnsmasq as their dns.

After a long time dig, I found the problem and here is the fix.
open the file /etc/nsswitch.conf, search the line with “hosts:” and change the setting to “files dns” (without quotes).

Web Deploy for Visual Studio

While I’m trying to deploy my site from Visual Studio to a Windows Server 2012 R2 instance, I have to dig a lot for troubleshooting about Web Deploy related problems.

 

Here are some not listed in Microsoft documents:

  • Visual Studio need both Web Deploy (Web Deployment Agent Service, MsDepSvc) and Web Management Service (WMSVC).
  • You cannot install Web Management Service separately.
  • The easy way to install the both is choose “Web Deploy 3.6 for Hosting Servers” Web Platform Installer.
  • If you selected “Web Deploy 3.6” in another release, without “for Hosting Servers”, Web Management Service will not be installed.
  • If you leave Web Deploy installed without Web Management Service, you could add it by install “Web Deploy 3.6 for Hosting Servers” but IT WILL NEVER WORK! You have to UNINSTALL existed Web Deploy first and then reinstall the “Web Deploy 3.6 for Hosting Servers”.

Install ubuntu server into Soekris net6501 by using USB boot

This will guide you to prepare a installation media from Windows and install ubuntu into Soekris net6501.

Stuffs you need:

  • a USB stick that CAN BE RECOGNIZED by net6501 device.
    Not sure? Attach it to net6501 and reboot. If you can read the SIZE of this stick while booting, it is.
  • an ISO file for ubuntu server 32-bit. Version 16.10 is tested.
  • rufus. Guide and download point at https://www.ubuntu.com/download/desktop/create-a-usb-stick-on-windows.
  • a working serial connection from your workstation to your net6501, software included.

Prepare installation media:

  • Use rufus to build the USB stick. Check the guide above.

Install:

  1. Link net6501 to your computer using serial connection and start the link. In this case, I’m using the default setting (Speed=19200, Data bits=8, Stop bits=1, Parity=None, Flow control=XON/XOFF).
  2. Boot your net6501 with USB stick attached.
  3. Hit ctrl-p to go into bios.
  4. Type “boot 81” (without quotes) and press enter to boot from USB.
  5. You will see that something wrong with gfxboot.
  6. Type this text below SLOWLY and press enter. DO NOT COPY AND PASTE:
    install console=ttyS0,19200
  7. Do not press any key related to screen mode selection. It will be passed.
  8. Choose a language and continue installation steps as normal.

Finalize:

Grub configuration need to be adjusted for using serial as console.

Edit /etc/default/grub and change to these lines below:
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="console=ttyS0,19200 noplymouth"

Then run update-grub.

 

Hope this help you.