Ubuntu upgrading kicks the dotnet apps failed.

I manages many server using Ubuntu Server for hosting dotnet apps as service. Recently, they failed one by one. The dotnet services on them cannot start. When typing dotnet –version, it reports

A fatal error occurred. The folder [/usr/share/dotnet/host/fxr] does not exist.

The whole problem is we installed the dotnet very early. In that time, we need to install the packages-microsoft-prod due to lack of official support of dotnet from ubuntu. But now it’s changed. Recently, dotnet is listed in the Ubuntu package manager feeds. That leads into the conflict which cause the services cannot start.

The way to fix is easy:

First, lists all installed packages using

sudo apt list --installed | grep dotnet

You may see some like dotnet-host dotnet-hostfxr-6.0 dotnet-runtime-6.0.

Now, removes those package listed using

sudo apt purge dotnet-host dotnet-hostfxr-6.0 dotnet-runtime-6.0

Also, packages-microsoft-prod is need to be purged also. To be mentioned, it must be purged instead of removed, which will remove the config file which is very the reason cause the conflicts.

sudo dpkg -P packages-microsoft-prod

Now, we can install the dotnet runtime from ubuntu official feeds using some like

sudo apt update && \
  sudo apt install -y aspnetcore-runtime-6.0

Hope this can save your day.

MessageBus

Message Bus is a component provided from SecretNest.info, providing publishing subscription model support within one app.

Source code: https://github.com/SecretNest/MessageBus

License: MIT

Doc: https://messagebus.docs.secretnest.info/

Standard: netstandard 1.3

Packages:

  • SecretNest.MessageBus.Abstractions: Abstractions for Message Bus from SecretNest.info. Imports this package when composing a component intended to be used to connect to Message Bus but do not hold the instance of the Message Bus itself. When the project contains the instance of Message Bus itself, imports SecretNest.MessageBus directly.
  • SecretNest.MessageBus: Provides publishing subscription model support within one app. Imports this package when composing an app managing the instance of Message Bus. This package references SecretNest.MessageBus.Abstractions.

Copyright: SecretNest.info / Allen Cui

No way to stop FileStream.Read

Recently, I need to read the data from a code scanner, which is recognized as a keyboard, in the dotnet program development of Linux. Because the program is running background, hosted in systemd, there is no way to get the entered text from console. All devices are presented as files in Linux. Reading the event from keyboard device file /dev/input/eventX is a good choice.

To read from the device in Linux, FileStream need to be created on the device file. Using the method Read() form the instance of FileStream, all key events can be processed one by one. For make it easy, I post the code in GitHub. Nuget package is also presented.

The sad thing is, there is no way to stop the FileStream.Read method except quit the app. The Read method is designed to read at least one byte. If there is no data available currently but not reaching the end of file, it will block and wait. And there is no way to break the waiting. I also tried to use BinaryReader — “Stream was not readable” reported. ReadByte failed in the same way. Cancel the CancellationToken cannot break the ReadAsync either. Setting read timeout is not supported in FileStream. And finally, the Thread.Abort is obsoleted and no use any more.

Is there any approach to cancel the running Read method of FileStream without exiting application?

Use GZipStream as response in plain Asp.Net Core 5

Recently, I’m struggle with asp.net core 5 working with GZipStream. My request is easy: using GZipStream to compress a large text before sending it as response of asp.net content.

Here is my first guess code, which is not working:

//This code is not working.
public static async Task WriteWithGZipAndCompleteAsync(this HttpResponse response, string text)
{
    response.StatusCode = 200;
    response.ContentType = "application/gzip";

    await using var gzip = new GZipStream(response.Body, CompressionLevel.Optimal, true);
    await using var streamWriter = new StreamWriter(gzip, Encoding.UTF8, -1, true);

    await streamWriter.WriteAsync(text);
    await streamWriter.FlushAsync();
    streamWriter.Close();

    await gzip.FlushAsync();
    gzip.Close();

    await response.CompleteAsync();
}

It does send the text to the client but the gunzip reports “unexpected end of file” after decompressed all texts.

The second version is worse, it sends nothing at all:

//This code is not working either.
public static async Task WriteWithGZipAndCompleteAsync(this HttpResponse response, string text)
{
    response.StatusCode = 200;
    response.ContentType = "application/gzip";

    using (var compressed = new MemoryStream())
    {
        using (var gzip = new GZipStream(compressed, CompressionLevel.Optimal, true))
        {
            var uncompressed = Encoding.UTF8.GetBytes(text);
            gzip.Write(uncompressed);

            gzip.Flush();
            gzip.Close();
        }

        var result = compressed.ToArray();
        response.Body.Write(result);
    }

    await response.CompleteAsync();
}

Finally, I get it work by this code below:

public static async Task WriteWithGZipAndCompleteAsync(this HttpResponse response, string text)
{
    response.StatusCode = 200;
    response.ContentType = "application/gzip";

    await using var gzip = new GZipStream(response.Body, CompressionLevel.Optimal, true);
    await using var streamWriter = new StreamWriter(gzip, Encoding.UTF8, -1, true);

    await streamWriter.WriteAsync(text);
    await streamWriter.FlushAsync();
    await gzip.FlushAsync();
    await response.Body.FlushAsync();

    //WARNING: DO NOT CALL CompleteAsync, which will thrown an exception.
}

Conclusion:

  • When calling Close() on instances of StreamWriter and GZipStream, the underlying stream will be closed, NO MATTER the value of leaveOpen specified. Uhh? Weird? But it’s true.
  • Do not call response.CompleteAsync() after write data to response.Body.
    • If Close() called on streams before, calling response.CompleteAsync() will thrown an ObjectDisposedException: Cannot access a closed stream.
    • If Close() does not present, like the code above, calling response.CompleteAsync() will thrown an InvalidOperationException: Writing is not allowed after writer was completed.

A bash script for copying data from SD card

When I hang out, GoPro and drone is usually taken. Sometimes it’s necessary to copy data from the SD card to a hard drive frequently when traveling for several days. There are some product I owned for this scenario, like My Passport Wireless Pro. But none of them are good enough. Take that WD disk for example, it is very slow, with bad app, and the worst thing is the built-in battery. Many designers didn’t consider that how to take their products on a plane.

Finally, I decide to build a Raspberry Pi with all tools and scripts I need inside.

This is the script for copying data from SD card to a USB disk. It is tested with Ubuntu 20.04.1 ARM64 version on Raspberry Pi 4.

Before using this script, you need to prepare your disk for saving data for creating a folder named Target. Of cause, the file system of the partition should be writable on your device.

You can change the folder settings by editing the # define block.

  • MountPoint settings are the paths used in this script.
  • TargetFolder is the path as the target folder. The partition contains this folder will be detected as target. Default value is “/Target”.
  • SourceTestFolder is the folder for detecting as source. Note: All files, not only within this folder, will be copied. Default value is “/DCIM”. All SD cards from DC and drone should contain this folder.

To use this script, you need to connect your target disk and sd card to your device (mounting is not required) and run this script. Both source and target will be detected automatically and a name of the sub folder will be asked. Then all files from the source will be copy to the sub folder you inputted in the TargetFolder of the target disk. The files on the sd will NOT be deleted after copying.

If an argument is provided, the value will be used as the sub folder.

Source is licensed under MIT license. Click here to get it.

Forward Client Certificate to .net Core App through Nginx

When dotnet core app is deployed in Linux with Kestrel, nginx works in front as a proxy. Usually, Nginx will handle all https related issue and forward a plain http request to core app. Some app may require clients to use certificate for authentication. In this case, client certificate need to be transferred to core app.

Core App

First, the core app need to be prepared to receive and check the client certificate.

public class StartUp
{
    ...
    public void ConfigureServices(IServiceCollection services)
    {
        ...
        //Add code here
        services.AddCertificateForwarding(options => "X-ARR-ClientCert");
        //PointA - for later reference
    }
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        ...
        //Add code here
        app.UseAuthorization();
        app.UseCertificateForwarding();
        ...
    }
}

Note: UseHttpsRedirection() cannot be used because the core app is set to use http only. UseCertificateForwarding() may expose a security issue, you could set a switch to open it only when required. The header name X-ARR-ClientCert can be changed as your wish.

By default, core app will validate client certificate with local trusted CA. For additional tuning, add this code to the PointA position above.

services.AddAuthentication(CertificateAuthenticationDefaults.AuthenticationScheme)
    .AddCertificate(options =>
    {
        options.Events = new CertificateAuthenticationEvents
        {
            OnCertificateValidated = aMethod,
            OnAuthenticationFailed = anotherMethod
        };

It is not required to present OnCertificateValidated and OnAuthenticationFailed at the same time. Check this doc for details.

Nginx

Now, in the Nginx setting, some lines need to be added.

Check Certificate with Nginx

When need to check the client certificate by Nginx,

ssl_client_certificate file;
ssl_verify_client on;

is required. The file should contain trusted CA certificates in PEM format. When using multiple CA certificates, write all of them into the same file. When client certificate is not forcible, change on to optional.

Not Check Certificate with Nginx

If we need Nginx to leave the certificate checking to core app, simply use the code

ssl_verify_client optional_no_ca;

in site file. This will let Nginx transfer the client certificate to proxy without touch.

Pass to Proxy

After processing one of those above, the client certificate is ready to be passed into the proxy app — core app. This code below will do that.

proxy_set_header X-ARR-ClientCert $ssl_client_escaped_cert;

If you changed the name X-ARR-ClientCert above, use the same value here. This code can be placed into location block too.

Now you can enjoy your dirty job by checking everything about client certificate in your core app. 😀