A bash script for copying data from SD card

When I hang out, GoPro and drone is usually taken. Sometimes it’s necessary to copy data from the SD card to a hard drive frequently when traveling for several days. There are some product I owned for this scenario, like My Passport Wireless Pro. But none of them are good enough. Take that WD disk for example, it is very slow, with bad app, and the worst thing is the built-in battery. Many designers didn’t consider that how to take their products on a plane.

Finally, I decide to build a Raspberry Pi with all tools and scripts I need inside.

This is the script for copying data from SD card to a USB disk. It is tested with Ubuntu 20.04.1 ARM64 version on Raspberry Pi 4.

Before using this script, you need to prepare your disk for saving data for creating a folder named Target. Of cause, the file system of the partition should be writable on your device.

You can change the folder settings by editing the # define block.

  • MountPoint settings are the paths used in this script.
  • TargetFolder is the path as the target folder. The partition contains this folder will be detected as target. Default value is “/Target”.
  • SourceTestFolder is the folder for detecting as source. Note: All files, not only within this folder, will be copied. Default value is “/DCIM”. All SD cards from DC and drone should contain this folder.

To use this script, you need to connect your target disk and sd card to your device (mounting is not required) and run this script. Both source and target will be detected automatically and a name of the sub folder will be asked. Then all files from the source will be copy to the sub folder you inputted in the TargetFolder of the target disk. The files on the sd will NOT be deleted after copying.

If an argument is provided, the value will be used as the sub folder.

Source is licensed under MIT license. Click here to get it.

Forward Client Certificate to .net Core App through Nginx

When dotnet core app is deployed in Linux with Kestrel, nginx works in front as a proxy. Usually, Nginx will handle all https related issue and forward a plain http request to core app. Some app may require clients to use certificate for authentication. In this case, client certificate need to be transferred to core app.

Core App

First, the core app need to be prepared to receive and check the client certificate.

public class StartUp
{
    ...
    public void ConfigureServices(IServiceCollection services)
    {
        ...
        //Add code here
        services.AddCertificateForwarding(options => "X-ARR-ClientCert");
        //PointA - for later reference
    }
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        ...
        //Add code here
        app.UseAuthorization();
        app.UseCertificateForwarding();
        ...
    }
}

Note: UseHttpsRedirection() cannot be used because the core app is set to use http only. UseCertificateForwarding() may expose a security issue, you could set a switch to open it only when required. The header name X-ARR-ClientCert can be changed as your wish.

By default, core app will validate client certificate with local trusted CA. For additional tuning, add this code to the PointA position above.

services.AddAuthentication(CertificateAuthenticationDefaults.AuthenticationScheme)
    .AddCertificate(options =>
    {
        options.Events = new CertificateAuthenticationEvents
        {
            OnCertificateValidated = aMethod,
            OnAuthenticationFailed = anotherMethod
        };

It is not required to present OnCertificateValidated and OnAuthenticationFailed at the same time. Check this doc for details.

Nginx

Now, in the Nginx setting, some lines need to be added.

Check Certificate with Nginx

When need to check the client certificate by Nginx,

ssl_client_certificate file;
ssl_verify_client on;

is required. The file should contain trusted CA certificates in PEM format. When using multiple CA certificates, write all of them into the same file. When client certificate is not forcible, change on to optional.

Not Check Certificate with Nginx

If we need Nginx to leave the certificate checking to core app, simply use the code

ssl_verify_client optional_no_ca;

in site file. This will let Nginx transfer the client certificate to proxy without touch.

Pass to Proxy

After processing one of those above, the client certificate is ready to be passed into the proxy app — core app. This code below will do that.

proxy_set_header X-ARR-ClientCert $ssl_client_escaped_cert;

If you changed the name X-ARR-ClientCert above, use the same value here. This code can be placed into location block too.

Now you can enjoy your dirty job by checking everything about client certificate in your core app. 😀

Thread-Safe calling support in Remote Agency 2

In the next release of Remote Agency, the thread safe calling support will be added.

In the current version, all accessing to assets is from the thread which sending message to the Remote Agency Manager. Due to network transportation, this may cause multithread calling on the target service object. Without special treatment, some error may be caused when accessing object without thread safe designing.

In the next release, a new attribute is introduced. User can specify the behavior of thread using for each interface or the class of service object: Free — like now, use SynchronizationContext — useful on form based program, one certain free thread or one task schedule. A new task scheduler is builtin with Remote Agency which always use one thread to execute all jobs one by one. The task scheduler also support user passing a thread in as the working one inside. Therefore, user code can use the same thread to initialize some object and then turn it into a task scheduler to run all accessing on the same object.

Serializer for Remote Agency 2

In the next major release of Remote Agency, the default serializer will be changed to JSON, from Json.net, instead of DataContractSerializer shipped with dotnet runtime.

This change is taken because the type support, especially the generic type support, is too weak in DataContractSerializer, and many user defined class is not marked with [DataMember] correctly. This change will also make it possible to serialize and deserialize data in one phase, instead of 2 phases in version 1, because the serializer will recognize the generic type automatically, without code specified generic types to be extracted in advanced.

New version plan of Remote Agency

I am excited to announce that Remote Agency will have one major update, version 2 in late this year.

Key features will be included in new version:

  • Speed up by combining serializing.
  • DataContractSerializer will be merged into main library. You can still use your own extended serializer, but this one will be shipped within the main library. Therefore, DataContractSerializer.EasyEncapsulation will be removed from the new version.
  • Reference to Roslyn directly, without CSharpRoslynAgency package.

Compatible issues:

  • If you have your own serializer, you need to rebuild one following new standard.
  • Facade classes will have some minor changes. You need to change your code working with them. In general, the amount of changes in each project will not exceed 10 lines of code.
  • There will be a new event for routing of the messages. The old one still works, but with forcible serialization, like in version 1.

There are many changes to be taken care within Remote Agency. I hope all my rest time in this year can make this change born.

Cheers,
Allen

Warning: When Upgrading to Ubuntu Server 20.04

Ubuntu, including server, 20.04 is launched several days ago. Many servers with 19.10 or earlier system are planed to upgrade soon. There are some issues should be noticed before you process.

1. If do-release-upgrade returns nothing found, you may want to use

do-release-upgrade -d

instead.

2. Some servers will be stuck with “Updating database of manual pages” process. In my tests on about 20 servers, about 20% servers will pass it with no more than 2 minutes delay. More than half servers, mainly on Azure, stuck there for 20-40 minutes. During this time, around 100 GB data are read from disk with 100% CPU using, reported by Azure portal. I don’t know what it read. All my files on the server cost no more than 20GB disk space. One of my server, hosted on Azure, costs more than half day so far (and not finished yet), more than 200GB data read. One server failed in that step, SSH drop and cannot reconnect at all. Before this problem solved, I strongly NOT recommend to upgrade your product server.

Update: The one stuck on Azure resume to idle after 14 hrs, 626.14GB reading and 27GB writing. Due to dead SSH process, I have to try to reboot the server. But the server never come back online. A kernel panic captured by Azure console while booting:

Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-1010-azure #10-Ubuntu
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
Call Trace:
  dump_stack+0x57/0x7a
  ? rest_init+0x50/0xb0
  panic+0xfb/0x2d7
  ? do_execve+0x25/0x30
  ? rest_init+0xb0/0xb0
  kernel_init+0xfb/0x100
  ret_from_fork+0x35/0x40
 Kernel Offset: 0x30000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

3. Upgrading will replace php 7.3 with 7.4. If some site is powered by php-fpm with Nginx, you may need to change your web site configuration file in Nginx available sites folder. All packages installed by pip3 will be dropped, not uninstalled properly. You may want to uninstall them if you don’t want to use it any more before upgrading system. Or, you can reinstall them after upgrading.