A bash script for copying data from SD card

When I hang out, GoPro and drone is usually taken. Sometimes it’s necessary to copy data from the SD card to a hard drive frequently when traveling for several days. There are some product I owned for this scenario, like My Passport Wireless Pro. But none of them are good enough. Take that WD disk for example, it is very slow, with bad app, and the worst thing is the built-in battery. Many designers didn’t consider that how to take their products on a plane.

Finally, I decide to build a Raspberry Pi with all tools and scripts I need inside.

This is the script for copying data from SD card to a USB disk. It is tested with Ubuntu 20.04.1 ARM64 version on Raspberry Pi 4.

Before using this script, you need to prepare your disk for saving data for creating a folder named Target. Of cause, the file system of the partition should be writable on your device.

You can change the folder settings by editing the # define block.

  • MountPoint settings are the paths used in this script.
  • TargetFolder is the path as the target folder. The partition contains this folder will be detected as target. Default value is “/Target”.
  • SourceTestFolder is the folder for detecting as source. Note: All files, not only within this folder, will be copied. Default value is “/DCIM”. All SD cards from DC and drone should contain this folder.

To use this script, you need to connect your target disk and sd card to your device (mounting is not required) and run this script. Both source and target will be detected automatically and a name of the sub folder will be asked. Then all files from the source will be copy to the sub folder you inputted in the TargetFolder of the target disk. The files on the sd will NOT be deleted after copying.

If an argument is provided, the value will be used as the sub folder.

Source is licensed under MIT license. Click here to get it.

Forward Client Certificate to .net Core App through Nginx

When dotnet core app is deployed in Linux with Kestrel, nginx works in front as a proxy. Usually, Nginx will handle all https related issue and forward a plain http request to core app. Some app may require clients to use certificate for authentication. In this case, client certificate need to be transferred to core app.

Core App

First, the core app need to be prepared to receive and check the client certificate.

public class StartUp
{
    ...
    public void ConfigureServices(IServiceCollection services)
    {
        ...
        //Add code here
        services.AddCertificateForwarding(options => "X-ARR-ClientCert");
        //PointA - for later reference
    }
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        ...
        //Add code here
        app.UseAuthorization();
        app.UseCertificateForwarding();
        ...
    }
}

Note: UseHttpsRedirection() cannot be used because the core app is set to use http only. UseCertificateForwarding() may expose a security issue, you could set a switch to open it only when required. The header name X-ARR-ClientCert can be changed as your wish.

By default, core app will validate client certificate with local trusted CA. For additional tuning, add this code to the PointA position above.

services.AddAuthentication(CertificateAuthenticationDefaults.AuthenticationScheme)
    .AddCertificate(options =>
    {
        options.Events = new CertificateAuthenticationEvents
        {
            OnCertificateValidated = aMethod,
            OnAuthenticationFailed = anotherMethod
        };

It is not required to present OnCertificateValidated and OnAuthenticationFailed at the same time. Check this doc for details.

Nginx

Now, in the Nginx setting, some lines need to be added.

Check Certificate with Nginx

When need to check the client certificate by Nginx,

ssl_client_certificate file;
ssl_verify_client on;

is required. The file should contain trusted CA certificates in PEM format. When using multiple CA certificates, write all of them into the same file. When client certificate is not forcible, change on to optional.

Not Check Certificate with Nginx

If we need Nginx to leave the certificate checking to core app, simply use the code

ssl_verify_client optional_no_ca;

in site file. This will let Nginx transfer the client certificate to proxy without touch.

Pass to Proxy

After processing one of those above, the client certificate is ready to be passed into the proxy app — core app. This code below will do that.

proxy_set_header X-ARR-ClientCert $ssl_client_escaped_cert;

If you changed the name X-ARR-ClientCert above, use the same value here. This code can be placed into location block too.

Now you can enjoy your dirty job by checking everything about client certificate in your core app. 😀

Thread-Safe calling support in Remote Agency 2

In the next release of Remote Agency, the thread safe calling support will be added.

In the current version, all accessing to assets is from the thread which sending message to the Remote Agency Manager. Due to network transportation, this may cause multithread calling on the target service object. Without special treatment, some error may be caused when accessing object without thread safe designing.

In the next release, a new attribute is introduced. User can specify the behavior of thread using for each interface or the class of service object: Free — like now, use SynchronizationContext — useful on form based program, one certain free thread or one task schedule. A new task scheduler is builtin with Remote Agency which always use one thread to execute all jobs one by one. The task scheduler also support user passing a thread in as the working one inside. Therefore, user code can use the same thread to initialize some object and then turn it into a task scheduler to run all accessing on the same object.

Serializer for Remote Agency 2

In the next major release of Remote Agency, the default serializer will be changed to JSON, from Json.net, instead of DataContractSerializer shipped with dotnet runtime.

This change is taken because the type support, especially the generic type support, is too weak in DataContractSerializer, and many user defined class is not marked with [DataMember] correctly. This change will also make it possible to serialize and deserialize data in one phase, instead of 2 phases in version 1, because the serializer will recognize the generic type automatically, without code specified generic types to be extracted in advanced.

New version plan of Remote Agency

I am excited to announce that Remote Agency will have one major update, version 2 in late this year.

Key features will be included in new version:

  • Speed up by combining serializing.
  • DataContractSerializer will be merged into main library. You can still use your own extended serializer, but this one will be shipped within the main library. Therefore, DataContractSerializer.EasyEncapsulation will be removed from the new version.
  • Reference to Roslyn directly, without CSharpRoslynAgency package.

Compatible issues:

  • If you have your own serializer, you need to rebuild one following new standard.
  • Facade classes will have some minor changes. You need to change your code working with them. In general, the amount of changes in each project will not exceed 10 lines of code.
  • There will be a new event for routing of the messages. The old one still works, but with forcible serialization, like in version 1.

There are many changes to be taken care within Remote Agency. I hope all my rest time in this year can make this change born.

Cheers,
Allen