Simple Cryptography With C#

One of the applications I’m working on has the need to store Privately Identifiable Information (PII) in it’s databases. We have chosen to store this information using the RSA Public Key Cryptography provider included in the .NET Core base libraries.

We all use Public Key Cryptography (PKC) every day. If you are reading this you are using PKC as the underpinnings of HTTPS use public key cryptography. Basically PKC is implemented using two individual keys the PUBLIC one and the PRIVATE one. The public key is used to encrypt data and the private key is used to decrypt data. As it name implies you can distribute the PUBLIC key … well… publicly. Meaning you can give it out freely. The PRIVATE key is yours and yours alone and is the only key that can decrypt your data.

For our needs we created a simple library for managing our PKC needs. This library has 3 parts:

1) Function to generate PUBLIC and PRIVATE keys
2) Function to encrypt a string
3) Function to decrypt a string.

Lets take a look at generating keys first. To to this you need to create a class that will be used to return the PUBLIC and PRIVATE key strings. The class looks like this:

public class KeyPair
{
	public string PublicKey { get; set; } = "";
	public string PrivateKey { get; set; } = "";
}

Next you need to create an instance of the RSACryptoServiceProvider class. When creating an instance of this class you need to pass it the size of your keys.

The size of the key specifies the strength of the key you are using. Meaning how difficult will it be to crack. The larger the more difficult it is to crack. Our code stores the size of the key in a CONST called Keysize. After creating an instance of the provider your an call the ToXmlString() of the provide which will return your PUBLIC and PRIVATE keys depending on the boolean you pass to the function. Passing a false value returns the PUBLIC key, passing a true value returns the private key. The following code shows how to create and return a PUBLIC/PRIVATE key pair.

using System;
using System.Security.Cryptography;
using System.Text;

namespace CryptoLibrary
{
	public class CryptoTools
	{
		private const int KeySize = 2048;

		public KeyPair GetKeyPair()
		{
			var provider = new RSACryptoServiceProvider(KeySize);
			var retVal = new KeyPair()
			{
				PublicKey = provider.ToXmlString(false),
				PrivateKey = provider.ToXmlString(true)
			};
			return retVal;
		}

	public class KeyPair
	{
		public string PublicKey { get; set; } = "";
		public string PrivateKey { get; set; } = "";
	}

}

Now you can call the library to generate your keypair.

var tools = new CryptoTools();
var keys = tools.GetKeyPair();

This code returns your keys in an XML string that looks like this:

<RSAKeyValue>
  <Modulus>vx53GKAPG02.......yQgyoVlAwYHZxP7jVTyQ==</Modulus>
  <Exponent>AQAB</Exponent>
</RSAKeyValue>

With these two keys you can write some simple functions to Encrypt and Decrypt strings. The steps to encrypt a string are:

1) Create an instance of the RSACryptoServiceProvider class.
2) Load the PUBLIC key using the FromXmlString() function.
3) Convert your string to a byte[] array
4) Encrypt the string (returned as byte[] array)
5) Turn the byte[] array into a base 64 string.
6) Return the string.

The steps to decrypt a string are:

1) Create an instance of the RSACryptoServiceProvider class.
2) Load the PRIVATE key using the FromXmlString() function.
3) Convert your string to a byte[] array
4) Decrypt the string (returned as byte[] array)
5) Turn the byte[] array into a UTF8 string.
6) Return the string.

The following code represents this functionality:

public string Encrypt(string publicKeyXML, string itemToEncrypt)
{
	var provider = new RSACryptoServiceProvider(KeySize);
	provider.FromXmlString(publicKeyXML);

	byte[] bytes = Encoding.UTF8.GetBytes(itemToEncrypt);
	var encryptedData = provider.Encrypt(bytes,true);
	var retval = Convert.ToBase64String(encryptedData);
	return retval;
}

public string Decrypt(string privateKeyXML, string itemToDecrypt)
{
	var provider = new RSACryptoServiceProvider(KeySize);
	provider.FromXmlString(privateKeyXML);

	byte[] bytes = Convert.FromBase64String(itemToDecrypt);
	var encryptedData = provider.Decrypt(bytes, true);
	var retval = Encoding.UTF8.GetString(encryptedData);
	return retval;
}

The following code shows how to generate keys, encrypt the data and descrypt the data.

var tools = new CryptoTools();
var keys = tools.GetKeyPair();
			
Console.WriteLine(keys.PublicKey);
Console.WriteLine(keys.PublicKey);

var encrypted = tools.Encrypt(keys.PublicKey, "555-12-3456");
var decrypted = tools.Decrypt(keys.PrivateKey, encrypted);

var encrypted2 = tools.Encrypt(keys.PublicKey, "4000-0000-0000-0002");
var decrypted2 = tools.Decrypt(keys.PrivateKey, encrypted);

The complete library for these tools:

using System;
using System.Security.Cryptography;
using System.Text;

namespace CryptoLibrary
{
	public class CryptoTools
	{
		private const int KeySize = 2048;

		public KeyPair GetKeyPair()
		{
			var provider = new RSACryptoServiceProvider(KeySize);

			var retVal = new KeyPair()
			{
				PublicKey = provider.ToXmlString(false),
				PrivateKey = provider.ToXmlString(true)
			};
			
			return retVal;

		}

		public string Encrypt(string publicKeyXML, string itemToEncrypt)
		{
			var provider = new RSACryptoServiceProvider(KeySize);
			provider.FromXmlString(publicKeyXML);
			byte[] bytes = Encoding.UTF8.GetBytes(itemToEncrypt);
			var encryptedData = provider.Encrypt(bytes,true);
			var retval = Convert.ToBase64String(encryptedData);
			return retval;
		}

		public string Decrypt(string privateKeyXML, string itemToDecrypt)
		{
			var provider = new RSACryptoServiceProvider(KeySize);
			provider.FromXmlString(privateKeyXML);
			byte[] bytes = Convert.FromBase64String(itemToDecrypt);
			var encryptedData = provider.Decrypt(bytes, true);
			var retval = Encoding.UTF8.GetString(encryptedData);
			return retval;
		}
	}

	public class KeyPair
	{
		public string PublicKey { get; set; } = "";
		public string PrivateKey { get; set; } = "";
	}
}

I hope that you find these tools useful. In my next post I’ll show how to combine this solution and the hashing library shown in my Building a Simple C# Hashing Utility post to build a bit more holistic solution to handling encrypted data.

Console Apps in Python and C# (.Net Core) Part 4

At this point we have a pair of programs written in Python and C#. These programs are used to run the ccextractor program with extension and path parameters. The next step in our evolution is to run our code on other platforms namely macOS and Linux. This post will demonstrate running code on both of those platforms.

Running on macOS

Before you start working on the code, you’ll need to get your Mac set up to 1) Install the ccextractor application and 2) Python 3 code.

Installing the extractor is simple and is done via the Homebrew infrastructure used by Mac Developers. To install the ccextractor do the following :

Install Homebrew if it’s not already installed. Simply run this script (copied from https://brew.sh/ ) from a terminal window.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Once you have installed Homebrew, you can install the ccextractor program by issuing the following command:

brew install ccextractor

After installing the ccextractor, test it by typing ccextractor from the terminal window. You should see a screen full of help information. Now insure Python3 is installed. From a terminal window type: python3

If Python 3 is installed you’ll see the Python’s interactive window. If no,t you may be promoted to install the Command Line tools for OSX. If so, run that installer. If the Command Line Tools installer does not run directions for installing Python 3 can be found here: https://docs.python-guide.org/starting/install3/osx/

Now it’s time to test the code. Clone this repo:

https://github.com/rjpaddock/ExtractorRunner.git

Now from a terminal window change into the folder where you cloned that repo and run the following command:

python3 run_cc.py --extension mpg --directory [[INSERT YOUR DIRECTORY HERE]]

You will be presented with the following error information:

Traceback (most recent call last):
  File "run_cc.py", line 15, in <module>
    subprocess.run([extractor_name,  os.path.join(args.directory, file)])        
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ccextractorwin'

This error is because the name of the ccextractor application is different in the Windows environment. Check out the last line. What is the fix for this ?

To fix this you need to call a different executable based on operating system. Lucky for us Python has a built in library for just such a thing. To check what platform your code is running import the platform library at the top of your python file

import platform

Next add the following code to your script:

extractorname = ''
if platform.system() == 'Windows':
  extractor_name = 'ccextractorwin'
elif platform.system() == 'Darwin':
  extractor_name = 'ccextractor'
elif platform.system() == "Linux":
  extractor_name = 'ccextractor'

Now run the application. Your script should start processing files with no error.

NOTE: The code in the Repository already has this change applied. You’re welcome 🙂

The next step is to get the C# code up and running on macOS. This process was much easier than I anticipated, as Microsoft has created a macOS version of Visual Studio. The first step is to install Visual Studio Mac from the Microsoft website https://visualstudio.microsoft.com/vs/mac/

When installing the application make sure to install it with the .NET Core option selected:

Once the installer completes, open the ExtractorRunner solution from the folder you pulled code into. Open the options dialog for the project and set the command line parameters you have been using to test:

Run your code now. You will now see an error in the console window of your application:

This is very similar to the Python error and requires the same solution. .NET Core also included a set or libraries to determine your operating system. Add the following snippet to the top of your program:

using System.Runtime.InteropServices;

Now add the following block of code to your C# program:

var extractor_exe_path = "";
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
  {
  extractor_exe_path = "ccextractorwin";  
  }
else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
  {
  extractor_exe_path = "ccextractorwin";
  }
else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
  {
  extractor_exe_path = "ccextractor";  
  }

Now run your code and you should see proper output in the runner window.

Now you have the same command line functionality for both the Python and C# versions of this program and can run the code on the Mac and Windows. Now lets take a look at the process of running thus under Ubuntu.

Running on Linux (Ubuntu)

Before modifying the our runner programs we need to install the ccextractor application on our Linux server. Directions for installing ccextractor on Linux can be found here: https://github.com/CCExtractor/ccextractor/wiki/Installation

Basically you pull the code from Github, and run the typical process of building applications in the Linux world. I was lucky as the code “just compiled” and ran using the instructions provided. Once I did that I had to make one simple change to the script and was able to execute our runner application. The branch of code to determine the proper program to run looks like this:

if platform.system() == 'Windows':
  extractor_name = 'ccextractorwin' 
elif platform.system() == 'Darwin':
  extractor_name = 'ccextractor'
elif platform.system() == "Linux":
  extractor_name = '/home/azureuser/data/projects/ccextractor/linux/ccextractor'

Now I was able to run the code using the same command line options we used on the Mac.

python3 run_cc.py --extension mpg --directory /home/azureuser/data/sampledata/

Now that the Python code is up and running you can turn your sites onto running the C# code next. To do this you need to first install the .NET Core SDK on your Ubuntu instance. This is done by following the directions from this page: https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu

If you are running a different flavor of Linux you can find directions on this page: https://docs.microsoft.com/en-us/dotnet/core/install/linux

Once you have the SDK installed , change into the folder where you cloned the GitHub repository and run the following command:

dotnet build

This will build an executable file and put it in a sub-folder (off the root of your code) in this location /bin/Debug/netcoreapp3.1 There is one more step though. Before you can run the code you need to change your Program.cs file to use the following executable selection code:

var extractor_exe_path = "";
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
  {
	  extractor_exe_path = "ccextractorwin";
	}
else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
	{
		extractor_exe_path = "ccextractor";
	}
else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
	{
		extractor_exe_path = "/home/azureuser/data/projects/ccextractor/linux/ccextractor";
	}

Run the dotnet build command again and change into that folder and run the following command:

./ExtractorRunner --extension mpg --directory /home/azureuser/data/sampledata/

The following screen shows the ccextractor running on Ubuntu via the c# Extractorrunner program.

That’s how you create a totally cross platform application in Python and C#. I was pleasantly surprised how simple it was to build and run the C# code on Mac and Linux which is a testament to the work the Microsoft team has done over the last few years. I hope you enjoyed this series. Next week I hope to return with a recap of this series with a few additional tips. Then it’s on to writing about other programming items.

Console Apps in Python and C# (.Net Core) Part 3

In my last post, I showed you how to add named parameters to a Python script. These parameters –extension and –directory gave us the ability to run our programs with different extensions and located in different directories. This article will add the same functionality to the C# version of this program.

Where Python has an argument parser built into its native libraries, the .NET platform does not. Not to fear, there is a third-party library that you can install to add this needed functionality. This library is called CommandLineParser and can be installed via a Nuget package. You can install this library via the Nuget console by issuing the following command:

Install-Package CommandLineParser -Version 2.8.0

Once you have installed this library you need to build a class that will hold your parsed command line parameters. This class will be augmented with Attributes provided by the command line parser. The first parameter to add is the dynamic extension. To do this add the following class code to your program:

public class Options
{
  [Option(longName:"extension",HelpText = "Extension of files to convert",Default = ".mpg")]
  public string Extension { get; set; } = "";
}

This code has a string property called Extension. When you pass in the –extension parameter it will be stored on this property. The more interesting aspect of this class is the [Option] attribute.

[Option(longName:"extension",HelpText = "Extension of files to convert",Default = ".mpg")]

The longName property tells the CommandLIneParser library to parse an argument with the name –extension onto the Extension parameter. The HelpText and Default properties are self explanatory based solely on their name.

Now that you have created this class you can call the command line parser to populate your arguments onto an instance of the Options class. This code demonstrates how to do this:

var parsed = Parser.Default.ParseArguments<Options>(args);
var options = ((Parsed<Options>) parsed).Value;

This code takes the args collection passed to your program, parses them, and returns a parsed object. After parsing the argument collection you need to cast the .Value property of the parsed object into an instance you can use in your programming code. Your processing logic will now look like this:

var directory_to_import = "D:/Data/clients/RodPaddock/CCExtractor/";
var extractor_exe_path = "D:/Data/clients/RodPaddock/CCExtractor/ccextractorwin";
foreach (var fileName in Directory.GetFiles(options.Directory,$"*{options.Extension}"))
{
  Console.WriteLine(fileName);
  var process = new Process()
  {
    StartInfo = new ProcessStartInfo
	  {
  	    FileName = $"{extractor_exe_path}",
  	    Arguments = $"{fileName}",
	    UseShellExecute = true,
	   }
   };
	process.Start();
 }

Notice that the GetFiles() function now uses the Extension property of your Options class.

The next step is to add the directory to your Options class. To do this, simply add another property to your class with the appropriate name and options.  Your class code will now look like this:

public class Options
{
  [Option(longName:"extension",HelpText = "Extension of files to convert",Default = ".mpg")]
  public string Extension { get; set; } = "";

  [Option(longName: "directory", HelpText = "Directory to process", Default = ".")]
  public string Directory { get; set; } = ".";
}

Notice that the DefaultValue property is a single period (.). This tells the get files routine to simple process the current directory.

Now you can incorporate your new Directory option into your application code. This is what the final version will look like:

using System;
using System.Diagnostics;
using System.IO;
using CommandLine;

namespace ExtractorRunner
{
	class Program
	{
		static void Main(string[] args)
		{
			
			var parsed = Parser.Default.ParseArguments<Options>(args);
			var options = ((Parsed<Options>) parsed).Value;
			
			var extractor_exe_path = "ccextractorwin";
			foreach (var fileName in Directory.GetFiles(options.Directory,$"*{options.Extension}"))
			{
				Console.WriteLine(fileName);
				var process = new Process()
				{
					StartInfo = new ProcessStartInfo
					{
						FileName = $"{extractor_exe_path}",
						Arguments = $"{fileName}",
						UseShellExecute = true,
					}
				};
				process.Start();
			}

		}
		public class Options
		{
			[Option(longName:"extension",HelpText = "Extension of files to convert",Default = ".mpg")]
			public string Extension { get; set; } = "";

			[Option(longName: "directory", HelpText = "Directory to process", Default = ".")]
			public string Directory { get; set; } = ".";
		}
	}
}

One item of note is the path to the EXE is just the name of the application. This is because in the last post we decided to add the Ccexteactorwin.exe file to our system PATH via the System Environment variables screen..

You can now run your code from visual studio. When testing your code you can call your application with arguments by opening your Project Properties Window, selecting the Debug section and passing specifying your parameters in the Arguments section. The following screen shows that:

Your running program will now spawn a new process which looks like this:

At this point we have two programs one in Python and one in C#. They both accept similar named parameters and now call the ccextractor program based on global PATHs. Next step is to get the code running on platforms other than Windows.

Console Apps in Python and C# (.NET Core) Part 1

Earlier this week I got an e-mail from a friend. Here’s the gist of the e-mail (names have been excluded to protect the innocent LOL)

I have a command line tool installed through homebrew on my laptop running high sierra. The command is just ccextractor <filepath> and it runs fine in a standard bash terminal. I was hoping to use automator to be able to run it on batches of files, but i'm struggling with the syntax for the Run Shell Script command.
It just keeps saying ccextractor command not found. 
Also the command line tool can only do one file at a time so I guess I need some way to loop the request so it does the first file, then runs the command again on the second file etc?

My friend is a fellow movie geek who wants to run CCExtractor on a batch of movie files.

CCExtractor https://www.ccextractor.org/ is an application used to extract closed captions from video files.

The problem was my friend could not figure out how to use Automator (a Mac tool) to run this command on a directory of files. An attempt was made to use bash as well with no luck. Hence the e-mail.

I replied back that I could probably whip something up in Python if that would work. “Are you sure that’s not too much work?” my friend replied. “Nah it should be pretty simple to whip up.”, I replied.

Here’s the gist what I did.

  1. Traveled to the https://www.ccextractor.org/ site and downloaded the binaries and some 3.x GB sample files to my drive.
  2. Then I opened my trusty text editor (https://www.sublimetext.com/) is my editor of choice and started a new .py (python) prorgam.
  3. After a bit of google-fu I came up with this set of code:

import os
import subprocess
directory_to_import = 'D:/Data/clients/RodPaddock/CCExtractor/'
extractor_exe_path = 'D:/Data/clients/RodPaddock/CCExtractor/ccextractorwin'
for file in os.listdir(directory_to_import):
  if file.endswith(".mpg"):
    print(os.path.join(directory_to_import, file))
    subprocess.run([extractor_exe_path, os.path.join(directory_to_import, file)])        

This code was built, debugged and run on my Windows development box. The goal was to get it working as fast as possible on my main development box before moving it onto a Mac.

Here’s a link a Gist of the code: https://gist.github.com/rjpaddock/d53956767dd4a1fe267dee08c995c956.js

Getting the code to run on the Mac was simple. Here’s that version:

import os
import subprocess
directory_to_import = '/Users/rodpaddock/ccextractor'
extractor_exe_path = 'ccextractor'
for file in os.listdir(directory_to_import):
  if file.endswith(".mpg"):
    print(os.path.join(directory_to_import, file))
    subprocess.run([extractor_exe_path,   os.path.join(directory_to_import, file)])   

As you can see the changes were minimal at best. I changed the path to my user directory on the Mac and git rid of the specific path to the executable. I used brew to install the CCExtractor on my mac so it was in the PATH already. After installing Python version 3.x on my old Mac I was able to run the application as-is. No operating specific issues.

After getting it to work I sent it off to my fiend who simply changed the path to the files to decode and BOOM it just worked.

After marveling at how much could be accomplished with so few lines of code, I became curious to see how complex it would be to build the same application in C#. I’m using .NET Core to do this, as I want to run it cross platform as well.

Here’s the same functionality in C#

using System;
using System.Diagnostics;
using System.IO;

namespace ExtractorRunner
{
  class Program
  {
  static void Main(string[] args)
   {
    var directory_to_import = "D:/Data/clients/RodPaddock/CCExtractor/";
    var extractor_exe_path = "D:/Data/clients/RodPaddock/CCExtractor/ccextractorwin";
    foreach (var fileName in Directory.GetFiles(directory_to_import,"*.mpg"))
    {
      Console.WriteLine(fileName);
      var process = new Process()
	{
	  StartInfo = new ProcessStartInfo
		{
		FileName = $"{extractor_exe_path}",
		Arguments = $"{fileName}",
		UseShellExecute = true,
	 } 
	};
	 process.Start();
   }
  }
 }
}

Not too bad .NET core. It was pretty simple to build this application and get it running in a console application.

Here’s a Gist to the C# code: https://gist.github.com/rjpaddock/be601db3995082949071121d8aa992d7

Now that I have this code, I think it would be fun to explore making it a bit more useful. I’m doing this as an exercise to learn a few more things about building more robust Python and C# console applications. Here’s a set of features I plan on adding:

  • Accept an extension parameter (I started with .mpg files) my friend had to change the extension to .mp4 files.
  • Accept the path to decode as a parameter.
  • Accept the path to the executable as a parameter
  • Parameters should be named vs positional if possible.
  • Run this code on Windows, Mac and Linux.

Building a Simple C# Hashing Utility

One of the most interesting aspects of Cryptography is the ability to generate unique hash values from strings or files.

Wikipedia Defines Cryptographic as follows:

A cryptographic hash function is a hash function that is suitable for use in cryptography. It is a mathematical algorithm that maps data of arbitrary size to a bit string of a fixed size and is a one-way function, that is, a function which is practically infeasible to invert

https://en.wikipedia.org/wiki/Cryptographic_hash_function

The most common use of hashes is to store hashed (vs plain text) values of passwords in databases.

Another use case of hashes is to detect changes to files. A few years back we built a script runner that would only run new or changed files from a given scripts folder. The basic algorithm for this was:

1) Read all file names from a folder.

2) Look for for a file same name with a .HASH extension.
    If .HASH file was missing:  
      a) Run the script file
      b) Create a new file with a .HASH extension. 
      c) Hash the script file and store the result in 
         the .HASH file

3) If the .HASH file already existed:
    a) Generate a hash value of the script file 
    b) Compare value the hash value stored in the .HASH file. 
    c) If they were the same we ignore the file. 
    d) If they were different Run the script, 
       re-generate the hash and store the results 
       in the .HASH file. 

The core of these processed is a little nugget of code I built up a few years back .I thought it would be good to share my hashing function with the world. The following code generates a SHA512 has for a given string. The code then takes the generated byte array converts it to a string:

public static string GenerateHash(string stringToHash)
{
  var crypt = new SHA512Managed();
  var hash = new StringBuilder();
  var crypto = crypt.ComputeHash(Encoding.UTF8.GetBytes(stringToHash), 0,
    Encoding.UTF8.GetByteCount(stringToHash));

    foreach (var bit in crypto)
    {
      hash.Append(bit.ToString("x2"));
    }

    return hash.ToString();
}

I built a little command line utility that implements this function and can generate a hash for a passed in string or file. You can find this utility on Github:

https://github.com/rjpaddock/HashUtility.git

Description of SHA Hashing: https://en.wikipedia.org/wiki/Secure_Hash_Algorithms