Console Apps in Python and C# (.NET Core) Part 2

Sometimes time is of essence when building little utility programs. Once the job of your utility is completed it may be useful to make that program more useful and more universal. Or it just might be a good way to learn something new for a blog post or just your own education.

I decided that this would be a cool opportunity to add some features to the batch runner I created for my friend. One item that was immediately needed changing was the ability to choose the extension of the files to process. My initial choice was to process mpg files as the default extension. My friend immediately changed it to mp4. This is a useful extension point and the first thing to parameterize.

This post will focus on the Python version first. We’ll look at the C# version in our next post. One way we could hack this together would be to use Python’s sys.argv[] array which provides positional arguments to Python programs. For instance if we called our program with the following statement:

python copy run_cc.py .mp4

we could access the .mp4 with sys.arg[0]
While this works it will cause problems in the long haul if we add or remove parameters. It is also not very intuitive. It would be better to call the with a named parameter. For example:

run_cc.py –extension .mp4

Python has a built-in library to do this exact thing. This library is known as argparse. To implement our first option we need to do the following:

  1. Add an import argparse to the imports section of our program
  2. Create an argument parser object and add an argument to it. Your code will look like this:
parser = argparse.ArgumentParser()
parser.add_argument("--extension", help="Extension of files to convert", default='.mpg')

args = parser.parse_args()

There is a lot going on with just these few lies of code. What this set of code does is 1) Create an argument parser, 2)Adds a parameter called –extension to the command line. This parameter will be added the args array as a property with the name extension. parameter. Finally this code specifies a help description and a default parameter value. Our program code now looks like:

import os
import subprocess
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--extension", help="Extension of files to convert", default='.mpg')
args = parser.parse_args()

directory_to_import = 'D:/Data/clients/RodPaddock/CCExtractor/'
extractor_exe_path = 'D:/Data/clients/RodPaddock/CCExtractor/ccextractorwin'
for file in os.listdir(directory_to_import):
  if file.endswith(args.extension):
    print(os.path.join(directory_to_import, file))
    subprocess.run([extractor_exe_path,   os.path.join(directory_to_import, file)])

The next step is to add a parameter to specify the directory you wish to read files from. We’ll call the parameter –directory and will default it to (.) the current working directory. A sample call would be as follows:

python run_cc.py 
    --extension .mp4 
    --directory "D:/Data/clients/RodPaddock/CCExtractor/"

Your Python code will now looks like this:

import os
import subprocess
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--extension", help="Extension of files to convert", default='.mpg')
parser.add_argument("--directory", help="Directory to process", default='.')
args = parser.parse_args()

extractor_exe_path = 'D:/Data/clients/RodPaddock/CCExtractor/ccextractorwin'
for file in os.listdir(args.directory):
  if file.endswith(args.extension):
    print(os.path.join(args.directory, file))
    subprocess.run([extractor_exe_path,   os.path.join(args.directory, file)])        



Finally lets get rid of the EXE path. We are going to “cheat” a bit on this one. We are simply going to add that directory to the PATH statement on the machine. This will sync its operation with the behavior on the Mac.

To change your PATH statement in Windows open the Environmental Variables from the Windows Start menu find PATH in the System variables and add the path to wherever you extracted the ccextractor application. The following screen demonstrates how this should look:

Image of changes to PATH statement in Environmental Variables Screen.
Changes to PATH statement in Environmental Variables Screen.

Now your final Python program will look like this:

import os
import subprocess
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--extension", help="Extension of files to convert", default='.mpg')
parser.add_argument("--directory", help="Directory to process", default='.')
args = parser.parse_args()

# should be added to the system PATH statement
extractor_name = 'ccextractorwin' 
for file in os.listdir(args.directory):
  if file.endswith(args.extension):
    print(os.path.join(args.directory, file))
    subprocess.run([extractor_name,  os.path.join(args.directory, file)])        


This completes part 2 of this series. In part 3 we will learn how to accomplish the same feature using C# across multiple platforms.

Console Apps in Python and C# (.NET Core) Part 1

Earlier this week I got an e-mail from a friend. Here’s the gist of the e-mail (names have been excluded to protect the innocent LOL)

I have a command line tool installed through homebrew on my laptop running high sierra. The command is just ccextractor <filepath> and it runs fine in a standard bash terminal. I was hoping to use automator to be able to run it on batches of files, but i'm struggling with the syntax for the Run Shell Script command.
It just keeps saying ccextractor command not found. 
Also the command line tool can only do one file at a time so I guess I need some way to loop the request so it does the first file, then runs the command again on the second file etc?

My friend is a fellow movie geek who wants to run CCExtractor on a batch of movie files.

CCExtractor https://www.ccextractor.org/ is an application used to extract closed captions from video files.

The problem was my friend could not figure out how to use Automator (a Mac tool) to run this command on a directory of files. An attempt was made to use bash as well with no luck. Hence the e-mail.

I replied back that I could probably whip something up in Python if that would work. “Are you sure that’s not too much work?” my friend replied. “Nah it should be pretty simple to whip up.”, I replied.

Here’s the gist what I did.

  1. Traveled to the https://www.ccextractor.org/ site and downloaded the binaries and some 3.x GB sample files to my drive.
  2. Then I opened my trusty text editor (https://www.sublimetext.com/) is my editor of choice and started a new .py (python) prorgam.
  3. After a bit of google-fu I came up with this set of code:

import os
import subprocess
directory_to_import = 'D:/Data/clients/RodPaddock/CCExtractor/'
extractor_exe_path = 'D:/Data/clients/RodPaddock/CCExtractor/ccextractorwin'
for file in os.listdir(directory_to_import):
  if file.endswith(".mpg"):
    print(os.path.join(directory_to_import, file))
    subprocess.run([extractor_exe_path, os.path.join(directory_to_import, file)])        

This code was built, debugged and run on my Windows development box. The goal was to get it working as fast as possible on my main development box before moving it onto a Mac.

Here’s a link a Gist of the code: https://gist.github.com/rjpaddock/d53956767dd4a1fe267dee08c995c956.js

Getting the code to run on the Mac was simple. Here’s that version:

import os
import subprocess
directory_to_import = '/Users/rodpaddock/ccextractor'
extractor_exe_path = 'ccextractor'
for file in os.listdir(directory_to_import):
  if file.endswith(".mpg"):
    print(os.path.join(directory_to_import, file))
    subprocess.run([extractor_exe_path,   os.path.join(directory_to_import, file)])   

As you can see the changes were minimal at best. I changed the path to my user directory on the Mac and git rid of the specific path to the executable. I used brew to install the CCExtractor on my mac so it was in the PATH already. After installing Python version 3.x on my old Mac I was able to run the application as-is. No operating specific issues.

After getting it to work I sent it off to my fiend who simply changed the path to the files to decode and BOOM it just worked.

After marveling at how much could be accomplished with so few lines of code, I became curious to see how complex it would be to build the same application in C#. I’m using .NET Core to do this, as I want to run it cross platform as well.

Here’s the same functionality in C#

using System;
using System.Diagnostics;
using System.IO;

namespace ExtractorRunner
{
  class Program
  {
  static void Main(string[] args)
   {
    var directory_to_import = "D:/Data/clients/RodPaddock/CCExtractor/";
    var extractor_exe_path = "D:/Data/clients/RodPaddock/CCExtractor/ccextractorwin";
    foreach (var fileName in Directory.GetFiles(directory_to_import,"*.mpg"))
    {
      Console.WriteLine(fileName);
      var process = new Process()
	{
	  StartInfo = new ProcessStartInfo
		{
		FileName = $"{extractor_exe_path}",
		Arguments = $"{fileName}",
		UseShellExecute = true,
	 } 
	};
	 process.Start();
   }
  }
 }
}

Not too bad .NET core. It was pretty simple to build this application and get it running in a console application.

Here’s a Gist to the C# code: https://gist.github.com/rjpaddock/be601db3995082949071121d8aa992d7

Now that I have this code, I think it would be fun to explore making it a bit more useful. I’m doing this as an exercise to learn a few more things about building more robust Python and C# console applications. Here’s a set of features I plan on adding:

  • Accept an extension parameter (I started with .mpg files) my friend had to change the extension to .mp4 files.
  • Accept the path to decode as a parameter.
  • Accept the path to the executable as a parameter
  • Parameters should be named vs positional if possible.
  • Run this code on Windows, Mac and Linux.

The Importance of Banking Locally

I relocated my family from Seattle, WA to Austin, TX, in August of 2008. At that time, my bank accounts (personal and business) were with Bank of America.  The nice thing about this arrangement was the Bank of America branch right up the street from my new house.  This seemed like total win. That is until several things converged that made me re-evaluate this relationship.

If you recall, August of 2008 was the height of the financial crisis. As the banks were getting bailed out by taxpayers, they were simultaneously cutting off access to credit for their customers. I had a number of credit lines that were paid off but would have provided a lifeline should the need ever arise. Well that lifeline was withdrawn arbitrarily by several banks including BOA.

The second thing that happened was my move to Austin where a number of stores participated in an initiative called GO LOCAL. The idea behind GO LOCAL was that monies spent with local businesses.stayed in the communities where they were spent. This made total sense. Wherever possible I did my best to shop at local stores and restaurants (not chains).

The last straw was the total lack of customer service with the local branch of BOA. Every week or so I would show up, in person, to deposit checks into my account. Every time I went in I was treated like a complete stranger vs a valued customer. Now I get that as a person new to that branch it might take some time to become familiar with me and my small company. Well this familiarity was never achieved.  Almost every time I was at the store I banked with THE SAME PERSON. This person NEVER even hinted at a sign of recognition. This was over a period of 6 or more months.

It was this total lack of disregard that made me act. I decided that the best thing to do was to GO LOCAL. I would open new personal and business accounts with a local institution. In this case it was UFCU (University Federal Credit Union) which, lucky for me was located right around the corner from my house too.

I was NOT looking forward to the task of moving numerous auto draft transactions attached to my current bank accounts.  This included payroll, credit cards, house payments, utilities, etc. How would I do this with as little disruption as possible. My strategy was turned out to be simple: keep my old accounts open while I move everything to the new accounts.  I started by examining each bank statement and seeing what vendors were being paid from my accounts. I opened each vendor’s respective site and changed the auto billing information. This took a few months as some bills were not monthly. After a few months I left a reasonable amount in each account, you know, just in case 😊

Now you might have a few concerns about banking with small local banks. For example you may thing: the small banks do not have all of the conveniences of the big banks. I have a one-word answer to this concern: BLAHBLAH :),

These small banks have access to the same technology as the big banks and in many cases are much more efficient at deploying it. For instance, we have purchased several cars using our credit union. The way they do this is unique. They give us a “blank check” that we take to the car dealer and purchase our car. This is not technically a blank check, but a check that with a limit up to the max amount we were approved to finance.  When we finished our negotiations to purchase our vehicles, we fill in the amount of the purchase and sign the check. A few days later, after the car dealer cashes the check we get a call from the credit union and we sign the final contract for our loan. Oh, and this is all done via Docusign. This is a very cool experience and works well. The added benefit is that the interest paid on the loan is reasonable and STAYS IN THE COMMUNITY.

One added benefit is the RELATIONSHIP I have with my credit union. I know the people I bank with, and when I have issues, or need help they are there for me. And in the year 2020 this was especially true. As the Covid virus took over the world and caused a disruption to the economy (including my business) I heard about the Payroll Protection Program which would provide a two months of assistance in making payroll. I was LUCKY enough to secure one of these SBA PPP loans in the first round. I feel that this was due to the fact I had a relationship with my credit union and that they were agile enough to implement their loan program quickly. I have heard directly that some of the larger banks outsourced their work and failed to get the loans their customers really needed.

My credit union was the opposite. They treated their customers with respect and concern and helped secure these desperately needed loans. This isreally assuring and gives me confidence that my decision to bank locally was a good one.

Building a Simple C# Hashing Utility

One of the most interesting aspects of Cryptography is the ability to generate unique hash values from strings or files.

Wikipedia Defines Cryptographic as follows:

A cryptographic hash function is a hash function that is suitable for use in cryptography. It is a mathematical algorithm that maps data of arbitrary size to a bit string of a fixed size and is a one-way function, that is, a function which is practically infeasible to invert

https://en.wikipedia.org/wiki/Cryptographic_hash_function

The most common use of hashes is to store hashed (vs plain text) values of passwords in databases.

Another use case of hashes is to detect changes to files. A few years back we built a script runner that would only run new or changed files from a given scripts folder. The basic algorithm for this was:

1) Read all file names from a folder.

2) Look for for a file same name with a .HASH extension.
    If .HASH file was missing:  
      a) Run the script file
      b) Create a new file with a .HASH extension. 
      c) Hash the script file and store the result in 
         the .HASH file

3) If the .HASH file already existed:
    a) Generate a hash value of the script file 
    b) Compare value the hash value stored in the .HASH file. 
    c) If they were the same we ignore the file. 
    d) If they were different Run the script, 
       re-generate the hash and store the results 
       in the .HASH file. 

The core of these processed is a little nugget of code I built up a few years back .I thought it would be good to share my hashing function with the world. The following code generates a SHA512 has for a given string. The code then takes the generated byte array converts it to a string:

public static string GenerateHash(string stringToHash)
{
  var crypt = new SHA512Managed();
  var hash = new StringBuilder();
  var crypto = crypt.ComputeHash(Encoding.UTF8.GetBytes(stringToHash), 0,
    Encoding.UTF8.GetByteCount(stringToHash));

    foreach (var bit in crypto)
    {
      hash.Append(bit.ToString("x2"));
    }

    return hash.ToString();
}

I built a little command line utility that implements this function and can generate a hash for a passed in string or file. You can find this utility on Github:

https://github.com/rjpaddock/HashUtility.git

Description of SHA Hashing: https://en.wikipedia.org/wiki/Secure_Hash_Algorithms

Adventures in Writing Part 2: The Amateur Years

My last post Adventured in Writing Part 1 talked about my experiences as a fledgling writer and how I published my 1st article. That experience was truly life affirming. Unfortunately, that was my 1st and last D&D related article. But this did happen:

Met Gary Gygax at a conference in Evansville, Indiana.

Not too long after the D&D article was published, my family moved to Bend Oregon where I registered at Central Oregon Community College. When I built my schedule, I took as many classes as possible that related to writing and publishing. Here’s my schedule (Yes, I have a lot of this crap in my binders)

I had every plan on becoming a full-time writer. As a matter of fact, one of my first papers for writing comp was called “Tales of a freelance writer”. Check out the red typewriter ink on this baby:

Term Paper For The Win!

During my tenure at C.O.C.C. I kept the dream of writing alive and the college newspaper became my outlet. I wrote few profiles of a couple of my instructors. Presented for your enjoyment: my profiles of Art Sanchez and Gene Taylor

Profile of Art Sanchez

One item I would like to note is that these two professors were in the Business Software track at the community college. I had changed my focus from becoming a writer to becoming a computer programmer. If you look at my schedule, you’ll see that in the 2nd and following semesters I took more and more technology programs. After a little more than 2 years I had succeeded in becoming a programmer. But what happened to my writing? Honestly my writing took a back seat to my coding. I had traded one passion for another.

After spending a few years as a programmer for a local vacation resort I set out for the big city: SEATTLE! After a brief stint working at CSC (Computer Sciences Corp), I went to work for a company called The Juiceman. You remember the Juiceman:

You know you’ve made it as a company when Jim Carry mocks your founder 😊 At the Juiceman I worked on a kick ass team where we toiled building innovative order entry, fulfillment and payment processing systems. For the time this was large scale application capable of supporting literally hundreds of users (LOL)

Our company was exploding at the seams and our team was in a race to keep up with the growth. One night we were deploying our software and things went horribly wrong. A conversion application we had written was seriously flawed and we had to roll back our deployment. This recovery was done with a partial backup I had miraculously created. It was not a planned backup, it was a lucky backup. After a very LONG night another LONG day after we managed to recover.

After some sorely needed rest we took time analyze what went wrong. I took copious notes. These notes rekindled something that had gone dormant: my desire to be a writer. “This could be a cool article” I thought to myself. But where to publish this article. In the next post I’ll talk about getting published professionally.

Adventures in Writing Part 1

A good friend of mine C. Robert Cargill  is a screen writer who spends a lot of his time on Twitter (@massawrym) mentoring writers. The biggest themes in his mentoring include: being disciplined and never giving up. I wanted to share my story to help emphasize theme of never giving up.

My story begins in the early 80’s. In the early 80’s I, like many other writers, discovered the game Dungeons & Dragon. I was a player and eventually became a dungeon master. As the mid 80’s rolled around I got deeper into the game and eventually started to go to D&D conventions that occurred throughout the Los Angeles area. Along with attending D&D conventions I felt the urge to write about the game I truly loved. I had a goal: I wanted to be published in either Dragon (the O.G. of D&D magazines) or Dungeon (a new magazine dedicated to short one adventures).

My first submission was an adventure for the new magazine called Dungeon: 

Here you can see my 1st submission being SOUNDLY REJECTED…

Being the ever diligent adventurer (writer) I didn’t give up. I started work almost immediately on my next article. This one would be for Dragon Magazine and had the title “Before First Level”. This one took a long time and I was proud of the work. When I was done I printed it on the good old dot-matrix printer in computer science class (APPLE IIe and submitted it to Dragon Magazine c/o TSR Hobbies.

It took forever to get a response. Oh no, it was another rejection…. But WAIT! This time the rejection letter had a NOTE. Check it out…

Yup I received praise from the Editor in Chief…. The one and only Roger Moore (no not James Bond) He said I had talent. WOW this was EXACTLY what I needed to hear. I persevered. While I waited to hear from Dragon Magazine, I became a member of the RPGA (Role Playing Gamers Association). The RPGA had a newsletter called Polyhedron.  Oh boy yet another place I could possibly get published….

I soon went to work putting together a submission for Polyhedron. My 1st article for Poly was called “Solutions to the Monty Haul” campaign.

I soon received a letter from Jean Rabe the editor in chief of Polyhedron. She rejected the article:

She also suggested I flesh out one of the threads in the article… I chose an idea related to taxes. I called it “The Role of Taxes” This one was put together on a typewriter and submitted to Polyhedron. Sometime later I received a note from Jean… CONGRATULATIONS we will be printing your article in a future issue of Polyhedron.

My 1st (non-professional) article was published in Issue #38 of Polyhedron. Here’s the table of contents with a shot of the cover and the text of the printed article..

This publication occurred in 1987 and it took me nearly 5 more years before I got paid to write. Now I didn’t stop writing I just had stuff like college take over a bit. In my next part I’ll talk about college and how I became a professional writer.