Running Python on the Command Line

Fixing MacOS error with Python3

Have you ever run into a problem executing a python3 file where you would get the following error:

• OSError: cannot open resource
• env: python3: Operation not permitted
• python3: bad interpreter: Operation not permitted.

Easy Fix

Changes are you have the following header in that Python file:

To make it work, change the header to:

Reasoning

This is probably happening because Python3 is not pointing to the right version of Python3.

The easy fix, is put in the full path of Python3. For most installations, it should be /usr/bin/python3. (Apple does a good job of keeping up with the current Python version.)

File Permissions

One last thing...

If your still having problems, make sure that the file permissions are set correctly.

Permalink

Variables in Bash

Using variables in Bash is easy to learn if you're familiar with PHP programming.

Four Simple tips when using variables in Bash

1. Keep it simple
2. Use variable parameters to expand the script capabilities
3. Watch for extra spacing when using the command expansion ${
4. When using characters as values make sure to use double quotes.

Learn by Example

One of the first Web programming books that I found useful was "Perl by Example." It's a great book to learn various Perl functionality. ( I still have the book!)

I got one of the early editions which was a much smaller book and a lot easier to understand

So here are some Variable examples in BASH:

Simple HelloWorld.sh Example

Basics of assigning a text value to a BASH variable:

#!/bin/bash 
STR="Hello World!" 
echo $STR 

BASH script with Variable Parameters

You can pass additional parameters to you BASH script at run time:

#!/bin/sh
#
# Echo a couple of script input values
# usage:
#    name   
#
# The result will be an echo of the first and last
if [ $# != 3 ]; then
 echo "usage: name.sh first middle last"
 exit 0
fi
FIRST="$1"
MIDDLE="$2"
LAST="$3"
echo $FIRST . " " . $LAST

Execute Command Lines via BASH variables

You can assign the results of an command line to a BASH variable

#!/bin/bash
# Get the last day of March
lastday=$(date -v+3w -v1d -v+1m -v-1d +"%B %e, %Y")
echo $lastday

Simple Math in Bash

#!/bin/bash
num1=2017
num2=1975
total=$(($num1 - $num2))
echo "Did you know 1975 was $total years ago!"

Miscellaneous Notes

In Bash $$ is the process ID, as noted in the comments it is not safe to use as a temp filename for a variety of reasons.

Permalink

Printable view of Man pages

The manual for many Unix commands are available via the 'man' functionality.

For example, If I want to find out information about the Date functionality, I simply type in:

man date

Useful information, but this a Macintosh and we can do better. There has to be a better way to view the man page.

View like its 2017

The problem with this way is that you only see a screen size view of the document. To go to the next page, you simply type in the spacebar.

This is 2017, there's no reason to view man pages like this anymore.

Plaintext file View

To get a plain text version of a man page, without backspaces and underscores, try

man calendar | col -b > cal.txt

Now you can view this plain text file in your favorite text editor. You can also view this in Chrome simply by drag and drop file the icon to Chrome or Safari.

Open in a different terminal window

Mac OS comes with a built-in Man viewer that might be useful. Simply type this command

open x-man-page://ls

You get a yellow reference guide. This is nice to have so that you can display the info side-by-side your working terminal:

BBEdit view

If BBEdit is your favorite editor, simply send a plain text man page directly to BBEdit

man man | col -b | bbedit

Update your .bashrc

Here's a suggested code snippet to add to your bash profile:

function mman() { man $@ | col -b | bbedit; }

So the next time you need to view a man page type in:

mman perl

Then you can search using BBEdit for the feature you are looking for.

Permalink

Jot

Jot is a Unix program that prints sequential or random data. It's useful when you need a bunch of random words from a file or need a large numeric list.

This is just another way to generate a list of random words.

Random Group of Words

#! /bin/bash
jot -r -c 1000000 a z | rs -g 0 6 | sort | join /usr/share/dict/words - | uniq

Examples of using Jot

Sometimes it's easier to learn a command when you see several examples:

List the Years

If you needed to list the years in a decade or a large data range:

jot - 1950 1960
jot - 1970 2016

Multiple Text

Bart Simpsons would have loved this jot script:

jot -b "I will not cheat and output this text 20 times." 20

Hours in a Day

List out the hours in a day. (Could be done in Excel just as Easier)

jot -w '%01d:00' 12

Twitter Limit

Permalink

Date - Unix Command Line

The Unix Date command is a very handy tool to do date displays and calculations.

The basic of displaying a date:

date

You can format the output by simply passing a few parameters:

date +"%B %d, %Y"

April 05, 2017

Common Display Formats

Standard Date	date +"%A, %B %d, %Y"	Wednesday, April 05, 2017
Standard Time	date +"%I:%M %p"	03:29 PM
MySQL Format	date +"%Y-%m-%d %H:%M:%S"	2017-04-05 13:21:13
United States format date +"%m/%d/%Y"	04/05/2017
European date format	date +"%d %B, %Y"	05 April, 2017
ISO 8601 date format	date +"%Y-%m-%d"	2017-04-05
ISO 8601 Combined Format	date +"%Y-%m-%dT%R:%S"	2017-04-05T13:3252

Find Non-Release Weeks

At my work we do release every Thursday. However, if the Thursday is within 2 days of the end of the month we skip doing a release that week.

It would be handy to know all the weeks that we will not be shipping and let people know in advance. I thought it would be a handy thing for a Bash script.

Here's a simple script that does that. This will loop through all the Thursdays in 2017 and see if it follows the business rule. If it's not a release week print out the date.

#!/bin/BASH
max=50
for (( start = 1; start <= $max; start++ )) do
# Get the Next Thursday
check=$(date -v+"$start"w -v+1d +"%e") 
check_fmt=$(date -v+"$start"w -v+1d +"%B %e, %Y")
# Get the last day of the Month
lastday=$(date -v+"$start"w -v1d -v+1m -v-1d +"%e")
# Check the number of days to the End of the Month
daystoendofmonth=$(($lastday - $check));
# Apply the 2 Day Business Rule and echo the no release weeks
if [ "$daystoendofmonth" -lt "2" ]
then
echo "Don't release on: " $check_fmt
fi
done

The above script is written to be run on a Wednesday.

When I run this script, I discovered:

• Don't release on: June 29, 2017
• Don't release on: August 31, 2017
• Don't release on: November 30, 2017

Looks like we'll have a pretty quiet summer this year.

Permalink

Say Terminal Command

The Terminal 'Say" command is a utility that converts text to audible speech. This is the same speech technology that MacOS uses when you select a text and ask to speak.

With this command and some voice files, you can create some unique audio clips. The MacOS comes with over 48 speech files.

Simple Execution

To run the command, you tell "Say" the voice you want to use and then what text to speak:

Say -v Bruce "Hello World"

Instead of entering a text at the command line, you could reference a text file by using the "-f" command.

Say -v Bruce speech.txt

If you which to output the speech to a file use the "-o" and the name of the file. If you don't include a file extension, the file format will be AIFF which is basically no audio compression - and you end up with a very large sound file. The best format to use, at least in my testing, is m4p.

Say -v Bruce "Hello World" -o hello.m4p

Bash Script Example

You can sample all the audio clips by using this simple Bash script:

#! /bin/bash
array=("Agnes" "Albert" "Alex" "Alice" "Alva" "Amelie" "Anna" "Bad News" "Bahh" "Bells" "Boing" "Bruce" "Bubbles" "Carmit" "Cellos" "Damayanti" "Daniel" "Deranged" "Diego" "Ellen" "Fiona" "Fred" "Good News" "Hysterical" "Ioana" "Joana" "Jorge" "Juan" "Junior" "Kanya" "Karen" "Kathy" "Katya" "Kyoko" "Laura" "Lekha" "Luca" "Luciana" "Maged" "Mariska" "Mei-Jia" "Melina" "Milena" "Moira" "Monica" "Nora" "Paulina" "Pipe Organ" "Princess" "Ralph" "Samantha" "Sara" "Satu" "Sin-ji" "Tessa" "Thomas" "Ting-Ting" "Trinoids" "Veena" "Vicki" "Victoria" "Whisper" "Xander" "Yelda" "Yuna" "Yuri" "Zarvox" "Zosia" "Zuzana")
for i in "${array[@]}"
do
   say -v $i -f stock.txt -o $i.m4p
done

What this file does

This Bash script will loop through each speech file and output the speech to an M4p file. You can then listen to each clip and find the perfect audio for your needs.

There is an individual file for each sound so you know the name of the voice that you are listening to.

Just make sure to put this script in a place where all the audio files are going. The folder size will be 279 MB.

Halloween Scary Fun

Some Macintosh users find the Whisper voice to be creepy. Whisper doesn't come by default. You will need to install it. Simple Install Instructions

• Go to the System Preferences
• Select Accessibility
• Look for the Speech icon
• Next to System Voice, select the pull down and select Customize
• Under the 'English (United States) - Novelty' select Whisper
• Click 'OK'
• Note: Downloading the Whisper will also update the Samantha and Alex voices to higher quality.

Once you have it installed, Try this example:

say -v Whisper "We are watching you"

More Voices Available

If your really into Text-To-Speech, check out the collection of voices at CereProc. They have a rather large collection of voices in various accents and languages.

The voices cost ~ $30, however right now they have a sale for 50% for personal use. The voices are really cool and very natural sounding. There's a demo on the top of the page to try it out.

Permalink

Grep

Grep is a great powerful tool that print lines matching certain patterns.

Some example Regular Expressions using Grep:

Remove BOTs from the website access.log file:

grep -Ev '|spider|slurp|yandex|bingbot|majestic12' access.log > access2.log

Show only City/Towns in Massachusetts that have at least 19 characters

grep -x '.{10}' MassachusettsCityTowns.txt

Highlight the Search Results

export GREP_OPTIONS='--color=auto' GREP_COLOR='100;8'
grep '.{10}' MassachusettsCityTowns.txt

Using Grep in Bash Script

This is a script that I wrote a long time ago, that I used to check to see how many times a certain error would appear in a log file.

#!/bin/bash
itoday=$(date +"%d/%b/%Y")
todayerrors=$(cat grep $itoday /server/debug.log | grep $itoday | grep "Exception" | wc -l)
echo "# ------------------------------------------------------------"
echo "# Number of Errors Found on $itoday : $todayerrors"
echo "# ------------------------------------------------------------"
echo "# Raw data"
grep -C 10 "Exception" /mnt/$i/server/debug.log

This script will count out the total number of errors that were generated today. Then display the RAW lines. This is useful to have handy to see if a particular error is occurring quite often.

Quick RegEx Table

This is a quick regular expression table that I found many years ago. It's a quick reminder of some of the common grep commands:

#######################################################
# RegExpression Table
#######################################################
Reg-expr   | Description
-----------+---------------------------------------------------
.          | Matches any character except newline
[a-z0-9]   | Matches any single character of the set
[^a-z0-9]  | Matches any single character not in set
d         | Matches a digit,                   i.e., [0-9]
w         | Matches a alpha-numeric character, i.e., [a-zA-Z0-9_]
W         | Matches a non-word character,      i.e., [^a-zA-Z0-9_]
metachar  | Matches the character itself,      i.e., |, *, +
x?         | Matches 0 or 1 x's, where x is any of the above
x*         | Matches 0 or more x's
x+         | Matches 1 or more x's
x{m,n}     | Matches at least m x's but no more than n
foo|bar    | Matches one of foo or bar
(x)        | Brackets a regular expression (this is a bit of a lie :-)
b         | Matches a word boundary

Permalink

Directory Cleanup via Bash Script

This is a simple BASH script that I used to sort a folder that has a lot of images. I wanted to put photos in a folder based on the date it was created.

This script will work in any environment, but I wrote this specifically to handle a large set of photos in a folder on my Macintosh.

#!/bin/bash
for f in *; do
dir=$(stat -f%SB -t%Y-%m-%d "$f")
echo $f '->' $dir
[ -d "$dir" ] || mkdir "$dir"
mv "$f" "$dir"/
done

What this Script Does

When you execute this script in any directory, it will use the UNIX stat command to check on the time stamp on the file. It will then move the file to a folder with the same. If the directory doesn't exist then one will be created.

Script Execution

Create a file with the above content. I call my file sort.sh. Move the file to any directory that you want to clean up. To execute, simply type in:

./sort.sh

In a few minutes you'll see the contents of the folder be replace with a bunch of date folders. (Yes even the sort.sh file will get moved to a folder.)

Permalink

Bashmarks

None

When you SSH to a remote box do you change to the same set of directories? Wouldn't it be great to bookmark the locations to make it easy to get to? You can with Bashmarks!

Bashmarks gives you the ability to create simple bookmarks to any location on your server. Now you can jump to whatever directory you want wherever you are on the box.

As my former boss would say, "Now your cooking with gas."

Setting up Bashmarks

Bashmarks is simply a shell script that performs actions.

Download bashmarks.sh from GitHub. Put the file someplace where you won't accidentally delete it. (~/bin is a safe place, Macintosh users may want to keep it in /usr/local/bin)

If it isn't easy to move the file, simply copy and paste the contents of the file into a new file on the server that you want to use. (The file size is small.)

Update the .bash_profile with the following line:

# Make sure the directory and filename is correct.
source ~/bin/bashmarks.sh

Exit out and log back in the server.

Type:
bookmarksshow

If it's installed correctly, you should get back an empty line.

Using BashMarks

It's super easy to use BashMarks. When your at a directory that you want to bookmark simply type in:

bookmark staging

Where "staging" is the name of the bookmark - Tip: keep it simple and short.

When you want to go to that directory simply type in:

go staging

If you want to see all the Bashmarks that you have installed, type in:

bookmarksshow

Additional Notes

Currently Bashmarks only works with directories and not applications. However, you have the bashmarks.sh source file and it should be easy to manipulate it to do what works best in your environment. For example, you could shorten the word 'bookmark' to simply 'book.'

Enjoy playing around with Bashmarks!

Permalink

Emoji on the Command Line

In today's environment, Developers and QA engineers quite often have to log in to a remote AWS server that has virtual servers to perform tasks. Sometimes they are busy and may forget which server they are monitoring. One way to fix this is to use the emoji name in the Macintosh prompt. This way you know that your back on your computer and not on any production AWS server.

You will need to use Apple's Terminal app to add Emoji to the prompt, as iTerm doesn't really have good emoji support. This isn't an issue since your not going to be using emoji all the time.

Enable Emoji at the Command Line

Open Terminal app and use nano to modify the .bash_profile file:

nano .bash_profile

Tech Note: This will not work using BBEdit. This functionality appears to be an exclusive functionality of the Terminal app.

Add a new line like the following:

PS1=" "

PS1 is the Bash variable for Prompt.

Now pull down the "Edit" menu and choose "Special Characters", then select "Emoji" from the special character menu

Find the Emoji you want to use in the shell prompt, I would suggest an Apple or the House. Then drag & drop it into the PS1=" " line so that it's contained within the quotes.

Save the .bash_profile change with Control+O then exit out of nano with Control+X

Open a new Terminal window to see the emoji as the prompt

What is Nano?

nano is a small, free and friendly editor which aims to replace Pico, the default editor included in the non-free Pine package. Rather than just copying Pico's look and feel, nano also implements some missing (or disabled by default) features in Pico, such as "search and replace" and "go to line and column number".

Bash Generator

If you want more out of your Bash Prompt, check out the Bash Generator where you can create a productive Bash Prompt.

Permalink