## Merging two files in linux using paste

Posted: 9th November 2015 by Tim in Bash, Linux, paste
Tags: , , , , , , , , , ,

Imagine you have two text files, one with a list of names and another with a list of birth dates, which each name corresponding to the date of birth on the same line number in the other file, like so:

 names.txt Anthony Kiedis Flea Chad Smith John Frusciante dob.txt 1-Nov-1962 16-Oct-1962 25-Oct-1961 5-Mar-1970

You would like to join these files together into one file with the format <name>:<date_of_birth>. In linux, this can be done with paste, like so:

paste -d':' names.txt dob.txt

Here, -d is the delimiter option. This specifies which character separates data from file 1 and file 2. Running the command above produces this output:

Anthony Kiedis:1-Nov-1962 Flea:16-Oct-1962 Chad Smith:25-Oct-1961 John Frusciante:5-Mar-1970

## Rotating axis labels in Gnuplot

Posted: 27th October 2015 by Tim in Gnuplot
Tags: , , , , , , , ,

In gnuplot, labels on the X and Y axes are aligned horizontally by default, and written over the top of each other if the labels are too long. To fix this, you can rotate the labels using the set xtics rotate [by <angle>] [left|right] command. For example, you can rotate x-axis labels by 60 degrees with the following code:

set xtics rotate by 60 right

We use right alignment here, otherwise the labels run from the base of the graph upwards.

The following full gnuplot code:

set terminal svg set format x '%+-.6f' # to make the labels longer set xtics rotate by 60 right plot cos(x)

will generate this graph:

## Bash gotcha: function variable scope

Posted: 11th October 2015 by Tim in Bash, Linux
Tags: , , , , , , ,

Consider the following bash script:

#!/usr/bin/env bash

function myFunc {     myvar=123     echo "myFunc: setting myvar=$myvar" } myvar=1 echo "before myFunc: myvar=$myvar" myFunc echo "after myFunc: myvar=$myvar" The code here is fairly simple – we set a variable myvar, call a function and print the value of myvar to the terminal. However, even though we don’t pass myvar to the function, that value is still changed: before myFunc: myvar=1 myFunc: setting myvar=123 after myFunc: myvar=123 Somehow, myFunc changed the value of myvar. This is because, unless specified otherwise, this variable is in the global scope. This can sometimes be useful, but it can also be confusing for programmers used to the scoping rules of other languages. To keep myvar contained to function scope only, we can use the local keyword like so: function myFunc { local myvar=123 echo "myFunc: setting myvar=$myvar" }

With this specified, the code behaves more like we would expect:

before myFunc: myvar=1 myFunc: setting myvar=123 after myFunc: myvar=1

## How to get a Makefile directory path

Posted: 27th September 2015 by Tim in Linux, Make
Tags: , , , , , ,

Sometimes in a Makefile you need to reference something from the directory where the Makefile lives. $(PWD) won’t work in this instance, since that will return the path from where make was called which may be different if, for example, you use the --directory option. In this case, we can use the following code to find Makefile’s directory: $(dir $(realpath$(firstword $(MAKEFILE_LIST)))) Let’s pull this apart and look at each component. For this example, our Makefile path is /tmp/path/to/my/Makefile and contains the following: include other/Makefile default: @echo$(PWD)
@echo $(dir$(realpath $(firstword$(MAKEFILE_LIST))))

First, MAKEFILE_LIST will contain all of the Makefiles which have been loaded. If you havent’t included any others, only one will be listed here:
$(MAKEFILE_LIST) = Makefile other/Makefile Since we’re only interested in the current Makefile, we strip off the includes: $(firstword $(MAKEFILE_LIST)) = Makefile Now that we have one file, find the absolute path: $(realpath $(firstword$(MAKEFILE_LIST))) = /tmp/path/to/my/Makefile

Finally, strip off the file name, leaving only the directory path:
$(dir$(realpath $(firstword$(MAKEFILE_LIST)))) = /tmp/path/to/my/

So, if we call make --directory=/tmp/path/to/my default from the /tmp directory, the following will be printed:

/tmp /tmp/path/to/my/

If you’re interested in the path of the last included Makefile, use lastword instead of firstword. For example, you may use lastword from other/Makefile to find that directory.

## Drawing power crosses in LaTeX

Posted: 10th September 2015 by Tim in LaTeX, Optics
Tags: , , , , , , , ,

Power crosses are useful diagrams for doing lens power calculations in optics. Drawing the power crosses in LaTeX can be tedious, especially if you need to draw multiple power crosses in the same document. The code below creates a new command, \powercross{front@90}{front@180}{back@90}{back@180}, which will draw the power crosses for you and calculate the final lens power based on the powers of the front and back lens surfaces. All values will be rounded to two decimal places.

Note: you need have \usepackage{fp} in the preamble in order for this code to work.

% usage: \powercross{front@90}{front@180}{back@90}{back@180} \newcommand{\powercross}[4]{     \setlength{\unitlength}{1mm}     \vspace{1em}     \begin{picture}(135,35)         % crosses         \multiput(0,17.5)(50,0){3}{\line(1,0){35}}         \multiput(17.5,0)(50,0){3}{\line(0,1){35}}

        % arithmetic symbols         \put(41.5,16.5){\makebox{$+$}}         \put(91.5,16.5){\makebox{$=$}}

        % calculate resulting lens (requires fp)         \FPeval{\frontninety}{round(#1:2)}         \FPeval{\frontzero}{round(#2:2)}         \FPeval{\backninety}{round(#3:2)}         \FPeval{\backzero}{round(#4:2)}         \FPeval{\lensninety}{round(\frontninety+\backninety:2)}         \FPeval{\lenszero}{round(\frontzero+\backzero:2)}

        % labels         \put(18.5,33){\makebox{{\frontninety}D}}         \put(30.5,19){\makebox{{\frontzero}D}}         \put(68.5,33){\makebox{{\backninety}D}}         \put(80.5,19){\makebox{{\backzero}D}}         \put(118.5,33){\makebox{{\lensninety}D}}         \put(130.5,19){\makebox{{\lenszero}D}}     \end{picture}     \newline }

For example, this code:

\powercross{3.25}{3.25}{-10.25}{-7}

will produce the following:

## Converting a string to lowercase in linux with tr

Posted: 26th August 2015 by Tim in Linux, tr
Tags: , , , , , ,

If you’re using a linux terminal and need to convert all upper case characters to lowercase, there are a number of ways you can do it. One of the easiest is to use tr:

tr '[:upper:]' '[:lower:]'

For example:

$echo "HeLlo WOrlD" | tr '[:upper:]' '[:lower:]' hello world :upper: and :lower: can be reversed if you want to create an upper case string instead: $ echo "HeLlo WOrlD" | tr '[:lower:]' '[:upper:]' HELLO WORLD

## Accessing fields by index in Awk

Posted: 11th August 2015 by Tim in Awk
Tags: , , , , ,

In awk, fields are accessed by number; $1 for the first field, $2 for the second, etc. But sometimes the field number you want to access is not known until run time. In these cases, you can access the field using the $(<index>) syntax. The constant NF contains the number of fields available. For example, you can print all fields with their field index like so: echo "Hello, World" | awk '{ for (i = 1; i <= NF; ++i) { print i,$(i); } }'

The code above will print:

1 Hello, 2 World

## Reading a file line by line in Bash

Posted: 26th July 2015 by Tim in Bash, Linux
Tags: , , , , , , ,

There are a few ways to read a file in bash, each with their own caveats. If you’re looking to read a file line-by-line verbatim, including blank lines, then using a simple while loop should do the trick. For example, the following code will print the contents of a file with line numbers:

 line_no=0 while IFS='' read -r line do     ((++line_no))     echo "$line_no:$line" done < /etc/kderc

This will print something like:

1: [Directories] 2: kioskAdmin=root: 3: profileDirsPrefix=/usr/share/kde-settings/kde-profile/ 4: userProfileMapFile=/etc/kde-user-profile 5: 6: [Directories-default] 7: prefixes=/usr/share/kde-settings/kde-profile/default/

## Creating temporary files in Java

Posted: 9th July 2015 by Tim in Java
Tags: , , , , ,

Sometimes it is necessary to use temporary files in your program. One concern when creating these files is thread safety; two processes creating a temporary file with the same name. This concern can be largely ignored using Java’s File.createTempFile, which guarantees the file path is unique amongst all java processes running on that JVM.

For example:

import java.io.File; import java.io.IOException;

class TempFile {     public static void main(String[] args) throws IOException {         File tempFile = File.createTempFile("prefix_", "_suffix");         System.out.println(tempFile);     } }

will print something like:
/var/folders/5s/wh643h2j3fqf4d7_zy_hl4fc0000gn/T/prefix_1987572090443392628_suffix

## Pause in Bash

Posted: 27th June 2015 by Tim in Bash, Linux
Tags: , , , , , , ,

Sometimes it is useful to pause a script until the user is ready to proceed. In Windows Batch scripting, this can be done with the PAUSE command. In Bash, the same thing can be done using read with the right parameters:

read -rn1 -p "Press any key to continue"

This command is doing the following:

• -r disables backslash escaping. Without this option, typing only \ will not continue execution.
• -n1 instructs the command to read only one character.
• -p "Press any key..." is the prompt, or message printed to the screen.

Note that, like Batch’s PAUSE command, some keys which don’t result in a character being typed, like ALT, CTRL, SHIFT and CAPS LOCK, will not continue execution. Most other keys will work though.

## Small Angle Approximation and Skinny Triangles

Posted: 13th June 2015 by Tim in Mathematics
Tags: , , , ,

Small Angle Approximations and Skinny Triangles are concepts which simplify trigonometric calculations. The concept is based on right-angle or isosceles triangles where one side is very small compared to the other two. In these cases, some approximations can be made in place of trigonometric calculations, which may be used for estimations when information required to compute an exact value is missing, or where an estimated value is good enough.

Consider the following triangle:

On this right-angle triangle, the blue arc a represents the the edge of a circle if the shape was a slice of a full circle (ie: the shape bounded by the green, blue and bottom black lines would be a portion of a circle). From this, we see that r represents the radius of the circle, y is the triangle height and h is the hypotenuse. Θ is the angle from the middle of the circle, which we represent in radians for this approximation.

When Θ = π/4 (0.785 radians / 45°) as shown above, we can’t see any obvious relationship between these values:
when r = 100:
h = 141.4
y = 100
a = 78.5
sinΘ = 0.707
tanΘ = 1
cosΘ = 0.707
1 – (Θ²/2) = 0.692

But watch what happens as Θ decreases (all triangles drawn to scale).

when r = 100:
h = 111.8
y = 50
a = 46.4
sinΘ = 0.448
tanΘ = 0.500
cosΘ = 0.894
1 – (Θ²/2) = 0.892

when r = 100:
h = 103.1
y = 25
a = 24.4
sinΘ = 0.242
tanΘ = 0.249
cosΘ = 0.970
1 – (Θ²/2) = 0.970

As you can see, for small values of Θ:

• rh
• ya
• sinΘtanΘΘ
• cosΘ ≈ 1 – (Θ²/2)

As Θ approaches zero, the errors in these approximations also approach zero. As you can see, Θ doesn’t need to be that small for these approximations to be accurate enough for quick calculations. These approximations are used in a number of fields, as diverse as astronomy, optics and aircraft navigation to name a few.

## Executable awk scripts in Linux

Posted: 26th May 2015 by Tim in Awk, Linux
Tags: , , , , , , ,

Awk is a useful language for many command line tasks. Often awk is used on the command line on its own or with strings piped to it, but it is possible to turn that awk code into an executable script.

Consider the following script. This file contains awk code with a shebang of awk -f. The -f is important here; without it, the script won’t work. The code itself is the same as you would use on the command line.

#!/usr/bin/awk -f

# This code will print out the users who have bash as their default shell.

BEGIN {     FS=":"; }

{     if (substr($7, length($7)-3) == "bash")     {         print $1, "is using",$7;     } }

This script can then be called with an /etc/passwd style file to print the users who have bash as their default shell. For example:

$./passwd.awk /etc/passwd root is using /bin/bash tim is using /bin/bash ## Fetch more depth in a shallow git clone Posted: 6th May 2015 by Tim in Git Tags: , , , , If you’ve made a shallow clone of a repository but suddenly find yourself needing more history, you can fetch this history using one of these two commands: All history: git fetch --unshallow Only the last 123 updates: git fetch --depth=123 For the second command, 123 can be replaced with any number. If the depth requested is larger than the history of the repository, all records will be fetched. ## Exit command in R Posted: 26th April 2015 by Tim in R Tags: , , , , , , R is quite different from most other programming languages. One common feature of other languages is the exit routine, or something with a similar name. In R, that function is stop, and takes an error message string as an argument. For example: stop("I don't work on weekends") will print: Error: I don't work on weekends and exit the program. ## Drawing chemistry diagrams in LaTeX using ChemFig Posted: 10th April 2015 by Tim in Chemistry Tags: , , , , , , , , , , , , , , There are a number of software packages available for creating chemistry diagrams in LaTeX. One of those packages is ChemFig. ChemFig is a powerful package, with many features which aren’t available in other packages. The syntax is fairly easy to follow once you understand the structure, but going in blind can be a bit daunting. This post will describe the basic syntax. For all examples below, \usepackage{chemfig} needs to be in the preamble. No other configuration needs to be set, and no other packages are required. Let’s take a basic example, the water molecule. This can be drawn in a number of ways: \chemfig{H-O-H} will create an image that looks much like the ASCII code: This is useful for simple molecules, but it only shows the covalent bonds. If we also want to represent the angles between the bonds, we can do this in one of three ways: 1. Add an angle step; a number between 0 and 7, where 0 means an upward (north) bond, 2 means a bond to the right (east), and so on. Each step adds 45 degrees. \chemfig{H-[1]O-[7]H} 2. Set the angle explicitly. This angle is added to the horizontal (east) in an anti-clockwise direction, and can be negative. For example, an upward bond would have an angle of 90. \chemfig{H-[:37.75]O-[:-37.75]H} (angle here is 104.5 degrees) 3. Set the angle relative to the previous angle (or 0 if no branches have been added). \chemfig{H-[::37.75]O-[::-75.5]H} (angle here is also 104.5 degrees) Some molecules contain a double covalent bond. These bonds can be represented using an equals sign: \chemfig{O=O} You may have noticed that each branch is taken from the last atom/sub-molecule listed, but sometimes that’s not what you want. You can use brackets to indicate that no connection should be made from a branch. For example, methane can be drawn like so: \chemfig{C(-[0]H)(-[2]H)(-[4]H)(-[6]H)} With these features, you should be able to create most simple chemistry diagrams in LaTeX, such as D-glucose: \chemfig{ H-[7]C(=[1]O) -[6] C(-[4]H)(-OH) -[6] C(-[4]OH)(-H) -[6] C(-[4]H)(-OH) -[6] C(-[4]H)(-OH) -[6] CH_2OH } ChemFig is much more powerful than described here. The docs on the project website are very thorough and clear, should you wish to do more than what is described here. ## Removing Git Tags Posted: 26th March 2015 by Tim in Git Tags: , , , , , , , , , , , , If you’ve accidentally added a git tag, or if you want to remove old tags, these tags can be removed with the following three commands: git tag -d <my_tag> git push origin :<my_tag> git push --tags For example, if you want to remove the tag bad_tag, you would use these commands: git tag -d bad_tag git push origin :bad_tag git push --tags ## Linux Terminal Clock Posted: 9th March 2015 by Tim in Bash, Linux Tags: , , , , , , , , , Ever wanted to have a real-time clock on your linux terminal? We can create one with a single line of bash, like so: while echo -en "$(date)\r"; do sleep 1; done

Let’s look at how this works:

• date is a common unix tool used to print the current date and time. The $(...) means that the date command is run, and the output is placed here instead. For example, if logged in as root, echo "I am$(whoami)!" is the same as echo "I am root!".
• echo -en will print the output of $(date). -e allows escaped characters (the \r in this case), and -n means echo will not add a newline character to the end of the line. • \r is a carriage return character. This returns the cursor to the beginning of the current line, which means the next thing to print will overwrite anything on that line. • while <condition>; do <commands>; done is a standard while loop; <commands> will be run while <condition> returns true (ie: execution is successful). • sleep 1 will pause program execution for 1 second. Altogether, it means we will print the date every second, overwriting the previously printed date, until we end the program (ctrl-c) or until the echo command fails (which is unlikely to happen). ## Running multiple programs in a Docker container from the command line Posted: 27th February 2015 by Tim in Docker, Ubuntu Tags: , , , , , , , , With Docker, you can specify the command to run inside the container on the command line. But what if you want to run multiple commands? You can’t escape the && syntax, or wrap the command in quotes, as Docker won’t recognise it. The trick here is to use sh -c '<command1 && command2 [&& command3 [...]]>' For example, to run date and whoami in a vanilla ubuntu container, we would run the following: $ docker run ubuntu:latest sh -c 'date && whoami' Wed Feb 18 00:42:53 UTC 2015 root

This works because sh -c '<date && whoami>' is one call; the commands will be split and executed from within sh.

## Listing directories only in linux

Posted: 9th February 2015 by Tim in Linux
Tags: , , , , , ,

The ls command will list files and directories in the current directory. But what if you only want to list the directories? There are a few ways to do this. One of the easiest ways is this:

ls -d */

The -d flag instructs ls to display directories instead of displaying directory contents. The */ instructs ls to perform the listing on subdirectories only. The output will look something like this:

$ls -d */ hello_world/ test_project/ ## Test if a directory is empty in Bash Posted: 29th January 2015 by Tim in Bash, Linux Tags: , , , , , , Using Bash, there are a number of ways to test if a directory is empty. One of those ways is to use ls -A to list all files, including those starting with . and see if anything is printed. This can be done like so: if [ ! "$(ls -A <path>)" ] then     echo "<path> is empty!" else     echo "<path> is not empty" fi

Or inline:

if [ ! "$(ls -A <path>)" ]; then echo "empty!"; fi Or, for a slightly less readable version: test "$(ls -A <path>)" || echo "empty!"

This works because test "" is true, and any other string value is false. If the directory is empty, ls -A will return an empty string.

## Printing the nth word on each line using awk

Posted: 7th January 2015 by Tim in Awk, Bash, Linux
Tags: , , , , , , , , ,

There are many ways in the linux terminal to print the nth word of a given file or output. One way to do this without worrying about tabs, extra spaces or word length is to use awk. With awk, this can be done on one line by using the {print $<n>} syntax. For example, the ps command may print this: $ ps   PID TTY          TIME CMD  3677 pts/1    00:00:00 bash  3699 pts/1    00:00:00 ps  3700 pts/1    00:00:00 awk

To print only the fourth column (CMD), we can pipe the output to awk like so:

$ps | awk '{print$4}' CMD bash ps awk

## Printing zero-padded numbers in Python

Posted: 28th December 2014 by Tim in Python
Tags: , , , , , ,

There are a number of ways to print a number with zero-padding (leading zeros), depending on the variable type you are wanting to print. One easy way to add zero padding to any type is to use the str(<number>).zfill(<length>). This will convert any number type to a string before adding the extra zeros. For example:

def zeroPad(number, length):     return str(number).zfill(length)

This function can be called using many different types. We can use this function like this:

print zeroPad(123, 5) print zeroPad(12.3, 5) print zeroPad('123', 10)

to print this:

00123 012.3 0000000123

## Managing disk partitions in Windows 8

Posted: 12th December 2014 by Tim in Windows 8
Tags: , , , , , , , ,

Windows 8 comes with a disk partition utility, but it’s difficult to find. There are a few ways to run the utility, the easiest being:

Open the windows run prompt (windows key + r), and run diskmgmt.msc

Alternatively, for the graphically inclined, open the control panel and search for partition

## Highlighting search results in Vim

Posted: 29th November 2014 by Tim in Vim
Tags: , , , , ,

To highlight any results found when searching in Vim, you can use :set hlsearch. To turn off the highlighting, use :nohlsearch. If search highlighting is something you always want turned on, you can add set hlsearch (no colon) to your .vimrc file.

If, like me, you like to turn off the search highlighting after you’ve found what you’re looking for, you can map :nohlsearch to one of the F keys. For example adding the following to your .vimrc:

set hlsearch noremap <silent> <F8> :nohlsearch<CR>

will highlight search results, and remove that highlighting when you press F8. Highlighting will be turned back on when you start your next search. When adding this to the .vimrc, be sure to hit the F8 key instead of typing in “<F8>”, or the shortcut won’t work.

## Querying a HSQL database on the command line with hsqldb-sqltool

Posted: 12th November 2014 by Tim in HyperSQL
Tags: , , , , , ,

If you have a HyperSQL (HSQL) database stored in a file, it is often useful to be able to query that database from the command line. This can be done using the following command:

hsqldb-sqltool --inlineRc url=jdbc:hsqldb:file:<db_name>,user=<username>,password=[<password>]

By default, user SA with no password will exist for each database file. If the database does not exist, it will be created for you.