Awk is a useful language for many command line tasks. Often awk is used on the command line on its own or with strings piped to it, but it is possible to turn that awk code into an executable script.

Consider the following script. This file contains awk code with a shebang of awk -f. The -f is important here; without it, the script won’t work. The code itself is the same as you would use on the command line.

#!/usr/bin/awk -f

# This code will print out the users who have bash as their default shell.

BEGIN {
    FS=":";
}

{
    if (substr($7, length($7)-3) == "bash")
    {
        print $1, "is using", $7;
    }
}

This script can then be called with an /etc/passwd style file to print the users who have bash as their default shell. For example:

$ ./passwd.awk /etc/passwd
root is using /bin/bash
tim is using /bin/bash




Fetch more depth in a shallow git clone

Posted: 6th May 2015 by Tim in Git
Tags: , , , ,

If you’ve made a shallow clone of a repository but suddenly find yourself needing more history, you can fetch this history using one of these two commands:

All history:
git fetch --unshallow

Only the last 123 updates:
git fetch --depth=123

For the second command, 123 can be replaced with any number. If the depth requested is larger than the history of the repository, all records will be fetched.

Exit command in R

Posted: 26th April 2015 by Tim in R
Tags: , , , , , ,

R is quite different from most other programming languages. One common feature of other languages is the exit routine, or something with a similar name. In R, that function is stop, and takes an error message string as an argument. For example:

stop("I don't work on weekends")

will print:

Error: I don't work on weekends

and exit the program.

There are a number of software packages available for creating chemistry diagrams in LaTeX. One of those packages is ChemFig. ChemFig is a powerful package, with many features which aren’t available in other packages. The syntax is fairly easy to follow once you understand the structure, but going in blind can be a bit daunting. This post will describe the basic syntax.

For all examples below, \usepackage{chemfig} needs to be in the preamble. No other configuration needs to be set, and no other packages are required.

Let’s take a basic example, the water molecule. This can be drawn in a number of ways:

\chemfig{H-O-H} will create an image that looks much like the ASCII code:

ChemFig H2O molecule inline

This is useful for simple molecules, but it only shows the covalent bonds. If we also want to represent the angles between the bonds, we can do this in one of three ways:

  1. Add an angle step; a number between 0 and 7, where 0 means an upward (north) bond, 2 means a bond to the right (east), and so on. Each step adds 45 degrees.

    \chemfig{H-[1]O-[7]H}

    ChemFig H2O molecule with angle step
  2. Set the angle explicitly. This angle is added to the horizontal (east) in an anti-clockwise direction, and can be negative. For example, an upward bond would have an angle of 90.

    \chemfig{H-[:37.75]O-[:-37.75]H}

    ChemFig H2O molecule with angle set at 104.5 degrees
    (angle here is 104.5 degrees)
  3. Set the angle relative to the previous angle (or 0 if no branches have been added).

    \chemfig{H-[::37.75]O-[::-75.5]H}

    ChemFig H2O molecule with angle set at 104.5 degrees
    (angle here is also 104.5 degrees)

Some molecules contain a double covalent bond. These bonds can be represented using an equals sign:

\chemfig{O=O}
ChemFig oxygen molecule

You may have noticed that each branch is taken from the last atom/sub-molecule listed, but sometimes that’s not what you want. You can use brackets to indicate that no connection should be made from a branch. For example, methane can be drawn like so:

\chemfig{C(-[0]H)(-[2]H)(-[4]H)(-[6]H)}

ChemFig methane molecule

With these features, you should be able to create most simple chemistry diagrams in LaTeX, such as D-glucose:

\chemfig{
    H-[7]C(=[1]O)
    -[6]
    C(-[4]H)(-OH)
    -[6]
    C(-[4]OH)(-H)
    -[6]
    C(-[4]H)(-OH)
    -[6]
    C(-[4]H)(-OH)
    -[6]
    CH_2OH
}


ChemFig D-glucose molecule

ChemFig is much more powerful than described here. The docs on the project website are very thorough and clear, should you wish to do more than what is described here.

Removing Git Tags

Posted: 26th March 2015 by Tim in Git
Tags: , , , , , , , , , , , ,

If you’ve accidentally added a git tag, or if you want to remove old tags, these tags can be removed with the following three commands:

git tag -d <my_tag>
git push origin :<my_tag>
git push --tags

For example, if you want to remove the tag bad_tag, you would use these commands:

git tag -d bad_tag
git push origin :bad_tag
git push --tags

Linux Terminal Clock

Posted: 9th March 2015 by Tim in Bash, Linux
Tags: , , , , , , , , ,

Ever wanted to have a real-time clock on your linux terminal? We can create one with a single line of bash, like so:

while echo -en "$(date)\r"; do sleep 1; done

Let’s look at how this works:

  • date is a common unix tool used to print the current date and time. The $(...) means that the date command is run, and the output is placed here instead. For example, if logged in as root, echo "I am $(whoami)!" is the same as echo "I am root!".
  • echo -en will print the output of $(date). -e allows escaped characters (the \r in this case), and -n means echo will not add a newline character to the end of the line.
  • \r is a carriage return character. This returns the cursor to the beginning of the current line, which means the next thing to print will overwrite anything on that line.
  • while <condition>; do <commands>; done is a standard while loop; <commands> will be run while <condition> returns true (ie: execution is successful).
  • sleep 1 will pause program execution for 1 second.

Altogether, it means we will print the date every second, overwriting the previously printed date, until we end the program (ctrl-c) or until the echo command fails (which is unlikely to happen).

With Docker, you can specify the command to run inside the container on the command line. But what if you want to run multiple commands? You can’t escape the && syntax, or wrap the command in quotes, as Docker won’t recognise it. The trick here is to use

sh -c '<command1 && command2 [&& command3 [...]]>'

For example, to run date and whoami in a vanilla ubuntu container, we would run the following:

$ docker run ubuntu:latest sh -c 'date && whoami'
Wed Feb 18 00:42:53 UTC 2015
root

This works because sh -c '<date && whoami>' is one call; the commands will be split and executed from within sh.

Listing directories only in linux

Posted: 9th February 2015 by Tim in Linux
Tags: , , , , , ,

The ls command will list files and directories in the current directory. But what if you only want to list the directories? There are a few ways to do this. One of the easiest ways is this:

ls -d */

The -d flag instructs ls to display directories instead of displaying directory contents. The */ instructs ls to perform the listing on subdirectories only. The output will look something like this:

$ ls -d */
hello_world/
test_project/

Test if a directory is empty in Bash

Posted: 29th January 2015 by Tim in Bash, Linux
Tags: , , , , , ,

Using Bash, there are a number of ways to test if a directory is empty. One of those ways is to use ls -A to list all files, including those starting with . and see if anything is printed. This can be done like so:

if [ ! "$(ls -A <path>)" ]
then
    echo "<path> is empty!"
else
    echo "<path> is not empty"
fi

Or inline:

if [ ! "$(ls -A <path> ]; then echo "empty!"; fi

Or, for a slightly less readable version:

test "$(ls -A <path>)" || echo "empty!"

This works because test "" is true, and any other string value is false. If the directory is empty, ls -A will return an empty string.

There are many ways in the linux terminal to print the nth word of a given file or output. One way to do this without worrying about tabs, extra spaces or word length is to use awk. With awk, this can be done on one line by using the {print $<n>} syntax.

For example, the ps command may print this:

$ ps
  PID TTY          TIME CMD
 3677 pts/1    00:00:00 bash
 3699 pts/1    00:00:00 ps
 3700 pts/1    00:00:00 awk

To print only the fourth column (CMD), we can pipe the output to awk like so:

$ ps | awk '{print $4}'
CMD
bash
ps
awk

There are a number of ways to print a number with zero-padding (leading zeros), depending on the variable type you are wanting to print. One easy way to add zero padding to any type is to use the str(<number>).zfill(<length>). This will convert any number type to a string before adding the extra zeros. For example:

def zeroPad(number, length):
    return str(number).zfill(length)

This function can be called using many different types. We can use this function like this:

print zeroPad(123, 5)
print zeroPad(12.3, 5)
print zeroPad('123', 10)

to print this:

00123
012.3
0000000123

Windows 8 comes with a disk partition utility, but it’s difficult to find. There are a few ways to run the utility, the easiest being:

Open the windows run prompt (windows key + r), and run diskmgmt.msc

Alternatively, for the graphically inclined, open the control panel and search for partition

To highlight any results found when searching in Vim, you can use :set hlsearch. To turn off the highlighting, use :nohlsearch. If search highlighting is something you always want turned on, you can add set hlsearch (no colon) to your .vimrc file.

If, like me, you like to turn off the search highlighting after you’ve found what you’re looking for, you can map :nohlsearch to one of the F keys. For example adding the following to your .vimrc:

set hlsearch
noremap <silent> <F8> :nohlsearch<CR>

will highlight search results, and remove that highlighting when you press F8. Highlighting will be turned back on when you start your next search. When adding this to the .vimrc, be sure to hit the F8 key instead of typing in “<F8>”, or the shortcut won’t work.

If you have a HyperSQL (HSQL) database stored in a file, it is often useful to be able to query that database from the command line. This can be done using the following command:

hsqldb-sqltool --inlineRc url=jdbc:hsqldb:file:<db_name>,user=<username>,password=[<password>]

By default, user SA with no password will exist for each database file. If the database does not exist, it will be created for you.

Hide grep command from ps output

Posted: 28th October 2014 by Tim in Linux
Tags: , , , ,

When searching for a running process on a *nix machine, it is common to use ps together with grep, like so:

ps -ef | grep vim

This works well enough for quick searches, but will always return the grep command since grep vim will always contain the string “vim”. The pgrep utility can work around this, but if that’s not available you can use this grep trick:

ps -ef | grep [v]im

Putting square brackets around one letter will still instruct grep to match “vim” in the ps output, but the grep command itself will not match since it does not contain this string exactly.

Disabling auto indent in Vim

Posted: 12th October 2014 by Tim in Vim
Tags: , , , , , , ,

Auto indent is disabled by default in Vim, but some systems have this feature enabled in the system-wide vimrc file (found in /usr/share/vim/vimrc or a similar location). There are two ways to disable this: remove the system-wide setting, or remove the setting for your user only.

To remove the setting system-wide, search for and remove or comment out this line in the system-wide vimrc file (the ‘plugin’ word may or may not appear):

filetype plugin indent on

To remove the setting for your user only, add this line to ~/.vimrc:

filetype indent off

You will need to restart your Vim session to pick up the new settings.

HTTP GET request in Python

Posted: 28th September 2014 by Tim in Python
Tags: , , , , , ,

There are a number of ways to make a GET request in Python, but the easiest (in my opinion) is via urllib2. With this library, you can make a request with only one line of code, storing the result for use later.

For example:

import urllib2
data = urllib2.urlopen("http://timmurphy.org").read()
print "Website size (bytes): " + str(len(data))

Will print:

Website size (bytes): 114209

HTML link coloring with CSS

Posted: 12th September 2014 by Tim in CSS, HTML
Tags: , , , , , , , , ,

HTML link colors can be changed easily using CSS. The properties which can be set are:

  • link – a link to a page which has not been visited
  • visited – a visited link
  • hover – a link which has the mouse hovering over it
  • active – a link which is being clicked (mouse button held down)

Each of these properties can be set like any other CSS tag, so to change the color we can use the color tag.

For example, the following CSS:

#coloredlink a:link { color: blue; }
#coloredlink a:visited { color: red; }
#coloredlink a:hover { color: black; }
#coloredlink a:active { color: green; }

with the following HTML:

<div id="coloredlink">
<a href="http://timmurphy.org/2010/02/28/my-first-latex-document/" target="_blank">LaTeX Tutorial</a>
</div>

will produce this link:

Passing function pointers as a parameter to another function can be tedious work. The function pointer definitions can be long and cumbersome to write, and obscure to read. Using pointers to member functions can be even more ambiguous. Fortunately, we can leverage the power of templates to make this work easier for us by making the compiler figure out the function pointer type for us. If we take this a step further, we can pass in both the object and it’s method as templated parameters, effectively allowing us to use the same code for multiple objects and multiple methods.

Consider the following code:

#include <iostream>

struct MyFuncs
{
    int smallInt() { return 1; }
    unsigned long bigULong() { return 1000; }
};

class OtherFuncs
{
public:
    unsigned long aNumber() { return 555; }
};

template <typename OBJECT, typename FUNC>
long long getNum(OBJECT obj, FUNC getNumber)
{
    return (obj.*getNumber)();
}

int main()
{
    MyFuncs funcs;
    std::cout << "small: " << getNum(funcs, &MyFuncs::smallInt) << std::endl;
    std::cout << "big: " << getNum(funcs, &MyFuncs::bigULong) << std::endl;

    OtherFuncs other;
    std::cout << "other: " << getNum(other, &OtherFuncs::aNumber) << std::endl;

    return 0;
}

The code above prints the following output:

small: 1
big: 1000
other: 555

This program creates a getNum(...) function which can take any object and that object’s (public) method which takes no arguments and returns a value which can be cast to a long long. If written without templates, we would need to write separate getNum(...) methods for each of the calls in main. Templates allow us to avoid such code duplication while keeping the code manageable.

Creating bar charts with gnuplot

Posted: 11th August 2014 by Tim in Gnuplot
Tags: , , , , ,

Bar charts are very easy to create with gnuplot. Very little setup is required; just a data file with labels in one column and data in another. From here, the graph can be drawn with the following line:

plot <data_file> using <label_column>:xtic(<value_column>) with boxes

For example, the following two files:

barchart.gnuplot

set terminal pngcairo font "arial,10" size 500,500
set output 'barchart.png'
set boxwidth 0.75
set style fill solid
set title "Population of Australian cities (millions), as of June 2012"
plot "population.dat" using 2:xtic(1) with boxes

population.dat

Adelaide    1.277174
Brisbane    2.189878
Canberra    0.374658
Darwin      0.131678
Hobart      0.216959
Melbourne   4.246345
Sydney      4.667283

Will create this graph:

gnuplot bar chart

If you have a JAR file and want to print the details from MANIFEST.MF, this can be done with one command in linux, using the unzip utility. For example:

$ unzip -p /usr/share/java/hsqldb.jar META-INF/MANIFEST.MF
Manifest-Version: 1.0
Created-By: 1.7.0_03-b147 (Oracle Corporation)
Specification-Title: HSQLDB
Implementation-Title: Standard runtime
Class-Path: /usr/share/java/servlet-api-3.0.jar
Main-Class: org.hsqldb.util.SqlTool
Ant-Version: Apache Ant 1.8.2
Implementation-Vendor: buildd
Implementation-Version: private-2012/07/12-02:29:31
Specification-Version: 1.8.0.10
Specification-Vendor: The HSQLDB Development Group

PHP is able to communicate with PostgreSQL databases using some relatively simple calls. In a similar manner to other database systems, the script needs to do the following:

  1. connect to the database using pg_connect
  2. execute queries using pg_query and pg_free_result
  3. close the database connection using pg_close

For example, consider the following script:

<?PHP
// database connection
$dbhost = "localhost";
$dbname = "everyone";
$dbuser = "phptest";
$dbpass = "testpassword";
$db = pg_connect("host=$dbhost dbname=$dbname user=$dbuser password=$dbpass")
    or die("Could not connect to database $dbname on host $dbhost!");

// execute the SQL query
$query = "SELECT lanname, lanpltrusted FROM pg_language;";
$result = pg_query($query)
    or die ("Query failed: " . pg_last_error());

// print the results
echo "<table style=\"width: 500px; border: 1px black solid;\">\n";
echo "\t<tr>\n";
echo "\t\t<th>Language</th>\n";
echo "\t\t<th>Trusted</th>\n";
echo "\t</tr>\n";

while ($row = pg_fetch_array($result, NULL, PGSQL_ASSOC))
{
    echo "\t<tr>\n";
    echo "\t\t<td>${row['lanname']}</td>\n";
    echo "\t\t<td>${row['lanpltrusted']}</td>\n";
    echo "\t</tr>\n";
}

echo "</table>\n";

// clean up
pg_free_result($result);
pg_close($db);
?>

This script fetches all languages supported by this PostgreSQL installation, and notes whether it is ‘trusted’ (ie: whether non-superusers can create scripts using that language). The script above will generate the following HTML table:

Language Trusted
internal f
c f
sql t
plpgsql t

Note that this code does not only work for web environments; it can be used for standalone PHP scripts too.

LaTeX style (.sty) files

Posted: 27th June 2014 by Tim in LaTeX
Tags: , , , , , ,

When writing LaTeX documents, you may find yourself copying and pasting some common settings such as margins, fonts and paragraph indentation. This is not only tedious, it can be a real headache if you’re writing multiple documents that you want to look the same. To solve this problem, you can use a style (.sty) file.

A style file uses the same syntax as a LaTeX file, but uses the .sty suffix. To use this file in your LaTeX document, load it using the \usepackage{<filename_without_sty_suffix>} syntax.

Consider the following two files:

timstyle.sty

% timstyle.sty
% This file contains common document settings

% Page margins (2cm wider, 2cm longer)
\addtolength{\textwidth}{2cm}
\addtolength{\hoffset}{-1cm}
\addtolength{\textheight}{2cm}
\addtolength{\voffset}{-1cm}

% Font (Times New Roman)
\usepackage{times}

% No paragraph indentation
\setlength{\parindent}{0in}

example.tex

\documentclass[11pt, a4paper]{article}
\usepackage{timstyle} % note: no .sty suffix here
\begin{document}
Hello World! This is the first paragraph in the document.
The paragraph is not very long, but it spans multiple lines.
As you can see, the first line of the paragraph is not indented.\\

This is the second paragraph. Look --- still no indentation!
\end{document}

The code above will produce this document. As you can see, the margin, font and paragraph indentation settings are in timstyle.sty, which simplifies the example.tex file. The style file can now be reused in other files too.

Command Line Arguments in Bash

Posted: 14th June 2014 by Tim in Bash, Linux
Tags: , , , , ,

In Bash, arguments passed in on the command line are stored in numbered variables. For example, the first argument is $1, the second argument is $2, and so on. The total number of arguments passed to the program is stored in $#

$0 contains the path to the program. This path may be an absolute path or a relative path, depending on how you called the script. $@ and $* will return all of the arguments passed to the program.

For example:

#/bin/bash
echo "Execution command: '$0 $@' ($# args)"
echo "First 3 arguments:"
if [ $# -ge 1 ]
then
    echo "  \$1 = $1"
fi
if [ $# -ge 2 ]
then
    echo "  \$2 = $2"
fi
if [ $# -ge 3 ]
then
    echo "  \$3 = $3"
fi

This script will print this if called using a relative path:
Execution command: './command_line_args.sh one two three' (3 args)
First 3 arguments:
  $1 = one
  $2 = two
  $3 = three

Or, if called using an absolute path:
Execution command: '/tmp/command_line_args.sh ichi ni san' (3 args)
First 3 arguments:
  $1 = ichi
  $2 = ni
  $3 = san

Using tools such as ps or top, you are able to see the processes running on a machine. However, you can’t see the directory from which the process was started. Knowing the working directory can be useful if, for example, you need to move a script or program to stop a fork bomb, if you want to see where a script or program lives or, if a script or program reads files using a relative path, to see which files are being read.

This working directory can be found using the pwdx <pid> [<pid> ...] utility. For example, consider the following output from ps:

UID        PID  PPID  C STIME TTY          TIME CMD
everyone  2646     1  2 21:21 ?        00:00:00 terminal
everyone  2651  2646  0 21:21 pts/0    00:00:00 bash

We can see that terminal and bash are running, but we don’t know where these processes were started. Using pwdx, we can easily find out:

$ pwdx 2646 2651
2646: /home/everyone
2651: /home/everyone