\chapter{System programming}

\section{The sys module and argv}

The \pythoninline{sys} module contains functions and variables which
provide access to the \myidx{environment} in which the python
interpreter runs. The following example shows the values of a few of
these variables on one of our systems:

\begin{minipage}{\textwidth}
\begin{pythonInteractive}
>>> import sys
>>> sys.platform
'darwin'
>>> sys.path
['/',
'/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python25.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5',
 '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload',
 '/Library/Python/2.5/site-packages',
 '/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/PyObjC'] 
>>> sys.version
'2.5.1 (r251:54863, Jan 13 2009, 10:26:13) \n[GCC 4.0.1 (Apple Inc. build 5465)]'
\end{pythonInteractive}
\end{minipage}

Starting \myidx{Jython} on vex produces different values for the same
variables:

\begin{minipage}{\textwidth}
\begin{pythonInteractive}
>>> import sys
>>> sys.platform
'java1.6.0_04'
>>> sys.path
['', '.', '/usr/share/jython/Lib', '__classpath__']
>>> sys.version
'2.2b1'
\end{pythonInteractive}
\end{minipage}
The results will be different on your machine of course.

The \pythoninline{argv} variable holds a list of strings read in from the
\myidx{command line} when a Python script is run. These
\myidx{command line arguments} can be used to pass information into
a program at the same time it is invoked.

\begin{minipage}{\textwidth}
\begin{python}
#
# demo_argv.py
#
import sys

print sys.argv
\end{python}
\end{minipage}
Running this program from the unix command prompt demonstrates how
\pythoninline{sys.argv} works:
\begin{shell}
$ python demo_argv.py this and that 1 2 3
['demo_argv.py', 'this', 'and', 'that', '1', '2', '3']
$ 
\end{shell}
\pythoninline{argv} is a list of strings. Notice that the first
element is the name 
of the program. Arguments are separated by white space, and separated
into a list in the same way that \pythoninline{string.split} operates.
If you want an argument with white space in it, use quotes: 
\begin{shell}
$ python demo_argv.py "this and" that "1 2" 3
['demo_argv.py', 'this and', 'that', '1 2', '3']
$ 
\end{shell}
With \pythoninline{argv} we can write useful programs that take their
input directly from the command line. For example, here is a program
that finds the sum of a series of numbers: 

\begin{minipage}{\textwidth}
\begin{python}
#
# sum.py
#
from sys import argv

nums = argv[1:]

for index, value in enumerate(nums):
    nums[index] = float(value)

print sum(nums)
\end{python}
\end{minipage}
In this program we use the
\pythoninline{from~$<$module$>$~import~$<$attribute$>$} 
style of importing, so \pythoninline{argv} is brought into the
module's \pythoninline{main} namespace.
We can now run the program from the command prompt like this:
\begin{shell}
$ python sum.py 3 4 5 11
23
$ python sum.py 3.5 5 11 100
119.5
\end{shell}
You are asked to write similar programs as exercises.

\section{The \pythoninline{os} and \pythoninline{glob} module}
The \pythoninline{os} module contains functions for interacting with
the underlying operating system. It includes many standard operating
system functions including, but not limited to:
\begin{itemize}
\item directory listings (\pythoninline{os.listdir})
\item moving to a different directory (\pythoninline{os.chdir})
\item executing other processes (\pythoninline{os.execv} and
\pythoninline{os.system})
\item moving files (\pythoninline{os.rename})
\item deleting files (\pythoninline{os.remove})
\item getting information about a file (\pythoninline{os.stat})
\end{itemize}

Let's have a look at a simple example:

\begin{minipage}{\textwidth}
\begin{pythonInteractive}
>>> import os
>>> photos =
os.listdir('/home/cshome/coursework/COMP150/coursefiles150/Photos/')
>>> photos
['112_1286.jpg', '112_1287.jpg', '112_1284.jpg', '112_1285.jpg',
'112_1282.jpg', '112_1283.jpg', '112_1280.jpg', '112_1281.jpg',
'112_1288.jpg', '112_1276.jpg', '112_1277.jpg', '112_1274.jpg',
'112_1275.jpg', '112_1272.jpg', '112_1273.jpg', '112_1271.jpg',
'112_1278.jpg', '112_1279.jpg']
>>> os.stat('/home/cshome/coursework/COMP150/coursefiles150/Photos/' + photos[0])
(33216, 597894501L, 436207622L, 1, 15391, 1026, 1089680L, 1239768034,
1239768034, 1239768034)
\end{pythonInteractive}
\end{minipage}

The \pythoninline{os.stat} command returns several pieces of
information about the file in question, including its size
(\pythoninline{1089680L}). 

The \pythoninline{\myidx{glob}} module is another useful module that contains
several functions, but the most important one is the
\pythoninline{glob} function. Globbing is a term for converting a
generic name which includes wildcard characters, into specific names
of files or directories that match the generic name. Here are some
simple examples:

\brenNote{Need to change this}

\begin{minipage}{\textwidth}
\begin{pythonInteractive}
>>> import glob
>>> glob.glob("/home/cshome/coursework/COMP150/coursefiles150/Photos/*.jpg")
['/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1272.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1286.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1280.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1274.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1275.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1281.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1287.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1288.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1273.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1285.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1271.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1277.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1278.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1283.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1282.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1276.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1279.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1284.jpg']
>>> glob.glob("/home/cshome/coursework/COMP150/coursefiles150/Photos/*2.jpg")
['/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1272.jpg',
'/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1282.jpg'] 
\end{pythonInteractive}
\end{minipage}

The combination of a little bit of programming knowledge, plus the
\pythoninline{os} and \pythoninline{glob} modules really allows us to
remove the drudgery from a lot of mundane and laborious computer
tasks.

\section{A mini-case study}

In this mini-case study, we are going to look at using
\pythoninline{os} and \pythoninline{glob} and the external ImageMagick
program called \pythoninline{convert} to do some useful work. Our
original photos are rather large, and I'd like to convert them to
smaller versions and thumbnails so I can make a visual directory to
put on the web. First, let's have a look at some of the capabilities
of ImageMagick.

\subsection{ImageMagick}

ImageMagick is a suite of stand-alone programs that are often run from
the command line which makes them ideal for scripting or batch
processing. The most useful of the programs is the utility called
\pythoninline{convert}. If you open a terminal
shell and type \pythoninline{convert} at the prompt, you will be
greeted with a dizzying array of options (not reproduced here for
space reasons). There are lots of things you can do with images using
\pythoninline{convert}. Let's try out a few simple ones.

To convert one of our pictures to a charcoal drawing, try the
following command:
\begin{shell}
$ convert \\
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1286.jpg \\
-charcoal 5 charcoal.jpg
\end{shell}
In a finder window, double-click on \pythoninline{charcoal.jpg} to see
the result. The result should look similar to Figure
\ref{fig-charcoal}.
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{Photos/charcoal.jpg}
\caption{A charcoaled image \label{fig-charcoal}}
\end{figure}

How about adding a little swirl to a picture:
\begin{shell}
$ convert \\
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1271.jpg \\
-swirl 90 swirl.jpg
\end{shell}
The result should look similar to Figure \ref{fig-swirl}. Too much to
drink perhaps? A bad trip maybe?
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{Photos/swirl.jpg}
\caption{A swirled image \label{fig-swirl}}
\end{figure}

Try out a few other operations if you feel so inclined. However,
enough frivolity, on to the real work.

\subsection{Scripting ImageMagick}
First, create a file called \pythoninline{magick.py}.
\brenNote{replace the directory with one which students can access and
change the written file to a directory they can write to (the last is
trickier because we need to do some string processing)} Let's start
with listing all the photos in the shared Photos folder. Add the
following code to \pythoninline{magick.py} and execute it.

\begin{minipage}{\textwidth}
\begin{python}
import os, glob

photos = glob.glob("/home/cshome/coursework/COMP150/coursefiles150/Photos/*.jpg")

for p in photos:
  print(p)

print("Done")
\end{python}
\end{minipage}

The output should look something like the following:

\begin{minipage}{\textwidth}
\begin{pythonOutput}
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1272.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1286.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1280.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1274.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1275.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1281.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1287.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1288.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1273.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1285.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1271.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1277.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1278.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1283.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1282.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1276.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1279.jpg
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1284.jpg
Done
\end{pythonOutput}
\end{minipage}

That's a good start because now we know we can specify the input
filenames. What about the output files? Well, we can't just use the
input filenames as the output filenames for two reasons: we don't want
to overwrite the original files; and you don't have write access to the
above directory/folder. So, we need to do some processing on the
filenames. To do this, the \pythoninline{re} (for \myidx{regular
expressions})
package is useful. We could spend several lectures on regular
expressions, but we won't. We'll try to give you a little taste of
them here --- if you don't fully understand what's going on, don't
panic, the most important thing is that you recognise that regular
expressions are an incredibly powerful way of manipulating
strings. 

Regular expressions are a bit like globbing, except the rules
are a bit different. Have a close look at the format of the filenames
listed above. What we would like to do is maintain the filename part
of the output file so it's easy to associate the input and output, but
save the file in the current directory. To do that, we need to chop
out all the leading directory names and structure. To do that, we use
the \pythoninline{re.sub} command. Here's how:

\begin{minipage}{\textwidth}
\begin{python}
import os, glob, re

photos = glob.glob("/home/cshome/coursework/COMP150/coursefiles150/Photos/*.jpg")

for p in photos:
    newp = re.sub('.*/([a-z_0-9]+)\.jpg', r'\1', p)
    print("input name: " + p + ", output name: " + newp)

print("Done")
\end{python}
\end{minipage}

The \pythoninline{re.sub} command (sub is short for substitute) is
performing a bit of regular 
expression magic. Let's have a quick look at what all those parts are
doing. The first parameter is a regular expression (RE), the second a
replacement string, and the third, the string to do the matching and
replacing on. The RE in this case is
\verb|.*/([a-z_0-9]+)\.jpg|. Let's look at different parts of
the RE:
\begin{itemize}
\item {\bf \verb|jpg|:} Characters that occur unadorned are simply matched
directly to the input string. So in this case, the characters
\verb|jpg| will only match to the end of the input strings
(the file type).
\item {\bf \verb|\.|:} A \verb|.| is a special character
(described below). When we actually want to match with a
\verb|.| in the input string, we need to precede it with a
\verb|\|. So the final part of the RE, \verb|\.jpg|,
matches the end of the filenames.
\item {\bf \verb|[a-z_0-9]|:} Square braces indicate that the RE
should match any of the characters inside the braces. Rather than
typing out all the characters in the alphabet and all the numerals, we
can abbreviate those with the range operator \verb|-|. So
\verb|a-z| matches all lowercase characters,
\verb|0-9| matches all numerals, and \verb|_| matches
the underscore character. The sub-expression \verb|[a-z_0-9]|
therefore matches any single lower case character, underscore or
numeral. If we add a \verb|+| to the end, then we can match
any sequence of one or more of those characters.
\item {\bf \verb|([a-z_0-9]+)|:} The round brackets around this
sub-expression, essentially saves the matched part to a variable for
later use. In this case, the first part of the filename is saved to
the special variable \verb|\1|.
\item {\bf \verb|.*/|:} In the first part of the RE, the
\verb|.| is a special character that matches any character,
and the \verb|*| is a special character that says to match 0
or more instances of the previous matched character. So,
\verb|.*| matches any sequence of characters. If we add a
\verb|/|, then \verb|.*/| matches any sequence of
characters followed by a \verb|/|. In other words, all the
leading directory information.
\end{itemize}
The final result of the RE is that we'll match the complete filename
(including the directory structure) of each file, and more
importantly, the filename, minus the directory structure and file
type, will be saved in the special variable \verb|\1|. We
then use the special variable in the replacement parameter (in this
case, \verb|r'\1'| of the function. The final result 
is that we replace the entire name, with just the important part of
the filename.

Now that we have useful input and output names, it is a simple matter
to create a customised command to pass to the \pythoninline{os.system}
function:


\begin{minipage}{\textwidth}
\begin{python}
import os, glob, re

photos = glob.glob("/home/cshome/coursework/COMP150/coursefiles150/Photos/*.jpg")

for p in photos:
    newp = re.sub('.*/([a-z_0-9]+)\.jpg', r'\1', p)
    print("input name: " + p + ", output name: " + newp)
    command = "convert " + p + " -resize 50x50 " + newp + "_50.jpg"
    print("doing: " + command)
    os.system(command)

print("Done")
\end{python}
\end{minipage}
Which should produce output that looks similar to:

\begin{minipage}{\textwidth}
\begin{pythonOutput}
input name:
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1272.jpg,
output name: 112_1272 
doing: convert
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1272.jpg
-resize 50x50 112_1272_50.jpg 
input name:
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1286.jpg,
output name: 112_1286 
doing: convert
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1286.jpg
-resize 50x50 112_1286_50.jpg 
input name:
/home/cshome/coursework/COMP150/coursefiles150/Photos/112_1280.jpg,
output name: 112_1280 
...
Done
\end{pythonOutput}
\end{minipage}

For this particular problem, application programs, such as PhotoShop,
offer a batch processing alternative within the application. However,
the power of such approaches is limited to what the application
provides --- which depends on what the programmer thinks the user
might need. The power of the scripting approach is that it is limited
only by the users skill and imagination --- all command line driven
programs can be scripted and many GUI programs too.

\section{Glossary}
\begin{description}
\item[environment:] The operating context in which a program runs.
  This would include things such as the file system, current working
  directory, network environment etc.
\item[Jython:] An implementation of Python for the Java Virtual
  Machine.
\item[command line:] Many programs can be executed by typing in a
  command in a system interpreter rather than by double clicking. Such
  a system interpreter is a command line system. The ``command line''
  then is the text you type in to execute a command.
\item[command line arguments:] Any arguments passed to a program from
  the command line.
\item[glob:] A common method for selecting multiple files from the
  command line.
\item[regular expression:] A special type of expression that is used
  for string matching.
\end{description}
\newpage
\section{Laboratory exercises}
\begin{enumerate}
\item Write a program named \pythoninline{mean.py} that takes a
sequence of numbers on the command line and returns the mean of their values.
\begin{shell}
$ python mean.py 3 4
3.5
$ python mean.py 3 4 5
4.0
$ python mean.py 11 15 94.5 22
35.625
\end{shell}
A session of your program running on the same input should produce
the same output as the sample session above.
\item Write a program named \pythoninline{median.py} that takes a sequence of
numbers on the command line and returns the median of their values.
\begin{shell}
$ python median.py 3 7 11
7
$ python median.py 19 85 121
85
$ python median.py 11 15 16 22
15.5
\end{shell}
A session of your program running on the same input should produce
the same output as the sample session above.
\item Modify the program you wrote for Exercise \ref{file-histogram}
  in Section \ref{sec-labs-dicts}
  so that it takes the file to open as a command line argument. 

\item \emph{Extension Exercise:} Modify the final version of
\pythoninline{magick.py}, so that it first resizes each input image to
200x200, then produces a morph sequence between
each pair of input images. To morph between two input images you can
use the \pythoninline{convert} program as follows:
\begin{shell}
$ convert -morph 5 image1.jpg image2.jpg out.jpg
\end{shell}
The parameter $5$ indicates how many in between frames to
use.
\end{enumerate}
