\chapter{Files and modules}
\label{lec-files-modules}

Remember at the start of the course we said that programs are made up
of instructions which only perform a handful of
operations\footnote{See the list back on
  page~\pageref{basic-instructions} to refresh your memory.}.  The
first two of these operations were input and output.  Input involves
getting data from the keyboard, a file, or some other device.  
Output is concerned with displaying data on the screen or sending data
to a file or other device.

We have already seen how to generate output to the screen and get
input from the keyboard, in this lecture we are going to look at
getting input from a file and generating output to a file.

\section{Files}

While a program is running, its data is stored in \emph{random access
  memory} (\myidx{RAM}). RAM is fast and inexpensive, but it is also
\myidx{volatile}, which means that when the program ends, or the
computer shuts down, data in RAM disappears. To make data available
the next time you turn on your computer and start your program, you
have to write it to a \myidx{non-volatile} storage medium, such as a
hard drive, USB drive, or optical disk.  Data on non-volatile storage
media is stored in named locations called
\myidx{files}. By reading and writing files, programs can save
information between program runs.

Working with files is a lot like working with a notebook. To use a
notebook, you have to open it. When you're done, you have to close it.
While the notebook is open, you can either write in it or read from
it. In either case, you know where you are in the notebook. You can
read the whole notebook in its natural order or you can skip around.
All of this applies to files as well. To open a file, you specify its
name and indicate whether you want to read or write.


Opening a file creates a file object. In this example, the variable
\pythoninline{myfile} refers to the new file object.

\begin{pythonInteractive}
>>> myfile = open('test.txt', 'w')
>>> print myfile
<open file 'test.txt', mode 'w' at 0x87bb608>
\end{pythonInteractive}

The \pythoninline{open} function takes two arguments. The first is the
name of the file, and the second is the \myidx{mode}.  Mode
\pythoninline{'w'} means that we are opening the file for writing.  If
there is no file named \pythoninline{test.txt}, it will be created. If
there already is one, it will be replaced by the file we are writing.
When we print the file object, we see the name of the file, the mode,
and the location of the object in memory.  To put data in the file we
invoke the \pythoninline{write} method on the file object:

\noindent
\begin{minipage}{\textwidth}
\begin{pythonInteractive}
>>> myfile.write("Now is the time")
>>> myfile.write("to close the file")
\end{pythonInteractive}
\end{minipage}

Closing the file tells the system that we are done writing and makes
sure that everything actually gets written to disk.  

\begin{pythonInteractive}
>>> myfile.close()
\end{pythonInteractive}

Now we can open the file again, this time for reading, and read the
contents into a string. This time, the mode argument is
\pythoninline{'r'} for reading:

\begin{pythonInteractive}
>>> myfile = open('test.txt', 'r')
\end{pythonInteractive}

Actually the second argument is optional, since the default mode when
opening a file is \pythoninline{'r'}.  So we could have just done:

\begin{pythonInteractive}
>>> myfile = open('test.txt')
\end{pythonInteractive}

If we try to open a file for reading that doesn't exist, we get an error:

\begin{pythonInteractive}
>>> myfile = open('toast.txt')
IOError: [Errno 2] No such file or directory: 'toast.txt'
\end{pythonInteractive}

Not surprisingly, the \pythoninline{read} method of a file reads data
from the file. With no arguments, it reads the entire contents of the
file into a single string:

\begin{pythonInteractive}
>>> text = myfile.read()
>>> print text
Now is the timeto close the file
\end{pythonInteractive}

There is no space between \pythoninline{time} and \pythoninline{to}
because we did not write a space between the strings.

The \pythoninline{read} method can also take an argument that
indicates how many characters to read:

\begin{pythonInteractive}
>>> myfile = open('test.txt')
>>> print myfile.read(5)
Now i
\end{pythonInteractive}

If not enough characters are left in the file, \pythoninline{read}
returns the remaining characters. When we get to the end of the file,
\pythoninline{read} returns the empty string:

\begin{pythonInteractive}
>>> print myfile.read(1000006)
s the timeto close the file
>>> print myfile.read()

>>>
\end{pythonInteractive}

\section{Processing things from a file}

You would normally read information from a file in a Python program
because you want to do something with it.  Let's try counting the
number of words in a file.  Here is a short file for us to work with.

\begin{pythonOutput}
Mary had a little lamb
little lamb little lamb
Mary had a little lamb
its fleece was white as snow
and everywhere that Mary went
Mary went Mary went
everywhere that Mary went
the lamb was sure to go
\end{pythonOutput}

We can start be opening the file and reading its contents into a
string.

\begin{pythonInteractive}
>>> myfile = open('mary.txt')
>>> text_string = myfile.read()
>>> text_string
'Mary had a little lamb\nlittle lamb little lamb\nMary had a little
lamb\nits fleece was white as snow\nand everywhere that Mary
went\nMary went Mary went\neverywhere that Mary went\nthe lamb was
sure to go\n'
\end{pythonInteractive}

Those funny little \pythoninline{$\backslash$n} characters which you can see in
\pythoninline{text\_string} represent newlines in the file.

Now that we have the entire file in \pythoninline{text\_string} we can
use \pythoninline{split} to get a list of words from the string.  Then
we can use the \pythoninline{len} function to tell us how many items
are in the list.  The number of items in the list will be exactly the
same as the number of words in the file.

\begin{pythonInteractive}
>>> word_list = text_string.split()
>>> word_list
['Mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little',
'lamb', 'Mary', 'had', 'a', 'little', 'lamb', 'its', 'fleece', 'was',
'white', 'as', 'snow', 'and', 'everywhere', 'that', 'Mary', 'went',
'Mary', 'went', 'Mary', 'went', 'everywhere', 'that', 'Mary', 'went',
'the', 'lamb', 'was', 'sure', 'to', 'go']
>>> len(word_list)
39
\end{pythonInteractive}

What if we wanted to create a new file which contained a poem about
John instead of Mary?  We already have the poem about Mary in
\pythoninline{text\_string} so all we need to do is change every
occurrence of Mary to John and then write the new poem to a file.

\begin{pythonInteractive}
>>> new_poem = text_string.replace('Mary', 'John')
>>> new_poem
'John had a little lamb\nlittle lamb little lamb\nJohn had a little
lamb\nits fleece was white as snow\nand everywhere that John
went\nJohn went John went\neverywhere that John went\nthe lamb was
sure to go\n'
>>> myfile = open('john.txt', 'w')
>>> myfile.write(new_poem)
>>> myfile.close()
\end{pythonInteractive}

\section{Directories}
Files on non-volatile storage media are organized by a set of rules
known as a \myidx{file system}. File systems are made up of files
and \myidx{directories}, which are containers for both files and
other directories.
When you create a new file by opening it and writing, the new file
goes in the current directory (wherever you were when you ran the
program). Similarly, when you open a file for reading, Python looks
for it in the current directory.

If you open a python interpreter with IDLE, the current working
directory may not be where you think. To find out what directory we're
currently in, we need to use the \pythoninline{os} module. Here is
what happens on my Linux machine:
\begin{pythonInteractive}
>>> import os
>>> os.getcwd()
'/home/cshome/m/mccane/'
\end{pythonInteractive}
\pythoninline{/home/cshome/m/mccane/} is Brendan's home directory
and happens to be where I started IDLE from.

If you want to open a file somewhere else, you have to specify the
\myidx{path} to the file, which is the name of the directory (or
folder) where the file is located:
\begin{pythonInteractive}
>>> wordsfile = open('/usr/share/dict/words', 'r')
>>> wordlist = wordsfile.readlines()
>>> print wordlist[:5]
['\n', 'A\n', "A's\n", 'AOL\n', "AOL's\n", 'Aachen\n']
\end{pythonInteractive}
This example opens a file named \pythoninline{words} that resides in a
directory named \pythoninline{dict}, which resides in
\pythoninline{share}, which resides in \pythoninline{usr}, which
resides in the top-level directory of the system, called
\pythoninline{/}. It then reads in each line into a list using
readlines, and prints out the first 5 elements from that list. 
You cannot use \pythoninline{/} as part of a filename; it is reserved as a
\myidx{delimiter} between directory and filenames.
The file \pythoninline{/usr/share/dict/words} should exist on unix
based systems, and contains a list of words in alphabetical order.

Alternatively, you can change the current working directory using the
method \pythoninline{os.chdir(DIRECTORYNAME)}, where
\pythoninline{DIRECTORYNAME} is a string indicating the new directory.
For example, to change the current working directory to your home
directory, try the following:
\begin{pythonInteractive}
>>> import os
>>> os.chdir(os.path.expanduser("~"))
'/home/cshome/m/mccane'
>>> os.getcwd()
'/home/cshome/m/mccane'
\end{pythonInteractive}
The method \pythoninline{os.path.expanduser("\~{}")} returns a string
representing your home directory.

\section{Modules}

A \myidx{module} is a file containing Python definitions and
statements intended for use in other Python programs. There are many
Python modules that come with Python as part of the
\myidx{standard~library} . We have seen two of these already, the
\pythoninline{doctest} module and the \pythoninline{math} module.


We can use the \pythoninline{help} function to see the functions and
data contained within modules. The keyword module
contains a single function, \pythoninline{iskeyword}, which as its
name suggests is a boolean function that returns \pythoninline{True}
if a string passed to it is a keyword:
\begin{pythonInteractive}
>>> from keyword import *
>>> iskeyword('for')
True
>>> iskeyword('all')
False
>>>
\end{pythonInteractive}
The data item, \pythoninline{kwlist} contains a list of all the
current keywords in Python:
\begin{pythonInteractive}
>>> from keyword import *
>>> print kwlist
['and', 'as', 'assert', 'break', 'class', 'continue', 'def', 'del', 'elif',
'else', 'except', 'exec', 'finally', 'for', 'from', 'global', 'if', 'import',
'in', 'is', 'lambda', 'not', 'or', 'pass', 'print', 'raise', 'return', 'try',
'while', 'with', 'yield']
>>>
\end{pythonInteractive}
We encourage you to check out \url{http://docs.python.org/library/}
to explore the extensive libraries that 
come with Python. There are so many treasures to discover!

\section{Creating modules}
All we need to create a module is a text file with a .py extension on
the filename:
\begin{python}
#  mymodule.py
#
def greet(name):
    return 'Hello ' + name + ', how are you?'
\end{python}
We can now use our module in both scripts and the Python shell. To do
so, we must first \emph{import} the module. There are three ways to do
this:
\begin{pythonInteractive}
>>> from mymodule import greet
>>> greet('Bob')
'Hello Bob, how are you?'
\end{pythonInteractive}
or:
\begin{pythonInteractive}
>>> import mymodule
>>> mymodule.greet('Bob')
'Hello Bob, how are you?'
\end{pythonInteractive}
or:
\begin{pythonInteractive}
>>> from mymodule import *
>>> greet('Bob')
'Hello Bob, how are you?'
\end{pythonInteractive}

In the first example, \pythoninline{greet} is called just like
the functions we 
have seen previously. In the second example the name of the module
and a dot (.) are written before the function name.
Notice that in either case we do not include the .py file extension
when importing. Python expects the file names of Python modules to
end in .py, so the file extension is not included in the
\myidx{import statement}.
The use of modules makes it possible to break up very large programs
into manageable sized parts, and to keep related parts together.

Note: the naming convention for python modules allows only
alphanumeric characters plus the underscore '\_'.

\section{Namespaces}
A \myidx{namespace} is a syntactic container which permits the same
name to be used in different modules or functions.
Each module determines its own namespace, so we can use the same name
in multiple modules without causing an identification problem.
\begin{python}
# module1.py
question = "What is the meaning of life, the Universe, and everything?"
answer = 42
\end{python}
\begin{python}
# module2.py
question = "What is your goal?"
answer = "To get an A+."
\end{python}
We can now import both modules and access question and answer in
each:
\begin{pythonInteractive}
>>> import module1
>>> import module2
>>> print module1.question
What is the meaning of life, the Universe, and everything?
>>> print module2.question
What is your goal?
>>> print module1.answer
42
>>> print module2.answer
To get an A+.
>>>
\end{pythonInteractive}
If we had used \pythoninline{from~module1~import~*} and
\pythoninline{from~module2~import~*} 
instead, we would have a \myidx{naming collision} and would not be
able to access \pythoninline{question} and \pythoninline{answer} from
\pythoninline{module1}. Functions also have their own namespace.

\section{Attributes and the dot operator}

Variables defined inside a module are called \myidx{attributes} of the
module. They are accessed by using the \myidx{dot operator}
(\pythoninline{.}). The \pythoninline{question} attribute of
\pythoninline{module1} and \pythoninline{module2} are accessed using
\pythoninline{module1.question} and \pythoninline{module2.question}.
Modules contain functions as well as attributes, and the dot operator
is used to access them in the same way.
\pythoninline{seqtools.remove\_at} refers to
the \pythoninline{remove\_at} function in the \pythoninline{seqtools} module.

\section{Glossary}
\begin{description}

\item[volatile memory:] Memory which requires an electrical current to
  maintain state. The \emph{main memory} or RAM of a computer is
  volatile. Information stored in RAM is lost when the computer is
  turned off.
\item[non-volatile memory:] Memory that can maintain its state without
  power. Hard drives, flash drives, and optical disks (CD or DVD) are
  each examples of non-volatile memory.
\item[file:] A named entity, usually stored on a hard drive, floppy
  disk, or CD-ROM, that contains a stream of characters.
\item[mode:] A distinct method of operation within a computer
  program. Files in Python can be opened in one of three modes: read
  ('r'), write ('w'), and append ('a').
\item[path:] A sequence of directory names that specifies the exact
  location of a file.
\item[text file:] A file that contains printable characters organized
  into lines separated by newline characters.

\item[module:]A file containing Python definitions and statements
   intended for use in other Python programs. The contents of a module
   are made available to the other program by using the import
   statement.
\item[standard library:]A library is a collection of software used as
   tools in the development of other software. The standard library of
   a programming language is the set of such tools that are
   distributed with the core programming language. Python comes with
   an extensive standard library.
\item[import statement:]A statement which makes the objects contained
   in a module available for use. There are three forms for the import
   statement. Using a hypothetical module named \pythoninline{mymod} containing
   functions \pythoninline{f1} and \pythoninline{f2}, and variables
\pythoninline{v1} and \pythoninline{v2}, examples of these three 
   forms include:
\begin{python}
import mymod
\end{python}
and:
\begin{python}
from mymod import f1, f2, v1, v2
\end{python}
and:
\begin{python}
from mymod import *
\end{python}

\item[namespace:]A syntactic container providing a context for names
   so that the same name can reside in different namespaces without
   ambiguity. In Python, modules, classes, functions and methods all
   form namespaces.
\item[naming collision:]A situation in which two or more names in a
   given namespace cannot be unambiguously resolved. Using
\begin{python}
import mymodule
\end{python}
instead of
\begin{python}
from mymodule import *
\end{python}
prevents naming collisions.

\item[attribute:] A variable defined inside a module.  Module
  attributes are accessed by using the \myidx{dot operator} (.).
\item[dot operator:] The dot operator (.) permits access to
   attributes and functions of a module.
\end{description}

\section{Laboratory Test}

This lab is the first mastery test lab.

\subsection*{Note: Please make sure you bring your Student ID Card
  with you.}

There are no new exercises corresponding to the lecture associated with
this lab.

\subsection*{Assessment}

There are four parts to complete for this lab.  Each part is worth
2.5\% giving a total of 10\% for the entire lab.  On your Desktop you
will find a folder called \texttt{Lab-Test-1} which contains a file
for each part of the test.  You must add code to make the doctests
pass for each of the four files.  Please do each of the parts in
order. A demonstrator
will mark off each part as you complete it.  If you are unable to
complete one of the parts you may still continue on and attempt to
complete the later parts.  If you have any questions about this test
and the way in which it will be assessed please come and see Nick
Meek.

\subsection*{Writing code unaided}

In the previous labs you could complete the work using whatever
resources you wished to, including help from demonstrators, lecture
notes, previous work, on-line resources. In this lab you must complete
the task during a 90-minute time slot, with minimal resources at your
disposal. Using only IDLE and the four provided files,
you must write all of the code needed to pass the given tests without
any other help.

At first you might think this sounds a bit daunting; however we are
not asking you to write all of the code for the first time during your
allocated lab.  We encourage you to write these functions initially
using whatever resources you need. Once you have done this, and are
sure that your code is working correctly, try to write it again
without using any resources (you might need to peek
occasionally). Keep doing this until you can write it all completely
unaided. If you prepare well for this lab then you will find it pretty
straightforward. Solving problems like this, without any outside
assistance, will build your confidence to tackle more demanding
programming tasks.

\subsection*{Note: You are not permitted to access your home
  directory, or any other files or computers, nor may you use the
  Internet during this lab.}

Listings of the files which you will be provided with in the
\pythoninline{Lab-Test-1} folder are given on pages
\pageref{test1-start} to \pageref{test1-end} so that you can prepare
for the lab test.  Note that some of the functions require only one
line of code and wouldn't normally be wrapped in a function, but
in this case it is convenient for testing purposes.


\newpage
\label{test1-start}
\begin{python}
# Mastery test 1 - part 1

def hello():
    """
    >>> hello()
    Hello, world!
    """


def is_divisible(a, b):
    """
    >>> is_divisible(12, 4)
    True
    >>> is_divisible(2, 5)
    False
    """


def maximum(a, b, c):
    """
    >>> maximum(3, 12, 7)
    12
    >>> maximum(-9, -33, -7)
    -7
    """


def my_add(a, b):
    """
    >>> my_add(2, 3)
    5
    >>> my_add('Jim', 'Bob')
    JimBob
    """


def what_am_i(something):
    """
    >>> what_am_i('bob')
    <type 'str'>
    >>> what_am_i(3)
    <type 'int'>
    >>> what_am_i(str)
    <type 'type'>
    """


if __name__ == '__main__':
    import doctest
    doctest.testmod(verbose=True)

\end{python}

\newpage

\begin{python}
# Mastery test 1 - part 2

def compare(a, b):
    """
    >>> compare(3, 1)
    1
    >>> compare(2, 5)
    -1
    >>> compare(4, 4)
    0
    """


def count_vowels(sentence):
    """
    count the number of vowels (a,e,i,o,u) in a sentence
    
    >>> count_vowels("Hello, world!")
    3
    >>> count_vowels("baa baa do baa baa")
    9
    >>> count_vowels("zzz")
    0
    """


def count_words(sentence):
    """
    >>> count_words("The quick brown fox jumps over the lazy dog")
    9
    >>> count_words("Hello")
    1
    >>> count_words("")
    0
    """


def emphasise(sentence):
    """
    >>> emphasise("come here please")
    'COME HERE PLEASE!'
    >>> emphasise("leave me alone")
    'LEAVE ME ALONE!'
    """


if __name__ == '__main__':
    import doctest
    doctest.testmod(verbose=True)

\end{python}

\newpage

\begin{python}
# Mastery test 1 - part 3

def add_nums(numlist):
    """
    >>> add_nums([2, 7, 9, 10, 13, 1, 5, 12])
    59
    >>> add_nums([3, 7, 2.5, -4])
    8.5
    """


def compare(a, b):
    """
    >>> compare(3, 1)
    '3 is greater than 1'
    >>> compare(2, 5)
    '2 is less than 5'
    >>> compare(4, 4)
    '4 and 4 are equal'
    """


def hypotenuse(a, b):
    """
    Get the hypotenuse (longest side) of a right angled triangle with
    other sides of length a and b.  It is the square root of a squared
    plus b squared.
    >>> hypotenuse(3, 4)
    5.0
    >>> hypotenuse(7, 5)
    8.6023252670426267
    """

    
def sortstuff(mylist):
    """
    >>> sortstuff([3, 5, 1, 9, 13, 2])
    [1, 2, 3, 5, 9, 13]
    >>> sortstuff(["one", "two", "three", "four"])
    ['four', 'one', 'three', 'two']
    """


def sortstuff_reverse(mylist):
    """
    >>> sortstuff_reverse([3, 5, 1, 9, 13, 2])
    [13, 9, 5, 3, 2, 1]
    >>> sortstuff_reverse(["one", "two", "three", "four"])
    ['two', 'three', 'one', 'four']
    """


if __name__ == '__main__':
    import doctest
    doctest.testmod(verbose=True)
\end{python}

\newpage

\begin{python}
# Mastery test 1 - part 4

def average(numbers):
    """
    >>> average([4, 5, 1, 9, 13, 2, -6])
    4.0
    >>> average([9, 17, -5, 8])
    7.25
    """


# remember that f = c * 9/5 + 32
def convert_c_to_f(celsius):
    """
    >>> convert_c_to_f(0)
    32
    >>> convert_c_to_f(100)
    212
    >>> convert_c_to_f(-40)
    -40
    >>> convert_c_to_f(12)
    54
    >>> convert_c_to_f(18)
    64
    >>> convert_c_to_f(-48)
    -54
    """

def is_divisible(x, y):
    """
    >>> is_divisible(12, 4)
    'Yes, 12 is divisible by 4'
    >>> is_divisible(7, 3)
    'No, 7 is not divisible by 3'
    """


def score(numbers):
    """
    give the average of the numbers excluding the biggest and smallest one

    >>> score([2, 7, 9, 10, 13, 1, 5, 12])
    7.5
    >>> score([3, 7, 2.5, -4])
    2.75
    """


if __name__ == '__main__':
    import doctest
    doctest.testmod(verbose=True)

\end{python} \label{test1-end}
