__name__ == '__main_'

The line if __name__ == '__main__' may be an unfamiliar idiom if you have not done much c­ommand-line scripting (or come across it in other use cases). In Python, each module has a __name__ attribute, which is always automatically set to the name of the module that is currently being executed.

The value __main__ is special. When a module is invoked directly (such as by running it on the command line), the __name__ attribute is set to this value.

Why test for this? Nearly every .py file in Python is importable as a module, and, therefore, could be imported. In CLI scripts, you probably do not want the code to directly run in this case. CLI scripts sometimes contain code such as calls to sys.exit() that would terminate the entire program. This module's option and argument-parsing behavior really only makes sense if it is invoked directly. Therefore, this type of code should be placed beneath the if __name__ == '__main__' test.

Note that there is nothing magic about the if block; it is simply a top-level if statement. Other ­top-level code will still be executed even if the if test fails. Additionally, note that it is traditional that such tests be placed at the bottom of the file.

OptionParser

Next, consider the creation of an OptionParser instance, followed by the call to its parse_args method. The OptionParser class is the primary class in the optparse module used for taking the arguments and options sent to a CLI command, and making sense of them.

The fundamental way that this works is that you tell the OptionParser instance what options you expect and know how to address. Options are strings that start with - or --, such as -v or --verbose. (You learn more about these shortly.) The call to parse_args iterates over all of the options that the parser recognizes, and places them in the first variable that parse_args returns (which is named options in the previous example). Any arguments left over are considered to be positional arguments, and are placed in the second variable (args in the previous example), which is a list.

The previous example uses no options, so everything that the parser receives is considered to be a positional argument. The script then takes that list, joins it into a string, converts it to uppercase, and prints it.

One thing to note is that any argument that begins with hyphens is expected to be an option, and optparse raises an exception if you try to send an option that the parser does not recognize. Furthermore, the exception is internally handled within optparse and calls sys.exit, so there is no real way to catch these errors yourself.

$ python echo_upper.py --foo
Usage: echo_upper.py [options]

echo_upper.py: error: no such option: --foo

Types of Options

A CLI script can accept two common types of options.

One type is sometimes called a flag or a switch, and is an option that does not require or accept a value along with the option. Essentially, in these cases, it is the presence or absence of the option that determines the script behavior.

Two common examples of such switches are --verbose and --quiet (often also provided as -v and -q, respectively). The script executes normally if these options are absent, but does something different (provides more or less output) if they are present. Note that you generally specify this as --quiet, as opposed to --quiet=true or something similar. The value is implied by the presence of the switch.

Another type of option is one that does expect a value. Most database clients accept options such as --host, --port, and the like. These do not make sense as switches. You do not simply provide --host and expect the database client to infer what the actual host is. You must provide the hostname or IP address that you are connecting to.

Adding Options to OptionParser

Once you have an OptionParser instance, you can add an option to it using the add_option method. This comes after the OptionParser instance is instantiated, but before parse_args, which is the final step in the chain.

Consider first the addition of a simple switch, which does not actually expect an argument.

import optparse


if __name__ == '__main__':
    parser = optparse.OptionParser()
    parser.add_option('-q', '--quiet',
        action='store_true',
        dest='quiet',
        help='Suppress output.',
    )

This would add support for a -q and --quiet switch. Note that, in CLI scripts, it is extremely ­common to have a long-form and short-form version of options, and so optparse supports this easily. By providing two different strings as positional arguments to add_option, the add_option method understands that they are supposed to be accepted, and that they are aliases of one another.

The action keyword argument is what specifies that the --quiet flag is a flag, and does not expect a variable. If you leave off the action keyword argument, the option is assumed to expect a value (more on that in a moment). Setting action to store_true or store_false means that no value is expected, and, if the flag is provided at all, the value is True or False, respectively.

The dest keyword argument is what decides the name of the option in Python. The name of this particular option within the options variable is quiet. In many cases, you do not have to set this. OptionParser infers an appropriate name based on the name of the option itself. However, it is a good idea to always set it explicitly for readability and maintainability.

Finally, the help keyword argument sets the help text for this option. It is what a user will see if he or she invokes your script with --help. It is wise to always provide this.

It is worth noting that optparse automatically adds a --help option, and handles it automatically. If you call a script with only the example option and provide --help, you get useful output.

$ python cli_script.py --help
Usage: cli_script.py [options]

Options:
  -h, --help   show this help message and exit
  -q, --quiet  Suppress output.

Options with Values

In addition to switches, sometimes you need options that actually expect values to be provided along with the option. This does add some complexity. The biggest reason for this is that values have types in Python, and the CLI does not have a robust concept of types. Essentially, everything is a string.

First, consider an option that accepts a string, such as a --host flag that might be sent to a database client. This option should probably be optional. The biggest use case for database clients is connecting to databases on the same machine, so localhost makes for an entirely sensible default.

Here is a complete script that does nothing but reprint the host to standard out:

import optparse


if __name__ == '__main__':
    parser = optparse.OptionParser()
    parser.add_option('-H', '--host',
        default='localhost',
        dest='host',
        help='The host to connect to. Defaults to localhost.',
        type=str,
    )
    options, args = parser.parse_args()

If you call this script with no arguments, you will see that the default of localhost is applicable.

$ python optparse_host.py
The host is localhost.

By adding a --host option, you override this default.

$ python optparse_host.py --host 0.0.0.0
The host is 0.0.0.0.

If you fail to provide an option, optparse will complain.

$ python optparse_host.py --host
Usage: optparse_host.py [options]

optparse_host.py: error: --host option requires an argument

Focus on the call to add_option. Several things are different from your --quiet flag. First, you omitted the action keyword argument. The default for this (store) simply stores the value provided. You can specify this manually if you choose to do so.

Second, you provided a type. The OptionParser instance actually infers this from the type of the default value in most cases (although this does not work if your default value is None), so providing it is often optional. Explicitly providing it often makes the code easier to read later. The default for type is also str.

Finally, you provided a default. Most options should be optional, which means they must have a sensible default. In many cases, this default may be None. In the case of the host value, you chose localhost as a sensible default because having your client and server on the same machine is a ­common use case.

One other thing is worth pointing out explicitly. The way you read the value off of the options variable is not what you might expect—the host value is read as options.host. You may have expected the options value to be provided as a dictionary, in which case options['host'] would have been correct. However, the options variable is provided using its own special class (called Values), and the individual options exist on this object as attributes. Note that, if you want a dictionary, options.__dict__ will provide you with the corresponding dictionary.

Non-String Values

What about values that are not strings? For example, continuing the example of a database client of some sort, what if the script should accept a port number? Most databases run on a default port (PostgreSQL uses 5432, MySQL uses 3306, and so on), but sometimes such services run on alternate ports.

An option for a port looks similar to an option for a host.

    parser.add_option('-p', '--port',
        default=5432,
        dest='port',
        help='The port to connect to. Defaults to 5432.',
        type=int,
    )

The crucial difference here is that the type is now specified as int. Again, OptionParser would infer this from the fact that the default value is the integer 5432.

In this case, OptionParser performs the type conversion for you, and raises an appropriate error if it is not able to. Consider a script that takes a host and port, as shown here:

import optparse


if __name__ == '__main__':
    parser = optparse.OptionParser()
    parser.add_option('-H', '--host',
        default='localhost',
        dest='host',
        help='The host to connect to. Defaults to localhost.',
        type=str,
    )
    parser.add_option('-p', '--port',
        default=5432,
        dest='port',
        help='The port to connect to. Defaults to 5432.',
        type=int,
    )
    options, args = parser.parse_args()

    print('The host is %s, and the port is %d.' %
          (options.host, options.port))

Again, invoking the script without arguments provides both default values. Because the format string uses %d rather than %s, you know that options.port is an integer under the hood.

$ python optparse_host_and_port.py
The host is localhost, and the port is 5432.

If you try to specify a port value that is not an integer, you get an error.

$ python optparse_host_and_port.py --port=foo
Usage: optparse_host_and_port.py [options]

optparse_host_and_port.py: error: option --port: invalid integer value: 'foo'
$ echo $?
2

And, of course, if you specify a valid integer, it overrides the default.

$ python3 optparse_host_and_port.py --port=8000
The host is localhost, and the port is 8000.

Specifying Option Values

Several different idioms exist for how to specify option values on the command line. The optparse module attempts to support all of them.

$ python optparse_host_and_port.py -H localhost
The host is localhost, and the port is 5432.
$ python optparse_host_and_port.py -H "localhost"
The host is localhost, and the port is 5432.
$ python optparse_host_and_port.py -Hlocalhost
The host is localhost, and the port is 5432.
$ python optparse_host_and_port.py -H"localhost"
The host is localhost, and the port is 5432.

$ python optparse_host_and_port.py -H=localhost
The host is =localhost, and the port is 5432.

$ python optparse_host_and_port.py -H="localhost"
The host is =localhost, and the port is 5432.

$ python cli_script.py --host localhost
The host is localhost, and the port is 5432.
$ python cli_script.py --host "localhost"
The host is localhost, and the port is 5432.
$ python cli_script.py --host=localhost
The host is localhost, and the port is 5432.
$ python cli_script.py --host="localhost"
The host is localhost, and the port is 5432.

Positional Arguments

It is also possible to send positional arguments to optparse. Actually, any argument that is not attached to an option will be considered by the parser to be a positional argument, and is sent to the args variable that is returned from parser.parse_args().

import optparse


if __name__ == '__main__':
    parser = optparse.OptionParser()
    options, args = parser.parse_args()

    print('The sum of the numbers sent is: %d' %
          sum([int(i) for i in args]))

Any arguments sent to this script are part of the args variable, and the script tries to convert them to integers and add them together.

$ python optparse_sum.py 1 2 5
The sum of the numbers sent is: 8

Of course, if you sent an argument that cannot be converted to an integer, you will get an exception.

$ python optparse_sum.py 1 2 foo
Traceback (most recent call last):
  File "optparse_sum.py", line 8, in <module>
    print('The sum of the numbers sent is: %d' % sum([int(i) for i in args]))
ValueError: invalid literal for int() with base 10: 'foo'

Counters

You can use a small number of other types of options besides simple flags and direct value storage. One type that is infrequently used but is sometimes useful is a counter flag.

Most flags simply set a Boolean value to True or False, based on the presence or absence of the flag. A related idiom, however, is to allow specifying a flag multiple times to intensify the effect.

Consider a -v flag that causes a script to be more verbose. Some programs allow -v to be specified repeatedly in order to make the script become even more verbose. For example, a popular configuration tool called Ansible allows you to specify-v up to four times to provide increasingly verbose output.

You do this through a different action value that you can provide to add_option. Consider this script:

import optparse


if __name__ == '__main__':
    parser = optparse.OptionParser()
    parser.add_option('-v',
        action='count',
        default=0,
        dest='verbosity',
        help='Be more verbose. This flag may be repeated.',
    )
    options, args = parser.parse_args()

    print('The verbosity level is %d, ah ah ah.' % options.verbosity)

Notice that the call to add_option now specifies action='count'. This means that the value will be incremented by one every time the flag is sent.

You can invoke the script to easily see this in action.

$ python count_script.py
The verbosity level is 0, ah ah ah.
$ python count_script.py -v
The verbosity level is 1, ah ah ah.
$ python count_script.py -v -v
The verbosity level is 2, ah ah ah.
$ python count_script.py -vvvvvvvvvvv
The verbosity level is 11, ah ah ah.

Notice that you have two valid ways to specify the short-form option in this case: -v -v and -vv are equivalent. This is actually true for distinct short-form options as well, provided they do not expect a value.

It is also worth noting that explicitly specifying the default value of 0 is important. If you do not specify it explicitly, OptionParser uses a default value of None, which is usually not what you want. (In this case, the script would raise TypeError when it tries to do the string interpolation on the last line.)

Finally, note that if you choose a default value other than 0, the flag functions as an increment, not a flat count. So, if your default value is 1, and you provide two -v flags, the value would be 3 (not 2).

List Values

Sometimes, you may want to accept multiple values for the same option, and provide them to your script as a list. This is fundamentally similar to a count option, except that it takes a value each time, rather than simply incrementing an integer variable.

The following script prints usernames, one at a time:

import optparse


if __name__ == '__main__':
    parser = optparse.OptionParser()
    parser.add_option('-u', '--user',
        action='append',
        default=[],
        dest='users',
        help='The username to be printed. Provide this multiple times to '
             'print the username for multiple users.',
    )
    options, args = parser.parse_args()

    for user in options.users:
        print('Username: %s.' % user)

Running this with no -u or --user options provided generates no output.

$ python echo_usernames.py
$

However, you can provide one or more -u or --user options to the script, and regardless of how many, the OptionParser makes them available as a list:

$ python echo_usernames.py -u me
Username: me.
$ python echo_usernames.py -u me -u myself
Username: me.
Username: myself.

Option Flags

The first kind of option is a flag, such as -v or --verbose for a verbose mode, or -q or --quiet for a mode that suppresses most or all output. These options do not expect a value. The presence or absence of the option determines the appropriate Boolean in the parser.

The syntax for specifying a flag looks like this:

parser.add_argument('-q', '--quiet',
    action='store_true',
    dest='quiet',
    help='Suppress output.',
)

If you are familiar with optparse (or read the section on optparse earlier in this chapter), this will look very familiar to you. Other than the method name, not much has changed so far.

First, note the action variable. This is set to store_true, which is the reason why the parser will not expect a value. Most options do not need an action to be specified (the default is store, which stores the value it receives). The specification of store_true or store_false is the most common way to indicate that an option is a flag and should not accept a value.

The dest keyword argument determines how to look up the parsed value (in this case, True or False) on the object you get back when you call parse_args. The string used here will be the attribute name on the object. (So, you would look up this one using args.quiet.) In many cases, the dest keyword argument is optional. ArgumentParser determines an intuitive name based on the name of the option itself. However, it is useful to explicitly provide this for readability and maintainability.

The help keyword argument determines what users get if they call your script with -h or --help. The ArgumentParser implicitly provides a help screen attached to these switches, so you should always specify a help on your arguments.

Alternate Prefixes

Most CLI scripts use the hyphen (-) character as the prefix for options, and this is what ArgumentParser expects by default. However, some scripts may use different characters. For example, a script that is only intended to be run in Windows environments may prefer to use the / character, which is consistent with many Windows command-line programs.

You can change which characters are used for prefixes by providing the prefix_chars keyword argument to the ArgumentParser constructor, as shown here:

import argparse


if __name__ == '__main__':
    parser = argparse.ArgumentParser(prefix_chars='/')
    parser.add_argument('/q', '//quiet',
        action='store_true',
        dest='quiet',
        help='Suppress output.',
    )
    args = parser.parse_args()

    print('Quiet mode is %r.' % args.quiet)

In this example, you changed the prefix character to /. Note that this also means that the argument itself (the one passed to add_argument) must change accordingly.

Calling this script is still straightforward. You simply must use /q or //quiet (rather than -q or --quiet).

$ python argparse_quiet.py
Quiet mode is False.
$ python argparse_quiet.py /q
Quiet mode is True.

Viewing the help reflects this:

$ python argparse_quiet.py /h
usage: argparse_quiet.py [/h] [/q]

optional arguments:
  /h, //help  show this help message and exit
  /q          Suppress output.

Note that, because you changed the prefix character to /, the automatically registered help command is changed along with it.

Options with Values

Options that accept values are fundamentally similar. Consider the following example of a script that accepts a host value (such as a database client), translated into argparse:

import argparse


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-H', '--host',
        default='localhost',
        dest='host',
        help='The host to connect to. Defaults to localhost.',
        type=str,
    )
    args = parser.parse_args()

    print('The host is %s.' % args.host)

Again, if you are already familiar with optparse, you will likely notice just how similar this is. The keyword arguments are the same, and they do the same thing.

The important argument to focus on here is type, which controls what Python type the value is ­ultimately expected to be. It is common for this to be int or float, and a small number of other types may also make sense.

Parsing arguments when you use argparse is slightly different from when you use optparse. Regardless of whether you use the short-form or the long-form syntax, you can separate the option from the value using a space or an equal sign. The short-form syntax (and only the short-form syntax) also supports not separating the value from the option at all. Both the short-form and the long-form syntax allow quotes around the value.

Therefore, all of these are equivalent:

$ python argparse_args.py -Hlocalhost
The host is localhost.
$ python argparse_args.py -H"localhost"
The host is localhost.
$ python argparse_args.py -H=localhost
The host is localhost.
$ python argparse_args.py -H="localhost"
The host is localhost.
$ python argparse_args.py -H localhost
The host is localhost.
$ python argparse_args.py -H "localhost"
The host is localhost.
$ python argparse_args.py --host=localhost
The host is localhost.
$ python argparse_args.py --host="localhost"
The host is localhost.
$ python argparse_args.py --host localhost
The host is localhost.
$ python argparse_args.py --host "localhost"
The host is localhost.

import argparse


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--cheese',
        choices=('american', 'cheddar', 'provolone', 'swiss'),
        default='swiss',
        dest='cheese',
        help='The kind of cheese to use',
    )
    args = parser.parse_args()

    print('You have chosen %s cheese.' % args.cheese)

$ python argparse_choices.py
You have chosen swiss cheese.

$ python argparse_choices.py --cheese provolone
You have chosen provolone cheese.

$ python argparse_choices.py --cheese pepperjack
usage: argparse_choices.py [-h] [--cheese {american,cheddar,provolone,swiss}]
argparse_choices.py: error: argument --cheese: invalid choice: 'pepperjack'
     (choose from 'american', 'cheddar', 'provolone', 'swiss')

import argparse


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--madlib',
        default=['fox', 'dogs'],
        dest='madlib',
        help='Two words to place in the madlib.',
        nargs=2,
    )
    args = parser.parse_args()

    print('The quick brown {0} jumped over the '
          'lazy {1}.'.format(*args.madlib))

$ python argparse_multiargs.py
The quick brown fox jumped over the lazy dogs.

$ python argparse_multiargs.py --madlib pirate ninjas
The quick brown pirate jumped over the lazy ninjas.

$ python argparse_multiargs.py --madlib pirate
usage: argparse_multiargs.py [-h] [--madlib MADLIB MADLIB]
argparse_multiargs.py: error: argument --madlib: expected 2 arguments
$ python argparse_multiargs.py --madlib pirate ninjas cowboy
usage: argparse_multiargs.py [-h] [--madlib MADLIB MADLIB]
argparse_multiargs.py: error: unrecognized arguments: cowboy

import argparse


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--addends',
        dest='addends',
        help='Integers to provide a sum of',
        nargs='+',
        required=True,
        type=int,
    )
    args = parser.parse_args()

    print('%s = %d' % (
        ' + '.join([str(i) for i in args.addends]),
        sum(args.addends),
    ))

$ python argparse_sum.py --addends 1 2 5
1 + 2 + 5 = 8
$ python argparse_sum.py --addends 1 2
1 + 2 = 3

$ python argparse_sum.py --addends 1
1 = 1

Positional Arguments

With argparse (unlike with optparse), you must declare your positional arguments explicitly. If you do not, the parser expects to have no arguments left over after it completes parsing, and it raises an error if arguments still remain.

The declaration for positional arguments is equivalent to the declaration for options, except that the leading hyphen is omitted. As an example, it seems bad form for the --addends option in the previous example to be an option at all. Options should be optional.

It is easy to provide the same thing as a positional argument.

import argparse


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('addends',
        help='Integers to provide a sum of',
        nargs='+',
        type=int,
    )
    args = parser.parse_args()

    print('%s = %d' % (
        ' + '.join([str(i) for i in args.addends]),
        sum(args.addends),
    ))

This is mostly the same, except that the --addends argument has been replaced with addends, without the double-hyphen prefix. This causes the parser to expect a positional argument instead.

Why provide a name for positional arguments? (After all, optparse does not need positional argument names,) The answer is that the name you provide is used in the program's --help output.

$ python cli_script.py --help
usage: cli_script.py [-h] addends [addends ...]

positional arguments:
  addends     Integers to provide a sum of

optional arguments:
  -h, --help  show this help message and exit

Notice that the word addends is used in the usage line near the top of the help. This provides slightly more insight into what is being expected. Additionally, unlike in help provided by optparse, the positional arguments are documented as part of the help screen.

You can invoke this script the same way, except without the --addends option.

$ python cli_script.py 1 2 5
1 + 2 + 5 = 8

Reading Files

A common need when writing CLI applications is to read files. The argparse module provides a special class that can be sent to the type keyword argument of add_argument, which is argparse.FileType.

The argparse.FileType class expects the arguments that would be sent to Python's open function, excluding the filename (which is what is being provided by the user invoking the program). If you are opening the file for reading, this may be nothing. open defaults to opening files only for reading. However, any arguments after the initial positional argument to open can be provided to FileType, and they will be passed on to open.

Consider the following program that may read a configuration file from a non-default location:

import argparse


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-c', '--config-file',
        default='/etc/cli_script',
        dest='config',
        help='The configuration file to use.',
        type=argparse.FileType('r')
    )
    args = parser.parse_args()

    print(args.config.read())

This would read from /etc/cli_script by default, but allow you to specify a different file to read from using the -c or --config-file options. Rather than providing these options as text and forcing you to open the file yourself, you will simply be provided with an open file object:

$ echo "This is my config file." > foo.txt
$ python cli_script.py --config-file foo.txt
This is my config file.

Note that the file is expected to exist. If it does not, you get an error.

$ python cli_script.py --config-file bar.txt
usage: cli_script.py [-h] [-c CONFIG]
cli_script.py: error: argument -c/--config-file: can't open 'bar.txt':
     [Errno 2] No such file or directory: 'bar.txt'