Using type Directly

This can best be illustrated by simply creating a class using type directly, rather than using the Python class keyword. This is syntactically quite ugly, but it offers a clear view into what is going on under the hood.

Therefore, consider the following simple set of classes:

class Animal(object):
    """A class representing an arbitrary animal."""

    def __init__(self, name):
        self.name = name

    def eat(self):
        pass

    def go_to_vet(self):
        pass


class Cat(Animal):
    def meow(self):
        pass

    def purr(self):
        pass

The Animal class obviously represents an animal, and defines certain things that the animal is capable of doing, such as eating and being taken to the vet. The Cat subclass additionally knows how to meow and purr, functions not available to other animals. (The method bodies are stubbed, and left to the reader's intuition.)

What happens here is that when the Python interpreter gets to the top statement in the code, class Animal(object), it invokes the type constructor under the hood. As alluded to earlier, type is a built-in class in Python, which is the default class for other class objects. It is the default class that creates other classes—or, the default metaclass.

However, nothing stops you from simply doing this directly. The type constructor takes three positional arguments: name, bases, and attrs. The name argument (a string) is simply the name of the class. The bases argument is a tuple of the superclasses for that class. Python supports multiple inheritance, which is why this is a tuple. If you are only inheriting from a single class, just send a tuple with a single element. Finally, the attrs argument is a dictionary of all the attributes on the class.

def init(self, name):
    self.name = name

def eat(self):
    pass

def go_to_vet(self):
    pass


Animal = type('Animal', (object,), {
    '__doc__': 'A class representing an arbitrary animal.',
    '__init__': init,
    'eat': eat,
    'go_to_vet': go_to_vet,
})

def meow(self):
    return None

def purr(self):
    return None


Cat = type('Cat', (Animal,), {
    'meow': meow,
    'purr': purr,
})

The type Chain

Consider the following instance of the Cat class:

louisoix = Cat(name='Louisoix')

Notice the three things that are on deck. louisoix is an object, and an instance of Cat. The Cat class is also an object (because classes are objects), and is an instance of type. Finally, type is the top of the chain.

You can also observe this in another way. Passing a single object to type returns its class, as shown here:

>>> type(5)
<type 'int'>

So, observe the following chain:

>>> type(louisoix)
<class '__main__.Cat'>

>>> type(Cat)
<class 'type'>
>>> type(type)
<class 'type'>

The type class is the base case here. It is the top of the chain, and, therefore, type(type) returns itself.

The Role of type

type is the primary metaclass in Python. Ordinary classes that are created with the class keyword, by default, have type as their metaclass.

Colloquially, you can refer to type as the metaclass for both the class (Cat) and its instances (louisoix).

Additionally, type is also the superclass from which other metaclasses inherit. This is analogous to object being the class from which other classes inherit. Just as object is the top of the class hierarchy, type is the top of the metaclass hierarchy.

The __new__ Method

The most important method that custom metaclasses must define is the __new__ method. This method actually handles the creation of the class, and must return the new class.

The __new__ method is a class method (that does not need to be explicitly decorated as such). The arguments sent to __new__ in custom metaclasses must mirror the arguments sent to type's __new__ method, which takes four positional arguments.

The first argument is the metaclass itself, prepended to arguments in a manner similar to that of bound methods. By convention, this argument is called cls.

Beyond this, __new__ expects three positional arguments:

• First, the desired name of the class as a string (name)

• Second, a tuple of the class's superclasses (bases)

• Third, a dictionary of attributes that the class should contain (attrs)

Most custom implementations of __new__ in metaclasses should ensure that they call the superclass implementation, and perform whatever work is needed in the code around that.

__new__ Versus __init____

Recall at this point the distinction between the __new__ method and the __init__ method. In a class or a metaclass, the __new__ method is responsible for creating and returning the object. Conversely, the __init__ method is responsible for customizing the object after it has been created, and returns nothing.

In ordinary classes, you generally do not define a custom __new__ method at all. By contract, defining a custom __init__ method is extremely common. This is because the implementation of __new__ provided by object is essentially always sufficient, but it is also necessary. Overriding it (even in direct subclasses of object) would require calling the superclass method and being careful to return the result (the new instance). By contrast, overriding __init__ is easy and relatively risk-free. An object's implementation of __init__ is a no-op, and the method does not return anything at all.

When you're writing custom metaclasses, this behavior changes. Custom metaclasses generally should override the __new__ method, and generally do not implement an __init__ method at all. When doing this, keep in mind that you almost always must call the superclass implementation. type's implementation of __new__ will actually provide you with the object you need to do work on and return.

A Trivial Metaclass

Before diving into a metaclass that customizes behavior, consider a custom metaclass that does nothing but check all the boxes that have been covered thus far.

class Meta(type):
    """A custom metaclass that adds no actual functionality."""

    def __new__(cls, name, bases, attrs):
        return super(Meta, cls).__new__(cls, name, bases, attrs)

This discussion has not yet explored how to assign a metaclass within class creation using the class keyword (more on that shortly). But you can create a class that uses the Meta metaclass by calling Meta directly, similar to the direct invocation of type earlier.

>>> C = Meta('C', (object,), {})

This creates a class, C, which is an instance of Meta rather than an instance of type. Observe the following:

>>> type(C)
<class '__main__.Meta'>

This is distinct from what you observe from a “normal” class, as shown here:

>>> class N(object):
...     pass
...
>>> type(N)
<class 'type'>

Metaclass Inheritance

It is worth noting that metaclasses are inherited. Therefore, subclasses of C will be instances of Meta, rather than being direct instances of type as shown in the following code and illustrated in Figure 5.1.

>>> class D(C):
...     pass
...
>>> type(D)
<class '__main__.Meta'>

In this case, D is an instance of Meta not because it has an explicit metaclass declared, or because you called Meta to create it, but rather because its superclass is an instance of Meta, and, therefore, it is also.

It is important to note here that classes may only have one metaclass. Under most circumstances, this is fine, even in scenarios where multiple inheritance is in play. If a class subclasses two or more distinct classes with distinct metaclasses, the Python interpreter will try to resolve this by checking the ancestry of the metaclasses. If they are direct ancestors, the subclass will be used.

Consider the following class that subclasses both C (an instance of Meta) and N (an instance of type)

>>> class Z(C, N):
...     pass
...
>>> type(Z)
<class '__main__.Meta'>

Figure 5.2 shows what is happening in this code.

What is going on here? The Python interpreter is told to create class Z, and that it should subclass both C and N. This would be the equivalent of type('Z', (C, N), {}).

First, the Python interpreter examines C, and realizes that it is an instance of Meta. Then it examines N, and realizes that it is an instance of type. This is a potential conflict. The two superclasses have different metaclasses. However, the Python interpreter also realizes that Meta is a direct subclass of type. Therefore, it knows it can safely use Meta, and does so.

What happens if you have two metaclasses where one is not a direct descendent of the other? Now there is a conflict, and the Python interpreter does not know how to solve it. And it will cowardly refuse to try, as shown here:

>>> class OtherMeta(type):
...     def __new__(cls, name, bases, attrs):
...         return super(OtherMeta, cls).__new__(cls, name, bases, attrs)
...
>>> OtherC = OtherMeta('OtherC', (object,), {})
>>>
>>> class Invalid(C, OtherC):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in __new__
TypeError: Error when calling the metaclass bases
    metaclass conflict: the metaclass of a derived class must be a (non-
    strict) subclass of the metaclasses of all its bases

This happens because Python can only have one metaclass for each class, and will not try to guess which metaclass to use in an ambiguous case.

Python 3

In Python 3, metaclasses are declared alongside the superclasses (if any). The syntax resembles that of a keyword argument in a function declaration or a function call, and the “keyword argument” is metaclass.

Earlier, you created the C class by calling Meta directly. Here is the preferred way to do this in Python 3:

class C(metaclass=Meta):
    pass

This class keyword call does the exact same thing as creating the class by directly calling Meta. This, however, is the preferred style.

One thing to note here is that you did not explicitly specify object as the superclass. In most of the examples used in this book, you have explicitly specified object as the superclass. This is because this book intends examples to be run on either Python 2 or Python 3. In Python 2, specifying this matters, because subclassing object is what makes the class be a “new-style class,” which is a construct introduced a long time ago (Python 2.2) that altered Python's method-resolution order, as well as some of the other guts of how Python classes work. The direct subclassing of object was used as a way to ensure backward-compatibility, forcing developers to “opt-in” to new-style classes, rather than to opt out of them.

In Python 3, which was a backward-incompatible release, all classes are new-style, and directly subclassing object is no longer necessary, and thus is not done here. That said, the previous code is exactly equivalent to the following:

class C(object, metaclass=Meta):
    pass

This style allows you to observe more explicitly the distinction between superclasses, which are declared here using a syntax akin to positional arguments in a function declaration, as opposed to the metaclass that is declared with the keyword argument syntax. They must be specified in this order, with metaclass last, just like function arguments.

When directly subclassing object in Python 3, either style (explicitly including it or omitting it) is acceptable.

Python 2

Python 2 has an entirely different syntax for metaclass declaration. The Python 2 syntax is not ­supported under Python 3, and the Python 3 syntax is not supported under Python 2. (Skip down a ­section to see how to declare a metaclass in a way that does the right thing on both.)

The Python 2 syntax for declaring a metaclass is to assign a __metaclass__ attribute to the class. Consider the earlier creation of class C using a call to Meta. Following is the equivalent code in Python 2:

class C(object):
    __metaclass__ = Meta

In this case, the metaclass is being assigned in the class body. This is fine. The Python interpreter looks for this when the class keyword is invoked, and uses Meta rather than type to create the new class.

What About Code That Might Run on Either Version?

Because Python 3 introduced backward-incompatible changes to the Python language, Python developers have come up with strategies for running the same set of code under either the Python 3 interpreter or the Python 2 interpreter with similar results.

One of the most popular ways to do this involves using a tool called six, which was written by Benjamin Peterson and is available from PyPI.

six provides two ways to declare a metaclass: by creating a stand-in class and using it as a direct superclass, or by using a decorator to add the metaclass.

The first method (which is the stand-in class method) looks like this:

import six

class C(six.with_metaclass(Meta)):
    pass

What is happening here? six.with_metaclass creates a dummy class of sorts that subclasses object, and has Meta as its metaclass, but which does nothing else. By applying this class as the superclass to C, and based on how metaclasses interact with class inheritance (discussed previously), C is now an instance of Meta, regardless of which Python version is in use.

Depending on exactly what the metaclass in question does, sometimes this solution will not actually work. Because six.with_metaclass actually instantiates a class, some metaclasses may want to do work, and it is possible that said work would not be compatible with having an abstract superclass.

six provides one other way to assign a metaclass to a class, which is using a decorator: @six.add_metaclass. The syntax for that looks like this:

import six

@six.add_metaclass(Meta)
class C(object):
    pass

The result here becomes the same to the Python 2- or Python 3-specific implementations. Class C is created, using the class keyword, and the Meta metaclass, rather than using type. The decorator does this without instantiating an abstract class.

When Is Cross-Compatibility Important?

Because there are two incompatible syntaxes for Python 2 as opposed to Python 3, it's important to explore at this point when it is better to use the “pure” language approach, and when it is the right time to introduce six.

Without delving too deeply into the theory, a good rule of thumb here is that if you are running Python 2, assume that you may at some point want to migrate to Python 3, and try to write cross-compatible code. This will entail using six for any number of things (this among them), and so probably introducing six into your codebase is wise. By contrast, if you are already exclusively in a Python 3 environment, it is unlikely that you will ever want to shift backward, and just writing Python 3 code should be fine.

Declarative Class Declaration

The most common reason to use a custom metaclass is to create a delineation between class declaration and class structure, particularly when you're creating APIs for other developers to use.

from django.db import models

class Book(models.Model):
    author = models.CharField(max_length=100)
    title = models.CharField(max_length=250)
    isbn = models.CharField(max_length=20)
    publication_date = models.DateField()
    pages = models.PositiveIntegerField()

Class Verification

Another key use for metaclasses is for class verification. If a class must conform to a particular interface, a metaclass can be a very effective way to enforce this. Usually, it is preferable that this sort of problem be handled by a sensible default. Occasionally, however, this is not possible.

For example, consider a class that requires either one or another attribute to be set, but not both. This is difficult to handle with a sensible default if it is important that one attribute be unset (as opposed to set to None).

This concept can be handled using a metaclass. The following simple metaclass requires classes to contain either a foo attribute or a bar attribute:

class FooOrBar(type):
    def __new__(cls, name, bases, attrs):
        if 'foo' in attrs and 'bar' in attrs:
            raise TypeError('Class %s cannot contain both `foo` and '
                            '`bar` attributes.' % name)
        if 'foo' not in attrs and 'bar' not in attrs:
            raise TypeError('Class %s must provide either a `foo` '
                            'attribute or a `bar` attribute.' % name)
        return super(FooOrBar, cls).__new__(cls, name, bases, attrs)

The following Python 3 class uses this metaclass and conforms to this interface:

>>> class Valid(metaclass=FooOrBar):
...     foo = 42
...
>>>

Everything here works fine. What happens if you try to set both attributes, or neither?

>>> class Invalid(metaclass=FooOrBar):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 9, in __new__
TypeError: Class Invalid must provide either a ‘foo‘ attribute or a ‘bar‘
    attribute.
>>>
>>> class Invalid(metaclass=FooOrBar):
...     foo = 42
...     bar = 42
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 6, in __new__
TypeError: Class Invalid cannot contain both ‘foo‘ and ‘bar‘ attributes.

This particular implementation has a problem. It will not work well continuing down the subclass chain. The reason for this is because the metaclass examines the attrs dictionary directly, but this only contains attributes set for the class being declared. It does not know anything about attributes that are inherited from superclasses.

>>> class Valid(metaclass=FooOrBar):
...     foo = 42
...
>>> class AlsoValid(Valid):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in __new__
TypeError: Class AlsoValid must provide either a ‘foo‘ attribute or a ‘bar‘
    attribute.

This is a problem. After all, your AlsoValid class is also valid. It contains a foo attribute. An alternate approach to the FooOrBar metaclass is necessary.

class FooOrBar(type):
    def __new__(cls, name, bases, attrs):
        answer = super(FooOrBar, cls).__new__(cls, name, bases, attrs)
        if hasattr(answer, 'foo') and hasattr(answer, 'bar'):
            raise TypeError('Class %s cannot contain both `foo` and '
                            '`bar` attributes.' % name)
        if not hasattr(answer, 'foo') and not hasattr(answer, 'bar'):
            raise TypeError('Class %s must provide either a `foo` '
                            'attribute or a `bar` attribute.' % name)
        return answer

What is the difference here? This time, you are checking for the attributes on the instantiated class before it is returned, rather than looking at the attrs dictionary.

The new class will get all the attributes from the superclass as part of the call to type's constructor on the first line of the __new__ method. Therefore, the hasattr calls work, regardless of whether the attribute is declared on this class or inherited from a superclass.

Could this be handled without a metaclass? Absolutely. Nothing prevents writing a simple method that receives the class as an argument and does this same check. In fact, this is an excellent use for a decorator. However, the class must be manually sent to the verification method. With a metaclass, this is just handled when the class is created. Sometimes, an explicit opt-in is preferable; other times, it is not. It simply depends on the use case.

Non-Inheriting Attributes

Metaclasses can also be used as a tool to cause certain attributes of a class to not automatically inherit. The most common scenario in which you might want to do this is in conjunction with other metaclass behavior. For example, suppose that a metaclass provides functionality for its classes, but some classes will be created as abstract classes, and you do not want said functionality to run in this case.

An obvious way to go about this would be to allow the class to set an abstract attribute, and only perform the special functionality of the metaclass if its abstract is either not set or set to False.

class Meta(type):
    def __new__(cls, name, bases, attrs):
        # Sanity check: If this is an abstract class, then we do not
        # want the metaclass functionality here.
        if attrs.get('abstract', False):
            return super(Meta, cls).__new__(cls, name, bases, attrs)

        # Perform actual metaclass functionality.
        [...]

There is one problem with this approach, however. The abstract attribute, like any other attribute, will be inherited by subclasses. That means that any subclass would have to explicitly declare itself not to be abstract, which seems strange.

class AbstractClass(metaclass=Meta):
    abstract = True

class RegularClass(AbstractClass):
    abstract = False

Intuitively, you want abstract to have to be declared on all abstract classes, but for that attribute not to be inherited. It turns out that this is very easy, because instead of just reading the attrs dictionary like your metaclass is doing, it can modify it, disposing of the abstract attribute once it is no longer necessary.

In this case, you can do this by just popping the abstract value off of the attrs dictionary, as shown here:

class Meta(type):
    def __new__(cls, name, bases, attrs):
        # Sanity check: If this is an abstract class, then we do not
        # want the metaclass functionality here.
        if attrs.pop('abstract', False):
            return super(Meta, cls).__new__(cls, name, bases, attrs)

        # Perform actual metaclass functionality.
        [...]

The difference here is subtle, but important. The abstract attribute is being removed entirely from the actual class being created. In this example, AbstractClass would not get the metaclass functionality, but the actual abstract attribute would be gone. Most importantly, this means that subclasses do not inherit the attribute, which is exactly the behavior you want.