top

A Guide To Object Introspection in Python

IntroductionIn this post, we will be looking into Python's remarkable powers of introspection. Introspection is the ability of a program to examine its own structure and state, a process of looking inward to perform the self-examination. We’ll review and use some of the tools and bring them under the area of introspection.Why introspection and how it helps?In computer programming, the flexibility to look at one thing in depth and to examine what it holds and what it can do is known as introspection. In Python, it is one of its strength. Here, introspection helps in determining the type of an object at runtime. Since in Python everything is an object, it helps in leveraging the introspection properties to examine those objects. It provides us with some modules and a few built-in functions to achieve this. One should always use introspection because it gives programmers a great flexibility and management to examine the object with respect to its state and structure and this help us in using the important properties of the object and unnecessarily not instantiating the other unexamined objects. So, let us understand how to inspect objects and how could this benefit us.Object Types in DepthPerhaps the simplest introspective tool known is the built-in function type. We generally use this extensively for displaying object types. For example, if we create an integer i = 7, we can use type(i) to return its type int. Now, if we enter int and press return. This indicates that type int is just a representation of int as produced by repr, which is what the REPL does when displaying the result of an expression. We can confirm this by evaluating this expression repr(int). So, type 7 is actually returning an int, which is the class type of integers. We can even call the constructor on the return type directly by doing type(i)(78). But what type does type return? The type of type is the type. Every object in Python has an associated type object, which is retrieved using the type function.Using introspection we can see that type is itself an object because the type is a subclass of object. And the type of object is the type. What this circular dependency shows is that both type and object are fundamental to the Python object model, and neither can stand alone without the other. Also, the issubclass performs introspection. It answers a question about an object in the program as does the isinstance method. We should prefer to use the isinstance or issubclass functions rather than a direct comparison of type objects.Introspecting ObjectsAn important function for object introspection in Pythonis dir, which returns a list of attribute names for the instance. Given an object and an attribute name, we can retrieve the corresponding attribute object using the built-in getattr function.  Let's retrieve the denominator attribute using getattr. This returns the same value as accessing the denominator directly. Trying to retrieve an attribute that does not exist will result in an AttributeError. So, we determine whether a particular object has an attribute of a given name using the built-in hasattr function, which returns True if a particular attribute exists. For example, our integer i have an attribute bit_length but does not have an attribute index. The programs using hasattr get cluttered, particularly if we need to test for the existence of many different attributes, and perhaps using try/except is faster than using hasattr because internally hasattr uses an exception handler anyway. Here's a module, numerals.py, which given an object supporting the numerator and denominator attributes for rational numbers returns a so-called mixed numeral containing the separate whole number and fractional parts. For example, it will convert 17/3 into 5 2/3.At the beginning of the function, we use two calls to hasattr to check whether the supplied object supports the rational number interface. We get a TypeError when we try to pass a float because float supports neither numerator nor denominator. We can have the best version by using an exception handler to raise a more appropriate exception type of TypeError chaining to it the original AttributeError to provide the details. This approach yields the maximum amount of information about why and what when wrong. Now we see that the AttributeError was the direct cause of the TypeError.from fractions import Fraction def mixed_numeral(vulgar):     try:         integer = vulgar.numerator // vulgar.denominator         fraction = Fraction(vulgar.numerator - integer * vulgar.denominator,                             vulgar.denominator)         return integer, fraction     except AttributeError as e:         raise TypeError("{} is not a rational number".format(vulgar)) from e                                                            numerals.pyIntrospecting ScopesPython contains two built-in functions for examining the content of scopes. The first function is globals(). This returns a dictionary which represents the global namespace. Let’s define a variable a = 42 and call globals() again, and we can see that the binding of the name 'a' to the value of 42 has been added to the namespace. In fact, the dictionary returned by globals() is the global namespace. Let's create a variable tau and assign value 6.283185. We can now use this variable just like any other variables. The second function is locals().To really see locals() in action, we're are going to create another local scope, which we can do by defining a function that accepts a single argument, defines a variable X to have a value of 496, and then prints the locals() dictionary with a width of 10 characters. When run, we see that this function has the expected three entries in its local namespace. By using locals() to provide the dictionary, we can easily refer to local variables in format strings.Duck Tail: An Object Introspection ToolWe'll now build a tool to introspect objects by leveraging the interesting techniques we've discussed and bring them together into a useful program. Our objective is to create a function, which when passed a single object prints out a nicely formatted dump of that object's attributes with rather more clarity. This is a small tool that we are going to create and the main use case of this tool is to help us identify a different aspect of the objects with respect to its type, methods, attributes, and documentation. It will help us in getting more clarity on the object.import inspect import reprlib import itertools from sorted_set import SortedSet def full_sig(method):     try:         return method.__name__ + inspect.signature(method)     except ValueError:         return method.__name__ + '(...)' def brief_doc(obj):     doc = obj.__doc__     if doc is not None:         lines = doc.splitlines()         if len(lines) > 0:             return lines[0]     return '' def print_table(rows_of_columns, *headers):     num_columns = len(rows_of_columns[0])     num_headers = len(headers)     if len(headers) != num_columns:         raise TypeError("Expected {} header arguments, "                         "got {}".format(num_columns, num_headers))     rows_of_columns_with_header = itertools.chain([headers], rows_of_columns)     columns_of_rows = list(zip(*rows_of_columns_with_header))     column_widths = [max(map(len, column)) for column in columns_of_rows]     column_specs = ('{{:{w}}}'.format(w=width) for width in column_widths)     format_spec = ' '.join(column_specs)     print(format_spec.format(*headers))     rules = ('-' * width for width in column_widths)     print(format_spec.format(*rules))     for row in rows_of_columns:         print(format_spec.format(*row)) def dump(obj):     print("Type")     print("====")     print(type(obj))     print()     print("Documentation")     print("=============")     print(inspect.getdoc(obj))     print()     print("Attributes")     print("==========")     all_attr_names = SortedSet(dir(obj))     method_names = SortedSet(         filter(lambda attr_name: callable(getattr(obj, attr_name)),                all_attr_names))     assert method_names <= all_attr_names     attr_names = all_attr_names - method_names     attr_names_and_values = [(name, reprlib.repr(getattr(obj, name)))                              for name in attr_names]     print_table(attr_names_and_values, "Name", "Value")     print()     print("Methods")     print("=======")     methods = (getattr(obj, method_name) for method_name in method_names)     method_names_and_doc = [(full_sig(method), brief_doc(method))                             for method in methods]     print_table(method_names_and_doc, "Name", "Description")     print()                                                                introspector.pyLet's start by creating a module, intropector.py, and defining a dump method that prints the outline of the object dump. We print four section headings. For Type, the only object detail is retrieved using the type built-in function. In the details of the documentation section, we use a function getdoc from the inspect module that will combine the operations of retrieving and tidying up the docstring. Now for the attributes and methods. For this, we need to produce a list of attributes and their values. We have an opportunity to apply some of the techniques we've picked up. We’ll start by list of attributes using the built-in dir. We'll put the attribute names into one of the SortedSet collections. Now we'll produce another SortedSet of all method_names by determining which of those attributes when retrieved getattr is callable. And use the filter built-in function to select those attributes which are callable, and our predicate which acts on the attribute name will be a lambda function.from bisect import bisect_left from collections.abc import Sequence, Set from itertools import chain class SortedSet(Sequence, Set):     def __init__(self, items=None):         self._items = sorted(set(items)) if items is not None else []     def __contains__(self, item):         try:             self.index(item)             return True         except ValueError:             return False     def __len__(self):         return len(self._items)     def __iter__(self):         return iter(self._items)     def __getitem__(self, index):         result = self._items[index]         return SortedSet(result) if isinstance(index, slice) else result     def __repr__(self):         return "SortedSet({})".format(             repr(self._items) if self._items else ''         )     def __eq__(self, rhs):         if not isinstance(rhs, SortedSet):             return NotImplemented         return self._items == rhs._items     def __ne__(self, rhs):         if not isinstance(rhs, SortedSet):             return NotImplemented         return self._items != rhs._items     def _is_unique_and_sorted(self):         return all(self[i] < self[i + 1] for i in range(len(self) - 1))     def index(self, item):         assert self._is_unique_and_sorted()         index = bisect_left(self._items, item)         if (index != len(self._items)) and (self._items[index] == item):             return index         raise ValueError("{} not found".format(repr(item)))     def count(self, item):         assert self._is_unique_and_sorted()         return int(item in self)     def __add__(self, rhs):         return SortedSet(chain(self._items, rhs._items))     def __mul__(self, rhs):         return self if rhs > 0 else SortedSet()     def __rmul__(self, lhs):         return self * lhs     def issubset(self, iterable):         return self <= SortedSet(iterable)     def issuperset(self, iterable):         return self >= SortedSet(iterable)     def intersection(self, iterable):         return self & SortedSet(iterable)     def union(self, iterable):         return self | SortedSet(iterable)     def symmetric_difference(self, iterable):         return self ^ SortedSet(iterable)     def difference(self, iterable):         return self - SortedSet(iterable)                                                       sorted_set.py An attribute value could potentially have a huge text representation. So, we’ll print the attributes and their values by using the Python Standard Library reprlib module to cut the values down to a reasonable size in an intelligent way. Then we'll print a nice table of names and values using a print_table function. The function will need to accept a sequence of sequences representing rows and columns and the requisite number of column headers as strings. We now move onto methods. From our list of method names, we'll generate a series of method objects simply by retrieving each one with getattr. Then for each method object, we'll build a full method signature and a brief documentation string using functions and then, print the table of methods using our print_table function. We now implement the three functions full_sig, brief_doc, and print_table that are required by the dump function. So that completes our code and the dump function and we test if the function works.  We observe that using introspection we can get very comprehensive information even for a very simple object like the integer digit 7.ConclusionThere is various introspection that can be used in Python. We developed an object introspection tool, which uses all the techniques that we have learned, be it the type of objects using the type function, or the using the dir function, the getattr, and hasattr functions. This would be very great in getting deep into Python objects and performing introspection with more finesse. I hope this guide to Python introspection provides a great help and with more practice, we could discover how objects behave in Python.This guide to Python introspection will take you through the Python's remarkable powers of introspection. Deep dive into Python objects and  performing introspection with more finesse.
Rated 3.0/5 based on 11 customer reviews
Normal Mode Dark Mode

A Guide To Object Introspection in Python

Rohit Goyal
Blog
28th Sep, 2018
A Guide To Object Introspection in Python

Introduction

In this post, we will be looking into Python's remarkable powers of introspection. Introspection is the ability of a program to examine its own structure and state, a process of looking inward to perform the self-examination. We’ll review and use some of the tools and bring them under the area of introspection.


Why introspection and how it helps?

In computer programming, the flexibility to look at one thing in depth and to examine what it holds and what it can do is known as introspection. In Python, it is one of its strength. Here, introspection helps in determining the type of an object at runtime. Since in Python everything is an object, it helps in leveraging the introspection properties to examine those objects. It provides us with some modules and a few built-in functions to achieve this. One should always use introspection because it gives programmers a great flexibility and management to examine the object with respect to its state and structure and this help us in using the important properties of the object and unnecessarily not instantiating the other unexamined objects. So, let us understand how to inspect objects and how could this benefit us.


Object Types in Depth

Perhaps the simplest introspective tool known is the built-in function type. We generally use this extensively for displaying object types. For example, if we create an integer i = 7, we can use type(i) to return its type int. Now, if we enter int and press return. This indicates that type int is just a representation of int as produced by repr, which is what the REPL does when displaying the result of an expression. We can confirm this by evaluating this expression repr(int). So, type 7 is actually returning an int, which is the class type of integers. We can even call the constructor on the return type directly by doing type(i)(78). But what type does type return? The type of type is the type. Every object in Python has an associated type object, which is retrieved using the type function.

Using introspection we can see that type is itself an object because the type is a subclass of object. And the type of object is the type. What this circular dependency shows is that both type and object are fundamental to the Python object model, and neither can stand alone without the other. Also, the issubclass performs introspection. It answers a question about an object in the program as does the isinstance method. We should prefer to use the isinstance or issubclass functions rather than a direct comparison of type objects.


Introspecting Objects

An important function for object introspection in Pythonis dir, which returns a list of attribute names for the instance. Given an object and an attribute name, we can retrieve the corresponding attribute object using the built-in getattr function.  Let's retrieve the denominator attribute using getattr. This returns the same value as accessing the denominator directly. Trying to retrieve an attribute that does not exist will result in an AttributeError. So, we determine whether a particular object has an attribute of a given name using the built-in hasattr function, which returns True if a particular attribute exists. For example, our integer i have an attribute bit_length but does not have an attribute index. The programs using hasattr get cluttered, particularly if we need to test for the existence of many different attributes, and perhaps using try/except is faster than using hasattr because internally hasattr uses an exception handler anyway. Here's a module, numerals.py, which given an object supporting the numerator and denominator attributes for rational numbers returns a so-called mixed numeral containing the separate whole number and fractional parts. For example, it will convert 17/3 into 5 2/3.

At the beginning of the function, we use two calls to hasattr to check whether the supplied object supports the rational number interface. We get a TypeError when we try to pass a float because float supports neither numerator nor denominator. We can have the best version by using an exception handler to raise a more appropriate exception type of TypeError chaining to it the original AttributeError to provide the details. This approach yields the maximum amount of information about why and what when wrong. Now we see that the AttributeError was the direct cause of the TypeError.

from fractions import Fraction


def mixed_numeral(vulgar):
    try:
        integer = vulgar.numerator // vulgar.denominator
        fraction = Fraction(vulgar.numerator - integer * vulgar.denominator,
                            vulgar.denominator)
        return integer, fraction
    except AttributeError as e:
        raise TypeError("{} is not a rational number".format(vulgar)) from e

                                                           numerals.py


Introspecting Scopes

Python contains two built-in functions for examining the content of scopes. The first function is globals(). This returns a dictionary which represents the global namespace. Let’s define a variable a = 42 and call globals() again, and we can see that the binding of the name 'a' to the value of 42 has been added to the namespace. In fact, the dictionary returned by globals() is the global namespace. Let's create a variable tau and assign value 6.283185. We can now use this variable just like any other variables. The second function is locals().

To really see locals() in action, we're are going to create another local scope, which we can do by defining a function that accepts a single argument, defines a variable X to have a value of 496, and then prints the locals() dictionary with a width of 10 characters. When run, we see that this function has the expected three entries in its local namespace. By using locals() to provide the dictionary, we can easily refer to local variables in format strings.

Duck Tail: An Object Introspection Tool

We'll now build a tool to introspect objects by leveraging the interesting techniques we've discussed and bring them together into a useful program. Our objective is to create a function, which when passed a single object prints out a nicely formatted dump of that object's attributes with rather more clarity. This is a small tool that we are going to create and the main use case of this tool is to help us identify a different aspect of the objects with respect to its type, methods, attributes, and documentation. It will help us in getting more clarity on the object.

import inspect
import reprlib
import itertools
from sorted_set import SortedSet


def full_sig(method):
    try:
        return method.__name__ + inspect.signature(method)
    except ValueError:
        return method.__name__ + '(...)'


def brief_doc(obj):
    doc = obj.__doc__
    if doc is not None:
        lines = doc.splitlines()
        if len(lines) > 0:
            return lines[0]
    return ''


def print_table(rows_of_columns, *headers):
    num_columns = len(rows_of_columns[0])
    num_headers = len(headers)
    if len(headers) != num_columns:
        raise TypeError("Expected {} header arguments, "
                        "got {}".format(num_columns, num_headers))
    rows_of_columns_with_header = itertools.chain([headers], rows_of_columns)
    columns_of_rows = list(zip(*rows_of_columns_with_header))
    column_widths = [max(map(len, column)) for column in columns_of_rows]
    column_specs = ('{{:{w}}}'.format(w=width) for width in column_widths)
    format_spec = ' '.join(column_specs)
    print(format_spec.format(*headers))
    rules = ('-' * width for width in column_widths)
    print(format_spec.format(*rules))
    for row in rows_of_columns:
        print(format_spec.format(*row))


def dump(obj):
    print("Type")
    print("====")
    print(type(obj))
    print()

    print("Documentation")
    print("=============")
    print(inspect.getdoc(obj))
    print()

    print("Attributes")
    print("==========")
    all_attr_names = SortedSet(dir(obj))
    method_names = SortedSet(
        filter(lambda attr_name: callable(getattr(obj, attr_name)),
               all_attr_names))
    assert method_names <= all_attr_names
    attr_names = all_attr_names - method_names
    attr_names_and_values = [(name, reprlib.repr(getattr(obj, name)))
                             for name in attr_names]
    print_table(attr_names_and_values, "Name", "Value")
    print()

    print("Methods")
    print("=======")
    methods = (getattr(obj, method_name) for method_name in method_names)
    method_names_and_doc = [(full_sig(method), brief_doc(method))
                            for method in methods]
    print_table(method_names_and_doc, "Name", "Description")
    print()

                                                                introspector.py

Let's start by creating a module, intropector.py, and defining a dump method that prints the outline of the object dump. We print four section headings. For Type, the only object detail is retrieved using the type built-in function. In the details of the documentation section, we use a function getdoc from the inspect module that will combine the operations of retrieving and tidying up the docstring. Now for the attributes and methods. For this, we need to produce a list of attributes and their values. We have an opportunity to apply some of the techniques we've picked up. We’ll start by list of attributes using the built-in dir. We'll put the attribute names into one of the SortedSet collections. Now we'll produce another SortedSet of all method_names by determining which of those attributes when retrieved getattr is callable. And use the filter built-in function to select those attributes which are callable, and our predicate which acts on the attribute name will be a lambda function.

from bisect import bisect_left
from collections.abc import Sequence, Set
from itertools import chain


class SortedSet(Sequence, Set):

    def __init__(self, items=None):
        self._items = sorted(set(items)) if items is not None else []

    def __contains__(self, item):
        try:
            self.index(item)
            return True
        except ValueError:
            return False

    def __len__(self):
        return len(self._items)

    def __iter__(self):
        return iter(self._items)

    def __getitem__(self, index):
        result = self._items[index]
        return SortedSet(result) if isinstance(index, slice) else result

    def __repr__(self):
        return "SortedSet({})".format(
            repr(self._items) if self._items else ''
        )

    def __eq__(self, rhs):
        if not isinstance(rhs, SortedSet):
            return NotImplemented
        return self._items == rhs._items

    def __ne__(self, rhs):
        if not isinstance(rhs, SortedSet):
            return NotImplemented
        return self._items != rhs._items

    def _is_unique_and_sorted(self):
        return all(self[i] < self[i + 1] for i in range(len(self) - 1))

    def index(self, item):
        assert self._is_unique_and_sorted()
        index = bisect_left(self._items, item)
        if (index != len(self._items)) and (self._items[index] == item):
            return index
        raise ValueError("{} not found".format(repr(item)))

    def count(self, item):
        assert self._is_unique_and_sorted()
        return int(item in self)

    def __add__(self, rhs):
        return SortedSet(chain(self._items, rhs._items))

    def __mul__(self, rhs):
        return self if rhs > 0 else SortedSet()

    def __rmul__(self, lhs):
        return self * lhs

    def issubset(self, iterable):
        return self <= SortedSet(iterable)

    def issuperset(self, iterable):
        return self >= SortedSet(iterable)

    def intersection(self, iterable):
        return self & SortedSet(iterable)

    def union(self, iterable):
        return self | SortedSet(iterable)

    def symmetric_difference(self, iterable):
        return self ^ SortedSet(iterable)

    def difference(self, iterable):
        return self - SortedSet(iterable)


                                                      sorted_set.py 


An attribute value could potentially have a huge text representation. So, we’ll print the attributes and their values by using the Python Standard Library reprlib module to cut the values down to a reasonable size in an intelligent way. Then we'll print a nice table of names and values using a print_table function. The function will need to accept a sequence of sequences representing rows and columns and the requisite number of column headers as strings. We now move onto methods. From our list of method names, we'll generate a series of method objects simply by retrieving each one with getattr. Then for each method object, we'll build a full method signature and a brief documentation string using functions and then, print the table of methods using our print_table function. We now implement the three functions full_sig, brief_doc, and print_table that are required by the dump function. So that completes our code and the dump function and we test if the function works.  We observe that using introspection we can get very comprehensive information even for a very simple object like the integer digit 7.


Conclusion

There is various introspection that can be used in Python. We developed an object introspection tool, which uses all the techniques that we have learned, be it the type of objects using the type function, or the using the dir function, the getattr, and hasattr functions. This would be very great in getting deep into Python objects and performing introspection with more finesse. I hope this guide to Python introspection provides a great help and with more practice, we could discover how objects behave in Python.

This guide to Python introspection will take you through the Python's remarkable powers of introspection. Deep dive into Python objects and  performing introspection with more finesse.

Rohit

Rohit Goyal

Blog Author

Rohit is a Full Stack Developer and Programmer by profession, passionate about technologies and is constantly looking for better ways to write code. When not making loud noises on his mechanical keyboard, he can be found reading books or playing music.

Leave a Reply

Your email address will not be published. Required fields are marked *

Top comments

Charles

16 November 2018 at 4:14pm
I enjoy what you guys are usually up too. This sort of clever work and reporting! Keep up the good works.

SUBSCRIBE OUR BLOG

Follow Us On

Share on

other Blogs

20% Discount