top

The Truth About Context Managers In Python

Introduction - Why understanding context managers is important? If you are a Python programmer and if it is one of those things that you use without knowing what exactly it does and how it works then you are one among millions many. Don’t worry we’ll help you out. Context managers enable you to assign and release resources exactly when needed. The foremost wide use of context managers is that the with statement. So, what exactly is a context manager? Simply put, a context manager is an object that is designed to be used in a "with” statement. When a with statement is executed, the expression part of the statement, i.e.  the part following the with keyword, evaluates to a value. This value must be a context manager, and the underlying mechanics of the with statement uses this value in specific ways to implement the semantics of the with statement. Lets Implement Context Manager Conceptually, a context manager implements two methods, which are used by the with statement. The first method is called before the with statement's code block begins, and the second method is called when the with statement's code block ends even if the block exits with an exception.You can think of these operations as setup() and teardown(), construction() and destruction(), resource allocation() and deallocation() and such similar process functions. These methods of the context manager are __enter()__ and __exit()__ and are known popularly as dunder methods, as they are surrounded by double underscores. Hence, they are also called dunder-enter and dunder-exit respectively. Both the enter() and exit() methods are called every time the with statement is executed no matter how the code block terminates. So, basically a context manager ensures that resources are properly and automatically managed around the code that uses these resources The enter() method of a context manager ensures that the object is ready for use in the with block, and the exit() method ensures that the object is properly closed, shut down, or cleaned up when the block ends. For example let’s consider the files in Python .The benefit of using files in a with statement is that they are automatically closed at the end of the with block. This works because files are context managers, that is they have methods which are called by the with statement before the block is started and after the block exits. The exit() method for a file, that is the code executed after the with block exits, does the work of closing the file, and this is how files work with with statements to ensure proper resource management. The Context Manager Protocol For an object to be a context manager, it needs to support the context manager protocol, which consists of only two methods, __enter__ and __exit__ associated with a with statement. The first thing a with statement does is execute its expression i.e. the code immediately following the with keyword. This expression must evaluate to a context manager, it must produce an object which supports both the __enter__ and __exit__ methods. Once the expression is evaluated and we have a context manager object, the with statement then calls __enter__ on that context manager with no arguments. If __enter__ throws an exception, execution never enters the with block, and the with statement is done. If the __enter__ executes successfully, it returns a value. If the with statement includes an as clause, this return value is bound to the name in the as clause; otherwise, this return value is discarded. A simple interpretation of a with statement states that the result of the with statement expression is the one bound to the as variable while it is in fact the return value of the context manager's __enter__ that is bound to the as variable. Once __enter__ has been executed and its return value is potentially bound to the variable. The with block can now terminate in one of two fundamental ways, with an exception or by normal termination i.e. by running off the end of the block. In both cases the context manager's __exit__ method is called after the block. If the block exits normally, __exit__ is called with no extra information. If on the other hand the block exits exceptionally, then the exception information is passed to __exit__. __exit__() method and Handling Exceptions __exit__ is substantially more complex than __enter__ because it performs several roles in the execution of a with statement. Its first and primary role of course is that it is executed after the with block terminates, so it is responsible for cleaning up whatever resources the context manager controls. Its second role is to properly handle the case where the with block exits with an exception. To handle this case, __exit__ accepts three arguments: The type of the exception that was thrown, the value of the exception, that is the actual exception instance, and the traceback associated with the exception. When a with block exits without an exception, all three of these arguments are set to None, but when it exits exceptionally these arguments are bound to the exception, which terminated the block. If the type is None, then this means that no exception was thrown and a simple message is printed. Else the __exit__ prints a longer message, which includes the exception information. If __exit__ returns a value which evaluates to False in a Boolean context, then any exception that came out of the with block will be re-raised after the __exit__ call. You can think of it as if the with statement is asking __exit__ should I swallow the exception. If __exit__ says False, then the with statement re-raises the exception. If __exit__ says True, then the with statement will exit normally, that is without raising the exception. If your context manager's __exit__ function doesn't return anything, then exceptions would be propagated. This is because, a method that doesn't explicitly return a value will implicitly return None. Of course None evaluates to False, so a __exit__ which doesn't return anything is instructing the with statement to propagate exceptions. Lastly, __exit__ should only explicitly raise exceptions when it actually fails itself, that is when something goes wrong in the __exit__ method. The with statement machinery will interpret exceptions from __exit__ as a failure of __exit__, not as a propagation of the original exception. The contextlib.contextmanager module We can also implement Context Managers using decorators and generators and simplify context manager development. Python has a contextlib module for this very purpose which provides utilities for common tasks involving the with statement. Instead of a class, we can implement a Context Manager using a generator function.  Contextlib's contextmanager decorator is a very useful tool. The contextmanager decorator is, as its name suggests, a decorator that you can use to create new context managers. The concept behind context manager is simple. You define a generator, that is a function which uses yield instead of return and decorator it with the contextmanager decorator to create a context manager factory. This factory is nothing more than a callable object which returns context managers making it suitable for use in a with statement. The contextmanager decorator allows you to define context managers using normal control flow via the yield statement rather than breaking it up across two methods. Furthermore, since generators remember their state between calls to yield you don't need to define a new class just to create a stateful context manager. The new context manager works good for both normal exit and an exceptional exit. It doesn’t propagate the ValueError after it completed. Unlike standard context managers, those created with the contextmanager decorator must use normal exception handling to determine if exceptions are propagated from the with statement. So basically, the contextlib.contextmanager is a decorator used for creating context manager factories out of generator functions. A context manager generator yields a value, which will be bound to the name in the optional as clause. Execution moves from the context manager generator to the “with” block when the yield statement is executed. Execution returns from the with block to the yield call when the with block terminates. All of the code in a context manager generator before the yield is equivalent to the __enter__ method. If an exception is raised in the with block, it is re-raised at the yield statement in the context manager generator. The code executed after the yield in a context manager generator is equivalent to the __exit__ method. If a context manager generator wants to propagate an exception, it needs to explicitly re-raise it. Hopefully, now you could easily comprehend what a context manager is, how it works, and why it's helpful. As you may have noticed, there are a lot of useful things you can do with context managers. Their goal is to make working with resources and creating managed contexts easier. Now it's up to you to not solely use them, however produce new ones as well, thus making other people's lives easier.
Rated 3.5/5 based on 0 customer reviews
Normal Mode Dark Mode

The Truth About Context Managers In Python

Rohit Goyal
Blog
31st Jan, 2018
The Truth About Context Managers In Python

Introduction - Why understanding context managers is important?

If you are a Python programmer and if it is one of those things that you use without knowing what exactly it does and how it works then you are one among millions many. Don’t worry we’ll help you out.

Context managers enable you to assign and release resources exactly when needed. The foremost wide use of context managers is that the with statement.

So, what exactly is a context manager? Simply put, a context manager is an object that is designed to be used in a "with” statement. When a with statement is executed, the expression part of the statement, i.e.  the part following the with keyword, evaluates to a value. This value must be a context manager, and the underlying mechanics of the with statement uses this value in specific ways to implement the semantics of the with statement.

Lets Implement Context Manager

Conceptually, a context manager implements two methods, which are used by the with statement. The first method is called before the with statement's code block begins, and the second method is called when the with statement's code block ends even if the block exits with an exception.You can think of these operations as setup() and teardown(), construction() and destruction(), resource allocation() and deallocation() and such similar process functions.

These methods of the context manager are __enter()__ and __exit()__ and are known popularly as dunder methods, as they are surrounded by double underscores. Hence, they are also called dunder-enter and dunder-exit respectively.

Both the enter() and exit() methods are called every time the with statement is executed no matter how the code block terminates. So, basically a context manager ensures that resources are properly and automatically managed around the code that uses these resources

The enter() method of a context manager ensures that the object is ready for use in the with block, and the exit() method ensures that the object is properly closed, shut down, or cleaned up when the block ends.

For example let’s consider the files in Python .The benefit of using files in a with statement is that they are automatically closed at the end of the with block. This works because files are context managers, that is they have methods which are called by the with statement before the block is started and after the block exits. The exit() method for a file, that is the code executed after the with block exits, does the work of closing the file, and this is how files work with with statements to ensure proper resource management.

The Context Manager Protocol

For an object to be a context manager, it needs to support the context manager protocol, which consists of only two methods, __enter__ and __exit__ associated with a with statement.

The first thing a with statement does is execute its expression i.e. the code immediately following the with keyword. This expression must evaluate to a context manager, it must produce an object which supports both the __enter__ and __exit__ methods.

Once the expression is evaluated and we have a context manager object, the with statement then calls __enter__ on that context manager with no arguments. If __enter__ throws an exception, execution never enters the with block, and the with statement is done.

If the __enter__ executes successfully, it returns a value. If the with statement includes an as clause, this return value is bound to the name in the as clause; otherwise, this return value is discarded.

A simple interpretation of a with statement states that the result of the with statement expression is the one bound to the as variable while it is in fact the return value of the context manager's __enter__ that is bound to the as variable.

Once __enter__ has been executed and its return value is potentially bound to the variable. The with block can now terminate in one of two fundamental ways, with an exception or by normal termination i.e. by running off the end of the block. In both cases the context manager's __exit__ method is called after the block. If the block exits normally, __exit__ is called with no extra information. If on the other hand the block exits exceptionally, then the exception information is passed to __exit__.

__exit__() method and Handling Exceptions

__exit__ is substantially more complex than __enter__ because it performs several roles in the execution of a with statement. Its first and primary role of course is that it is executed after the with block terminates, so it is responsible for cleaning up whatever resources the context manager controls. Its second role is to properly handle the case where the with block exits with an exception. To handle this case, __exit__ accepts three arguments: The type of the exception that was thrown, the value of the exception, that is the actual exception instance, and the traceback associated with the exception. When a with block exits without an exception, all three of these arguments are set to None, but when it exits exceptionally these arguments are bound to the exception, which terminated the block.

If the type is None, then this means that no exception was thrown and a simple message is printed. Else the __exit__ prints a longer message, which includes the exception information.

If __exit__ returns a value which evaluates to False in a Boolean context, then any exception that came out of the with block will be re-raised after the __exit__ call.

You can think of it as if the with statement is asking __exit__ should I swallow the exception. If __exit__ says False, then the with statement re-raises the exception. If __exit__ says True, then the with statement will exit normally, that is without raising the exception.

If your context manager's __exit__ function doesn't return anything, then exceptions would be propagated. This is because, a method that doesn't explicitly return a value will implicitly return None. Of course None evaluates to False, so a __exit__ which doesn't return anything is instructing the with statement to propagate exceptions.

Lastly, __exit__ should only explicitly raise exceptions when it actually fails itself, that is when something goes wrong in the __exit__ method. The with statement machinery will interpret exceptions from __exit__ as a failure of __exit__, not as a propagation of the original exception.

The contextlib.contextmanager module

We can also implement Context Managers using decorators and generators and simplify context manager development. Python has a contextlib module for this very purpose which provides utilities for common tasks involving the with statement. Instead of a class, we can implement a Context Manager using a generator function. 

Contextlib's contextmanager decorator is a very useful tool. The contextmanager decorator is, as its name suggests, a decorator that you can use to create new context managers.

The concept behind context manager is simple. You define a generator, that is a function which uses yield instead of return and decorator it with the contextmanager decorator to create a context manager factory. This factory is nothing more than a callable object which returns context managers making it suitable for use in a with statement.

The contextmanager decorator allows you to define context managers using normal control flow via the yield statement rather than breaking it up across two methods. Furthermore, since generators remember their state between calls to yield you don't need to define a new class just to create a stateful context manager.

The new context manager works good for both normal exit and an exceptional exit. It doesn’t propagate the ValueError after it completed. Unlike standard context managers, those created with the contextmanager decorator must use normal exception handling to determine if exceptions are propagated from the with statement.

So basically, the contextlib.contextmanager is a decorator used for creating context manager factories out of generator functions. A context manager generator yields a value, which will be bound to the name in the optional as clause. Execution moves from the context manager generator to the “with” block when the yield statement is executed. Execution returns from the with block to the yield call when the with block terminates. All of the code in a context manager generator before the yield is equivalent to the __enter__ method. If an exception is raised in the with block, it is re-raised at the yield statement in the context manager generator. The code executed after the yield in a context manager generator is equivalent to the __exit__ method. If a context manager generator wants to propagate an exception, it needs to explicitly re-raise it.

Hopefully, now you could easily comprehend what a context manager is, how it works, and why it's helpful. As you may have noticed, there are a lot of useful things you can do with context managers. Their goal is to make working with resources and creating managed contexts easier. Now it's up to you to not solely use them, however produce new ones as well, thus making other people's lives easier.

Rohit

Rohit Goyal

Blog Author

Rohit is a Full Stack Developer and Programmer by profession, passionate about technologies and is constantly looking for better ways to write code. When not making loud noises on his mechanical keyboard, he can be found reading books or playing music.

Leave a Reply

Your email address will not be published. Required fields are marked *

Top comments

Twinkal Khanna

13 November 2018 at 10:29am
Very Impressive Python tutorial. The content seems to be pretty exhaustive and excellent and will definitely help in learning Python. I'm also a learner taken up Python training and I think your content has cleared some concepts of mine. While browsing for Python tutorials on YouTube i found this fantastic video on Python. Do check it out if you are interested to know more.:-https://www.youtube.com/watch?v=qgOXopu4n7c&

SUBSCRIBE OUR BLOG

Follow Us On

Share on

other Blogs

20% Discount