Python Context Manager Class | Writing Good Apis With Context Managers

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Context Managers and Python’s with Statement

The

with

statement in Python is a quite useful tool for properly managing external resources in your programs. It allows you to take advantage of existing context managers to automatically handle the setup and teardown phases whenever you’re dealing with external resources or with operations that require those phases.

Besides, the context management protocol allows you to create your own context managers so you can customize the way you deal with system resources. So, what’s the

with

statement good for?

In this tutorial, you’ll learn:

• What the Python
  
with

  statement is for and how to use it
• What the context management protocol is
• How to implement your own context managers

With this knowledge, you’ll write more expressive code and avoid resource leaks in your programs. The

with

statement helps you implement some common resource management patterns by abstracting their functionality and allowing them to be factored out and reused.

Free Download: Get a sample chapter from Python Tricks: The Book that shows you Python’s best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Take the Quiz: Test your knowledge with our interactive “Context Managers and Python’s with Statement” quiz. Upon completion you will receive a score so you can track your learning progress over time:

What is a Context Manager in Python?

According to the Python glossary, a context manager is —

An object which controls the environment seen in a

with

statement by defining

__enter__()

and

__exit__()

methods.

That may not be noticeably clear to you. Let me explain the concept with an example.

The

with

statement in Python lets you run a block of code within a runtime context defined by a context manager object.

Once the block of code has finished executing, the context manager object will take care of tearing down any external resources that are no longer needed.

You can rewrite the program by using the

with

statement as follows:


def main(): with open('books.txt', 'w') as my_file: my_file.write('If Tomorrow Comes by Sidney Sheldon') if __name__ == '__main__': main()


Since the

open()

function is paired with a

with

statement in this example, the function will create a context manager.

The file object will be accessible within the context of the indented code block, which means the file object doesn’t exist outside of that scope.

The

as

keyword is useful when you want to assign a target variable to a returned object. Here, the

my_file

variable is the target and will hold the file object.

You can do whatever you want within the indented block of code and don’t have to worry about closing the file.

Because once the block of code has finished executing the context manager will close the file automatically.

So, you have rewritten the entire

try...except...finally

ladder within two lines of code using the

with

statement and a context manager.

But how does that happen? How does a context manager object handle the task of setting up and closing resources?

And where are those

__enter__()

and

__exit__()

methods you read about on the Python documentation glossary?

Well, I’m so glad you asked 🙂

Summary

A context manager, used in a

with

statement, defines a temporary context for the given set of operations. It does so by injecting code at the beginning and at the end of the code block, setting up the context at the beginning, and tearing it down at the end.

One of the most common tasks that you’ll have to perform in your programs is working with external resources. These resources can be files on your computer’s storage or an open connection to third-party service on the internet.

For the sake of simplicity, imagine a program that opens a file, writes something to it, and then closes the file.

One way to implement this program in Python would be like this:


def main(): my_file = open('books.txt', 'w') my_file.write('If Tomorrow Comes by Sidney Sheldon') my_file.close() if __name__ == '__main__': main()


Given that you run this program with the right permissions on your computer, it’ll create a file called

books.txt

and write

If Tomorrow Comes by Sidney Sheldon

in it.

The

open()

function is one of the built-in functions in Python. It can open a file from a given path and return a corresponding file object.

A file object or file-like object, as it’s often called, is a useful way to encapsulate methods like

read()

,

write()

, or

close()

.

The

write()

method can be used write/send bytes-like object to an open stream, like a file.

Whenever you open an external resource, you must close it when its no longer needed, and the

close()

method does just that.

This program is functional, but it has a big flaw. If the program fails to close the file, it will remain open until the program itself closes.

You see, every program that you run on your computer gets a finite amount of memory allocated to it. All the variables you create or external resource you open from a program stay within the memory allocated to it by your computer.

If a program like this one, keeps opening new files without closing the previous ones, the allocated memory will keep shrinking.

At one point the program will inevitably run out of memory and crash ungracefully. This problem is referred to as a memory leak.

One way to prevent this from happening in Python is using a

try...except...finally

statement.


def main(): my_file = open('books.txt', 'w') try: my_file.write('If Tomorrow Comes by Sidney Sheldon') except Exception as e: print(f'writing to file failed: {e}') finally: my_file.close() if __name__ == '__main__': main()


The code inside the

finally

block will run no matter what. So even if the program fails on the right action, it’ll still be executed.

So, this solves the problem but imagine writing these lines of code every time you want to write something to a file.

It’s not very reusable. You will have to repeat yourself a lot and chances of skipping a portion of the

if...except...finally

ladder is also a possibility.

That’s where context managers come in.

27.Implementing a Context Manager as a Class:¶

At the very least a context manager has an

__enter__

and

__exit__

method defined. Let’s make our own file-opening Context
Manager and learn the basics.

class File(object): def __init__(self, file_name, method): self.file_obj = open(file_name, method) def __enter__(self): return self.file_obj def __exit__(self, type, value, traceback): self.file_obj.close()

Just by defining

__enter__

and

__exit__

methods we can use our new class in
a

with

statement. Let’s try:

with File(‘demo.txt’, ‘w’) as opened_file: opened_file.write(‘Hola!’)

Our

__exit__

method accepts three arguments. They are required by
every

__exit__

method which is a part of a Context Manager class.
Let’s talk about what happens under-the-hood.

1. The
   
with

   statement stores the
   
__exit__

   method of the
   
File

   class.
2. It calls the
   
__enter__

   method of the
   
File

   class.
3. The
   
__enter__

   method opens the file and returns it.
4. The opened file handle is passed to
   
opened_file

   .
5. We write to the file using
   
.write()

   .
6. The
   
with

   statement calls the stored
   
__exit__

   method.
7. The
   
__exit__

   method closes the file.

Using the async with Statement

The

with

statement also has an asynchronous version,

async with

. You can use it to write context managers that depend on asynchronous code. It’s quite common to see

async with

in that kind of code, as many IO operations involve setup and teardown phases.

For example, say you need to code an asynchronous function to check if a given site is online. To do that, you can use

aiohttp

,

asyncio

, and

async with

like this:


1# site_checker_v0.py 2 3import aiohttp 4import asyncio 5 6async def check(url): 7 async with aiohttp.ClientSession() as session: 8 async with session.get(url) as response: 9 print(f"{url}: status -> {response.status}") 10 html = await response.text() 11 print(f"{url}: type -> {html[:17].strip()}") 12 13async def main(): 14 await asyncio.gather( 15 check("https://realpython.com"), 16 check("https://pycoders.com"), 17 ) 18 19asyncio.run(main())


Here’s what this script does:

• Line 3 imports
  
aiohttp

  , which provides an asynchronous HTTP client and server for
  
asyncio

  and Python. Note that
  
aiohttp

  is a third-party package that you can install by running
  
python -m pip install aiohttp

  on your command line.
• Line 4 imports
  
asyncio

  , which allows you to write concurrent code using the
  
async

  and
  
await

  syntax.
• Line 6 defines
  
check()

  as an asynchronous function using the
  
async

  keyword.

Inside

check()

, you define two nested

async with

statements:

• Line 7 defines an outer
  
async with

  that instantiates
  
aiohttp.ClientSession()

  to get a context manager. It stores the returned object in
  
session

  .
• Line 8 defines an inner
  
async with

  statement that calls
  
.get()

  on
  
session

  using
  
url

  as an argument. This creates a second context manager and returns a
  
response

  .
• Line 9 prints the response status code for the
  
url

  at hand.
• Line 10 runs an awaitable call to
  
.text()

  on
  
response

  and stores the result in
  
html

  .
• Line 11 prints the site
  
url

  and its document type,
  
doctype

  .
• Line 13 defines the script’s
  
main()

  function, which is also a coroutine.
• Line 14 calls
  
gather()

  from
  
asyncio

  . This function runs awaitable objects in a sequence concurrently. In this example,
  
gather()

  runs two instances of
  
check()

  with a different URL for each.
• Line 19 runs
  
main()

  using
  
asyncio.run()

  . This function creates a new
  
asyncio

  event loop and closes it at the end of the operation.

If you run this script from your command line, then you get an output similar to the following:


$ python site_checker_v0.py https://realpython.com: status -> 200 https://pycoders.com: status -> 200 https://pycoders.com: type ->


https://realpython.com: type ->

Cool! Your script works and you confirm that both sites are currently available. You also retrieve the information regarding document type from each site’s home page.

Note: Your output can look slightly different due to the nondeterministic nature of concurrent task scheduling and network latency. In particular, the individual lines can come out in a different order.

The

async with

statement works similar to the regular

with

statement, but it requires an asynchronous context manager. In other words, it needs a context manager that is able to suspend execution in its enter and exit methods. Asynchronous context managers implement the special methods

.__aenter__()

and

.__aexit__()

, which correspond to

.__enter__()

and

.__exit__()

in a regular context manager.

The

async with ctx_mgr

construct implicitly uses

await ctx_mgr.__aenter__()

when entering the context and

await ctx_mgr.__aexit__()

when exiting it. This achieves

async

context manager behavior seamlessly.

Python3

Summarizing the with Statement’s Advantages

To summarize what you’ve learned so far, here’s an inexhaustive list of the general benefits of using the Python

with

statement in your code:

• Makes resource management safer than its equivalent
  
try

  …
  
finally

  statements
• Encapsulates standard uses of
  
try

  …
  
finally

  statements in context managers
• Allows reusing the code that automatically manages the setup and teardown phases of a given operation
• Helps avoid resource leaks

Using the

with

statement consistently can improve the general quality of your code and make it safer by preventing resource leak problems.

Python context manager protocol

Python context managers work based on the context manager protocol.

The context manager protocol has the following methods:

• 
__enter__()

  – setup the context and optionally return some object
• 
__exit__()

  – cleanup the object.

If you want a class to support the context manager protocol, you need to implement these two methods.

Suppose that

ContextManager

is a class that supports the context manager protocol.

The following shows how to use the

ContextManager

class:


with ContextManager() as ctx: # do something # done with the context

Code language: Python (python)

When you use

ContextManager

class with the

with

statement, Python implicitly creates an instance of the

ContextManager

class (

instance

) and automatically call

__enter__()

method on that instance.

The

__enter__()

method may optionally return an object. If so, Python assigns the returned object the

ctx

.

Notice that

ctx

references the object returned by the

__enter__()

method. It doesn’t reference the instance of the

ContextManager

class.

If an exception occurs inside the with block or after the

with

block, Python calls the

__exit__()

method on the

instance

object.

Functionally, the

with

statement is equivalent to the following

try...finally

statement:


instance = ContextManager() ctx = instance.__enter__() try: # do something with the txt finally: # done with the context instance.__exit__()

Code language: Python (python)

The __enter__() method

In the

__enter__()

method, you can carry the necessary steps to setup the context.

Optionally, you can returns an object from the

__enter__()

method.

The __exit__() method

Python always executes the

__exit__()

method even if an exception occurs in the

with

block.

The

__exit__()

method accepts three arguments: exception type, exception value, and traceback object. All of these arguments will be

None

if no exception occurs.


def __exit__(self, ex_type, ex_value, ex_traceback): ...

Code language: Python (python)

The

__exit__()

method returns a boolean value, either

True

or

False

.

If the return value is True, Python will make any exception silent. Otherwise, it doesn’t silence the exception.

Utilities¶

Functions and classes provided:

class contextlib.AbstractContextManager¶

An abstract base class for classes that implement


object.__enter__()

and

object.__exit__()

. A default implementation for

object.__enter__()

is provided which returns

self

while

object.__exit__()

is an abstract method which by default returns

None

. See also the definition of Context Manager Types.

New in version 3.6.

class contextlib.AbstractAsyncContextManager¶

An abstract base class for classes that implement


object.__aenter__()

and

object.__aexit__()

. A default implementation for

object.__aenter__()

is provided which returns

self

while

object.__aexit__()

is an abstract method which by default returns

None

. See also the definition of Asynchronous Context Managers.

New in version 3.7.

@contextlib.contextmanager¶

This function is a decorator that can be used to define a factory function for


with

statement context managers, without needing to create a class or separate

__enter__()

and

__exit__()

methods.

While many objects natively support use in with statements, sometimes a resource needs to be managed that isn’t a context manager in its own right, and doesn’t implement a


close()

method for use with

contextlib.closing


An abstract example would be the following to ensure correct resource management:

from contextlib import contextmanager @contextmanager def managed_resource(*args, **kwds): # Code to acquire resource, e.g.: resource = acquire_resource(*args, **kwds) try: yield resource finally: # Code to release resource, e.g.: release_resource(resource)

The function can then be used like this:

>>> with managed_resource(timeout=3600) as resource: … # Resource is released at the end of this block, … # even if code in the block raises an exception

The function being decorated must return a generator-iterator when called. This iterator must yield exactly one value, which will be bound to the targets in the


with

statement’s

as

clause, if any.

At the point where the generator yields, the block nested in the


with

statement is executed. The generator is then resumed after the block is exited. If an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred. Thus, you can use a

try

…

except

…

finally

statement to trap the error (if any), or ensure that some cleanup takes place. If an exception is trapped merely in order to log it or to perform some action (rather than to suppress it entirely), the generator must reraise that exception. Otherwise the generator context manager will indicate to the

with

statement that the exception has been handled, and execution will resume with the statement immediately following the

with

statement.

contextmanager()

uses

ContextDecorator

so the context managers it creates can be used as decorators as well as in

with

statements. When used as a decorator, a new generator instance is implicitly created on each function call (this allows the otherwise “one-shot” context managers created by

contextmanager()

to meet the requirement that context managers support multiple invocations in order to be used as decorators).

Changed in version 3.2: Use of


ContextDecorator

.

@contextlib.asynccontextmanager¶

Similar to


contextmanager()

, but creates an asynchronous context manager.

This function is a decorator that can be used to define a factory function for


async with

statement asynchronous context managers, without needing to create a class or separate

__aenter__()

and

__aexit__()

methods. It must be applied to an asynchronous generator function.

A simple example:

from contextlib import asynccontextmanager @asynccontextmanager async def get_connection(): conn = await acquire_db_connection() try: yield conn finally: await release_db_connection(conn) async def get_all_users(): async with get_connection() as conn: return conn.query(‘SELECT …’)

New in version 3.7.

Context managers defined with


asynccontextmanager()

can be used either as decorators or with

async with

statements:

import time from contextlib import asynccontextmanager @asynccontextmanager async def timeit(): now = time.monotonic() try: yield finally: print(f’it took {time.monotonic() – now}s to run’) @timeit() async def main(): # … async code …

When used as a decorator, a new generator instance is implicitly created on each function call. This allows the otherwise “one-shot” context managers created by


asynccontextmanager()

to meet the requirement that context managers support multiple invocations in order to be used as decorators.

Changed in version 3.10: Async context managers created with


asynccontextmanager()

can be used as decorators.

contextlib.closing(thing)¶

Return a context manager that closes thing upon completion of the block. This is basically equivalent to:

from contextlib import contextmanager @contextmanager def closing(thing): try: yield thing finally: thing.close()

And lets you write code like this:

from contextlib import closing from urllib.request import urlopen with closing(urlopen(‘https://www.python.org’)) as page: for line in page: print(line)

without needing to explicitly close


page

. Even if an error occurs,

page.close()

will be called when the

with

block is exited.

Note

Most types managing resources support the context manager protocol, which closes thing on leaving the


with

statement. As such,

closing()

is most useful for third party types that don’t support context managers. This example is purely for illustration purposes, as

urlopen()

would normally be used in a context manager.

contextlib.aclosing(thing)¶

Return an async context manager that calls the


aclose()

method of thing upon completion of the block. This is basically equivalent to:

from contextlib import asynccontextmanager @asynccontextmanager async def aclosing(thing): try: yield thing finally: await thing.aclose()

Significantly,


aclosing()

supports deterministic cleanup of async generators when they happen to exit early by

break

or an exception. For example:

from contextlib import aclosing async with aclosing(my_generator()) as values: async for value in values: if value == 42: break

This pattern ensures that the generator’s async exit code is executed in the same context as its iterations (so that exceptions and context variables work as expected, and the exit code isn’t run after the lifetime of some task it depends on).

New in version 3.10.

contextlib.nullcontext(enter_result=None)¶

Return a context manager that returns enter_result from


__enter__

, but otherwise does nothing. It is intended to be used as a stand-in for an optional context manager, for example:

def myfunction(arg, ignore_exceptions=False): if ignore_exceptions: # Use suppress to ignore all exceptions. cm = contextlib.suppress(Exception) else: # Do not ignore any exceptions, cm has no effect. cm = contextlib.nullcontext() with cm: # Do something

An example using enter_result:

def process_file(file_or_path): if isinstance(file_or_path, str): # If string, open file cm = open(file_or_path) else: # Caller is responsible for closing file cm = nullcontext(file_or_path) with cm as file: # Perform processing on the file

It can also be used as a stand-in for asynchronous context managers:

async def send_http(session=None): if not session: # If no http session, create it with aiohttp cm = aiohttp.ClientSession() else: # Caller is responsible for closing the session cm = nullcontext(session) async with cm as session: # Send http requests with session

New in version 3.7.

Changed in version 3.10: asynchronous context manager support was added.

contextlib.suppress(*exceptions)¶

Return a context manager that suppresses any of the specified exceptions if they occur in the body of a


with

statement and then resumes execution with the first statement following the end of the

with

statement.

As with any other mechanism that completely suppresses exceptions, this context manager should be used only to cover very specific errors where silently continuing with program execution is known to be the right thing to do.

For example:

from contextlib import suppress with suppress(FileNotFoundError): os.remove(‘somefile.tmp’) with suppress(FileNotFoundError): os.remove(‘someotherfile.tmp’)

This code is equivalent to:

try: os.remove(‘somefile.tmp’) except FileNotFoundError: pass try: os.remove(‘someotherfile.tmp’) except FileNotFoundError: pass

This context manager is reentrant.

If the code within the


with

block raises a

BaseExceptionGroup

, suppressed exceptions are removed from the group. If any exceptions in the group are not suppressed, a group containing them is re-raised.

New in version 3.4.

Changed in version 3.12:


suppress

now supports suppressing exceptions raised as part of an

BaseExceptionGroup

.

contextlib.redirect_stdout(new_target)¶

Context manager for temporarily redirecting


sys.stdout

to another file or file-like object.

This tool adds flexibility to existing functions or classes whose output is hardwired to stdout.

For example, the output of


help()

normally is sent to sys.stdout. You can capture that output in a string by redirecting the output to an

io.StringIO

object. The replacement stream is returned from the

__enter__

method and so is available as the target of the

with

statement:

with redirect_stdout(io.StringIO()) as f: help(pow) s = f.getvalue()

To send the output of


help()

to a file on disk, redirect the output to a regular file:

with open(‘help.txt’, ‘w’) as f: with redirect_stdout(f): help(pow)

To send the output of


help()

to sys.stderr:

with redirect_stdout(sys.stderr): help(pow)

Note that the global side effect on


sys.stdout

means that this context manager is not suitable for use in library code and most threaded applications. It also has no effect on the output of subprocesses. However, it is still a useful approach for many utility scripts.

This context manager is reentrant.

New in version 3.4.

contextlib.redirect_stderr(new_target)¶

Similar to


redirect_stdout()

but redirecting

sys.stderr

to another file or file-like object.

This context manager is reentrant.

New in version 3.5.

contextlib.chdir(path)¶

Non parallel-safe context manager to change the current working directory. As this changes a global state, the working directory, it is not suitable for use in most threaded or async contexts. It is also not suitable for most non-linear code execution, like generators, where the program execution is temporarily relinquished – unless explicitly desired, you should not yield when this context manager is active.

This is a simple wrapper around


chdir()

, it changes the current working directory upon entering and restores the old one on exit.

This context manager is reentrant.

New in version 3.11.

class contextlib.ContextDecorator¶

A base class that enables a context manager to also be used as a decorator.

Context managers inheriting from


ContextDecorator

have to implement

__enter__

and

__exit__

as normal.

__exit__

retains its optional exception handling even when used as a decorator.

ContextDecorator

is used by

contextmanager()

, so you get this functionality automatically.

Example of


ContextDecorator

:

from contextlib import ContextDecorator class mycontext(ContextDecorator): def __enter__(self): print(‘Starting’) return self def __exit__(self, *exc): print(‘Finishing’) return False

The class can then be used like this:

>>> @mycontext() … def function(): … print(‘The bit in the middle’) … >>> function() Starting The bit in the middle Finishing >>> with mycontext(): … print(‘The bit in the middle’) … Starting The bit in the middle Finishing

This change is just syntactic sugar for any construct of the following form:

def f(): with cm(): # Do stuff


ContextDecorator

lets you instead write:

@cm() def f(): # Do stuff

It makes it clear that the


cm

applies to the whole function, rather than just a piece of it (and saving an indentation level is nice, too).

Existing context managers that already have a base class can be extended by using


ContextDecorator

as a mixin class:

from contextlib import ContextDecorator class mycontext(ContextBaseClass, ContextDecorator): def __enter__(self): return self def __exit__(self, *exc): return False

Note

As the decorated function must be able to be called multiple times, the underlying context manager must support use in multiple


with

statements. If this is not the case, then the original construct with the explicit

with

statement inside the function should be used.

New in version 3.2.

class contextlib.AsyncContextDecorator¶

Similar to


ContextDecorator

but only for asynchronous functions.

Example of


AsyncContextDecorator

:

from asyncio import run from contextlib import AsyncContextDecorator class mycontext(AsyncContextDecorator): async def __aenter__(self): print(‘Starting’) return self async def __aexit__(self, *exc): print(‘Finishing’) return False

The class can then be used like this:

>>> @mycontext() … async def function(): … print(‘The bit in the middle’) … >>> run(function()) Starting The bit in the middle Finishing >>> async def function(): … async with mycontext(): … print(‘The bit in the middle’) … >>> run(function()) Starting The bit in the middle Finishing

New in version 3.10.

class contextlib.ExitStack¶

A context manager that is designed to make it easy to programmatically combine other context managers and cleanup functions, especially those that are optional or otherwise driven by input data.

For example, a set of files may easily be handled in a single with statement as follows:

with ExitStack() as stack: files = [stack.enter_context(open(fname)) for fname in filenames] # All opened files will automatically be closed at the end of # the with statement, even if attempts to open files later # in the list raise an exception

The


__enter__()

method returns the

ExitStack

instance, and performs no additional operations.

Each instance maintains a stack of registered callbacks that are called in reverse order when the instance is closed (either explicitly or implicitly at the end of a


with

statement). Note that callbacks are not invoked implicitly when the context stack instance is garbage collected.

This stack model is used so that context managers that acquire their resources in their


__init__

method (such as file objects) can be handled correctly.

Since registered callbacks are invoked in the reverse order of registration, this ends up behaving as if multiple nested


with

statements had been used with the registered set of callbacks. This even extends to exception handling – if an inner callback suppresses or replaces an exception, then outer callbacks will be passed arguments based on that updated state.

This is a relatively low level API that takes care of the details of correctly unwinding the stack of exit callbacks. It provides a suitable foundation for higher level context managers that manipulate the exit stack in application specific ways.

New in version 3.3.

enter_context(cm)¶

Enters a new context manager and adds its


__exit__()

method to the callback stack. The return value is the result of the context manager’s own

__enter__()

method.

These context managers may suppress exceptions just as they normally would if used directly as part of a


with

statement.

Changed in version 3.11: Raises


TypeError

instead of

AttributeError

if cm is not a context manager.

push(exit)¶

Adds a context manager’s


__exit__()

method to the callback stack.

As


__enter__

is not invoked, this method can be used to cover part of an

__enter__()

implementation with a context manager’s own

__exit__()

method.

If passed an object that is not a context manager, this method assumes it is a callback with the same signature as a context manager’s


__exit__()

method and adds it directly to the callback stack.

By returning true values, these callbacks can suppress exceptions the same way context manager


__exit__()

methods can.

The passed in object is returned from the function, allowing this method to be used as a function decorator.

callback(callback, /, *args, **kwds)¶

Accepts an arbitrary callback function and arguments and adds it to the callback stack.

Unlike the other methods, callbacks added this way cannot suppress exceptions (as they are never passed the exception details).

The passed in callback is returned from the function, allowing this method to be used as a function decorator.

pop_all()¶

Transfers the callback stack to a fresh


ExitStack

instance and returns it. No callbacks are invoked by this operation – instead, they will now be invoked when the new stack is closed (either explicitly or implicitly at the end of a

with

statement).

For example, a group of files can be opened as an “all or nothing” operation as follows:

with ExitStack() as stack: files = [stack.enter_context(open(fname)) for fname in filenames] # Hold onto the close method, but don’t call it yet. close_files = stack.pop_all().close # If opening any file fails, all previously opened files will be # closed automatically. If all files are opened successfully, # they will remain open even after the with statement ends. # close_files() can then be invoked explicitly to close them all.

close()¶

Immediately unwinds the callback stack, invoking callbacks in the reverse order of registration. For any context managers and exit callbacks registered, the arguments passed in will indicate that no exception occurred.

class contextlib.AsyncExitStack¶

An asynchronous context manager, similar to


ExitStack

, that supports combining both synchronous and asynchronous context managers, as well as having coroutines for cleanup logic.

The


close()

method is not implemented;

aclose()

must be used instead.

coroutine enter_async_context(cm)¶

Similar to


ExitStack.enter_context()

but expects an asynchronous context manager.

Changed in version 3.11: Raises


TypeError

instead of

AttributeError

if cm is not an asynchronous context manager.

push_async_exit(exit)¶

Similar to


ExitStack.push()

but expects either an asynchronous context manager or a coroutine function.

push_async_callback(callback, /, *args, **kwds)¶

Similar to


ExitStack.callback()

but expects a coroutine function.

coroutine aclose()¶

Similar to


ExitStack.close()

but properly handles awaitables.

Continuing the example for


asynccontextmanager()

:

async with AsyncExitStack() as stack: connections = [await stack.enter_async_context(get_connection()) for i in range(5)] # All opened connections will automatically be released at the end of # the async with statement, even if attempts to open a connection # later in the list raise an exception.

New in version 3.7.

Python3

Summary

A context manager, used in a

with

statement, defines a temporary context for the given set of operations. It does so by injecting code at the beginning and at the end of the code block, setting up the context at the beginning, and tearing it down at the end.

Sign in to your Python Morsels account to save your screencast settings.

Don’t have an account yet? Sign up here.

How can you create your own context manager in Python?

A context manager is an object that can be used in a

with

block to sandwich some code between an entrance action and an exit action.

File objects can be used as context managers to automatically close the file when we’re done working with it:


>>> with open("example.txt", "w") as file: ... file.write("Hello, world!") ... 13 >>> file.closed True


Context managers need a

__enter__

method and a

__exit__

method, and the

__exit__

method should accept three positional arguments:


class Example: def __enter__(self): print("enter") def __exit__(self, exc_type, exc_val, exc_tb): print("exit")


This context manager just prints

enter

when the

with

block is entered and

exit

when the

with

block is exited:


>>> with Example(): ... print("Yay Python!") ... enter Yay Python! exit


Of course, this is a somewhat silly context manager. Let’s look at a context manager that actually does something a little bit useful.

This context manager temporarily changes the value of an environment variable:


import os class set_env_var: def __init__(self, var_name, new_value): self.var_name = var_name self.new_value = new_value def __enter__(self): self.original_value = os.environ.get(self.var_name) os.environ[self.var_name] = self.new_value def __exit__(self, exc_type, exc_val, exc_tb): if self.original_value is None: del os.environ[self.var_name] else: os.environ[self.var_name] = self.original_value


The

USER

environment variable on my machine currently has the value of

Trey

:


>>> print("USER env var is", os.environ["USER"]) USER env var is trey


If we use this context manager, within its

with

block, the

USER

environment variable will have a different value:


>>> with set_env_var("USER", "akin"): ... print("USER env var is", os.environ["USER"]) ... USER env var is akin


But after the

with

block exits, the value of that environment variable resets back to its original value:


>>> print("USER env var is", os.environ["USER"]) USER env var is trey


This is all thanks to our context manager’s

__enter__

method and a

__exit__

method, which run when our context manager’s

with

block is entered and exited.


askeyword?


You’ll sometimes see context managers used with an

as

keyword (note the

as result

below):


>>> with set_env_var("USER", "akin") as result: ... print("USER env var is", os.environ["USER"]) ... print("Result from __enter__ method:", result) ...


The

as

keyword will point a given variable name to the return value from the

__enter__

method:

In our case, we always get

None

as the value of our

result

variable:


>>> with set_env_var("USER", "akin") as result: ... print("USER env var is", os.environ["USER"]) ... print("Result from __enter__ method:", result) ... USER env var is akin Result from __enter__ method: None


This is because our

__enter__

method doesn’t return anything, so it implicitly returns the default function return value of

None

.


__enter__


Let’s look at a context manager that does return something from

__enter__

.

Here we have a program called

timer.py

:


import time class Timer: def __enter__(self): self.start = time.perf_counter() return self def __exit__(self, exc_type, exc_val, exc_tb): self.stop = time.perf_counter() self.elapsed = self.stop - self.start


This context manager will time how long it took to run a particular block of code (the block of code in our

with

block).

We can use this context manager by making a

Timer

object, using

with

to run a block of code, and then checking the

elapsed

attribute on our

Timer

object:


>>> t = Timer() >>> with t: ... result = sum(range(10_000_000)) ... >>> t.elapsed 0.28711878502508625


But there’s actually an even shorter way to use this context manager.

We can make the

Timer

object and assign it to a variable, all on one line of code, using our

with

block and the

as

keyword:


>>> with Timer() as t: ... result = sum(range(10_000_000)) ... >>> t.elapsed 0.3115791230229661


This works because our context manager’s

__enter__

method returns

self

:


def __enter__(self): self.start = time.perf_counter() return self


So it’s returning the actual context manager object to us and that’s what gets assigned to the variable in our

with

block.

Since many context managers keep track of some useful state on their own object, it’s very common to see a context manager’s

__enter__

method return

self

.


__exit__


What about that

__exit__

method?


def __exit__(self, exc_type, exc_val, exc_tb): self.stop = time.perf_counter() self.elapsed = self.stop - self.start


What are those three arguments that it accepts? And does its return value matter?

If an exception occurs within a

with

block, these three arguments passed to the context manager’s

__exit__

method will be:

But if no exception occurs, those three arguments will all be

None

.

Here’s a context manager that uses all three of those arguments:


import logging class LogException: def __init__(self, logger, level=logging.ERROR, suppress=False): self.logger, self.level, self.suppress = logger, level, suppress def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): if exc_type is not None: info = (exc_type, exc_val, exc_tb) self.logger.log(self.level, "Exception occurred", exc_info=info) return self.suppress return False


This context manager logs exceptions as they occur (using Python’s

logging

module).

So we can use this

LogException

context manager like this:


import logging from log_exception import LogException logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger("example") with LogException(logger): result = 1 / 0 # This will cause a ZeroDivisionError print("That's the end of our program")


When an exception occurs in our code, we’ll see the exception logged to our console:


$ python3 log_example.py ERROR:example:Exception occurred Traceback (most recent call last): File "/home/trey/_/log_example.py", line 8, in

result = 1 / 0 # This will cause a ZeroDivisionError ~~^~~ ZeroDivisionError: division by zero Traceback (most recent call last): File "/home/trey/_/log_example.py", line 8, in

result = 1 / 0 # This will cause a ZeroDivisionError ~~^~~ ZeroDivisionError: division by zero




We see

ERROR

, the name of our logger (

example

),

Exception occurred

, and then the traceback.

In this example, we also a second traceback, which was printed by Python when our program crashed.

Because our program exited, it didn’t actually print out the last line in our program (

That's the end of our program

).


__exit__


If we had passed

suppress=True

to our context manager, we’ll see something different happen:


import logging from log_exception import LogException logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger("example") with LogException(logger, suppress=True): result = 1 / 0 # This will cause a ZeroDivisionError print("That's the end of our program")


Now when we run our program, the exception is logged, but then our program continues onward after the

with

block:


$ python3 log_example.py ERROR:example:Exception occurred Traceback (most recent call last): File "/home/trey/_/_/log_example.py", line 8, in

result = 1 / 0 # This will cause a ZeroDivisionError ~~^~~ ZeroDivisionError: division by zero That's the end of our program



We can see

That's the end of our program

actually prints out here!

What’s going on?

So this

suppress

argument, it’s used by our context manager to suppress an exception:


import logging class LogException: ... def __exit__(self, exc_type, exc_val, exc_tb): if exc_type is not None: info = (exc_type, exc_val, exc_tb) self.logger.log(self.level, "Exception occurred", exc_info=info) return self.suppress return False


If the

__exit__

method returns something true or truthy, whatever exception was being raised will actually be suppressed.

By default,

__exit__

returns

None

, just as every function does by default.
If we return

None

, which is falsey, or

False

, or anything that’s falsey,

__exit__

won’t do anything different from its default, which is to just continue raising that exception.

But if

True

or a truthy value is returned, the exception will be suppressed.


contextmanager?


Have you ever seen a generator function that somehow made a context manager?

Python’s

contextlib

module includes a decorator which allows for creating context managers using a function syntax (instead of using the typical class syntax we saw above):


from contextlib import contextmanager import os @contextmanager def set_env_var(var_name, new_value): original_value = os.environ.get(var_name) os.environ[var_name] = new_value try: yield finally: if original_value is None: del os.environ[var_name] else: os.environ[var_name] = original_value


Interestingly, this fancy decorator still involves

__enter__

and

__exit__

under the hood: it’s just a very clever helper for creating an object that has those methods.

This

contextmanager

decorator can sometimes be very handy, though it does have limitations!
I plan to record a separate screencast (and write a separate article) on

contextlib.contextmanager

.
Python Morsels subscribers will hear about this new screencast as soon as I publish it. 😉


__enter__&


__exit__


Context managers are objects that work in a

with

block.

You can make a context manager by creating an object that has a

__enter__

method and a

__exit__

method.

Python also includes a fancy decorator for creating context managers with a function syntax, which I’ll cover in a future screencast.

Need to fill-in gaps in your Python skills?

Sign up for my Python newsletter where I share one of my favorite Python tips every week.

Need to fill-in gaps in your Python skills? I send weekly emails designed to do just that.

Summary: in this tutorial, you’ll learn about the Python context managers and how to use them effectively

Managing Resources in Python

One common problem you’ll face in programming is how to properly manage external resources, such as files, locks, and network connections. Sometimes, a program will retain those resources forever, even if you no longer need them. This kind of issue is called a memory leak because the available memory gets reduced every time you create and open a new instance of a given resource without closing an existing one.

Managing resources properly is often a tricky problem. It requires both a setup phase and a teardown phase. The latter phase requires you to perform some cleanup actions, such as closing a file, releasing a lock, or closing a network connection. If you forget to perform these cleanup actions, then your application keeps the resource alive. This might compromise valuable system resources, such as memory and network bandwidth.

For example, a common problem that can arise when developers are working with databases is when a program keeps creating new connections without releasing or reusing them. In that case, the database back end can stop accepting new connections. This might require an admin to log in and manually kill those stale connections to make the database usable again.

Another frequent issue shows up when developers are working with files. Writing text to files is usually a buffered operation. This means that calling

.write()

on a file won’t immediately result in writing text to the physical file but to a temporary buffer. Sometimes, when the buffer isn’t full and developers forget to call

.close()

, part of the data can be lost forever.

Another possibility is that your application runs into errors or exceptions that cause the control flow to bypass the code responsible for releasing the resource at hand. Here’s an example in which you use

open()

to write some text to a file:


file = open("hello.txt", "w") file.write("Hello, World!") file.close()


This implementation doesn’t guarantee the file will be closed if an exception occurs during the

.write()

call. In this case, the code will never call

.close()

, and therefore your program might leak a file descriptor.

In Python, you can use two general approaches to deal with resource management. You can wrap your code in:

1. A
   
try

   …
   
finally

   construct
2. A
   
with

   construct

The first approach is quite general and allows you to provide setup and teardown code to manage any kind of resource. However, it’s a little bit verbose. Also, what if you forget any cleanup actions?

The second approach provides a straightforward way to provide and reuse setup and teardown code. In this case, you’ll have the limitation that the

with

statement only works with context managers. In the next two sections, you’ll learn how to use both approaches in your code.

The try … finally Approach

Working with files is probably the most common example of resource management in programming. In Python, you can use a

try

…

finally

statement to handle opening and closing files properly:


# Safely open the file file = open("hello.txt", "w") try: file.write("Hello, World!") finally: # Make sure to close the file after using it file.close()


In this example, you need to safely open the file

hello.txt

, which you can do by wrapping the call to

open()

in a

try

…

except

statement. Later, when you try to write to

file

, the

finally

clause will guarantee that

file

is properly closed, even if an exception occurs during the call to

.write()

in the

try

clause. You can use this pattern to handle setup and teardown logic when you’re managing external resources in Python.

The

try

block in the above example can potentially raise exceptions, such as

AttributeError

or

NameError

. You can handle those exceptions in an

except

clause like this:


# Safely open the file file = open("hello.txt", "w") try: file.write("Hello, World!") except Exception as e: print(f"An error occurred while writing to the file: {e}") finally: # Make sure to close the file after using it file.close()


In this example, you catch any potential exceptions that can occur while writing to the file. In real-life situations, you should use a specific exception type instead of the general

Exception

to prevent unknown errors from passing silently.

The with Statement Approach

The Python

with

statement creates a runtime context that allows you to run a group of statements under the control of a context manager. PEP 343 added the

with

statement to make it possible to factor out standard use cases of the

try

…

finally

statement.

Compared to traditional

try

…

finally

constructs, the

with

statement can make your code clearer, safer, and reusable. Many classes in the standard library support the

with

statement. A classic example of this is

open()

, which allows you to work with file objects using

with

.

To write a

with

statement, you need to use the following general syntax:


with expression as target_var: do_something(target_var)


The context manager object results from evaluating the

expression

after

with

. In other words,

expression

must return an object that implements the context management protocol. This protocol consists of two special methods:

1. 
.__enter__()

   is called by the
   
with

   statement to enter the runtime context.
2. 
.__exit__()

   is called when the execution leaves the
   
with

   code block.

The

as

specifier is optional. If you provide a

target_var

with

as

, then the return value of calling

.__enter__()

on the context manager object is bound to that variable.

Note: Some context managers return

None

from

.__enter__()

because they have no useful object to give back to the caller. In these cases, specifying a

target_var

makes no sense.

Here’s how the

with

statement proceeds when Python runs into it:

1. Call
   
expression

   to obtain a context manager.
2. Store the context manager’s
   
.__enter__()

   and
   
.__exit__()

   methods for later use.
3. Call
   
.__enter__()

   on the context manager and bind its return value to
   
target_var

   if provided.
4. Execute the
   
with

   code block.
5. Call
   
.__exit__()

   on the context manager when the
   
with

   code block finishes.

In this case,

.__enter__()

, typically provides the setup code. The

with

statement is a compound statement that starts a code block, like a conditional statement or a

for

loop. Inside this code block, you can run several statements. Typically, you use the

with

code block to manipulate

target_var

if applicable.

Once the

with

code block finishes,

.__exit__()

gets called. This method typically provides the teardown logic or cleanup code, such as calling

.close()

on an open file object. That’s why the

with

statement is so useful. It makes properly acquiring and releasing resources a breeze.

Here’s how to open your

hello.txt

file for writing using the

with

statement:


with open("hello.txt", mode="w") as file: file.write("Hello, World!")


When you run this

with

statement,

open()

returns an

io.TextIOBase

object. This object is also a context manager, so the

with

statement calls

.__enter__()

and assigns its return value to

file

. Then you can manipulate the file inside the

with

code block. When the block ends,

.__exit__()

automatically gets called and closes the file for you, even if an exception is raised inside the

with

block.

This

with

construct is shorter than its

try

…

finally

alternative, but it’s also less general, as you already saw. You can only use the

with

statement with objects that support the context management protocol, whereas

try

…

finally

allows you to perform cleanup actions for arbitrary objects without the need for supporting the context management protocol.

In Python 3.1 and later, the

with

statement supports multiple context managers. You can supply any number of context managers separated by commas:


with A() as a, B() as b: pass


This works like nested

with

statements but without nesting. This might be useful when you need to open two files at a time, the first for reading and the second for writing:


with open("input.txt") as in_file, open("output.txt", "w") as out_file: # Read content from input.txt # Transform the content # Write the transformed content to output.txt pass


In this example, you can add code for reading and transforming the content of

input.txt

. Then you write the final result to

output.txt

in the same code block.

Using multiple context managers in a single

with

has a drawback, though. If you use this feature, then you’ll probably break your line length limit. To work around this, you need to use backslashes () for line continuation, so you might end up with an ugly final result.

The

with

statement can make the code that deals with system resources more readable, reusable, and concise, not to mention safer. It helps avoid bugs and leaks by making it almost impossible to forget cleaning up, closing, and releasing a resource after you’re done with it.

Using

with

allows you to abstract away most of the resource handling logic. Instead of having to write an explicit

try

…

finally

statement with setup and teardown code each time,

with

takes care of that for you and avoids repetition.

Python with statement

Here is the typical syntax of the

with

statement:


with context as ctx: # use the the object # context is cleaned up

Code language: Python (python)

How it works.

• When Python encounters the
  
with

  statement, it creates a new context. The context can optionally return an
  
object

  .
• After the
  
with

  block, Python cleans up the context automatically.
• The scope of the
  
ctx

  has the same scope as the
  
with

  statement. It means that you can access the
  
ctx

  both inside and after the
  
with

  statement.

The following shows how to access the variable after the

with

statement:


with open('data.txt') as f: data = f.readlines() print(int(data[0])) print(f.closed) # True

Code language: Python (python)

27.Handling Exceptions¶

We did not talk about the

type

,

value

and

traceback

arguments of the

__exit__

method. Between the 4th and 6th step, if
an exception occurs, Python passes the type, value and traceback of the
exception to the

__exit__

method. It allows the

__exit__

method
to decide how to close the file and if any further steps are required.
In our case we are not paying any attention to them.

What if our file object raises an exception? We might be trying to access a method on the file object which it does not supports. For instance:

with File(‘demo.txt’, ‘w’) as opened_file: opened_file.undefined_function(‘Hola!’)

Let’s list the steps which are taken by the

with

statement when
an error is encountered:

1. It passes the type, value and traceback of the error to the
   
__exit__

   method.
2. It allows the
   
__exit__

   method to handle the exception.
3. If
   
__exit__

   returns
   
True

   then the exception was gracefully handled.
4. If anything other than
   
True

   is returned by the
   
__exit__

   method then the exception is raised by the
   
with

   statement.

In our case the

__exit__

method returns

None

(when no return
statement is encountered then the method returns

None

). Therefore,
the

with

statement raises the exception:

Traceback (most recent call last): File ”

“, line 2, in

AttributeError: ‘file’ object has no attribute ‘undefined_function’

Let’s try handling the exception in the

__exit__

method:

class File(object): def __init__(self, file_name, method): self.file_obj = open(file_name, method) def __enter__(self): return self.file_obj def __exit__(self, type, value, traceback): print(“Exception has been handled”) self.file_obj.close() return True with File(‘demo.txt’, ‘w’) as opened_file: opened_file.undefined_function() # Output: Exception has been handled

Our

__exit__

method returned

True

, therefore no exception was raised
by the

with

statement.

This is not the only way to implement Context Managers. There is another way and we will be looking at it in the next section.

Conclusion

Context managers in Python is one of those topics that a lot of programmers have used but do not understand clearly.

I hope this article has cleared up some of your confusions.

If you’d like to connect to me, I am always available on LinkedIn. Feel free to shoot a message and I’d be happy to respond. Also, if you think this was helpful, consider endorsing my relevant skills on the platform.

Until the next one, take care and keep exploring.

Context Managers in Python: Using the “with” statement

Context managers are used to set up and tear down temporary contexts, establish and resolve custom settings, and acquire and release resources. The

open()

function for opening files is one of the most familiar examples of a context manager.

Context managers sandwich code blocks between two distinct pieces of logic:

1. The enter logic – this runs right before the nested code block executes
2. The exit logic – this runs right after the nested code block is done.

The most common way you’ll work with context managers is by using the

with

statement.

The with statement

A

with

statement is the primary method used to call a context manager. The general syntax is as follows:

This code works in the following order:

1. 
SomeContextManager

   executes its enter (setup) logic before the indented code runs.
2. 
SomeContextManager

   binds a value to
   
context_variable

   , which can be used in the indented code
3. Inner code block runs:
   
# do stuff

   .
4. 
SomeContextManager

   executes its exit (cleanup) logic.

So why is this useful? In the next section, we’ll discuss why this programming pattern comes in handy and why it’s worth making your context managers from time to time.

The with statement

A

with

statement is the primary method used to call a context manager. The general syntax is as follows:

This code works in the following order:

1. 
SomeContextManager

   executes its enter (setup) logic before the indented code runs.
2. 
SomeContextManager

   binds a value to
   
context_variable

   , which can be used in the indented code
3. Inner code block runs:
   
# do stuff

   .
4. 
SomeContextManager

   executes its exit (cleanup) logic.

So why is this useful? In the next section, we’ll discuss why this programming pattern comes in handy and why it’s worth making your context managers from time to time.

Conclusion

The Python

with

statement is a powerful tool when it comes to managing external resources in your programs. Its use cases, however, aren’t limited to resource management. You can use the

with

statement along with existing and custom context managers to handle the setup and teardown phases of a given process or operation.

The underlying context management protocol allows you to create custom context managers and factor out the setup and teardown logic so you can reuse them in your code.

In this tutorial, you learned:

• What the Python
  
with

  statement is for and how to use it
• What the context management protocol is
• How to implement your own context managers

With this knowledge, you’ll write safe, concise, and expressive code. You’ll also avoid resource leaks in your programs.

Take the Quiz: Test your knowledge with our interactive “Context Managers and Python’s with Statement” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Context Managers and Python’s with Statement

Context Managers¶

Context managers allow you to allocate and release resources precisely
when you want to. The most widely used example of context managers is
the

with

statement. Suppose you have two related operations which
you’d like to execute as a pair, with a block of code in between.
Context managers allow you to do specifically that. For example:

with open(‘some_file’, ‘w’) as opened_file: opened_file.write(‘Hola!’)

The above code opens the file, writes some data to it and then closes it. If an error occurs while writing the data to the file, it tries to close it. The above code is equivalent to:

file = open(‘some_file’, ‘w’) try: file.write(‘Hola!’) finally: file.close()

While comparing it to the first example we can see that a lot of
boilerplate code is eliminated just by using

with

. The main
advantage of using a

with

statement is that it makes sure our file
is closed without paying attention to how the nested block exits.

A common use case of context managers is locking and unlocking resources and closing opened files (as I have already shown you).

Let’s see how we can implement our own Context Manager. This should allow us to understand exactly what’s going on behind the scenes.

Python3

27.Implementing a Context Manager as a Generator¶

We can also implement Context Managers using decorators and generators. Python has a contextlib module for this very purpose. Instead of a class, we can implement a Context Manager using a generator function. Let’s see a basic, useless example:

from contextlib import contextmanager @contextmanager def open_file(name): f = open(name, ‘w’) try: yield f finally: f.close()

Okay! This way of implementing Context Managers appear to be more intuitive and easy. However, this method requires some knowledge about generators, yield and decorators. In this example we have not caught any exceptions which might occur. It works in mostly the same way as the previous method.

Let’s dissect this method a little.

1. Python encounters the
   
yield

   keyword. Due to this it creates a generator instead of a normal function.
2. Due to the decoration, contextmanager is called with the function
   name (
   
open_file

   ) as its argument.
3. The
   
contextmanager

   decorator returns the generator wrapped by the
   
GeneratorContextManager

   object.
4. The
   
GeneratorContextManager

   is assigned to the
   
open_file

   function. Therefore, when we later call the
   
open_file

   function, we are actually calling the
   
GeneratorContextManager

   object.

So now that we know all this, we can use the newly generated Context Manager like this:

with open_file(‘some_file’) as f: f.write(‘hola!’)

Python context manager – bạn đã thực sự hiểu?

Bài đăng này đã không được cập nhật trong 2 năm

Bài viết gốc: https://manhhomienbienthuy.github.io/2017/05/12/python-context-managers.html (đã xin phép tác giả

)

Trong Python, context manager là một phương thức cho phép bạn cấp phát và sử dụng tài nguyên một cách hiệu quả. Context manager được sử dụng rộng rãi thông qua câu lệnh

with

. Ví dụ:


with open('foo', 'w') as f: f.write('Hora! We opened this file')


Đoạn code trên mở một file, ghi dữ liệu và đóng file lại. Nếu có bất kỳ lỗi gì xảy ra, thì file cũng luôn luôn được đảm bảo là đã đóng. Đoạn code trên nếu viết mà không sử dụng context manager thì sẽ trông như dưới đây:


f = open('foo', 'w') try: f.write('Hora! We opened this file') finally: f.close()


So sánh hai cách viết này thì chúng ta đã thấy rất rõ ràng rằng, context manager cho chúng ta cách viết code ngắn gọn hơn hẳn. Lệnh

with

cho chúng ta bảo đảm rằng file luôn luôn được đóng mà không cần biết những logic xử lý bên trong.

Hẳn là các bạn đã rất quen thuộc với những đoạn code trên, đặc biệt khi bạn đã từng nghe nói đến “Idiomatic Python”. Nhưng liệu bạn đã chắc chắn hiểu được cách làm việc chính xác với file và lý do tại sao đó lại là cách đúng không? Hoặc đơn giản hơn, bạn có biết khi nào thì mình đã thao tác sai không.

Nếu câu trả lời là không, thì bài viết này chính là dành cho bạn.

Context manager thường được sử dụng để lock các tài nguyên (trường hợp mở và đóng file là một ví dụ kinh điển cho việc này).

Quản lý tài nguyên

Tính năng quan trọng nhất và cũng là phổ biến nhất của context manager là để quản lý tài nguyên một cách chính xác. Quay lại với việc đọc và ghi file ở ví dụ trên, tại sao chúng ta phải sử dụng context manager. Mỗi khi mở một file để đọc hoặc ghi, một tài nguyên của hệ thống, trong trường hợp này là file descriptor sẽ đã bị tiêu tốn để chúng ta có thể thao tác. Thật không may là tài nguyên này lại là hữu hạn. Mỗi hệ điều hành đều có giới hạn nhất định cho số lượng file có thể mở cùng một lúc.

Không tin ư, bạn hãy xem ví dụ sau:


>>> files = [] >>> for _ in range(10000): ... files.append(open('foo', 'w')) ... Traceback (most recent call last): File "

", line 2, in

OSError: [Errno 24] Too many open files: 'foo'




Ngoài lề một chút, file descriptor thực chất là một số nguyên. Khi bạn mở một file, hệ điều hành sẽ tạo ra một entry (có thể ở kernel) lưu trữ những thông tin liên quan đến file được mở. Mỗi entry sẽ được gán với 1 số nguyên (và số này sẽ là duy nhất), cho phép người dùng thông qua đó để thao tác với file.

Thực chất người dùng đang thao tác một cách gián tiếp thông qua file descriptor (và thông qua entry của kernel) với các dữ liệu thật sự được ghi ở bộ nhớ ngoài. Việc này mang lại nhiều lợi ích như có thể chia sẻ file cho nhiều tiến trình khác nhau cũng như duy trì bảo mật cho các file đó.

Thực ra, hằng ngày bạn đều đang làm việc với file descriptor mà có thể bạn cũng không nhận ra. Hệ điều hành đã gán sẵn một số file descriptor như cho bàn phím, cho màn hình, v.v… Và mọi thao tác chúng ta làm với máy tính đều thông qua các file descriptor này. Bạn nghĩ sao về việc chuyển hướng của câu lệnh Linux như thế này:


$ sort < file_list > sorted_file_list 2>&1


Tương tự như vậy, khi bạn mở một socket, một socket descriptor cũng sẽ được sử dụng.

Quay trở lại với nội dung của bài viết. Vậy điều gì xảy khi code Python của bạn mở file mà không đóng nó lại. Rất hiển nhiên, chúng ta đã mất một file descriptor mà chúng ta sẽ không bao giờ cần đến nó nữa. Điều này đồng nghĩa với việc, số file chúng ta có thể thao tác sẽ ít dần đi, do số lượng của chúng bị giới hạn. Mà việc “quên” không đóng file nhiều khi xảy ra khá thường xuyên, và tích tiểu thành đại, đến một lúc nào đó bạn không thể mở thêm file nào nữa.

Bạn có thể dùng lệnh


ulimit -n

để kiểm tra xem hệ thống của mình cho phép mở tối đa bao nhiêu file cùng lúc.

Tất nhiên là mọi vấn đề đều có thể giải quyết được. Vẫn với ví dụ trên, chúng ta có thể xử lý bằng cách đóng từng file một như sau:


>>> files = [] >>> for _ in range(10000): ... f = open('foo', 'w') ... f.close() ... files.append(f) ... >>>


Quản lý tài nguyên hiệu quả hơn

Trên đây là một cách giải quyết tuy vẫn hoạt động tốt nhưng không được thông minh cho lắm. Trong những hệ thống phức tạp hơn, rất khó để đảm bảo rằng tất cả các file đã được đóng lại khi không dùng đến nữa.

Giả sử trong quá trình thao tác, chúng ta gặp phải một exception nào đó thì phải làm thế nào đây. Bắt exception và xử lý riêng sao? Phải bắt những exception nào mới thì gọi là đủ?

Hoặc kể cả không có exception nhưng hàm đã

return

trước khi file kịp close thì sao? Trong những trường hợp phức tạp như vậy, làm thế nào để chúng ta “nhớ” phải đóng file lại. Câu trả lời là khó tới gần như không thể (thật phũ phàng).

Trong nhiều ngôn ngữ, lập trình viên cần phải sử dụng cấu trúc kiểu như

try ... except ... finally ...

để đảm bảo rằng file sẽ được đóng. Rất may mắn, Python đã nghĩ đến những khó khăn này của chúng ta và đưa cho chúng ta một phương thức dễ dàng để làm những việc đó – context manager.

Nói một cách ngắn gọn, chúng ta cần một phương thức càng đơn giản càng tốt để đảm bảo các tài nguyên được dọn dẹp cẩn thận dù có xảy ra bất cứ chuyện gì đi chăng nữa. Và context manager sẽ cung cấp cho chúng ta tính năng này:


with something_that_returns_a_context_manager() as my_resource: do_something(my_resource) ... print('done using my_resource')


Đơn giản vậy đó. Bằng cách sử dụng

with

, chúng ta sẽ đưa mọi thứ vào trong một context manager. Chúng ta gán context manager này cho một biến, và biến đó chỉ tồn tại khi block sau đó được thực thi. Điều này giống như chúng ta tạo một hàm, nó sẽ gọi một số thao tác và khi kết thúc, nó sẽ tự dọn dẹp những gì nó tạo ra.

Một số context manager hữu ích khác

Context manager thực sự rất cần thiết trong Python, và nó đã có mặt trong thư việc chuẩn. Một số context manager có thể bạn đã từng làm việc là

zipfile.ZipFiles

,

subprocess.Popen

,

tarfile.TarFile

,

telnetlib.Telnet

,

pathlib.Path

, v.v… Thậm chí,

Lock

của

threading

cũng là context manager. Trên thực tế, tất cả những tài nguyên mà chúng ta cần

close

sau khi sử dụng đều (và rất nên) là context manager.

Việc sử dụng

Lock

tương đối đặc biệt một chút. Trong trường hợp này, tài nguyên là một mutex. Sử dụng context manager sẽ phòng tránh được deadlock trong lập trình multithread nếu chúng ta sử dụng khóa mà không bao giờ mở nó. Hãy xem xét ví dụ sau:


>>> from threading import Lock >>> lock = Lock() >>> def do_something_dangerous(): ... lock.acquire() ... raise Exception('OOPS! I forgot this code could raise exceptions') ... lock.release() ... >>> try: ... do_something_dangerous() ... except: ... print('Got an exception') ... Got an exception >>> lock.acquire()


Với code trên, rõ ràng là

lock.release()

sẽ không bao giờ được gọi, và do đó, mọi tiến trình sẽ gặp deadlock và chết cứng ở đó (

lock.acquire()

sẽ không bao giờ kết thúc). Rất may mắn, với context manager, điều này có thể sửa chữa được:


>>> from threading import Lock >>> lock = Lock() >>> def do_something_dangerous(): ... with lock: ... raise Exception('oops I forgot this code could raise exceptions') ... >>> try: ... do_something_dangerous() ... except: ... print('Got an exception') ... Got an exception >>> lock.acquire() True >>> print('We can get here') We can get here >>>


Trên thực tế, không có cách nào để gây ra deadlock nếu sử dụng context manager. Và đây là điều chúng ta đang cần.

Trong phần tiếp theo, chúng ta sẽ tìm hiểu cách cài đặt một context manager, và qua đó, chúng ta sẽ hiểu hơn về cách thức một context manager hoạt động.

Cài đặt context manager như một class

Có nhiều cách khác nhau để cài đặt một context manager. Cách đơn giản nhất là cài đặt một class với hai phương thức vô cùng đặc biệt:

__enter__

và

__exit__

. Phương thức

__enter__

sẽ trả về tài nguyên cần quản lý (ví dụ như file đang được mở) và

__exit__

sẽ làm việc dọn dẹp hệ thống.

Hãy xem xét ví dụ sau về một context manager khi làm việc với file:


>>> class File: ... def __init__(self, file_name, method): ... self.file_obj = open(file_name, method) ... def __enter__(self): ... return self.file_obj ... def __exit__(self, type, value, traceback): ... self.file_obj.close() ... >>>


Class trên cũng như nhiều class khác, phương thức

__init__

để để khởi tạo đối tượng, trong trường hợp này là khởi tạo tên file cần mở cùng với mode (đọc/ghi) của nó. Phương thức

__enter__

mở file và trả về đối tượng file để thao tác với file đó trong khi

__exit__

chỉ đơn giản là đóng file lại.

Với hai phương thức

__enter__

và

__exit__

, chúng ta có thể sử dụng class này cùng với

with

:


>>> with File('foo', 'w') as f: ... f.write('Hora! We opened this file') ... 25


Phương thức

__exit__

bắt buộc phải có 3 tham số. Dưới đây là những gì thực sự xảy ra khi chúng ta gọi context manager:

• Câu lệnh
  
with

  lưu phương thức
  
__exit__

  của class
  
File

• Câu lệnh này gọi phương thức
  
__enter__

  của class
  
File

• Phương thức
  
__enter__

  mở file và trả về object để thao tác với file đó
• Object được trả về được truyền cho biến
• Chúng ta thao tác với file bằng cách ghi dữ liệu
  
f.write

• Khi kết thúc block, câu lệnh
  
with

  gọi phương thức
  
__exit__

• Phương thức
  
__exit__

  đóng file cho chúng ta

Xử lý exception

Trong cài đặt đơn giản trên, chúng ta đã bỏ qua 3 tham số

type

,

value

,

traceback

của phương thức

__exit__

. Tuy nhiên, trong quá trình thực thi block lệnh ở trên, nếu xảy ra một exception, Python sẽ chuyển những thông tin

type

,

value

và

traceback

của exception này tới phương thức

__exit__

. Điều đó giúp chúng ta có thể tùy biến phương thức

__exit__

để xử lý những vấn đề có thể xảy ra trong quá trình thực thi. Trong trường hợp của chúng ta, chúng ta chỉ cần đóng file và không cần quan tâm đến exception này.

Nhưng chuyện gì sẽ xảy ra nếu bản thân file object gặp phải một exception? Ví dụ, khi chúng ta thử gọi một phương thức không tồn tại:


>>> with File('foo', 'w') as f: ... f.undefined_function('Oops! I called an unknown method') ... Traceback (most recent call last): File "

", line 2, in

AttributeError: '_io.TextIOWrapper' object has no attribute 'undefined_function'




Dưới đây là quy trình những gì đã xảy ra khi có lỗi xảy ra:

• 
type

  ,
  
value

  ,
  
traceback

  của lỗi đó được truyền cho
  
__exit__

• Trong phương thức
  
__exit__

  , chúng ta có thể tùy ý xử lý exception đó
• Nếu
  
__exit__

  trả về
  
True

  thì exception đã được xử lý hoàn toàn.
• Nếu không, exception sẽ tiếp tục được raise bởi lệnh
  
with


Trong trường hợp của chúng ta, phương thức

__exit__

không trả về bất cứ thứ gì, do đó, lệnh

with

sẽ raise exception.

Chúng ta có thể tạm xử lý exception như sau:


>>> class File: ... def __init__(self, file_name, method): ... self.file_obj = open(file_name, method) ... def __enter__(self): ... return self.file_obj ... def __exit__(self, type, value, traceback): ... print("Exception has been handled") ... self.file_obj.close() ... return True ... >>> with File('foo', 'w') as f: ... f.undefined_function() ... Exception has been handled >>>


Phương thức

__exit__

trả về

True

, do đó, không có exception nào được raise bởi lệnh

with

.

Có nhiều cách để cài đặt context manager. Trên đây là một cách đơn giản và dễ hiểu nhất. Trong phần tiếp theo, chúng ta sẽ tìm hiểu thêm một số phương pháp cài đặt nữa.

Sử dụng contextlib cài đặt context manager

Context manager quả là tiện lợi và hữu ích vô cùng. Do đó, trong thư viện chuẩn của Python có hẳn module

contextlib

với rất nhiều công cụ để tạo và làm việc với context manager.

Chúng ta có thể cài đặt context manager bằng cách sử dụng decorator và generator.

contextlib

cung cấp cho chúng ta decorator

@contextmanager

để decorate các hàm generator chỉ gọi

yield

đúng một lần duy nhất. Với decorator này, tất cả những gì diễn ra trước

yield

đều được coi là thao tác của phương thức

__enter__

. Những gì dễn ra sau đó được coi là của phương thức

__exit__

.

Hãy xem xét ví dụ của chúng ta về quản lý file khi dùng

contextlib

:


>>> from contextlib import contextmanager >>> @contextmanager ... def open_file(path, mode): ... f = open(path, mode) ... yield f ... f.close() ... >>> files = [] >>> for _ in range(10000): ... with open_file('foo', 'w') as f: ... files.append(f) ... >>> for f in files: ... if not f.closed: ... print('not closed') ... >>>


Như chúng ta đã thấy, việc cài đặt context manager đã ngắn gọn hơn rất nhiều. Chúng ta chỉ cần mở file,

yield

đối tượng đó và đóng nó lại. Mọi việc còn lại sẽ do decorator

@contextmanager

đảm nhiệm.

Và ví dụ thực tế cho thấy rằng các file của chúng ta đã được quản lý tốt, tất cả chúng đã được đóng lại đầy đủ. Tuy nhiên, cách thức cài đặt tiện lợi này yêu cầu chúng ta phải có chút hiểu biết về decorator, generator cũng như lệnh

yield

. Có thể tóm tắt quá trình tạo context manager trên như sau:

• Python tìm thấy
  
yield

  , hàm này là một generator chứ không phải hàm thông thường.
• Với decorator
  
@contextmanager

  , hàm
  
open_file

  sẽ được truyền là tham số cho hàm
  
contextmanager

  .
• Hàm
  
contextmanager

  trả về generator được bọc trong object của
  
GeneratorContextManager

  .
• Object
  
GeneratorContextManager

  được gán cho hàm
  
open_file

  . Do đó, khi chúng ta gọi hàm này, thực ra chúng ta đang làm việc với object
  
GeneratorContextManager

  .

Python docs còn một ví dụ khác thú vị hơn:


>>> from contextlib import contextmanager >>> @contextmanager ... def tag(name): ... print("<%s>" % name) ... yield ... print("

" % name) ... >>> with tag('h1'): ... print('foo') ...

foo

Trong tất cả các trường hợp trên, chúng ta cũng không hề xử lý exception, vì vậy, context manager của chúng ta sẽ hoạt động giống như code đầu tiên.

Một công cụ tiện lợi khác của

contextlib

là

ContextDecorator

. Nó cho phép chúng ta cài đặt các context manager theo kiểu class. Nhưng với việc kế thừa từ class

ContextDecorator

, bạn có thể sử dụng context manager với lệnh

with

thông thường, hoặc sử dụng nó như một decorator dùng để decorate các hàm khác. Chúng ta có thể xem xét ví dụ sau (tương tự như ví dụ tag HTML ở trên):


>>> from contextlib import ContextDecorator >>> class tag(ContextDecorator): ... def __init__(self, name): ... self.name = name ... def __enter__(self): ... print('<%s>' % self.name) ... return self ... def __exit__(self, *exc): ... print('

' % self.name) ... return False ... >>> with tag('h1'): ... print('this is not html') ...

this is not html

>>> @tag('h1') ... def content(): ... print('this is another non-html content') ... >>> content()

this is another non-html content

>>>


Kết luận

Bài viết trình bày những hiểu biết của tôi về context manager của Python, cách nó hoạt động và hỗ trợ cho chúng ta trong công việc lập trình. Như các bạn đã thấy, chúng ta có thể làm rất nhiều thứ với context manager. Mục đích cao nhất của nó thì không bao giờ thay đổi: quản lý hiệu quả các tài nguyên.

Chúng ta không chỉ có thể dùng context manager mà còn có thể tự cài đặt context manager cho riêng mình. Hãy sử dụng context manager và làm cho cuộc sống dễ chịu hơn.

All rights reserved

contextlib — Utilities for with-statement contexts¶

Source code: Lib/contextlib.py

This module provides utilities for common tasks involving the

with

statement. For more information see also Context Manager Types and
With Statement Context Managers.

Using Python context manager to implement the start and stop pattern

The following defines a

Timer

class that supports the context manager protocol:


from time import perf_counter class Timer: def __init__(self): self.elapsed = 0 def __enter__(self): self.start = perf_counter() return self def __exit__(self, exc_type, exc_value, exc_traceback): self.stop = perf_counter() self.elapsed = self.stop - self.start return False

Code language: Python (python)

How it works.

• First, import the
  
perf_counter

  from the
  
time

  module.
• Second, start the timer in the
  
__enter__()

  method
• Third, stop the timer in the
  
__exit__()

  method and return the elapsed time.

Now, you can use the

Timer

class to measure the time needed to calculate the Fibonacci of 1000, one million times:


def fibonacci(n): f1 = 1 f2 = 1 for i in range(n-1): f1, f2 = f2, f1 + f2 return f1 with Timer() as timer: for _ in range(1, 1000000): fibonacci(1000) print(timer.elapsed)

Code language: Python (python)

Examples and Recipes¶

This section describes some examples and recipes for making effective use of
the tools provided by

contextlib

.

Supporting a variable number of context managers¶

The primary use case for

ExitStack

is the one given in the class
documentation: supporting a variable number of context managers and other
cleanup operations in a single

with

statement. The variability
may come from the number of context managers needed being driven by user
input (such as opening a user specified collection of files), or from
some of the context managers being optional:

with ExitStack() as stack: for resource in resources: stack.enter_context(resource) if need_special_resource(): special = acquire_special_resource() stack.callback(release_special_resource, special) # Perform operations that use the acquired resources

As shown,

ExitStack

also makes it quite easy to use

with

statements to manage arbitrary resources that don’t natively support the
context management protocol.

Catching exceptions from __enter__ methods¶

It is occasionally desirable to catch exceptions from an

__enter__

method implementation, without inadvertently catching exceptions from
the

with

statement body or the context manager’s

__exit__

method. By using

ExitStack

the steps in the context management
protocol can be separated slightly in order to allow this:

stack = ExitStack() try: x = stack.enter_context(cm) except Exception: # handle __enter__ exception else: with stack: # Handle normal case

Actually needing to do this is likely to indicate that the underlying API
should be providing a direct resource management interface for use with

try

/

except

/

finally

statements, but not
all APIs are well designed in that regard. When a context manager is the
only resource management API provided, then

ExitStack

can make it
easier to handle various situations that can’t be handled directly in a

with

statement.

Cleaning up in an __enter__ implementation¶

As noted in the documentation of

ExitStack.push()

, this
method can be useful in cleaning up an already allocated resource if later
steps in the

__enter__()

implementation fail.

Here’s an example of doing this for a context manager that accepts resource acquisition and release functions, along with an optional validation function, and maps them to the context management protocol:

from contextlib import contextmanager, AbstractContextManager, ExitStack class ResourceManager(AbstractContextManager): def __init__(self, acquire_resource, release_resource, check_resource_ok=None): self.acquire_resource = acquire_resource self.release_resource = release_resource if check_resource_ok is None: def check_resource_ok(resource): return True self.check_resource_ok = check_resource_ok @contextmanager def _cleanup_on_error(self): with ExitStack() as stack: stack.push(self) yield # The validation check passed and didn’t raise an exception # Accordingly, we want to keep the resource, and pass it # back to our caller stack.pop_all() def __enter__(self): resource = self.acquire_resource() with self._cleanup_on_error(): if not self.check_resource_ok(resource): msg = “Failed validation for {!r}” raise RuntimeError(msg.format(resource)) return resource def __exit__(self, *exc_details): # We don’t need to duplicate any of our resource release logic self.release_resource()

Replacing any use of try-finally and flag variables¶

A pattern you will sometimes see is a

try-finally

statement with a flag
variable to indicate whether or not the body of the

finally

clause should
be executed. In its simplest form (that can’t already be handled just by
using an

except

clause instead), it looks something like this:

cleanup_needed = True try: result = perform_operation() if result: cleanup_needed = False finally: if cleanup_needed: cleanup_resources()

As with any

try

statement based code, this can cause problems for
development and review, because the setup code and the cleanup code can end
up being separated by arbitrarily long sections of code.


ExitStack

makes it possible to instead register a callback for
execution at the end of a

with

statement, and then later decide to skip
executing that callback:

from contextlib import ExitStack with ExitStack() as stack: stack.callback(cleanup_resources) result = perform_operation() if result: stack.pop_all()

This allows the intended cleanup up behaviour to be made explicit up front, rather than requiring a separate flag variable.

If a particular application uses this pattern a lot, it can be simplified even further by means of a small helper class:

from contextlib import ExitStack class Callback(ExitStack): def __init__(self, callback, /, *args, **kwds): super().__init__() self.callback(callback, *args, **kwds) def cancel(self): self.pop_all() with Callback(cleanup_resources) as cb: result = perform_operation() if result: cb.cancel()

If the resource cleanup isn’t already neatly bundled into a standalone
function, then it is still possible to use the decorator form of

ExitStack.callback()

to declare the resource cleanup in
advance:

from contextlib import ExitStack with ExitStack() as stack: @stack.callback def cleanup_resources(): … result = perform_operation() if result: stack.pop_all()

Due to the way the decorator protocol works, a callback function declared this way cannot take any parameters. Instead, any resources to be released must be accessed as closure variables.

Using a context manager as a function decorator¶


ContextDecorator

makes it possible to use a context manager in
both an ordinary

with

statement and also as a function decorator.

For example, it is sometimes useful to wrap functions or groups of statements
with a logger that can track the time of entry and time of exit. Rather than
writing both a function decorator and a context manager for the task,
inheriting from

ContextDecorator

provides both capabilities in a
single definition:

from contextlib import ContextDecorator import logging logging.basicConfig(level=logging.INFO) class track_entry_and_exit(ContextDecorator): def __init__(self, name): self.name = name def __enter__(self): logging.info(‘Entering: %s’, self.name) def __exit__(self, exc_type, exc, exc_tb): logging.info(‘Exiting: %s’, self.name)

Instances of this class can be used as both a context manager:

with track_entry_and_exit(‘widget loader’): print(‘Some time consuming activity goes here’) load_widget()

And also as a function decorator:

@track_entry_and_exit(‘widget loader’) def activity(): print(‘Some time consuming activity goes here’) load_widget()

Note that there is one additional limitation when using context managers
as function decorators: there’s no way to access the return value of

__enter__()

. If that value is needed, then it is still necessary to use
an explicit

with

statement.

Why use a context manager

Context managers keep our codebases much cleaner because they encapsulate administrative boilerplate and separate it from the business logic.

Additionally, context managers are structured to carry out their exit methods regardless of what happens in the code block they frame. So even if something goes wrong in the managed block, the context manager ensures the deallocations are performed and the default settings are restored.

Let’s give a solid example. Think about operating on a file without using

with

, like in the following block.

The first thing to note is that we must always close an open file. The

finally

block would perform the close even if an error occurred. If we had to do this try-except-finally logic every time we wanted to work with a file we’d have a lot of duplicate code.

Luckily, Python’s built-in

open()

is a context manager. Therefore, using a

with

statement, we can program the same logic like this:

Here,

open()

‘s enter method opens the file and returns a file object. The

as

keyword binds the returned value to , and we use to read the contents of

random.txt

. At the end of the execution of the inner code block, the exit method runs and closes the file.

We can check whether is actually closed (

with

does not define a variable scope, we can access the variables it created from outside the statement).

It’s evident from this simple example that context managers allow us to make our code cleaner and more reusable.

Python defines several other context managers in the standard library, but it also allows programmers to define context managers of their own.

In the next section, we will work on defining custom context managers. We will first work on the simple function-based implementation and later move on to the slightly more complicated class-based definitions.

Python context manager – bạn đã thực sự hiểu?

Danh mục:

2. Quản lý tài nguyên hiệu quả hơn

3. Một số context manager hữu ích khác

4. Cài đặt context manager như một class

6. Sử dụng contextlib cài đặt context manager

Context manager được sử dụng rộng rãi thông qua câu lệnh with. Ví dụ:


with open('foo', 'w') as f: f.write('Hora! We opened this file')


Đoạn code trên mở một file, ghi dữ liệu và đóng file lại. Nếu có bất kỳ lỗi gì xảy ra, thì file cũng luôn luôn được đảm bảo là đã đóng. Đoạn code trên nếu viết mà không sử dụng context manager thì sẽ trông như dưới đây:


f = open('foo', 'w') try: f.write('Hora! We opened this file') finally: f.close()


So sánh hai cách viết này thì chúng ta đã thấy rất rõ ràng rằng, context manager cho chúng ta cách viết code ngắn gọn hơn hẳn. Lệnh with cho chúng ta bảo đảm rằng file luôn luôn được đóng mà không cần biết những logic xử lý bên trong.

Hẳn là các bạn đã rất quen thuộc với những đoạn code trên, đặc biệt khi bạn đã từng nghe nói đến “Idiomatic Python“. Nhưng liệu bạn đã chắc chắn hiểu được cách làm việc chính xác với file và lý do tại sao đó lại là cách đúng không? Hoặc đơn giản hơn, bạn có biết khi nào thì mình đã thao tác sai không.

Nếu câu trả lời là không, thì bài viết này chính là dành cho bạn.

Context manager thường được sử dụng để lock các tài nguyên (trường hợp mở và đóng file là một ví dụ kinh điển cho việc này).

Tính năng quan trọng nhất và cũng là phổ biến nhất của context manager là để quản lý tài nguyên một cách chính xác. Quay lại với việc đọc và ghi file ở ví dụ trên, tại sao chúng ta phải sử dụng context manager. Mỗi khi mở một file để đọc hoặc ghi, một tài nguyên của hệ thống, trong trường hợp này là file descriptor sẽ đã bị tiêu tốn để chúng ta có thể thao tác. Thật không may là tài nguyên này lại là hữu hạn. Mỗi hệ điều hành đều có giới hạn nhất định cho số lượng file có thể mở cùng một lúc.

Không tin ư, bạn hãy xem ví dụ sau:


>>> files = [] >>> for _ in range(10000): ... files.append(open('foo', 'w')) ... Traceback (most recent call last): File "


“, line 2, in


OSError: [Errno 24] Too many open files: ‘foo’

Ngoài lề một chút, file descriptor thực chất là một số nguyên. Khi bạn mở một file, hệ điều hành sẽ tạo ra một entry (có thể ở kernel) lưu trữ những thông tin liên quan đến file được mở. Mỗi entry sẽ được gán với 1 số nguyên (và số này sẽ là duy nhất), cho phép người dùng thông qua đó để thao tác với file.

Thực chất người dùng đang thao tác một cách gián tiếp thông qua file descriptor (và thông qua entry của kernel) với các dữ liệu thật sự được ghi ở bộ nhớ ngoài. Việc này mang lại nhiều lợi ích như có thể chia sẻ file cho nhiều tiến trình khác nhau cũng như duy trì bảo mật cho các file đó.

Thực ra, hằng ngày bạn đều đang làm việc với file descriptor mà có thể bạn cũng không nhận ra. Hệ điều hành đã gán sẵn một số file descriptor như 0cho bàn phím, 1 cho màn hình, v.v… Và mọi thao tác chúng ta làm với máy tính đều thông qua các file descriptor này. Bạn nghĩ sao về việc chuyển hướngcủa câu lệnh Linux như thế này:


$ sort < file_list > sorted_file_list 2>&1


Tương tự như vậy, khi bạn mở một socket, một socket descriptor cũng sẽ được sử dụng.

Quay trở lại với nội dung của bài viết. Vậy điều gì xảy khi code Python của bạn mở file mà không đóng nó lại. Rất hiển nhiên, chúng ta đã mất một file descriptor mà chúng ta sẽ không bao giờ cần đến nó nữa. Điều này đồng nghĩa với việc, số file chúng ta có thể thao tác sẽ ít dần đi, do số lượng của chúng bị giới hạn. Mà việc “quên” không đóng file nhiều khi xảy ra khá thường xuyên, và tích tiểu thành đại, đến một lúc nào đó bạn không thể mở thêm file nào nữa.

BẠN CÓ THỂ DÙNG LỆNH ULIMIT -N ĐỂ KIỂM TRA XEM HỆ THỐNG CỦA MÌNH CHO PHÉP MỞ TỐI ĐA BAO NHIÊU FILE CÙNG LÚC.

Tất nhiên là mọi vấn đề đều có thể giải quyết được. Vẫn với ví dụ trên, chúng ta có thể xử lý bằng cách đóng từng file một như sau:


>>> files = [] >>> for _ in range(10000): ... f = open('foo', 'w') ... f.close() ... files.append(f) ... >>>


2. Quản lý tài nguyên hiệu quả hơn

Trên đây là một cách giải quyết tuy vẫn hoạt động tốt nhưng không được thông minh cho lắm. Trong những hệ thống phức tạp hơn, rất khó để đảm bảo rằng tất cả các file đã được đóng lại khi không dùng đến nữa.

Giả sử trong quá trình thao tác, chúng ta gặp phải một exception nào đó thì phải làm thế nào đây. Bắt exception và xử lý riêng sao? Phải bắt những exception nào mới thì gọi là đủ?

Hoặc kể cả không có exception nhưng hàm đã return trước khi file kịp close thì sao? Trong những trường hợp phức tạp như vậy, làm thế nào để chúng ta “nhớ” phải đóng file lại. Câu trả lời là khó tới gần như không thể (thật phũ phàng).

Trong nhiều ngôn ngữ, lập trình viên cần phải sử dụng cấu trúc kiểu như try … except … finally … để đảm bảo rằng file sẽ được đóng. Rất may mắn, Python đã nghĩ đến những khó khăn này của chúng ta và đưa cho chúng ta một phương thức dễ dàng để làm những việc đó – context manager.

Nói một cách ngắn gọn, chúng ta cần một phương thức càng đơn giản càng tốt để đảm bảo các tài nguyên được dọn dẹp cẩn thận dù có xảy ra bất cứ chuyện gì đi chăng nữa. Và context manager sẽ cung cấp cho chúng ta tính năng này:


with something_that_returns_a_context_manager() as my_resource: do_something(my_resource) ... print('done using my_resource')


Đơn giản vậy đó. Bằng cách sử dụng with, chúng ta sẽ đưa mọi thứ vào trong một context manager. Chúng ta gán context manager này cho một biến, và biến đó chỉ tồn tại khi block sau đó được thực thi. Điều này giống như chúng ta tạo một hàm, nó sẽ gọi một số thao tác và khi kết thúc, nó sẽ tự dọn dẹp những gì nó tạo ra.

3. Một số context manager hữu ích khác

Context manager thực sự rất cần thiết trong Python, và nó đã có mặt trong thư việc chuẩn. Một số context manager có thể bạn đã từng làm việc là zipfile.ZipFiles, subprocess.Popen, tarfile.TarFile, telnetlib.Telnet, pathlib.Path, v.v… Thậm chí, Lock của threadingcũng là context manager. Trên thực tế, tất cả những tài nguyên mà chúng ta cần close sau khi sử dụng đều (và rất nên) là context manager.

Việc sử dụng Lock tương đối đặc biệt một chút. Trong trường hợp này, tài nguyên là một mutex. Sử dụng context manager sẽ phòng tránh được deadlock trong lập trình multithread nếu chúng ta sử dụng khóa mà không bao giờ mở nó. Hãy xem xét ví dụ sau:


>>> from threading import Lock >>> lock = Lock() >>> def do_something_dangerous(): ... lock.acquire() ... raise Exception('OOPS! I forgot this code could raise exceptions') ... lock.release() ... >>> try: ... do_something_dangerous() ... except: ... print('Got an exception') ... Got an exception >>> lock.acquire()


Với code trên, rõ ràng là lock.release() sẽ không bao giờ được gọi, và do đó, mọi tiến trình sẽ gặp deadlock và chết cứng ở đó (lock.acquire() sẽ không bao giờ kết thúc). Rất may mắn, với context manager, điều này có thể sửa chữa được:


>>> from threading import Lock >>> lock = Lock() >>> def do_something_dangerous(): ... with lock: ... raise Exception('oops I forgot this code could raise exceptions') ... >>> try: ... do_something_dangerous() ... except: ... print('Got an exception') ... Got an exception >>> lock.acquire() True >>> print('We can get here') We can get here >>>


Trên thực tế, không có cách nào để gây ra deadlock nếu sử dụng context manager. Và đây là điều chúng ta đang cần.

Trong phần tiếp theo, chúng ta sẽ tìm hiểu cách cài đặt một context manager, và qua đó, chúng ta sẽ hiểu hơn về cách thức một context manager hoạt động.

4. Cài đặt context manager như một class

Có nhiều cách khác nhau để cài đặt một context manager. Cách đơn giản nhất là cài đặt một class với hai phương thức vô cùng đặc biệt: __enter__ và __exit__. Phương thức __enter__ sẽ trả về tài nguyên cần quản lý (ví dụ như file đang được mở) và __exit__ sẽ làm việc dọn dẹp hệ thống.

Hãy xem xét ví dụ sau về một context manager khi làm việc với file:


>>> class File: ... def __init__(self, file_name, method): ... self.file_obj = open(file_name, method) ... def __enter__(self): ... return self.file_obj ... def __exit__(self, type, value, traceback): ... self.file_obj.close() ... >>>


Class trên cũng như nhiều class khác, phương thức __init__ để để khởi tạo đối tượng, trong trường hợp này là khởi tạo tên file cần mở cùng với mode (đọc/ghi) của nó. Phương thức __enter__ mở file và trả về đối tượng file để thao tác với file đó trong khi __exit__ chỉ đơn giản là đóng file lại.

Với hai phương thức __enter__ và __exit__, chúng ta có thể sử dụng class này cùng với with:


>>> with File('foo', 'w') as f: ... f.write('Hora! We opened this file') ... 25


Phương thức __exit__ bắt buộc phải có 3 tham số. Dưới đây là những gì thực sự xảy ra khi chúng ta gọi context manager:

Câu lệnh with lưu phương thức __exit__ của class FileCâu lệnh này gọi phương thức __enter__ của class FilePhương thức __enter__ mở file và trả về object để thao tác với file đóObject được trả về được truyền cho biến fChúng ta thao tác với file bằng cách ghi dữ liệu f.writeKhi kết thúc block, câu lệnh with gọi phương thức __exit__Phương thức __exit__ đóng file cho chúng ta

Trong cài đặt đơn giản trên, chúng ta đã bỏ qua 3 tham số type, value, traceback của phương thức __exit__. Tuy nhiên, trong quá trình thực thi block lệnh ở trên, nếu xảy ra một exception, Python sẽ chuyển những thông tin type, value và traceback của exception này tới phương thức __exit__. Điều đó giúp chúng ta có thể tùy biến phương thức __exit__ để xử lý những vấn đề có thể xảy ra trong quá trình thực thi. Trong trường hợp của chúng ta, chúng ta chỉ cần đóng file và không cần quan tâm đến exception này.

Nhưng chuyện gì sẽ xảy ra nếu bản thân file object gặp phải một exception? Ví dụ, khi chúng ta thử gọi một phương thức không tồn tại:


>>> with File('foo', 'w') as f: ... f.undefined_function('Oops! I called an unknown method') ... Traceback (most recent call last): File "


“, line 2, in


AttributeError: ‘_io.TextIOWrapper’ object has no attribute ‘undefined_function’

Dưới đây là quy trình những gì đã xảy ra khi có lỗi xảy ra:

type, value, traceback của lỗi đó được truyền cho __exit__Trong phương thức __exit__, chúng ta có thể tùy ý xử lý exception đóNếu __exit__ trả về True thì exception đã được xử lý hoàn toàn.Nếu không, exception sẽ tiếp tục được raise bởi lệnh withTrong trường hợp của chúng ta, phương thức __exit__ không trả về bất cứ thứ gì, do đó, lệnh with sẽ raise exception.

Chúng ta có thể tạm xử lý exception như sau:


>>> class File: ... def __init__(self, file_name, method): ... self.file_obj = open(file_name, method) ... def __enter__(self): ... return self.file_obj ... def __exit__(self, type, value, traceback): ... print("Exception has been handled") ... self.file_obj.close() ... return True ... >>> with File('foo', 'w') as f: ... f.undefined_function() ... Exception has been handled >>>


Phương thức __exit__ trả về True, do đó, không có exception nào được raise bởi lệnh with.

Có nhiều cách để cài đặt context manager. Trên đây là một cách đơn giản và dễ hiểu nhất. Trong phần tiếp theo, chúng ta sẽ tìm hiểu thêm một số phương pháp cài đặt nữa.

6. Sử dụng contextlib cài đặt context manager

Context manager quả là tiện lợi và hữu ích vô cùng. Do đó, trong thư viện chuẩn của Python có hẳn module contextlib với rất nhiều công cụ để tạo và làm việc với context manager.

Chúng ta có thể cài đặt context manager bằng cách sử dụng decorator và generator. contextlib cung cấp cho chúng ta decorator @contextmanager để decorate các hàm generator chỉ gọi yield đúng một lần duy nhất. Với decorator này, tất cả những gì diễn ra trước yield đều được coi là thao tác của phương thức __enter__. Những gì dễn ra sau đó được coi là của phương thức __exit__.

Hãy xem xét ví dụ của chúng ta về quản lý file khi dùng contextlib:


>>> from contextlib import contextmanager >>> @contextmanager ... def open_file(path, mode): ... f = open(path, mode) ... yield f ... f.close() ... >>> files = [] >>> for _ in range(10000): ... with open_file('foo', 'w') as f: ... files.append(f) ... >>> for f in files: ... if not f.closed: ... print('not closed') ... >>>


Như chúng ta đã thấy, việc cài đặt context manager đã ngắn gọn hơn rất nhiều. Chúng ta chỉ cần mở file, yield đối tượng đó và đóng nó lại. Mọi việc còn lại sẽ do decorator @contextmanager đảm nhiệm.

Và ví dụ thực tế cho thấy rằng các file của chúng ta đã được quản lý tốt, tất cả chúng đã được đóng lại đầy đủ. Tuy nhiên, cách thức cài đặt tiện lợi này yêu cầu chúng ta phải có chút hiểu biết về decorator, generator cũng như lệnh yield. Có thể tóm tắt quá trình tạo context manager trên như sau:

Python tìm thấy yield, hàm này là một generator chứ không phải hàm thông thường.Với decorator @contextmanager, hàm open_file sẽ được truyền là tham số cho hàm contextmanager.Hàm contextmanager trả về generator được bọc trong object của GeneratorContextManager.Object GeneratorContextManager được gán cho hàm open_file. Do đó, khi chúng ta gọi hàm này, thực ra chúng ta đang làm việc với object GeneratorContextManager.Python docs còn một ví dụ khác thú vị hơn:


>>> from contextlib import contextmanager >>> @contextmanager ... def tag(name): ... print("" % name) ... yield ... print("%s>" % name) ... >>> with tag('h1'): ... print('foo') ...


foo

Trong tất cả các trường hợp trên, chúng ta cũng không hề xử lý exception, vì vậy, context manager của chúng ta sẽ hoạt động giống như code đầu tiên.

Một công cụ tiện lợi khác của contextlib là ContextDecorator. Nó cho phép chúng ta cài đặt các context manager theo kiểu class. Nhưng với việc kế thừa từ class ContextDecorator, bạn có thể sử dụng context manager với lệnh with thông thường, hoặc sử dụng nó như một decorator dùng để decorate các hàm khác. Chúng ta có thể xem xét ví dụ sau (tương tự như ví dụ tag HTML ở trên):


>>> from contextlib import ContextDecorator >>> class tag(ContextDecorator): ... def __init__(self, name): ... self.name = name ... def __enter__(self): ... print('' % self.name) ... return self ... def __exit__(self, *exc): ... print('%s>' % self.name) ... return False ... >>> with tag('h1'): ... print('this is not html') ...


this is not html

>>> @tag(‘h1’) … def content(): … print(‘this is another non-html content’) … >>> content()

this is another non-html content

>>>

Bài viết trình bày những hiểu biết của tôi về context manager của Python, cách nó hoạt động và hỗ trợ cho chúng ta trong công việc lập trình. Như các bạn đã thấy, chúng ta có thể làm rất nhiều thứ với context manager. Mục đích cao nhất của nó thì không bao giờ thay đổi: quản lý hiệu quả các tài nguyên.

Chúng ta không chỉ có thể dùng context manager mà còn có thể tự cài đặt context manager cho riêng mình. Hãy sử dụng context manager và làm cho cuộc sống dễ chịu hơn.

Bạn đang muốn tìm kiếm 1 công việc với mức thu nhập cao.✅ Hoặc là bạn đang muốn chuyển đổi công việc mà chưa biết theo học ngành nghề gì cho tốt.✅ Giới thiệu với bạn Chương trình đào tạo nhân sự dài hạn trong 12 tháng với những điều đặc biệt mà chỉ có tại IMIC và đây cũng chính là sự lựa chọn phù hợp nhất dành cho bạn:👉 Thứ nhất: Học viên được đào tạo bài bản kỹ năng, kiến thức chuyên môn lý thuyết, thực hành, thực chiến nhiều dự án và chia sẻ những kinh nghiệm thực tế từ Chuyên gia có nhiều năm kinh nghiệm dự án cũng như tâm huyết truyền nghề.👉 Thứ hai: Được ký hợp đồng cam kết chất lượng đào tạo cũng như mức lương sau tốt nghiệp và đi làm tại các đối tác tuyển dụng của IMIC. Trả lại học phí nếu không đúng những gì đã ký kết.👉 Thứ ba: Cam kết hỗ trợ giới thiệu công việc sang đối tác tuyển dụng trong vòng 10 năm liên tục.👉 Thứ tư: Được hỗ trợ tài chính với mức lãi suất 0 đồng qua ngân hàng VIB Bank.👉 Có 4 Chương trình đào tạo nhân sự dài hạn dành cho bạn lựa chọn theo học. Gồm có:1) Data Scientist full-stack2) Embedded System & IoT development full-stack3) Game development full-stack4) Web development full-stack✅ Cảm ơn bạn đã dành thời gian lắng nghe những chia sẻ của mình. Và tuyệt vời hơn nữa nếu IMIC được góp phần vào sự thành công của bạn.✅ Hãy liên hệ ngay với Phòng tư vấn tuyển sinh để được hỗ trợ về thủ tục nhập học.✅ Chúc bạn luôn có nhiều sức khỏe và thành công!

Managing Resources: In any programming language, the usage of resources like file operations or database connections is very common. But these resources are limited in supply. Therefore, the main problem lies in making sure to release these resources after usage. If they are not released then it will lead to resource leakage and may cause the system to either slow down or crash. It would be very helpful if users have a mechanism for the automatic setup and teardown of resources. In Python, it can be achieved by the usage of context managers which facilitate the proper handling of resources. The most common way of performing file operations is by using the keyword as shown below:

Output:

Traceback (most recent call last):
File “context.py”, line 3, in
OSError: [Errno 24] Too many open files: ‘test.txt’

An error message saying that too many files are open. The above example is a case of file descriptor leakage. It happens because there are too many open files and they are not closed. There might be chances where a programmer may forget to close an opened file.

Managing Resources using context manager: Suppose a block of code raises an exception or if it has a complex algorithm with multiple return paths, it becomes cumbersome to close a file in all the places. Generally in other languages when working with files try-except-finally is used to ensure that the file resource is closed after usage even if there is an exception. Python provides an easy way to manage resources: Context Managers. The with keyword is used. When it gets evaluated it should result in an object that performs context management. Context managers can be written using classes or functions(with decorators).

Creating a Context Manager: When creating context managers using classes, user need to ensure that the class has the methods: __enter__() and __exit__(). The __enter__() returns the resource that needs to be managed and the __exit__() does not return anything but performs the cleanup operations. First, let us create a simple class called ContextManager to understand the basic structure of creating context managers using classes, as shown below:

Writing Good APIs With Context Managers

Context managers are quite flexible, and if you use the

with

statement creatively, then you can define convenient APIs for your classes, modules, and packages.

For example, what if the resource you wanted to manage is the text indentation level in some kind of report generator application? In that case, you could write code like this:


with Indenter() as indent: indent.print("hi!") with indent: indent.print("hello") with indent: indent.print("bonjour") indent.print("hey")


This almost reads like a domain-specific language (DSL) for indenting text. Also, notice how this code enters and leaves the same context manager multiple times to switch between different indentation levels. Running this code snippet leads to the following output and prints neatly formatted text:


hi! hello bonjour hey


How would you implement a context manager to support this functionality? This could be a great exercise to wrap your head around how context managers work. So, before you check out the implementation below, you might take some time and try to solve this by yourself as a learning exercise.

Ready? Here’s how you might implement this functionality using a context manager class:


class Indenter: def __init__(self): self.level = -1 def __enter__(self): self.level += 1 return self def __exit__(self, exc_type, exc_value, exc_tb): self.level -= 1 def print(self, text): print(" " * self.level + text)


Here,

.__enter__()

increments

.level

by every time the flow of execution enters the context. The method also returns the current instance,

self

. In

.__exit__()

, you decrease

.level

so the printed text steps back one indentation level every time you exit the context.

The key point in this example is that returning

self

from

.__enter__()

allows you to reuse the same context manager across several nested

with

statements. This changes the text indentation level every time you enter and leave a given context.

A good exercise for you at this point would be to write a function-based version of this context manager. Go ahead and give it a try!

Creating an Asynchronous Context Manager

To create an asynchronous context manager, you need to define the

.__aenter__()

and

.__aexit__()

methods. The script below is a reimplementation of the original script

site_checker_v0.py

you saw before, but this time you provide a custom asynchronous context manager to wrap the session creation and closing functionalities:


# site_checker_v1.py import aiohttp import asyncio class AsyncSession: def __init__(self, url): self._url = url async def __aenter__(self): self.session = aiohttp.ClientSession() response = await self.session.get(self._url) return response async def __aexit__(self, exc_type, exc_value, exc_tb): await self.session.close() async def check(url): async with AsyncSession(url) as response: print(f"{url}: status -> {response.status}") html = await response.text() print(f"{url}: type -> {html[:17].strip()}") async def main(): await asyncio.gather( check("https://realpython.com"), check("https://pycoders.com"), ) asyncio.run(main())


This script works similar to its previous version,

site_checker_v0.py

. The main difference is that, in this example, you extract the logic of the original outer

async with

statement and encapsulate it in

AsyncSession

.

In

.__aenter__()

, you create an

aiohttp.ClientSession()

, await the

.get()

response, and finally return the response itself. In

.__aexit__()

, you close the session, which corresponds to the teardown logic in this specific case. Note that

.__aenter__()

and

.__aexit__()

must return awaitable objects. In other words, you must define them with

async def

, which returns a coroutine object that is awaitable by definition.

If you run the script from your command line, then you get an output similar to this:


$ python site_checker_v1.py https://realpython.com: status -> 200 https://pycoders.com: status -> 200 https://realpython.com: type ->


https://pycoders.com: type ->

Great! Your script works just like its first version. It sends

GET

requests to both sites concurrently and processes the corresponding responses.

Finally, a common practice when you’re writing asynchronous context managers is to implement the four special methods:

1. 
.__aenter__()

2. 
.__aexit__()

3. 
.__enter__()

4. 
.__exit__()


This makes your context manager usable with both variations of

with

.

Creating Custom Context Managers

You’ve already worked with context managers from the standard library and third-party libraries. There’s nothing special or magical about

open()

,

threading.Lock

,

decimal.localcontext()

, or the others. They just return objects that implement the context management protocol.

You can provide the same functionality by implementing both the

.__enter__()

and the

.__exit__()

special methods in your class-based context managers. You can also create custom function-based context managers using the

contextlib.contextmanager

decorator from the standard library and an appropriately coded generator function.

In general, context managers and the

with

statement aren’t limited to resource management. They allow you to provide and reuse common setup and teardown code. In other words, with context managers, you can perform any pair of operations that needs to be done before and after another operation or procedure, such as:

• Open and close
• Lock and release
• Change and reset
• Create and delete
• Enter and exit
• Start and stop
• Setup and teardown

You can provide code to safely manage any of these pairs of operations in a context manager. Then you can reuse that context manager in

with

statements throughout your code. This prevents errors and reduces repetitive boilerplate code. It also makes your APIs safer, cleaner, and more user-friendly.

In the next two sections, you’ll learn the basics of creating class-based and function-based context managers.

Creating Context Managers

Function-Based Implementation

The standard library provides

contextlib

, a module containing

with

statement utilities. One important helper is the

@contextmanager

decorator, which lets us define function-based context managers.

Let’s try it out by making a context manager function that swaps the case of a string:

The

@contextmanager

decorator expects a

yield

statement in the function body.

yield

divides the function into three parts:

1. The expressions above
   
yield

   are executed right before the code that is managed.
2. The managed code runs at
   
yield

   . Whatever is yielded makes up the context variable—
   
swapped_string

   in our case.
3. The expressions below
   
yield

   are executed after the managed code is run.

Notice that we have placed

yield

within a

try-except-finally

block. This is not enforced, yet, it is good practice. This way, if an error occurs in the managed code the context manager will carry out its exit logic no matter what.

Look what happens if we raise a

ValueError

inside the

with

statement:

The

except

block within

example_cm

handled the exception, and the

finally

block ensured we saw the exit message.

Notice that the “End of context manager” string is still printed. We’re specifically catching a

ValueError

, but let’s see what happens when an unhandled error is raised:

In this case, the

finally

block still executed regardless of the uncaught error, but the “End of context manager” printout didn’t make it. This exemplifies why a

finally

block is helpful in many situations.

For more flexibility in creating context managers, we’ll now introduce the class-based implementations.

Class-based implementation

Python defines a context management protocol that dictates that any class with an

__enter__()

and an

__exit__()

method can work as a context manager.

Double underscore methods are special methods that are not called but instead triggered. They are internally set to run at specific times or after certain events. For example,

__enter()__

runs when a

with

statement is entered and

__exit__

runs right before the

with

block is left.

We have seen an example of function-based context managers, which work great for quick and simple cases. For more complex use cases, we’ll define context management as an additional ability to an existing class.

Let’s create a

ListProtect

context manager class, which will operate on a copy of a list before returning the changes. This way, the original list would be restored unaltered if an error occurred during the operation.

Let’s see how that looks.

And we can use it like so:

The list was successfully modified. But what if an error occurs?

In this instance,

ListProtect

caught an error and the list remained unaltered. So how does this work?

Explanation

In the

ListProtect

class, we defined two required methods:


__enter__()

– this method defines what happens before the logic under the

with

statement runs. If the enter method returns anything, it is bound to the context variable. In our class, we create a clone of the list and returned it as the context variable—

the_copy

in the above examples.


__exit__()

– this method defines what happens when the

with

logic is complete or has raised an error. Besides

self

, this method takes three parameters:

exc_type

,

exc_val

,

exc_tb

, which can also be shortened to

*exc

. If no exception occurs in the managee, all these values are None. If we return a truthy value at the end of

__exit()__

—as we did in our class—the error is suppressed. Otherwise, if we return a falsy value, such as

None

,

False

, or blank, the error is propagated.

In short,

ListProtect

‘s

__exit()__

method first checks whether the exception type was

None

. If so, there were no errors, and it applies the changes to the original list; otherwise, it announces an error occurred.

Creating Context Managers

Function-Based Implementation

The standard library provides

contextlib

, a module containing

with

statement utilities. One important helper is the

@contextmanager

decorator, which lets us define function-based context managers.

Let’s try it out by making a context manager function that swaps the case of a string:

The

@contextmanager

decorator expects a

yield

statement in the function body.

yield

divides the function into three parts:

1. The expressions above
   
yield

   are executed right before the code that is managed.
2. The managed code runs at
   
yield

   . Whatever is yielded makes up the context variable—
   
swapped_string

   in our case.
3. The expressions below
   
yield

   are executed after the managed code is run.

Notice that we have placed

yield

within a

try-except-finally

block. This is not enforced, yet, it is good practice. This way, if an error occurs in the managed code the context manager will carry out its exit logic no matter what.

Look what happens if we raise a

ValueError

inside the

with

statement:

The

except

block within

example_cm

handled the exception, and the

finally

block ensured we saw the exit message.

Notice that the “End of context manager” string is still printed. We’re specifically catching a

ValueError

, but let’s see what happens when an unhandled error is raised:

In this case, the

finally

block still executed regardless of the uncaught error, but the “End of context manager” printout didn’t make it. This exemplifies why a

finally

block is helpful in many situations.

For more flexibility in creating context managers, we’ll now introduce the class-based implementations.

Class-based implementation

Python defines a context management protocol that dictates that any class with an

__enter__()

and an

__exit__()

method can work as a context manager.

Double underscore methods are special methods that are not called but instead triggered. They are internally set to run at specific times or after certain events. For example,

__enter()__

runs when a

with

statement is entered and

__exit__

runs right before the

with

block is left.

We have seen an example of function-based context managers, which work great for quick and simple cases. For more complex use cases, we’ll define context management as an additional ability to an existing class.

Let’s create a

ListProtect

context manager class, which will operate on a copy of a list before returning the changes. This way, the original list would be restored unaltered if an error occurred during the operation.

Let’s see how that looks.

And we can use it like so:

The list was successfully modified. But what if an error occurs?

In this instance,

ListProtect

caught an error and the list remained unaltered. So how does this work?

Explanation

In the

ListProtect

class, we defined two required methods:


__enter__()

– this method defines what happens before the logic under the

with

statement runs. If the enter method returns anything, it is bound to the context variable. In our class, we create a clone of the list and returned it as the context variable—

the_copy

in the above examples.


__exit__()

– this method defines what happens when the

with

logic is complete or has raised an error. Besides

self

, this method takes three parameters:

exc_type

,

exc_val

,

exc_tb

, which can also be shortened to

*exc

. If no exception occurs in the managee, all these values are None. If we return a truthy value at the end of

__exit()__

—as we did in our class—the error is suppressed. Otherwise, if we return a falsy value, such as

None

,

False

, or blank, the error is propagated.

In short,

ListProtect

‘s

__exit()__

method first checks whether the exception type was

None

. If so, there were no errors, and it applies the changes to the original list; otherwise, it announces an error occurred.

Python3

from pymongo import MongoClient class MongoDBConnectionManager(): def __init__( self , hostname, port): self .hostname hostname self .port port self .connection None def __enter__( self ): self .connection MongoClient( self .hostname, self .port) return self .connection def __exit__( self , exc_type, exc_value, exc_traceback): self .connection.close() with MongoDBConnectionManager( 'localhost' '27017' ) as mongo: collection mongo.connection.SampleDb.test data collection.find({ '_id' }) print (data.get( 'name' ))

Database connection management using context manager and with statement: On executing the with block, the following operations happen in sequence:

• A MongoDBConnectionManager object is created with localhost as the hostname name and 27017 as the port when the __init__ method is executed.
• The __enter__ method opens the MongoDB connection and returns the MongoClient object to variable mongo.
• The test collection in the SampleDb database is accessed and the document with _id=1 is retrieved. The name field of the document is printed.
• The __exit__ method takes care of closing the connection on exiting the with block(teardown operation).