9

In contextlib.py, I see the ExitStack class is calling __enter__() method via the type object (type(cm)) instead of direct method calls to the given object (cm).

I wonder why or why not.

e.g.,

  • does it give better exception traces when an error occurs?
  • is it just specific to some module author's coding style?
  • does it have any performance benefits?
  • does it avoid some artifacts/side-effects with complicated type hierarchies?
Andrea Corbellini
  • 17,339
  • 3
  • 53
  • 69
Achimnol
  • 1,551
  • 3
  • 19
  • 31

1 Answers1

14

First of all, this is what happens when you do with something, it's not just contextlib that looks up special method on the type. Also, it's worth noting that the same happens with other special methods too: e.g. a + b results in type(a).__add__(a, b).

But why does it happen? This is a question that is often fired up on the python-dev and python-ideas mailing lists. And when I say "often", I mean "very often".

The last one were these: Missing Core Feature: + - * / | & do not call getattr and Eliminating special method lookup.

Here are some interesting points:

The current behaviour is by design - special methods are looked up as slots on the object's class, not as instance attributes. This allows the interpreter to bypass several steps in the normal instance attribute lookup process.

(Source)

It is worth noting that the behavior is even more magical than this. Even when looked up on the class, implicit special method lookup bypasses __getattr__ and __getattribute__ of the metaclass. So the special method lookup is not just an ordinary lookup that happens to start on the class instead of the instance; it is a fully magic lookup that does not engage the usual attribute-access-customization hooks at any level.

(Source)

This behavior is also documented on the reference documentation: Special method lookup, which says:

Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).

In short, performance is the main concern. But let's take a closer look at this.

What's the difference between type(obj).__enter__() and obj.__enter__()?

When you write obj.attr, type(obj).__getattribute__('attr') gets called. The default implementation of __getattribute__() looks for attr into the instance dictionary (i.e. obj.__dict__) and into the class namespace and, failing that, calls type(obj).__getattr__('attr').

Now, this was a quick explanation and I have omitted some details, however it should give you an idea of how complicated an attribute lookup can be, and how slow it can become. Short circuiting special method lookup surely provides performance improvements, as looking up obj.__enter__() in the "classical" way may be too slow.

Andrea Corbellini
  • 17,339
  • 3
  • 53
  • 69
  • Thanks for your answer. Still, I wonder why `type(a).__add__(a, b)` instead of `a.__add__(b)`. Aren't they same? – Achimnol Dec 28 '15 at 09:49
  • 2
    @Achimnol: They're not the same if you assign something to `a.__add__`. – user2357112 Dec 28 '15 at 10:07
  • 1
    @Achimnol: I think I have answered that question as well, but please tell me if I was too vague – Andrea Corbellini Dec 28 '15 at 10:13
  • Thanks, I have noticed that Python has "a lot of" abstraction layers such like that. (So using `__slots__` for container-like classes could improve the performance and reduce memory usage.) In short, the reason is to bypass instance dictionary lookup since usually `__enter__` and other special methods are defined along with classes (=types). Am I right? – Achimnol Dec 28 '15 at 10:17
  • @Achimnol: The reason why the C layer does it is for performance; special methods have behaviors reserved by Python, and it's allowed to say "Nope, these *won't* be looked up on the instance because it's faster not to". The reason why `contextlib.py` does it, even though it's uglier and slower than the direct call (in cases where the behavior is otherwise equivalent), is for consistency with the C layer behaviors. You wouldn't want to have `__enter__` found on the class when using `with` directly, but on the instance when wrapping in `ExitStack`. That's a crazy subtle source of bugs. – ShadowRanger Nov 16 '22 at 17:09