Assigning vs. Defining Python Magic Methods

Question

Consider the following abhorrent class:

class MapInt:
    __call__ = int

    def __sub__(self, other):
        return map(self, other)

    __add__ = map

One can then call map(int, lst) via MapInt() - lst, i.e.

assert list(MapInt() - ['1','2','3'])) == [1,2,3]    # passes

However, addition is not so cooperative:

assert list(MapInt() + ['1','2','3'])) == [1,2,3]    # TypeError: map() must have at least two arguments.

This strangeness can be resolve by invoking the magic method directly:

assert list(MapInt.__add__(MapInt(), ['1','2','3']))   == [1,2,3]    # passes
assert list(MapInt().__add__(MapInt(), ['1','2','3'])) == [1,2,3]    # passes

So my question is, what gives? Assigning __add__ directly seems to "discard" the self argument, but invoking the method itself or defining it in the standard way works fine.

Answer 1

The transformation of instance methods is described in the Python Data Model (emphasis mine):

Note that the transformation from function object to instance method object happens each time the attribute is retrieved from the instance [...] Also notice that this transformation only happens for user-defined functions; other callable objects (and all non-callable objects) are retrieved without transformation.

Since map is a built-in, not a user-defined function, there is no transformation to an instance method, so the self argument is not added.

Answer 2

It doesn't have anything to do with the difference between "assigning" and/or "defining". You could even add functions dynamically after the creation of the class. They are all the same.

This is the behavior of a descriptor.

class Function:
    ...

    def __get__(self, obj, objtype=None):
        "Simulate func_descr_get() in Objects/funcobject.c"
        if obj is None:
            return self
        return MethodType(self, obj)

Your __sub__ is a descriptor. If it is called from an instance, it will give you a MethodType object whose first parameter is filled with the reference to the instance and that's why your first assert statement works.

Actually all user-defined functions are descriptors but built-in callables(functions/classes) like map in this case or other callable objects like classes are not so don't except that their first parameter is going to be filled with something.

Answer 3

@sj95126 has already given you the answer.

Here's some more insight

class Foo:
    pass

def wrapped(*args, **kwds):
    return dir(*args, **kwds)

Foo.dir = dir
Foo.wrapped = wrapped

f = Foo()

print(f'd()  : {dir()}')
print(f'w()  : {wrapped()}')
print(f'd(f) : {dir(f)}')
print(f'w(f) : {wrapped(f)}')
print(f'f.d(): {f.dir()}')
print(f'f.w(): {f.wrapped()}')

which gives the output

d()  : ['Foo', '__PYDOC_get_help', '__PYTHON_EL_eval', '__PYTHON_EL_eval_file', '__PYTHON_EL_native_completion_setup', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '__warningregistry__', 'dir_wrapped', 'dir_wrapped_in_pure_python_fn', 'f', 'wrapped']
w()  : ['args', 'kwds']
d(f) : ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'dir', 'wrapped']
w(f) : ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'dir', 'wrapped']
f.d(): ['Foo', '__PYDOC_get_help', '__PYTHON_EL_eval', '__PYTHON_EL_eval_file', '__PYTHON_EL_native_completion_setup', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '__warningregistry__', 'dir_wrapped', 'dir_wrapped_in_pure_python_fn', 'f', 'wrapped']
f.w(): ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'dir', 'wrapped']

Note that:

1. d(f), w(f) and f.w() give identical output.
2. d() and f.d() give identical output.
3. w() is the odd one out.

In group 1 above, f is passed as an argument to the function. In d(f) and w(f) it is done explicitly. In f.w() it is done implicitly by the binding behaviour of the pure-Python function wrapped.

Because dir is not a pure-Python function, this binding behaviour is absent, and f is not passed in, so its output is the same as that of d().

(Irrelevant to your question, but for completeness: wrapped() passes zero arguments to wrapped therefore it passes zero arguments to dir and calling dir with zero arguments makes it return the list of names in the scope in which it was called, that is the local scope of wrapped in which the only names are the parameters args and kwds.

Answer 4

This has to do with how python binds objects to an instance with the dot operator. When you create a class body, it contains a bunch of objects that become attributes living in the class dictionary. It does not matter how an attribute ends up in the class dictionary, the behavior will be the same. For example:

class MapInt:
    __call__ = int

    def __sub__(self, other):
        return map(self, other)

    __add__ = map

MapInt.test = property(lambda self: self.__dict__.get('test'))

mi = MapInt()

If you try to access those class attributes through an instance, one of three things will happen:

1. If the attribute has both __get__ and __set__, it's a data attribute. Data attributes shadow even instance dictionary items. A property object is a good example of a data attribute. Even if mi.test = 1 raises an AttributeError, MapInt.test has a __set__ method. As a consequence, mi.test will never access mi.__dict__['test'], which allow you to hide the property value there.

   >>> MapInt.test.__get__
   <method-wrapper '__get__' of property object at 0x7f264a09de50>
   >>> MapInt.test.__set__
   <method-wrapper '__set__' of property object at 0x7f264a09de50>
   

2. If the attribute only has a __get__ descriptor, it's a non-data attribute. Functions, lambdas, and some other objects are non-data descriptors. A function gets bounds to the instance when you access it. So mi.__sub__(...) gets resolved as MapInt.__sub__.__get__(mi)(...). A function's __get__ creates a partial that automatically passes self in the usual manner. Descriptors like classmethod and staticmethod do other things in __get__, as you can imagine.

   >>> MapInt.__sub__.__get__
   <method-wrapper '__get__' of function object at 0x7f2649ec4e50>
   >>> MapInt.__sub__.__set__
   ...
   AttributeError: 'function' object has no attribute '__set__'
   

3. If the attribute is not a descriptor, as determined by the presence of __get__ and __set__, then the object is returned as-is. In your particular case, mi.__call__ and mi.__add__ are int and map, respectively. Neither of those types has a __get__ method or is a descriptor, so it will not bind as a method.

   >>> MapInt.__call__.__get__
   ...
   AttributeError: type object 'int' has no attribute '__get__'
   >>> MapInt.__add__.__get__
   ...
   AttributeError: type object 'map' has no attribute '__get__'