How to unpickle an object whose class exists in a different namespace (python)?

Question

If I have a script that defines a class:

script = """

class myClass:
    def __init__(self):
        self.name = 'apple'
        self.color = 'green'

"""

and then exec this script in its own namespace dict:

NS = {}
exec script in NS

and then create an instance of the class and pickle it:

a = NS['myClass']()

import pickle

save = pickle.dumps(a)

Now if I try to unpickle it:

load = pickle.loads(save)

I get the error

AttributeError: 'module' object has no attribute 'myClass'

I gather that this doesn't work because python doesn't know where to find myClass in order to rebuild the object. But myClass does exist in the NS dict. Is there a way to tell pickle where to find the class for the object it is loading?

score 6 · Accepted Answer · answered Jan 09 '13 at 20:12

I discovered a solution this. It seems the problem is executing code in a dict prevents python from figuring out where the class is defined. The solution is to create an empty module, execute the code in the module, and then add the module to sys.modules so python knows about it.

script = """
class myClass:
    def __init__(self):
        self.name = 'apple'
        self.color = 'green'
"""

import imp, sys

moduleName = 'custom'

module = imp.new_module(moduleName)

exec script in module.__dict__

sys.modules[moduleName] = module

Now it is possible to pickle and unpickle an instance of the class:

import pickle
a = module.myClass()
s = pickle.dumps(a)
b = pickle.loads(s)

This solution is a bit hackish IMO. But it works! So..thank you. In my case, I dont have a control over the receiving side of my pickle. Otherwise Borealid's answer can be used — Joshua H, Sep 07 '18 at 05:58

score 5 · Answer 2 · edited Jun 20 '13 at 17:08

You can actually go one step further, and have the object reconstruct itself into whatever type you want.

import pickle
import copy_reg

class myClass(object):
    def __init__(self):
        self.apple = 'banana'

class otherclass(object):
    def __init__(self):
        self.apple = 'existential woe'

def pickle_an_object(o):
    print "pickling %s" % str(o)
    return otherclass, (o.apple,)

copy_reg.pickle(myClass, pickle_an_object)

foo = myClass()

s = pickle.dumps(foo)

del myClass
del otherclass

class otherclass(object):
    def __init__(self, appletype):
        self.apple = 'not %s' % appletype

o2 = pickle.loads(s)

print o2.apple

The basic idea is that you pack your class into a "trojan horse" of sorts, where its reconstruction causes an instantiation of a different class from what it originally was.

It does not matter what the otherclass on the pickling side contains. All that matters is that it exist at the same module path as the "destination" class - pickle is just putting a string representation of the module name into the serialized stream.

So, to break down what's happening in the above code in detail:

We register a custom pickler for myClass. This can be done via copy_reg or the __reduce_ex__ function.
Our custom pickler says "pickle this as an instance of otherclass" (which is a dummy. You do not need the "real" contents of otherclass on the pickling side, because all that goes into the pickle is the module/class name).
We pickle the object and "send it across the wire", to where the real version of otherclass exists.
On the remote side, otherclass is instantiated with the data from the tuple returned by the custom pickling function.

Python can be pretty powerful!

This is all very neat, but how does it answer the original question? — John Y, Jan 09 '13 at 16:19
@JohnY You can make the class unpickle into an instance of an arbitrary class from the unpickler's namespace. You set up `myClass` to unpickle as `foomodule.myClass`, and then you say `foomodule.myClass = NS['myClass']` before calling `pickle.loads`. — Borealid, Jan 09 '13 at 16:30

How to unpickle an object whose class exists in a different namespace (python)?

2 Answers2

Linked