Can pickle object only when part of dictionary?

Question

I've got a class instance (called case), in Python 2.7.6, subclassed from object. It has several methods but also stores a bunch of data. I'd like to pickle this thing. However, if I try to "just do it", I get an error:

import cPickle
saver = cPickle.Pickler(sfile)
saver.dump(case)
    File "/usr/lib64/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
    TypeError: can't pickle function objects

But then I realized that I can pickle case.__dict__ without problems, but one of its attributes is an instance of a similar class! Why can I pickle that but not this? So I tried the following:

casedict = {'testcase1':case}
saver.dump(casedict)

    <cPickle.Pickler object at 0x6fef398>

...so: apparently I cannot pickle the pure object but I can pickle it if it's part of a dictionary? What's the difference? I'm having a hard time understanding what can be pickled and what cannot, and this is extremely confusing to me.

(1) you should show your code, so others can help easily, and (2) you might want to try `dill` instead of `cPickle`. I'd wager that using `dill` will fix your pickling error. Most objects **CAN'T** be pickled with `cPickle`, however `dill` can pickle most objects in python. — Mike McKerns, Sep 18 '15 at 17:43
The actual classes I'm working with are too large to put in here, and I wouldn't be allowed to show them anyway. I've no idea which aspect is important, though. I managed to pickle one instance with dill, but not another (complains about ``Can't pickle : it's not the same object as numpy.int64``) when all instances should be the same apart for some numbers! => I was hoping someone could tell me where to look, e.g. why a dictionary would pickle but it's components would not, for a start — Zak, Sep 18 '15 at 19:30
Yes, this is a known issue, and doesn't work because a `numpy.int64` is not an `int`… it's an instance of a class that is partially built in C… but more importantly, `numpy.int64` has had some namespace mangling in the import that `dill` can't backtrack and solve. — Mike McKerns, Sep 18 '15 at 20:09
Does this mean I can't use numpy ``int``s if I want to pickle things? Do floats work better (I have plenty of either)? — Zak, Sep 21 '15 at 09:12
Alright, I found a solution, which is remove one of the instance's attributes (which was an instance of another class), then use dill to save the instance's dictionary (would still not work on the instance), and changing the class to have a function to reconstruct an instance from a dictionary. -- I was also not able to reconstruct the original problem, so the question is now somewhat useless ... should I delete it? Is there a standard procedure? — Zak, Sep 21 '15 at 17:02
I believe a good procedure would to post it as an answer, then decide if you want to accept your own answer or one to the others. — Mike McKerns, Sep 21 '15 at 23:48

score 1 · Answer 1 · answered Sep 18 '15 at 20:06

Look at pickling a dict. It's easy to do, until you put something in it that can't be pickled.

>>> import pickle
>>> pickle.dumps({})
'(dp0\n.'
>>> pickle.dumps({'a':1})
"(dp0\nS'a'\np1\nI1\ns."
>>> pickle.dumps({'a':1, 'b':(lambda x:x)})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <function <lambda> at 0x104c765f0>: it's not found as __main__.<lambda>
>>>

Now, let's use dill, with pickle trace turned on, so we can see the path the serializer takes. dill can serialize a lambda, so it doesn't fail.

>>> import dill
>>> 
>>> dill.detect.trace(True)
>>> pickle.dumps({})
D2: <dict object at 0x104c3a4b0>
# D2
'(dp0\n.'
>>> pickle.dumps({'a':1})
D2: <dict object at 0x104c3a4b0>
# D2
"(dp0\nS'a'\np1\nI1\ns."
>>> pickle.dumps({'a':1, 'b':(lambda x:x)})
D2: <dict object at 0x104c3a4b0>
F1: <function <lambda> at 0x105c729b0>
F2: <function _create_function at 0x105c0c938>
# F2
Co: <code object <lambda> at 0x104e70830, file "<stdin>", line 1>
F2: <function _unmarshal at 0x105c0c7d0>
# F2
# Co
D3: <dict object at 0x104b95168>
# D3
D2: <dict object at 0x105c30280>
# D2
# F1
# D2
"(dp0\nS'a'\np1\nI1\nsS'b'\np2\ncdill.dill\n_create_function\np3\n(cdill.dill\n_unmarshal\np4\n(S'c\\x01\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x01\\x00\\x00\\x00C\\x00\\x00\\x00s\\x04\\x00\\x00\\x00|\\x00\\x00S(\\x01\\x00\\x00\\x00N(\\x00\\x00\\x00\\x00(\\x01\\x00\\x00\\x00t\\x01\\x00\\x00\\x00x(\\x00\\x00\\x00\\x00(\\x00\\x00\\x00\\x00s\\x07\\x00\\x00\\x00<stdin>t\\x08\\x00\\x00\\x00<lambda>\\x01\\x00\\x00\\x00s\\x00\\x00\\x00\\x00'\np5\ntp6\nRp7\nc__main__\n__dict__\nS'<lambda>'\np8\nNN(dp9\ntp10\nRp11\ns."

You can see with the empty dict, just a dict (i.e. D2) is pickled, while with the lambda in the dict, there are several objects that get pickled…

the enclosing dict (D2)
the lambda (F1)
several helper functions (the F2s)
the code object from the lambda (Co)
the lambda's __dict__ (D2)
the __main__ __dict__ (i.e. globals()) (D3)

So, pickling a dict is easy… but if what's inside can't be pickled, then the dict can't be pickled.

Thanks for the explanation. It does make things clearer. The funny bit is that I could not pickle an object stand-alone but apparently could when it was part of a dictionary. -- I'm currently trying to figure out how to isolate the problem — Zak, Sep 21 '15 at 08:44
@Zak: you might want to try the methods for pickle debugging in `dill.detect`, see: http://stackoverflow.com/questions/17872056/how-to-check-if-an-object-is-pickleable/32206955#32206955 — Mike McKerns, Sep 21 '15 at 23:45

Can pickle object only when part of dictionary?

1 Answers1