1

I'm working on a game project that uses Pickle to implement savegames (I know the downsides of doing so - let's not discuss that here). This works excellently: unfortunately standard Pickle cannot handle some of the things I wish to do in future versions, so I'm transitioning to Dill instead. Unfortunately, it doesn't work: it gives _pickle.UnpicklingError: pickle exhausted before end of frame whenever it loads a save.

To reiterate: this code and the test cases I'm using work perfectly with Pickle. It only has issues with Dill.

I'm importing Dill like this:

try:
    import dill as pickle
except ImportError:
    print("Failed to load Dill serialization library: some features may not work correctly.")
    import pickle

And the full traceback is as such:

   Traceback (most recent call last):
  File "C:/Users/Schilcote/workspace/pyweek19/main.py", line 605, in game_init
    gamestate=pickle.load(open(os.path.join(_savedir,"save.sav"),"rb"))
  File "C:\Python34\lib\site-packages\dill\dill.py", line 199, in load
    obj = pik.load()
  File "C:\Python34\Lib\pickle.py", line 1036, in load
    dispatch[key[0]](self)
  File "C:\Python34\Lib\pickle.py", line 1321, in load_global
    module = self.readline()[:-1].decode("utf-8")
  File "C:\Python34\Lib\pickle.py", line 247, in readline
    "pickle exhausted before end of frame")
_pickle.UnpicklingError: pickle exhausted before end of frame

I don't know how to even begin diagnosing this. What's going on?

EDIT:

To clarify, the error comes upon deserialization.

Schilcote
  • 2,344
  • 1
  • 17
  • 35
  • How was the `save.sav` file that you use in your test generated? i.e. was it generated using `pickle` or `dill`? – Sebastian Dec 16 '15 at 19:27
  • @Sebastian With `dill`. – Schilcote Dec 16 '15 at 19:38
  • @Sebastian To clarify, I always test `pickle` against a `pickle` generated file and `dill` against a `dill` generated file. I haven't tried intermixing the two but I doubt anything good will happen. – Schilcote Dec 16 '15 at 19:45
  • ^1 because that title – brandonscript Dec 16 '15 at 20:27
  • I'm pretty sure this is a straight-up bug in Dill. @Mike McKerns? – Schilcote Dec 17 '15 at 02:09
  • I can't tell what you are doing from the traceback. You might want to (1) turn on `dill.detect.trace(True)` to see what is being serialized (and how), or (2) try serializing whatever causes problems with `dill.settings['byref'] = True` (or on a per object basis with the `byref` keyword in `dumps`). – Mike McKerns Dec 17 '15 at 05:45
  • @MikeMcKerns One of those showed me that I was trying to pickle some unpickleable objects, which I fixed, but now I'm having even weirder problems. The game has a REPL simulator object based on the ones in `code`, and sometimes when that gets serialized the resultant file causes the same "exaughsted before end of frame" error as before. What's really weird is that it depends what was entered in the console; "This will fail" works every time, but "foo" causes failure every time. Are there any other debugging options in Dill I can turn on? – Schilcote Dec 18 '15 at 16:48
  • `dill` has alternate serialization settings in `dill.settings`, and allows you to change how objects are serialized globally by changing the settings… or to change per-`dump` as a keyword in `dump`. For debugging, I use `trace` as suggested above, however, there are other functions in `dill.detect` that are for debugging, as well as `pickletools.dis` available in the standard library. For debugging, I typically use `dill.detect.trace` and `pickletools.dis`. – Mike McKerns Dec 20 '15 at 18:58
  • See: http://stackoverflow.com/a/32206955/2379433. – Mike McKerns Dec 20 '15 at 18:59
  • Also, if you are checking pickling on windows versus non-windows systems, or cross-process versus cross-computer, then check `dill.pickles` and `dill.check`. – Mike McKerns Dec 20 '15 at 19:03

1 Answers1

1

Maybe you are (or you were) using CPython 3.4? If so, there was a bug that should be fixed already.

I also had this problem, not in my computer, which has Python 3.4.4, but with Github's Travis, which is using 3.4.2.

Peque
  • 13,638
  • 11
  • 69
  • 105
  • I was indeed! 3.5 still doesn't work on my machine, but I'll try to install it again and see if that fixes the problem. – Schilcote Mar 15 '16 at 16:48