IndexError when appending to a list (no index used) in Python

Question

I'm appending an object to list images (initialized as empty) at every iteration of a for loop. I'm not explicitly accessing the list by index, I'm just appending an object to it, which is why to me inexplicable that I get an IndexError: list index out of range back.

Here's the stack trace:

  Traceback (most recent call last):
   File "/mnt/www-data/run/backend/lib/python2.7/site-packages/flask/app.py", line 1504, in wsgi_app
     response = self.full_dispatch_request()
   File "/mnt/www-data/run/backend/lib/python2.7/site-packages/flask/app.py", line 1264, in full_dispatch_request
     rv = self.handle_user_exception(e)
   File "/mnt/www-data/run/backend/lib/python2.7/site-packages/flask/app.py", line 1262, in full_dispatch_request
     rv = self.dispatch_request()
   File "/mnt/www-data/run/backend/lib/python2.7/site-packages/flask/app.py", line 1248, in dispatch_request
     return self.view_functions[rule.endpoint](**req.view_args)
   File "/mnt/www-data/run/backend/lib/python2.7/site-packages/newrelic-1.4.0.137-py2.7-linux-x86_64.egg/newrelic/api/object_wrapper.py", line 166, in __call__
     self._nr_instance, args, kwargs)
   File "/mnt/www-data/run/backend/lib/python2.7/site-packages/newrelic-1.4.0.137-py2.7-linux-x86_64.egg/newrelic/api/function_trace.py", line 81, in literal_wrapper
     return wrapped(*args, **kwargs)
   File "/mnt/www-data/run/backend/lib/python2.7/site-packages/newrelic-1.4.0.137-py2.7-linux-x86_64.egg/newrelic/api/name_transaction.py", line 58, in __call__
     return self._nr_next_object(*args, **kwargs)
   File "/mnt/www-data/run/backend/webservice/decorators/controller_wrapper.py", line 75, in decorated_function
     return _return(f(*args, body=body, **kwargs))
   File "/mnt/www-data/run/backend/webservice/decorators/controller_wrapper.py", line 90, in decorated_function
     r = f(*args, **kwargs)
   File "/mnt/www-data/run/backend/webservice/decorators/login_required.py", line 16, in decorated_function
     return f(*args, **kwargs)
   File "/mnt/www-data/run/backend/webservice/controllers/me/photos.py", line 28, in create_photo
     phs = photos.store_photos(g.user, data)
   File "/mnt/www-data/run/backend/evertale/tasks/photos.py", line 106, in store_photos
     images.append(entity)
  IndexError: list index out of range

And an excerpt of .../tasks/photos.py, the failing line is the last of the for loop:

def store_photos(user, photos, update=False):
    if type(photos) is dict:
        photos = [photos]

    images = []
    for pic in photos:
        if 'gps' in pic and pic['gps']:
            loc = Location(ts=long(pic['ts']), user=user, _latlon=pic['gps'])
            loc.save()
        image = None
        if 'data' in pic:
            image = _read_image_from_base64(pic['data'])
            image = _normalize_rotation(image)
        entity = _persist_entity(user, image, update, **pic)
        if image:
            file_name = _file_name(entity.id)
            _persist_file(image, file_name)
            thumb_photo(entity.id)
        images.append(entity)

    if len(images) > 1:
        return images
    elif len(images) == 1:
        return images[0]
    else:
        return []

Anyone having any idea why this happens? I can't debug this because the code is on the production server, and I've never experienced anything like it when testing on local machine.

This has nothing to do with your question, but your return type is strange. If there are no items, you return an empty list. If there is one item, you return the item. If there are multiple items, you again return a list. This makes it hard for someone to use the result of your function. — Steven Rumbalski, Sep 07 '12 at 16:50
I expected that somebody would find that suspicious :) The reason why The API method that triggers this function is decorated with a generic response handler that returns a serialized wrapper when the response is a list, and the serialized object itself when the response is an object. To play nicely with iPhone's RestKit, I need to return the same thing that was sent by the client, so an object when an object was sent, and a wrapper when multiple objects were sent. That's why the hack, I'm basically bypassing the decorator. Not very classy, indeed. — Matteo Danieli, Sep 07 '12 at 20:51

score 3 · Accepted Answer · answered Sep 07 '12 at 16:39

3

I suspect the sourcecode on your server has changed since you started it. In fact, I am almost certain of it.

Python byte-code includes a line number, and when a traceback occurs, this line number is used to look up and show you the non-compiled source line.

Once the bytecode is loaded into memory however, it is not reloaded if you were to change the source file. If you add or remove lines from the source file after your server has started, lines in a traceback will be wrong.

In other words, the line shown in your traceback is almost certainly not the line where the error occurs. I certainly would not expect a simple .append() to ever throw that exception. The specific exception (list index out of range) would only occur when trying to look up a specific entry in the list through __getattr__ (a lst[index] operation).

answered Sep 07 '12 at 16:39

Martijn Pieters

1,048,767
296
4,058
3,343

see http://stackoverflow.com/questions/11991756/which-line-is-chosen-to-be-reported-in-exception – Andy Hayden Sep 07 '12 at 16:41
1

@Hayden, You might want to accept the answer for the link that you provided – GodMan Sep 07 '12 at 16:42
For my edification, you're saying the .pyc code throws an error with a line number to something that generates the stack trace, which then looks up the lines in the .py code, which may or may not be the same code which the .pyc was built againt? And to end up in the situation the OP mentioned, photo.py was imported, but then modified later, though whatever code imported it has not stopped/re-imported it since it originally did so? – ernie Sep 07 '12 at 16:47

score 2 · Answer 2 · answered Sep 07 '12 at 16:45

2

Martijn may well be right, but there is another outside possibility.

If the last function called before the .append() was coded in C (either _persist_entity() or thumb_photo()) or the last thing one of those did was call something coded in C, and the C code failed to check and handle errors properly then you could end up in a state where an exception has been thrown but not recognised. In that case the append call could be recognising that an exception has been thrown even though it hasn't thrown it itself.

answered Sep 07 '12 at 16:45

Duncan

92,073
11
122
156

Indeed, also a distinct possibility. PIL or database adapters are possibly involved judging by the function names. – Martijn Pieters Sep 07 '12 at 16:46
I'm actually using PIL as you correctly guessed, but I tend to exclude this possibility, because the code in thumb_photo has run for months without change, and this is the first time this Exception appears. – Matteo Danieli Sep 07 '12 at 21:02

IndexError when appending to a list (no index used) in Python

2 Answers2