12

I am trying to understand when does it make sense to catch MemoryError in Python, I got two scenarios:

Scenario 1: MemoryError is successfully caught.

import numpy as np

try:
    a = np.ones(100000000000)
except MemoryError:
    print 'got memory error, plan B'
    a = np.ones(10) # this gets created

Scenario 2: My program freezes

silly = []
c = 0
try:
    while True:
        silly.append((str(c))) # just increasing the list
        c += 1
        if c % 1000000 == 0:
            print 'counter : {}'.format(c)
except MemoryError:
    print 'oops' # never get here
    silly.append('silly')

My guess is in the first case, python "knows" how much memory would need to be allocated, and thus raises a MemoryError exception. While in the second case python does not "know" how big I intent silly to be. However, list is a dynamic array; therefore, python should know that extending this array by a certain amount would cause a MemoryError, why is the exception not raised then?

I have looked at this question, and the relevant paragraph from the docs is:

exception MemoryError

Raised when an operation runs out of memory but the situation may still be rescued (by deleting some objects). The associated value is a string indicating what kind of (internal) operation ran out of memory. Note that because of the underlying memory management architecture (C’s malloc() function), the interpreter may not always be able to completely recover from this situation; it nevertheless raises an exception so that a stack traceback can be printed, in case a run-away program was the cause.

Though this helped me, I am still not very clear as to what is going on, and I did not get an exception as the docs seem to indicate.

My questions: Is my guess right on scenario 1? Why is MemoryException not raised in scenario 2?

I am using Python 2.7.5+, and I am on ubuntu 13.10

Community
  • 1
  • 1
Akavall
  • 82,592
  • 51
  • 207
  • 251

2 Answers2

1

In Python list is a linked list so the memory Exception MemoryError will be raised only when the system can't allocate additional memory.

This scenario occurs only when your RAM (main memory) is utilized to its maximum level most of the OS handles this scenario by moving less priority data to Hard Disk hence you don't get MemoryError in new OS but its better to handle this because the legacy OS don't have the above mentioned mechanism so the programer needs to handle it.

Mr. A
  • 1,221
  • 18
  • 28
  • 4
    Python list is certainly a dynamic array : https://docs.python.org/2/faq/design.html#how-are-lists-implemented – Akavall Mar 31 '14 at 13:20
  • The second part of your answer certainly makes sense, but how does python know when to raise a `MemoryError` and when to start moving data to hard dist? – Akavall Mar 31 '14 at 16:25
1

The MemoryError is raised by OS not by Python.

Legacy OS raise MemoryError immediately when it finds that memory is full, Where as Most of latest OS try to move the content to Hard Disk as it find that Main Memory is full and no room for new request, if its still can't be accommodated in Allocated space of Hard Disk or the current request size exceeds limit of data storage space allocated for this particular process in RAM(Main Memory) then it raise's MemoryError.

The MemoryError raised by OS can be captured by your python program using try exception.

The MemoryError is also raised by OS when you make a request for memory that exceeds limit of Main Memory to store data.

Example: If you are using 512MB RAM and your OS allocated 200MB in RAM for data and rest for code, In that 200MB for your process 50MB Data space is allocated by OS and you make a request to store 100MB which exceeds limit hence OS raises MemoryError in this scenario too.

Mr. A
  • 1,221
  • 18
  • 28