4

After a customer uploads an image I want to re-size an image to 8 different sizes and then send them up to S3 and save them to the file system. Obviously this could take a long time (seconds) and I would want to do it in a "multithreaded" way (I quote that because I don't care if it's actually multithreaded or if it's some other functionality). My understanding is that because of the GIL in Python you cannot successfully use multithreading for things like this because it uses the same process and therefore works exactly the same as if you did it in your code's normal flow (vs multithreading). Is there some other, better way of opening a new process in Python for this sort of task?

orokusaki
  • 55,146
  • 59
  • 179
  • 257
  • 1
    Duplicate of all of these: http://stackoverflow.com/search?q=%5Bpython%5D+subprocess+django. Specifically this: http://stackoverflow.com/questions/1619397/how-to-start-a-long-running-process-from-django-view – S.Lott Jan 20 '10 at 19:25
  • These posts don't help. Most of the answers mention using subprocess which doesn't do anything useful whatsoever for anyone besides complicate your code (unless you have large IO issues that can be solved with subprocess). Google "Problems with Python GIL". – orokusaki Jan 20 '10 at 19:42
  • "subprocess which doesn't do anything useful whatsoever for anyone". Yet, you accepted the answer with "subprocess". I don't get the comment mixed with the accepted answer. – S.Lott Jan 20 '10 at 20:05

1 Answers1

2

If you want to call an external process, use the subprocess module.

If on the other hand you want to side-step the GIL and spin of some Python task, use the multiprocessing module. It provides an interface very much like the threading package's but utilizes subprocesses so you are not bound by the constraints of the GIL.

jkp
  • 78,960
  • 28
  • 103
  • 104
  • 1
    What would be the reason for using subprocess (ie, what would be an external process)? – orokusaki Jan 20 '10 at 19:25
  • @orokusaki: say you want to process some file with some external utility, EG, you are processing video and you want ffmpeg to transcode it for you: in this case you'd use the subprocess module to fire up ffmpeg, pipe your input data to it and read it's output to get the processed data. If what you want is to run some long-running Python routine without being affected by the GIL however, the multiprocessing package is what you want. – jkp Jan 20 '10 at 19:28