2

I have a 30 node cluster, each node has 32 core, 240 G memory (AWS cr1.8xlarge instance). I have the following configurations:

--driver-memory 200g --driver-cores 30 --executor-memory 70g --executor-cores 8 --num-executors 90

I can see from the job tracker that I still have a lot of total storage memory left, but in one of the containers, I got the following message saying Storage limit = 28.3 GB. I am wondering where does this 28.3 GB came from? My memoryFraction for storage is 0.45

And how do I solve this Not enough space to cache rdd issue? Should I do more partition or change default parallelism ... since I still have a lot of total storage memory unused. Thanks!

15/12/05 22:39:36 WARN storage.MemoryStore: Not enough space to cache rdd_31_310 in memory! (computed 1326.6 MB so far)
15/12/05 22:39:36 INFO storage.MemoryStore: Memory use = 9.6 GB (blocks) + 18.1 GB (scratch space shared across 4 tasks(s)) = 27.7 GB. Storage limit = 28.3 GB.
15/12/05 22:39:36 WARN storage.MemoryStore: Not enough space to cache rdd_31_136 in memory! (computed 1835.8 MB so far)
15/12/05 22:39:36 INFO storage.MemoryStore: Memory use = 9.6 GB (blocks) + 18.1 GB (scratch space shared across 5 tasks(s)) = 27.7 GB. Storage limit = 28.3 GB.
15/12/05 22:39:36 INFO executor.Executor: Finished task 136.0 in stage 12.0 (TID 85168). 1272 bytes result sent to driver
Edamame
  • 23,718
  • 73
  • 186
  • 320
  • 4
    Only a fraction of the 70G of memory can be used for RDD storage. See http://stackoverflow.com/questions/26562033/how-to-set-apache-spark-executor-memory – Glennie Helles Sindholt Dec 06 '15 at 08:59
  • Thanks！So will the spark put the task to another executor if the current executor doesn't have enough memory？Or just fail ？ – Edamame Dec 06 '15 at 16:35
  • Well, since non of your executors will have more than ~28GB of memory, and at least one of the tasks seem to require more, I'm guessing that your job will eventually fail or maybe retry forever. I suggest that you try to repartition your `rdd` into more partitions to get the job to complete successfully. – Glennie Helles Sindholt Dec 06 '15 at 17:43
  • Would it help if I do: MEMORY_AND_DISK_SER_2 instead of all memories ? Thanks! – Edamame Dec 06 '15 at 20:47
  • 1
    Yes, that will help. – Glennie Helles Sindholt Dec 07 '15 at 07:55

0 Answers0