close Warning: Can't synchronize with repository "(default)" (Unsupported version control system "svn": /usr/lib/python2.7/dist-packages/libsvn/_fs.so: failed to map segment from shared object: Cannot allocate memory). Look in the Trac log for more information.

Changes between Version 5 and Version 6 of crawlzilla/stress_testing


Ignore:
Timestamp:
Sep 9, 2010, 1:50:32 PM (15 years ago)
Author:
rock
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • crawlzilla/stress_testing

    v5 v6  
    1717 * Ubuntu 10.04 (CPU1, RAM 512M) ('''''hadoop Heap 256M''''') in VM
    1818   * 同時執行以上3個爬取任務時，一樣會產生 out of memory 問題
    19    * error message (syslog)
     19   * error message ('''syslog''')
    2020{{{
    21 #!sh
    2221600     Sep  9 09:59:28 ubuntu-186 kernel: [ 3708.133724] Out of memory: kill process 3843 (go.sh) score 1775788 or a child
    2322601     Sep  9 09:59:28 ubuntu-186 kernel: [ 3708.133791] Killed process 4205 (counter.sh)
    … …  
    3938617     Sep  9 09:59:28 ubuntu-186 kernel: [ 3709.201865]  [<c058d0b3>] error_code+0x73/0x80
    4039}}}
     40   * error message ('''hadoop-crawler-jobtracker-ubuntu-186.log''')
     41{{{
     42869     2010-09-09 10:04:16,052 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201009090900_0026_m_000000_0: java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory
     43870       at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
     44871       at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
     45872       at org.apache.hadoop.util.Shell.run(Shell.java:134)
     46873       at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
     47874       at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:321)
     48875       at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
     49876       at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
     50877       at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:930)
     51878       at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:842)
     52879       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
     53880       at org.apache.hadoop.mapred.Child.main(Child.java:158)
     54881     Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
     55882       at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
     56883       at java.lang.ProcessImpl.start(ProcessImpl.java:65)
     57884       at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
     58885       ... 10 more
     59}}}