Whirr
  1. Whirr
  2. WHIRR-413

jobcache file is stored at /tmp/ folder so that it has out of storage error

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0, 0.7.0
    • Fix Version/s: 0.9.0
    • Component/s: build, service/hadoop
    • Labels:
    • Environment:

      Description

      when I run Hadoop to read/write data from/to HBase, I got the following error because of the less storage space at /tmp/.
      I guess whirr is supposed to use /data/tmp/ to store jobcache file such as taskTracker/jobcache/job_local_0001/attempt_local_0001_m_0000xx_0/output/file.out because /data/tmp/ has 335GB. However, it is stored at /tmp/ that has only 9.9G. Thus, some configuration xml file seems not correct. It generates errors both at 0.6.0 and 0.7.0

      -----Storage space check ---------------------------------------
      jongwook@ip-10-245-174-15:/tmp/hadoop-jongwook/mapred/local/taskTracker/jobcache/job_local_0001$ cd /tmp
      jongwook@ip-10-245-174-15:/tmp$ df -h .
      Filesystem Size Used Avail Use% Mounted on
      /dev/sda1 9.9G 9.1G 274M 98% /
      jongwook@ip-10-245-174-15:/tmp$ df -h
      Filesystem Size Used Avail Use% Mounted on
      /dev/sda1 9.9G 9.1G 274M 98% /
      none 846M 116K 846M 1% /dev
      none 879M 0 879M 0% /dev/shm
      none 879M 68K 878M 1% /var/run
      none 879M 0 879M 0% /var/lock
      none 879M 0 879M 0% /lib/init/rw
      /dev/sda2 335G 199M 318G 1% /mnt

      -----Error msg at the end of hadoop/hbase code -------------------------------------------------------

      11/10/27 03:33:09 INFO mapred.MapTask: Finished spill 61
      11/10/27 03:33:09 WARN mapred.LocalJobRunner: job_local_0001
      org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for taskTracker/jobcache/job_local_0001/attempt_local_0001_m_000016_0/output/file.out
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
      at org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:61)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1469)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154)
      at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
      at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
      11/10/27 03:33:09 INFO mapred.JobClient: Job complete: job_local_0001
      11/10/27 03:33:09 INFO mapred.JobClient: Counters: 8
      11/10/27 03:33:09 INFO mapred.JobClient: FileSystemCounters
      11/10/27 03:33:09 INFO mapred.JobClient: FILE_BYTES_READ=103074405254
      11/10/27 03:33:09 INFO mapred.JobClient: FILE_BYTES_WRITTEN=156390149579
      11/10/27 03:33:09 INFO mapred.JobClient: Map-Reduce Framework
      11/10/27 03:33:09 INFO mapred.JobClient: Combine output records=0
      11/10/27 03:33:09 INFO mapred.JobClient: Map input records=13248198
      11/10/27 03:33:09 INFO mapred.JobClient: Spilled Records=788109966
      11/10/27 03:33:09 INFO mapred.JobClient: Map output bytes=5347057080
      11/10/27 03:33:09 INFO mapred.JobClient: Combine input records=0
      11/10/27 03:33:09 INFO mapred.JobClient: Map output records=278212138
      It takes: 1966141 msec
      11/10/27 03:33:10 INFO zookeeper.ZooKeeper: Session: 0x13341a966cb000d closed

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jongwook Woo
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Remaining Estimate - 48h
              48h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development