Whirr
  1. Whirr
  2. WHIRR-413

jobcache file is stored at /tmp/ folder so that it has out of storage error

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0, 0.7.0
    • Fix Version/s: 0.9.0
    • Component/s: build, service/hadoop
    • Labels:
    • Environment:

      Description

      when I run Hadoop to read/write data from/to HBase, I got the following error because of the less storage space at /tmp/.
      I guess whirr is supposed to use /data/tmp/ to store jobcache file such as taskTracker/jobcache/job_local_0001/attempt_local_0001_m_0000xx_0/output/file.out because /data/tmp/ has 335GB. However, it is stored at /tmp/ that has only 9.9G. Thus, some configuration xml file seems not correct. It generates errors both at 0.6.0 and 0.7.0

      -----Storage space check ---------------------------------------
      jongwook@ip-10-245-174-15:/tmp/hadoop-jongwook/mapred/local/taskTracker/jobcache/job_local_0001$ cd /tmp
      jongwook@ip-10-245-174-15:/tmp$ df -h .
      Filesystem Size Used Avail Use% Mounted on
      /dev/sda1 9.9G 9.1G 274M 98% /
      jongwook@ip-10-245-174-15:/tmp$ df -h
      Filesystem Size Used Avail Use% Mounted on
      /dev/sda1 9.9G 9.1G 274M 98% /
      none 846M 116K 846M 1% /dev
      none 879M 0 879M 0% /dev/shm
      none 879M 68K 878M 1% /var/run
      none 879M 0 879M 0% /var/lock
      none 879M 0 879M 0% /lib/init/rw
      /dev/sda2 335G 199M 318G 1% /mnt

      -----Error msg at the end of hadoop/hbase code -------------------------------------------------------

      11/10/27 03:33:09 INFO mapred.MapTask: Finished spill 61
      11/10/27 03:33:09 WARN mapred.LocalJobRunner: job_local_0001
      org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for taskTracker/jobcache/job_local_0001/attempt_local_0001_m_000016_0/output/file.out
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
      at org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:61)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1469)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154)
      at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
      at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
      11/10/27 03:33:09 INFO mapred.JobClient: Job complete: job_local_0001
      11/10/27 03:33:09 INFO mapred.JobClient: Counters: 8
      11/10/27 03:33:09 INFO mapred.JobClient: FileSystemCounters
      11/10/27 03:33:09 INFO mapred.JobClient: FILE_BYTES_READ=103074405254
      11/10/27 03:33:09 INFO mapred.JobClient: FILE_BYTES_WRITTEN=156390149579
      11/10/27 03:33:09 INFO mapred.JobClient: Map-Reduce Framework
      11/10/27 03:33:09 INFO mapred.JobClient: Combine output records=0
      11/10/27 03:33:09 INFO mapred.JobClient: Map input records=13248198
      11/10/27 03:33:09 INFO mapred.JobClient: Spilled Records=788109966
      11/10/27 03:33:09 INFO mapred.JobClient: Map output bytes=5347057080
      11/10/27 03:33:09 INFO mapred.JobClient: Combine input records=0
      11/10/27 03:33:09 INFO mapred.JobClient: Map output records=278212138
      It takes: 1966141 msec
      11/10/27 03:33:10 INFO zookeeper.ZooKeeper: Session: 0x13341a966cb000d closed

        Activity

        Hide
        Jongwook Woo added a comment -

        Andrei, I don't have any clue what part of codes I need to look. Do you have any idea what part of codes or property files relate to this issue?

        Then, I may take a look at it.

        Show
        Jongwook Woo added a comment - Andrei, I don't have any clue what part of codes I need to look. Do you have any idea what part of codes or property files relate to this issue? Then, I may take a look at it.
        Hide
        Andrei Savu added a comment -

        Jongwook I'm glad this allows you to continue your work. It would be great if you could look for a more permanent fix because right now it's hard for us to replicate the issues. Thanks!

        Show
        Andrei Savu added a comment - Jongwook I'm glad this allows you to continue your work. It would be great if you could look for a more permanent fix because right now it's hard for us to replicate the issues. Thanks!
        Hide
        Jongwook Woo added a comment -

        Andrei:

        That sounds great. And, your suggestion is super. I made a symbolic link and it works great

        I can now continue my work.

        Show
        Jongwook Woo added a comment - Andrei: That sounds great. And, your suggestion is super. I made a symbolic link and it works great I can now continue my work.
        Hide
        Andrei Savu added a comment -

        This is a serious problem but I'm moving it to 0.8.0 because in the meantime you can work around it by symlinking /tmp/hadoop-jongwook/ to /data/something. Sounds reasonable? I can help you to implement that workaround.

        Show
        Andrei Savu added a comment - This is a serious problem but I'm moving it to 0.8.0 because in the meantime you can work around it by symlinking /tmp/hadoop-jongwook/ to /data/something. Sounds reasonable? I can help you to implement that workaround.
        Hide
        David Alves added a comment -

        Jongwook, thanks for trying that.

        So what your are saying is that /tmp/hadoop-jongwook/mapred/local gets used for HBase mr jobs, but not for plain Hadoop mr jobs, right?

        Thus, I wonder if whirr has any configuration file to increase the jobtracker host's '/tmp' storage.

        I think somehow "mapred.local.dir" is being overriden somewhere. Even if we could changes the mount mappings to put /tmp on sda2 this would not be a general solution as we would not be able to use the stock images (and we must).

        How are you executing your Hbase mr jobs, are you using "bin/hadoop jar ..."? If not could you try and run Hbase's peformance eval mr job (http://hbase.apache.org/docs/r0.89.20100621/apidocs/org/apache/hadoop/hbase/mapred/package-summary.html)

        Show
        David Alves added a comment - Jongwook, thanks for trying that. So what your are saying is that /tmp/hadoop-jongwook/mapred/local gets used for HBase mr jobs, but not for plain Hadoop mr jobs, right? Thus, I wonder if whirr has any configuration file to increase the jobtracker host's '/tmp' storage. I think somehow "mapred.local.dir" is being overriden somewhere. Even if we could changes the mount mappings to put /tmp on sda2 this would not be a general solution as we would not be able to use the stock images (and we must). How are you executing your Hbase mr jobs, are you using "bin/hadoop jar ..."? If not could you try and run Hbase's peformance eval mr job ( http://hbase.apache.org/docs/r0.89.20100621/apidocs/org/apache/hadoop/hbase/mapred/package-summary.html )
        Hide
        Jongwook Woo added a comment -

        David, I just run the hadoop only code to make sure and it does not generate any '/tmp/hadoop-jongwook' folder and it has the following storage usage.

        jongwook@domU-12-31-39-12-92-61:~$ df -h
        Filesystem Size Used Avail Use% Mounted on
        /dev/sda1 9.9G 2.0G 7.5G 21% /
        none 846M 116K 846M 1% /dev
        none 879M 0 879M 0% /dev/shm
        none 879M 64K 878M 1% /var/run
        none 879M 0 879M 0% /var/lock
        none 879M 0 879M 0% /lib/init/rw
        /dev/sda2 335G 196M 318G 1% /mnt

        Thus, I wonder if whirr has any configuration file to increase the jobtracker host's '/tmp' storage.

        Show
        Jongwook Woo added a comment - David, I just run the hadoop only code to make sure and it does not generate any '/tmp/hadoop-jongwook' folder and it has the following storage usage. jongwook@domU-12-31-39-12-92-61:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.9G 2.0G 7.5G 21% / none 846M 116K 846M 1% /dev none 879M 0 879M 0% /dev/shm none 879M 64K 878M 1% /var/run none 879M 0 879M 0% /var/lock none 879M 0 879M 0% /lib/init/rw /dev/sda2 335G 196M 318G 1% /mnt Thus, I wonder if whirr has any configuration file to increase the jobtracker host's '/tmp' storage.
        Hide
        David Alves added a comment -

        Jongwook, I'm having trouble replicating your issue.

        Does it (storing temp mapper mat. on /tmp) only for Hbase MR jobs or does it happen even for normal (hadoop only) mr jobs? Could you run the hadoop MR example and check?

        Show
        David Alves added a comment - Jongwook, I'm having trouble replicating your issue. Does it (storing temp mapper mat. on /tmp) only for Hbase MR jobs or does it happen even for normal (hadoop only) mr jobs? Could you run the hadoop MR example and check?
        Hide
        David Alves added a comment -

        1. Can you look at my first post? - in the folder of '/tmp/hadoop-jongwook/mapred/local/taskTracker/jobcache/job_local_0001'.

        Sorry I missed that.

        Thank you for the detailed post. I'll try to replicate your problem

        Show
        David Alves added a comment - 1. Can you look at my first post? - in the folder of '/tmp/hadoop-jongwook/mapred/local/taskTracker/jobcache/job_local_0001'. Sorry I missed that. Thank you for the detailed post. I'll try to replicate your problem
        Hide
        Jongwook Woo added a comment -

        David:

        As you have seen the config files, I cannot find any config files that specify "/tmp" to store jobcache data as a jobtracker host - I guess it is a jobtracker host server.

        Show
        Jongwook Woo added a comment - David: As you have seen the config files, I cannot find any config files that specify "/tmp" to store jobcache data as a jobtracker host - I guess it is a jobtracker host server.
        Hide
        Jongwook Woo added a comment -

        1) core-site.xml
        jongwook@domU-12-31-39-12-91-A6:/tmp$ more core-site.xml
        <configuration>
        <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/tmp/hadoop-$

        {user.name}</value>
        </property>
        <property>
        <name>io.file.buffer.size</name>
        <value>65536</value>
        </property>
        <property>
        <name>hadoop.rpc.socket.factory.class.default</name>
        <value>org.apache.hadoop.net.StandardSocketFactory</value>
        <final>true</final>
        </property>
        <property>
        <name>hadoop.rpc.socket.factory.class.ClientProtocol</name>
        <value></value>
        </property>
        <property>
        <name>hadoop.rpc.socket.factory.class.JobSubmissionProtocol</name>
        <value></value>
        </property>
        <property>
        <name>fs.trash.interval</name>
        <value>1440</value>
        </property>
        <property>
        <name>fs.default.name</name>
        <value>hdfs://ec2-184-72-193-67.compute-1.amazonaws.com:8020/</value>
        </property>
        </configuration>

        2) hbase-site.xml
        jongwook@domU-12-31-39-12-91-A6:/tmp$ more hbase-site.xml
        <configuration>
        <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
        </property>
        <property>
        <name>hbase.regionserver.handler.count</name>
        <value>100</value>
        </property>
        <property>
        <name>dfs.replication</name>
        <value>3</value>
        </property>
        <property>
        <name>zookeeper.session.timeout</name>
        <value>60000</value>
        </property>
        <property>
        <name>hbase.tmp.dir</name>
        <value>/data/tmp/hbase-${user.name}

        </value>
        </property>
        <property>
        <name>hbase.client.retries.number</name>
        <value>100</value>
        </property>
        <property>
        <name>hbase.zookeeper.recoverable.waittime</name>
        <value>600000</value>
        </property>
        <property>
        <name>hbase.rootdir</name>
        <value>hdfs://ec2-184-72-193-67.compute-1.amazonaws.com:8020/hbase</value>
        </property>
        <property>
        <name>hbase.zookeeper.quorum</name>
        <value>ec2-184-72-193-67.compute-1.amazonaws.com:2181</value>
        </property>
        </configuration>

        3) hdfs-site.xml
        jongwook@domU-12-31-39-12-91-A6:/tmp$ more hdfs-site.xml
        <configuration>
        <property>
        <name>dfs.block.size</name>
        <value>134217728</value>
        </property>
        <property>
        <name>dfs.data.dir</name>
        <value>/data/hadoop/hdfs/data</value>
        </property>
        <property>
        <name>dfs.datanode.du.reserved</name>
        <value>1073741824</value>
        </property>
        <property>
        <name>dfs.name.dir</name>
        <value>/data/hadoop/hdfs/name</value>
        </property>
        <property>
        <name>fs.checkpoint.dir</name>
        <value>/data/hadoop/hdfs/secondary</value>
        </property>
        </configuration>

        4) mapred-site.xml

        jongwook@domU-12-31-39-12-91-A6:/tmp$ more mapred-site.xml
        <configuration>
        <property>
        <name>mapred.local.dir</name>
        <value>/data/hadoop/mapred/local</value>
        </property>
        <property>
        <name>mapred.map.tasks.speculative.execution</name>
        <value>true</value>
        </property>
        <property>
        <name>mapred.reduce.tasks.speculative.execution</name>
        <value>false</value>
        </property>
        <property>
        <name>mapred.system.dir</name>
        <value>/hadoop/system/mapred</value>
        </property>
        <property>
        <name>mapreduce.jobtracker.staging.root.dir</name>
        <value>/user</value>
        </property>
        <property>
        <name>mapred.compress.map.output</name>
        <value>true</value>
        </property>
        <property>
        <name>mapred.output.compression.type</name>
        <value>BLOCK</value>
        </property>
        <property>
        <name>mapred.child.java.opts</name>
        <value>-Xmx550m</value>
        </property>
        <property>
        <name>mapred.child.ulimit</name>
        <value>1126400</value>
        </property>
        <property>
        <name>mapred.tasktracker.map.tasks.maximum</name>
        <value>2</value>
        </property>
        <property>
        <name>mapred.tasktracker.reduce.tasks.maximum</name>
        <value>2</value>
        </property>
        <property>
        <name>mapred.reduce.tasks</name>
        <value>10</value>
        </property>
        <property>
        <name>mapred.job.tracker</name>
        <value>ec2-184-72-193-67.compute-1.amazonaws.com:8021</value>
        </property>
        </configuration>

        Show
        Jongwook Woo added a comment - 1) core-site.xml jongwook@domU-12-31-39-12-91-A6:/tmp$ more core-site.xml <configuration> <property> <name>hadoop.tmp.dir</name> <value>/data/tmp/hadoop-$ {user.name}</value> </property> <property> <name>io.file.buffer.size</name> <value>65536</value> </property> <property> <name>hadoop.rpc.socket.factory.class.default</name> <value>org.apache.hadoop.net.StandardSocketFactory</value> <final>true</final> </property> <property> <name>hadoop.rpc.socket.factory.class.ClientProtocol</name> <value></value> </property> <property> <name>hadoop.rpc.socket.factory.class.JobSubmissionProtocol</name> <value></value> </property> <property> <name>fs.trash.interval</name> <value>1440</value> </property> <property> <name>fs.default.name</name> <value>hdfs://ec2-184-72-193-67.compute-1.amazonaws.com:8020/</value> </property> </configuration> 2) hbase-site.xml jongwook@domU-12-31-39-12-91-A6:/tmp$ more hbase-site.xml <configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.regionserver.handler.count</name> <value>100</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>zookeeper.session.timeout</name> <value>60000</value> </property> <property> <name>hbase.tmp.dir</name> <value>/data/tmp/hbase-${user.name} </value> </property> <property> <name>hbase.client.retries.number</name> <value>100</value> </property> <property> <name>hbase.zookeeper.recoverable.waittime</name> <value>600000</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://ec2-184-72-193-67.compute-1.amazonaws.com:8020/hbase</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>ec2-184-72-193-67.compute-1.amazonaws.com:2181</value> </property> </configuration> 3) hdfs-site.xml jongwook@domU-12-31-39-12-91-A6:/tmp$ more hdfs-site.xml <configuration> <property> <name>dfs.block.size</name> <value>134217728</value> </property> <property> <name>dfs.data.dir</name> <value>/data/hadoop/hdfs/data</value> </property> <property> <name>dfs.datanode.du.reserved</name> <value>1073741824</value> </property> <property> <name>dfs.name.dir</name> <value>/data/hadoop/hdfs/name</value> </property> <property> <name>fs.checkpoint.dir</name> <value>/data/hadoop/hdfs/secondary</value> </property> </configuration> 4) mapred-site.xml jongwook@domU-12-31-39-12-91-A6:/tmp$ more mapred-site.xml <configuration> <property> <name>mapred.local.dir</name> <value>/data/hadoop/mapred/local</value> </property> <property> <name>mapred.map.tasks.speculative.execution</name> <value>true</value> </property> <property> <name>mapred.reduce.tasks.speculative.execution</name> <value>false</value> </property> <property> <name>mapred.system.dir</name> <value>/hadoop/system/mapred</value> </property> <property> <name>mapreduce.jobtracker.staging.root.dir</name> <value>/user</value> </property> <property> <name>mapred.compress.map.output</name> <value>true</value> </property> <property> <name>mapred.output.compression.type</name> <value>BLOCK</value> </property> <property> <name>mapred.child.java.opts</name> <value>-Xmx550m</value> </property> <property> <name>mapred.child.ulimit</name> <value>1126400</value> </property> <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value>2</value> </property> <property> <name>mapred.tasktracker.reduce.tasks.maximum</name> <value>2</value> </property> <property> <name>mapred.reduce.tasks</name> <value>10</value> </property> <property> <name>mapred.job.tracker</name> <value>ec2-184-72-193-67.compute-1.amazonaws.com:8021</value> </property> </configuration>
        Hide
        Jongwook Woo added a comment -

        1. Can you look at my first post? - in the folder of '/tmp/hadoop-jongwook/mapred/local/taskTracker/jobcache/job_local_0001'.

        2. whirr-hadoop.properties, did you mean "hadoop-ec2.properties"?
        jongwook@ubuntu:~/src/whirr-trunk/recipes$ more hadoop-ec2.properties
        #

        1. Licensed to the Apache Software Foundation (ASF) under one or more
        2. contributor license agreements. See the NOTICE file distributed with
        3. this work for additional information regarding copyright ownership.
        4. The ASF licenses this file to You under the Apache License, Version 2.0
        5. (the "License"); you may not use this file except in compliance with
        6. the License. You may obtain a copy of the License at
          #
        7. http://www.apache.org/licenses/LICENSE-2.0
          #
        8. Unless required by applicable law or agreed to in writing, software
        9. distributed under the License is distributed on an "AS IS" BASIS,
        10. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        11. See the License for the specific language governing permissions and
        12. limitations under the License.
          #

        #

        1. Hadoop Cluster on AWS EC2
        1. Read the Configuration Guide for more info:
        2. http://whirr.apache.org/docs/latest/configuration-guide.html
        1. Change the cluster name here
          whirr.cluster-name=hadoop
        1. Change the number of machines in the cluster here
          whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,5 hadoop-datanode+hadoop-tasktracker
        1. Uncomment out these lines to run CDH
          #whirr.hadoop.install-function=install_cdh_hadoop
          #whirr.hadoop.configure-function=configure_cdh_hadoop
        1. For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
          whirr.provider=aws-ec2
          whirr.identity=$ {env:AWS_ACCESS_KEY_ID}

          whirr.credential=$

          {env:AWS_SECRET_ACCESS_KEY}
        1. The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
          whirr.hardware-id=c1.xlarge
        2. Ubuntu 10.04 LTS Lucid. See http://alestic.com/
          whirr.image-id=us-east-1/ami-da0cf8b3
        3. If you choose a different location, make sure whirr.image-id is updated too
          whirr.location-id=us-east-1
        1. You can also specify the spot instance price
        2. http://aws.amazon.com/ec2/spot-instances/
        3. whirr.aws-ec2-spot-price=0.15
        1. By default use the user system SSH keys. Override them here.
        2. whirr.private-key-file=$ {sys:user.home}

          /.ssh/id_rsa

        3. whirr.public-key-file=$ {whirr.private-key-file}.pub

          # Expert: override Hadoop properties by setting properties with the prefix
          # hadoop-common, hadoop-hdfs, hadoop-mapreduce to set Common, HDFS, MapReduce
          # site properties, respectively. The prefix is removed by Whirr, so that for
          # example, setting
          # hadoop-common.fs.trash.interval=1440
          # will result in fs.trash.interval being set to 1440 in core-site.xml.

          # Expert: specify the version of Hadoop to install.
          #whirr.hadoop.version=0.20.2
          #whirr.hadoop.tarball.url=http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr. hadoop.version}.tar.gz


          3. whirr-hbase.properties. did you mean "hbase-ec2.properties"?

          jongwook@ubuntu:~/src/whirr-trunk/recipes$ more ~/apache/whirr-0.6.0-incubating/hbase-ec2.properties
          #
          # Licensed to the Apache Software Foundation (ASF) under one or more
          # contributor license agreements. See the NOTICE file distributed with
          # this work for additional information regarding copyright ownership.
          # The ASF licenses this file to You under the Apache License, Version 2.0
          # (the "License"); you may not use this file except in compliance with
          # the License. You may obtain a copy of the License at
          #
          # http://www.apache.org/licenses/LICENSE-2.0
          #
          # Unless required by applicable law or agreed to in writing, software
          # distributed under the License is distributed on an "AS IS" BASIS,
          # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          # See the License for the specific language governing permissions and
          # limitations under the License.
          #

          #
          # HBase Cluster on AWS EC2
          #

          # Read the Configuration Guide for more info:
          # http://incubator.apache.org/whirr/configuration-guide.html

          # Change the cluster name here
          #whirr.cluster-name=test-cluster
          whirr.cluster-name=hbase

          # Change the number of machines in the cluster here
          whirr.instance-templates=1 zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,5 hadoop-datanode+hadoop-tasktr
          acker+hbase-regionserver


          # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
          whirr.provider=aws-ec2
          whirr.identity=${env:AWS_ACCESS_KEY_ID}
          whirr.credential=${env:AWS_SECRET_ACCESS_KEY}

          # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
          # $0.68 per High-CPU Extra Large Instance (c1.xlarge) instance-hour (or partial hour)
          #whirr.hardware-id=c1.xlarge
          # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
          # default 64bits

          # JW's; $0.17 per High-CPU Medium Instance (c1.medium) instance-hour (or partial hour)
          whirr.hardware-id=c1.medium
          whirr.image-id=us-east-1/ami-7000f019

          # If you choose a different location, make sure whirr.image-id is updated too
          whirr.location-id=us-east-1

          # By default use the user system SSH keys. Override them here.
          whirr.private-key-file=${sys:user.home}/.ssh/id_rsa2
          whirr.public-key-file=${whirr.private-key-file}

          .pub

        Show
        Jongwook Woo added a comment - 1. Can you look at my first post? - in the folder of '/tmp/hadoop-jongwook/mapred/local/taskTracker/jobcache/job_local_0001'. 2. whirr-hadoop.properties, did you mean "hadoop-ec2.properties"? jongwook@ubuntu:~/src/whirr-trunk/recipes$ more hadoop-ec2.properties # Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. # # Hadoop Cluster on AWS EC2 Read the Configuration Guide for more info: http://whirr.apache.org/docs/latest/configuration-guide.html Change the cluster name here whirr.cluster-name=hadoop Change the number of machines in the cluster here whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,5 hadoop-datanode+hadoop-tasktracker Uncomment out these lines to run CDH #whirr.hadoop.install-function=install_cdh_hadoop #whirr.hadoop.configure-function=configure_cdh_hadoop For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. whirr.provider=aws-ec2 whirr.identity=$ {env:AWS_ACCESS_KEY_ID} whirr.credential=$ {env:AWS_SECRET_ACCESS_KEY} The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/ whirr.hardware-id=c1.xlarge Ubuntu 10.04 LTS Lucid. See http://alestic.com/ whirr.image-id=us-east-1/ami-da0cf8b3 If you choose a different location, make sure whirr.image-id is updated too whirr.location-id=us-east-1 You can also specify the spot instance price http://aws.amazon.com/ec2/spot-instances/ whirr.aws-ec2-spot-price=0.15 By default use the user system SSH keys. Override them here. whirr.private-key-file=$ {sys:user.home} /.ssh/id_rsa whirr.public-key-file=$ {whirr.private-key-file}.pub # Expert: override Hadoop properties by setting properties with the prefix # hadoop-common, hadoop-hdfs, hadoop-mapreduce to set Common, HDFS, MapReduce # site properties, respectively. The prefix is removed by Whirr, so that for # example, setting # hadoop-common.fs.trash.interval=1440 # will result in fs.trash.interval being set to 1440 in core-site.xml. # Expert: specify the version of Hadoop to install. #whirr.hadoop.version=0.20.2 #whirr.hadoop.tarball.url= http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr. hadoop.version}.tar.gz 3. whirr-hbase.properties. did you mean "hbase-ec2.properties"? jongwook@ubuntu:~/src/whirr-trunk/recipes$ more ~/apache/whirr-0.6.0-incubating/hbase-ec2.properties # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # # HBase Cluster on AWS EC2 # # Read the Configuration Guide for more info: # http://incubator.apache.org/whirr/configuration-guide.html # Change the cluster name here #whirr.cluster-name=test-cluster whirr.cluster-name=hbase # Change the number of machines in the cluster here whirr.instance-templates=1 zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,5 hadoop-datanode+hadoop-tasktr acker+hbase-regionserver # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. whirr.provider=aws-ec2 whirr.identity=${env:AWS_ACCESS_KEY_ID} whirr.credential=${env:AWS_SECRET_ACCESS_KEY} # The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/ # $0.68 per High-CPU Extra Large Instance (c1.xlarge) instance-hour (or partial hour) #whirr.hardware-id=c1.xlarge # Ubuntu 10.04 LTS Lucid. See http://alestic.com/ # default 64bits # JW's; $0.17 per High-CPU Medium Instance (c1.medium) instance-hour (or partial hour) whirr.hardware-id=c1.medium whirr.image-id=us-east-1/ami-7000f019 # If you choose a different location, make sure whirr.image-id is updated too whirr.location-id=us-east-1 # By default use the user system SSH keys. Override them here. whirr.private-key-file=${sys:user.home}/.ssh/id_rsa2 whirr.public-key-file=${whirr.private-key-file} .pub
        Hide
        David Alves added a comment -

        Also: could you post/attach the *-site.xml files?

        Show
        David Alves added a comment - Also: could you post/attach the *-site.xml files?
        Hide
        David Alves added a comment -

        Hi, sorry for coming late to this.

        Jongwook Woo: Exactly where in /tmp is the folder that has the 9.5 GB's of data?
        Could u post the result of a "du -h" executed on /tmp (after the error) along with your complete whirr-hbase.properties and whirr-hadoop.properties?

        If the overwhelming amount of data is on "/tmp/mapred/local" this should be overrided by hadoop-mapreduce "mapred.local.dir" property and actually is by default so there must be a problem loading config properties (afaik the only thing that writes to $

        {hadoop.tmp.dir}

        /mapred/local is mapper intermediate materialization and this shouldn't have anything to do with hbase or its properties).

        Show
        David Alves added a comment - Hi, sorry for coming late to this. Jongwook Woo: Exactly where in /tmp is the folder that has the 9.5 GB's of data? Could u post the result of a "du -h" executed on /tmp (after the error) along with your complete whirr-hbase.properties and whirr-hadoop.properties? If the overwhelming amount of data is on "/tmp/mapred/local" this should be overrided by hadoop-mapreduce "mapred.local.dir" property and actually is by default so there must be a problem loading config properties (afaik the only thing that writes to $ {hadoop.tmp.dir} /mapred/local is mapper intermediate materialization and this shouldn't have anything to do with hbase or its properties).
        Hide
        Jongwook Woo added a comment -

        I also ran two times bigger data on HBase code without scan.setCaching(2000) and it has the same error as follows because of the space limit:
        11/11/13 04:51:34 WARN mapred.LocalJobRunner: job_local_0001
        org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:192)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:84)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:217)
        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1535)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
        Caused by: java.io.IOException: No space left on device
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:282)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:190)
        ... 15 more

        jongwook@ip-10-84-69-196:~$ df -h
        Filesystem Size Used Avail Use% Mounted on
        /dev/sda1 9.9G 9.4G 1.2M 100% /
        none 846M 116K 846M 1% /dev
        none 879M 0 879M 0% /dev/shm
        none 879M 64K 878M 1% /var/run
        none 879M 0 879M 0% /var/lock
        none 879M 0 879M 0% /lib/init/rw
        /dev/sda2 335G 200M 318G 1% /mnt

        Thus, the error is not mainly from "scan.setCaching(2000)". If there is a way to configure to increase the size of /dev/sda1, that is, /tmp, the issue could be resolved.

        Show
        Jongwook Woo added a comment - I also ran two times bigger data on HBase code without scan.setCaching(2000) and it has the same error as follows because of the space limit: 11/11/13 04:51:34 WARN mapred.LocalJobRunner: job_local_0001 org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:192) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:84) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:217) at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1535) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:282) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:190) ... 15 more jongwook@ip-10-84-69-196:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.9G 9.4G 1.2M 100% / none 846M 116K 846M 1% /dev none 879M 0 879M 0% /dev/shm none 879M 64K 878M 1% /var/run none 879M 0 879M 0% /var/lock none 879M 0 879M 0% /lib/init/rw /dev/sda2 335G 200M 318G 1% /mnt Thus, the error is not mainly from "scan.setCaching(2000)". If there is a way to configure to increase the size of /dev/sda1, that is, /tmp, the issue could be resolved.
        Hide
        Jongwook Woo added a comment -

        If I remove scan.setCaching(2000), it works fine with the space as follows but As you know, the code speed becomes too slow - /dev/sda1 is for /tmp and it still has good enough space as it only uses 21%:

        jongwook@ip-10-84-69-196:~$ df -h
        Filesystem Size Used Avail Use% Mounted on
        /dev/sda1 9.9G 2.0G 7.5G 21% /
        none 846M 116K 846M 1% /dev
        none 879M 0 879M 0% /dev/shm
        none 879M 64K 878M 1% /var/run
        none 879M 0 879M 0% /var/lock
        none 879M 0 879M 0% /lib/init/rw
        /dev/sda2 335G 198M 318G 1% /mnt

        Is there any way to change the configuration to store cached files to /data/tmp that has 335GB not /tmp that has only 9.9GB? Or way to increase /dev/sda1 to much bigger space? Then, it will resolve this issue. If we don't use setCaching, whirr may not become useful to use hbase applications on it.

        Show
        Jongwook Woo added a comment - If I remove scan.setCaching(2000), it works fine with the space as follows but As you know, the code speed becomes too slow - /dev/sda1 is for /tmp and it still has good enough space as it only uses 21%: jongwook@ip-10-84-69-196:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.9G 2.0G 7.5G 21% / none 846M 116K 846M 1% /dev none 879M 0 879M 0% /dev/shm none 879M 64K 878M 1% /var/run none 879M 0 879M 0% /var/lock none 879M 0 879M 0% /lib/init/rw /dev/sda2 335G 198M 318G 1% /mnt Is there any way to change the configuration to store cached files to /data/tmp that has 335GB not /tmp that has only 9.9GB? Or way to increase /dev/sda1 to much bigger space? Then, it will resolve this issue. If we don't use setCaching, whirr may not become useful to use hbase applications on it.
        Hide
        Andrei Savu added a comment -

        Jongwook is this failure still happening if you remove scan.setCaching(2000)?

        Show
        Andrei Savu added a comment - Jongwook is this failure still happening if you remove scan.setCaching(2000)?
        Hide
        Andrei Savu added a comment -

        Lars any ideas? Are there any HBase cache settings we are missing?

        Show
        Andrei Savu added a comment - Lars any ideas? Are there any HBase cache settings we are missing?
        Hide
        Jongwook Woo added a comment -

        I have the same error. I believe that no matter how to configure jobtracker or temp directory, the data is stored at "/tmp/hadoop-jongwook/mapred/local" that has only 9.9G. Thus, it causes "no space" error.

        And, "/data/tmp/hadoop-jongwook" folder is empty as follows:
        jongwook@ip-10-84-67-134:/data/tmp/hadoop-jongwook$ ls -al
        total 8
        drwxr-xr-x 2 jongwook jongwook 4096 2011-11-09 19:09 .
        drwxrwxrwt 3 root root 4096 2011-11-09 19:09 ..

        To read data from HBase I have used 'setCaching' for scan as in the below. Do you think setCaching is the function that store data to "/tmp/hadoop-jongwook/mapred/local"? If it is, which config file we need to change? If not, what other config files we could update to store data to "/data/tmp/hadoop-jongwook" not "/tmp/hadoop-jongwook"?:

        //For Mapper
        Scan scan = new Scan();
        scan.setCaching(2000);
        scan.setFilter(new FirstKeyOnlyFilter());
        scan.addFamily(Bytes.toBytes("items"));
        job.setOutputFormatClass(NullOutputFormat.class);
        TableMapReduceUtil.initTableMapperJob("market_trax", scan, MyMapper.class, Text.class, IntWritable.class, job);

        //For Reducer
        TableMapReduceUtil.initTableReducerJob("mba", MyReducer.class, job);

        Show
        Jongwook Woo added a comment - I have the same error. I believe that no matter how to configure jobtracker or temp directory, the data is stored at "/tmp/hadoop-jongwook/mapred/local" that has only 9.9G. Thus, it causes "no space" error. And, "/data/tmp/hadoop-jongwook" folder is empty as follows: jongwook@ip-10-84-67-134:/data/tmp/hadoop-jongwook$ ls -al total 8 drwxr-xr-x 2 jongwook jongwook 4096 2011-11-09 19:09 . drwxrwxrwt 3 root root 4096 2011-11-09 19:09 .. To read data from HBase I have used 'setCaching' for scan as in the below. Do you think setCaching is the function that store data to "/tmp/hadoop-jongwook/mapred/local"? If it is, which config file we need to change? If not, what other config files we could update to store data to "/data/tmp/hadoop-jongwook" not "/tmp/hadoop-jongwook"?: //For Mapper Scan scan = new Scan(); scan.setCaching(2000); scan.setFilter(new FirstKeyOnlyFilter()); scan.addFamily(Bytes.toBytes("items")); job.setOutputFormatClass(NullOutputFormat.class); TableMapReduceUtil.initTableMapperJob("market_trax", scan, MyMapper.class, Text.class, IntWritable.class, job); //For Reducer TableMapReduceUtil.initTableReducerJob("mba", MyReducer.class, job);
        Hide
        Jongwook Woo added a comment -

        Hi Tom:

        I guess you are talking about the property file "whirr-hadoop-default.properties" under the following path:
        jongwook@ubuntu:~/src/whirr-trunk/services/hadoop/src/main/resources$ ls
        functions META-INF whirr-hadoop-default.properties

        Let me add and run it. I will let you know how it works.

        Jongwook

        Show
        Jongwook Woo added a comment - Hi Tom: I guess you are talking about the property file "whirr-hadoop-default.properties" under the following path: jongwook@ubuntu:~/src/whirr-trunk/services/hadoop/src/main/resources$ ls functions META-INF whirr-hadoop-default.properties Let me add and run it. I will let you know how it works. Jongwook
        Hide
        Tom White added a comment -

        Sorry for coming to this late. Can you try adding the following to your configuration?

        hadoop-mapreduce.mapreduce.jobtracker.staging.root.dir=/data/tmp/mapred/staging
        hadoop-mapreduce.mapred.temp.dir=/data/mapred/temp
        
        Show
        Tom White added a comment - Sorry for coming to this late. Can you try adding the following to your configuration? hadoop-mapreduce.mapreduce.jobtracker.staging.root.dir=/data/tmp/mapred/staging hadoop-mapreduce.mapred.temp.dir=/data/mapred/temp
        Hide
        Andrei Savu added a comment -

        Jongwook sorry for not getting to this sooner - the queue for 0.7.0 RC1 is still long. Can you tell me a few things about your workload?

        Show
        Andrei Savu added a comment - Jongwook sorry for not getting to this sooner - the queue for 0.7.0 RC1 is still long. Can you tell me a few things about your workload?
        Hide
        Andrei Savu added a comment -

        I'm still busy doing something else. I'm planning to work on as many issues blocking the release over this weekend. Some help is highly appreciated.

        Show
        Andrei Savu added a comment - I'm still busy doing something else. I'm planning to work on as many issues blocking the release over this weekend. Some help is highly appreciated.
        Hide
        Jongwook Woo added a comment - - edited

        No clue yet, Andrei?

        Show
        Jongwook Woo added a comment - - edited No clue yet, Andrei?
        Hide
        Jongwook Woo added a comment -

        I just copy and paste the recipe but # is shown as number so that you could consider 1, 2 => #

        Show
        Jongwook Woo added a comment - I just copy and paste the recipe but # is shown as number so that you could consider 1, 2 => #
        Hide
        Jongwook Woo added a comment - - edited
        1. HBase Cluster on AWS EC2
        1. Read the Configuration Guide for more info:
        2. http://incubator.apache.org/whirr/configuration-guide.html
        1. Change the cluster name here
          #whirr.cluster-name=test-cluster
          whirr.cluster-name=hbase
        1. Change the number of machines in the cluster here
          whirr.instance-templates=1 zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,5 hadoop-datanode+hadoop-tasktracker+hbase-regionserver
        1. For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
          whirr.provider=aws-ec2
          whirr.identity=$ {env:AWS_ACCESS_KEY_ID}

          whirr.credential=$

          {env:AWS_SECRET_ACCESS_KEY}
        1. JW's; $0.17 per High-CPU Medium Instance (c1.medium) instance-hour (or partial hour)
          whirr.hardware-id=c1.medium
          whirr.image-id=us-east-1/ami-7000f019
        1. If you choose a different location, make sure whirr.image-id is updated too
          whirr.location-id=us-east-1
        1. By default use the user system SSH keys. Override them here.
          whirr.private-key-file=$ {sys:user.home}

          /.ssh/id_rsa2
          whirr.public-key-file=$

          {whirr.private-key-file}

          .pub

        Show
        Jongwook Woo added a comment - - edited HBase Cluster on AWS EC2 Read the Configuration Guide for more info: http://incubator.apache.org/whirr/configuration-guide.html Change the cluster name here #whirr.cluster-name=test-cluster whirr.cluster-name=hbase Change the number of machines in the cluster here whirr.instance-templates=1 zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,5 hadoop-datanode+hadoop-tasktracker+hbase-regionserver For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. whirr.provider=aws-ec2 whirr.identity=$ {env:AWS_ACCESS_KEY_ID} whirr.credential=$ {env:AWS_SECRET_ACCESS_KEY} JW's; $0.17 per High-CPU Medium Instance (c1.medium) instance-hour (or partial hour) whirr.hardware-id=c1.medium whirr.image-id=us-east-1/ami-7000f019 If you choose a different location, make sure whirr.image-id is updated too whirr.location-id=us-east-1 By default use the user system SSH keys. Override them here. whirr.private-key-file=$ {sys:user.home} /.ssh/id_rsa2 whirr.public-key-file=$ {whirr.private-key-file} .pub
        Hide
        Andrei Savu added a comment -

        Can you share the recipe you are using to start the cluster?

        Show
        Andrei Savu added a comment - Can you share the recipe you are using to start the cluster?
        Hide
        Jongwook Woo added a comment -

        Yes that is what I pointed out in this issue. Whirr 0.6.0 and 0.7.0 create /data/tmp but the jobcache files are stored at /tmp while running hbase code with hadoop. And, there is no file generated under /data/tmp.

        I don't test CDH. For whirr 0.6.0, I just downloaded and run it. For whirr 0.7.0, I just SVNed and run it.

        I may also take a look at whirr codes. If you have some idea in the following questions, it will be much easier to resolve the issue:
        (1) Is "whirr-hbase-default.properties" the only one that determines the location of jobcache files?
        (2) Do you remember any other configuration files or codes that specify the location of jobcache?
        (3) do you think my code - or any TableMapper hadoop code - to scan HBase also can specify the /tmp or /data/tmp folder to store jobcache files?

        Show
        Jongwook Woo added a comment - Yes that is what I pointed out in this issue. Whirr 0.6.0 and 0.7.0 create /data/tmp but the jobcache files are stored at /tmp while running hbase code with hadoop. And, there is no file generated under /data/tmp. I don't test CDH. For whirr 0.6.0, I just downloaded and run it. For whirr 0.7.0, I just SVNed and run it. I may also take a look at whirr codes. If you have some idea in the following questions, it will be much easier to resolve the issue: (1) Is "whirr-hbase-default.properties" the only one that determines the location of jobcache files? (2) Do you remember any other configuration files or codes that specify the location of jobcache? (3) do you think my code - or any TableMapper hadoop code - to scan HBase also can specify the /tmp or /data/tmp folder to store jobcache files?
        Hide
        Andrei Savu added a comment -

        Are you testing against CDH or a tarball based install?

        Show
        Andrei Savu added a comment - Are you testing against CDH or a tarball based install?
        Hide
        Andrei Savu added a comment -

        It seems like the temporary folder is configured as expected:

        hbase-site.hbase.tmp.dir=/data/tmp/hbase-\${user.name}
        

        See https://github.com/andreisavu/whirr/blob/trunk/services/hbase/src/main/resources/whirr-hbase-default.properties

        Show
        Andrei Savu added a comment - It seems like the temporary folder is configured as expected: hbase-site.hbase.tmp.dir=/data/tmp/hbase-\${user.name} See https://github.com/andreisavu/whirr/blob/trunk/services/hbase/src/main/resources/whirr-hbase-default.properties
        Hide
        Andrei Savu added a comment -

        All the code related to setting-up HBase is in services/cdh and services/hbase. Look into src/main/resources/functions for [install | configure]_hbase.sh. Ping me on IIRC #whirr - I can help you as needed.

        Show
        Andrei Savu added a comment - All the code related to setting-up HBase is in services/cdh and services/hbase . Look into src/main/resources/functions for [install | configure] _hbase.sh . Ping me on IIRC #whirr - I can help you as needed.
        Hide
        Jongwook Woo added a comment - - edited

        Just to make sure, it works well with hadoop alone. However, if I run a Hadoop code to read/write data to/from HBase - that is, hbase with hadoop, the less storage error occurs. Thus, it seems that hbase jobcache data is still stored in /tmp not /data/tmp.

        If you tell me which part of whirr code are related to configure hbase.tmp.dir and hadoop.tmp.dir, I could also take a look at them.

        Show
        Jongwook Woo added a comment - - edited Just to make sure, it works well with hadoop alone. However, if I run a Hadoop code to read/write data to/from HBase - that is, hbase with hadoop, the less storage error occurs. Thus, it seems that hbase jobcache data is still stored in /tmp not /data/tmp. If you tell me which part of whirr code are related to configure hbase.tmp.dir and hadoop.tmp.dir, I could also take a look at them.
        Hide
        Andrei Savu added a comment -

        Ok. I will start a cluster tomorrow and inspect the config files on the remote machines.

        Show
        Andrei Savu added a comment - Ok. I will start a cluster tomorrow and inspect the config files on the remote machines.
        Hide
        Jongwook Woo added a comment -

        I am pretty sure and I have run my codes at both 0.60.0 and 0.70.0.

        I have seen the property files so that I have checked the /tmp and /data/tmp folders several times with tens of code execution but as shown in the error message and 'df -h', jobcache folder is generated at /tmp so that it has the less storage error.

        Show
        Jongwook Woo added a comment - I am pretty sure and I have run my codes at both 0.60.0 and 0.70.0. I have seen the property files so that I have checked the /tmp and /data/tmp folders several times with tens of code execution but as shown in the error message and 'df -h', jobcache folder is generated at /tmp so that it has the less storage error.
        Hide
        Andrei Savu added a comment -

        And this seems to be also true for release 0.6.0. Are you sure Hadoop is filling the /tmp?

        Show
        Andrei Savu added a comment - And this seems to be also true for release 0.6.0. Are you sure Hadoop is filling the /tmp?
        Hide
        Andrei Savu added a comment -

        Are you sure you've tried with the latest trunk?

        It seems like Hadoop is using /data/tmp as expected:

        See services/hadoop/src/main/resources/whirr-hadoop-default.properties:

        hadoop-common.hadoop.tmp.dir=/data/tmp/hadoop-\${user.name}
        

        and services/hadoop/src/main/resources/functions/configure_hadoop.sh:

         if [ ! -e /data/tmp ]; then
            mkdir /data/tmp
            chmod a+rwxt /data/tmp
          fi
        

        and the same settings are used for CDH.

        Show
        Andrei Savu added a comment - Are you sure you've tried with the latest trunk? It seems like Hadoop is using /data/tmp as expected: See services/hadoop/src/main/resources/whirr-hadoop-default.properties : hadoop-common.hadoop.tmp.dir=/data/tmp/hadoop-\${user.name} and services/hadoop/src/main/resources/functions/configure_hadoop.sh : if [ ! -e /data/tmp ]; then mkdir /data/tmp chmod a+rwxt /data/tmp fi and the same settings are used for CDH.
        Hide
        Jongwook Woo added a comment -

        How to find and submit a patch?

        WHIRR-412 is resolved by your direction. I am using eclipse.

        Show
        Jongwook Woo added a comment - How to find and submit a patch? WHIRR-412 is resolved by your direction. I am using eclipse.
        Hide
        Andrei Savu added a comment -

        Good catch! Can you submit a patch? Is WHIRR-412 still affecting you? I'm using Intellij IDEA on a daily basis.

        Show
        Andrei Savu added a comment - Good catch! Can you submit a patch? Is WHIRR-412 still affecting you? I'm using Intellij IDEA on a daily basis.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jongwook Woo
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Remaining Estimate - 48h
              48h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development