Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1956

Multi RS HBase requires unique hbase.tmp.dir to be set for each RS on a node

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: hbase
    • Labels:
      None

      Description

      In a multi RS set-up of HBase, each RS instance need to have a unique "hbase.tmp.dir" without which starting of RS automatically at the same time will fail if HBase coprocessors are used due to contention on the temp directory.

      1. BIGTOP-1956-3.patch
        3 kB
        Biju Nair
      2. BIGTOP-1956-2.patch
        2 kB
        Biju Nair
      3. BIGTOP-1956-1.patch
        2 kB
        Biju Nair
      4. BIGTOP-1956.patch
        2 kB
        Biju Nair

        Activity

        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Biju Nair, 3.patch works fine on my VM Verified log files and JVM properties and everything looks good to me!

        $ ps aux | grep hbase
        hbase     6871  0.0  0.0   9236  1244 ?        S    01:08   0:00 bash /usr/lib/hbase/bin/hbase-daemon.sh --config /etc/hbase/conf foreground_start regionserver -D hbase.regionserver.port=60201 -D hbase.regionserver.info.port=60301
        hbase     6886  8.5  3.4 1633492 133796 ?      Sl   01:08   0:03 /usr/lib/jvm/java-openjdk/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC -Djava.io.tmpdir=/tmp/java_tmp_dir/1 -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-1-regionserver-bigtop1.vagrant.log -Dhbase.home.dir=/usr/lib/hbase -Dhbase.id.str=hbase-1 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/usr/lib/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer -D hbase.regionserver.port=60201 -D hbase.regionserver.info.port=60301 start
        hbase     6974  0.0  0.0   9236  1244 ?        S    01:08   0:00 bash /usr/lib/hbase/bin/hbase-daemon.sh --config /etc/hbase/conf foreground_start regionserver -D hbase.regionserver.port=60202 -D hbase.regionserver.info.port=60302
        hbase     6989  8.5  3.4 1647688 135096 ?      Sl   01:08   0:03 /usr/lib/jvm/java-openjdk/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC -Djava.io.tmpdir=/tmp/java_tmp_dir/2 -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-2-regionserver-bigtop1.vagrant.log -Dhbase.home.dir=/usr/lib/hbase -Dhbase.id.str=hbase-2 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/usr/lib/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer -D hbase.regionserver.port=60202 -D hbase.regionserver.info.port=60302 start
        hbase     7080  0.0  0.0   9236  1244 ?        S    01:08   0:00 bash /usr/lib/hbase/bin/hbase-daemon.sh --config /etc/hbase/conf foreground_start regionserver -D hbase.regionserver.port=60203 -D hbase.regionserver.info.port=60303
        hbase     7095  7.6  3.3 1648952 131608 ?      Sl   01:08   0:03 /usr/lib/jvm/java-openjdk/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC -Djava.io.tmpdir=/tmp/java_tmp_dir/3 -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-3-regionserver-bigtop1.vagrant.log -Dhbase.home.dir=/usr/lib/hbase -Dhbase.id.str=hbase-3 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/usr/lib/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer -D hbase.regionserver.port=60203 -D hbase.regionserver.info.port=60303 start
        
        

        Will commit it shortly. Thanks for your contribution, Biju Nair!

        Show
        warwithin YoungWoo Kim added a comment - - edited Biju Nair , 3.patch works fine on my VM Verified log files and JVM properties and everything looks good to me! $ ps aux | grep hbase hbase 6871 0.0 0.0 9236 1244 ? S 01:08 0:00 bash /usr/lib/hbase/bin/hbase-daemon.sh --config /etc/hbase/conf foreground_start regionserver -D hbase.regionserver.port=60201 -D hbase.regionserver.info.port=60301 hbase 6886 8.5 3.4 1633492 133796 ? Sl 01:08 0:03 /usr/lib/jvm/java-openjdk/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC -Djava.io.tmpdir=/tmp/java_tmp_dir/1 -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-1-regionserver-bigtop1.vagrant.log -Dhbase.home.dir=/usr/lib/hbase -Dhbase.id.str=hbase-1 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/usr/lib/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer -D hbase.regionserver.port=60201 -D hbase.regionserver.info.port=60301 start hbase 6974 0.0 0.0 9236 1244 ? S 01:08 0:00 bash /usr/lib/hbase/bin/hbase-daemon.sh --config /etc/hbase/conf foreground_start regionserver -D hbase.regionserver.port=60202 -D hbase.regionserver.info.port=60302 hbase 6989 8.5 3.4 1647688 135096 ? Sl 01:08 0:03 /usr/lib/jvm/java-openjdk/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC -Djava.io.tmpdir=/tmp/java_tmp_dir/2 -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-2-regionserver-bigtop1.vagrant.log -Dhbase.home.dir=/usr/lib/hbase -Dhbase.id.str=hbase-2 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/usr/lib/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer -D hbase.regionserver.port=60202 -D hbase.regionserver.info.port=60302 start hbase 7080 0.0 0.0 9236 1244 ? S 01:08 0:00 bash /usr/lib/hbase/bin/hbase-daemon.sh --config /etc/hbase/conf foreground_start regionserver -D hbase.regionserver.port=60203 -D hbase.regionserver.info.port=60303 hbase 7095 7.6 3.3 1648952 131608 ? Sl 01:08 0:03 /usr/lib/jvm/java-openjdk/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC -Djava.io.tmpdir=/tmp/java_tmp_dir/3 -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-3-regionserver-bigtop1.vagrant.log -Dhbase.home.dir=/usr/lib/hbase -Dhbase.id.str=hbase-3 -Dhbase.root.logger=INFO,RFA -Djava.library.path=/usr/lib/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer -D hbase.regionserver.port=60203 -D hbase.regionserver.info.port=60303 start Will commit it shortly. Thanks for your contribution, Biju Nair !
        Hide
        gsbiju Biju Nair added a comment - - edited

        YoungWoo Kim, thanks for the test. Can you confirm that the RS comes up even though the error message is thrown. The console out put shows all the 3 RS came up successfully. They should come up even when the directory doesn't exist but have the privilege to create the directories. We use an automated build process which brings up the RS and as far as they come-up successfully it ignores these messages and that being the reason this was not caught. I have updated the jira with a new patch. Hope this should resolve the misleading message displayed.

        Show
        gsbiju Biju Nair added a comment - - edited YoungWoo Kim , thanks for the test. Can you confirm that the RS comes up even though the error message is thrown. The console out put shows all the 3 RS came up successfully. They should come up even when the directory doesn't exist but have the privilege to create the directories. We use an automated build process which brings up the RS and as far as they come-up successfully it ignores these messages and that being the reason this was not caught. I have updated the jira with a new patch. Hope this should resolve the misleading message displayed.
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Added a following line to /etc/default/hbase:

        REGIONSERVER_OFFSETS="1 2 3"
        

        I got errors:

        $ sudo service hbase-regionserver start
        Starting regionserver daemon 1: OK
        Starting regionserver daemon 2: /etc/init.d/hbase-regionserver: line 215: -Djava.io.tmpdir=/tmp/java_tmp_dir/1: No such file or directory
        OK
        Starting regionserver daemon 3: /etc/init.d/hbase-regionserver: line 215: -Djava.io.tmpdir=/tmp/java_tmp_dir/2: No such file or directory
        OK
        

        Biju Nair, Please take a look. I tested it on my vagrant VM with new hbase packages and also I created the directories "/tmp/java_tmp_dir/{1,2,3}" manually but no differences.

        Show
        warwithin YoungWoo Kim added a comment - - edited Added a following line to /etc/default/hbase: REGIONSERVER_OFFSETS= "1 2 3" I got errors: $ sudo service hbase-regionserver start Starting regionserver daemon 1: OK Starting regionserver daemon 2: /etc/init.d/hbase-regionserver: line 215: -Djava.io.tmpdir=/tmp/java_tmp_dir/1: No such file or directory OK Starting regionserver daemon 3: /etc/init.d/hbase-regionserver: line 215: -Djava.io.tmpdir=/tmp/java_tmp_dir/2: No such file or directory OK Biju Nair , Please take a look. I tested it on my vagrant VM with new hbase packages and also I created the directories "/tmp/java_tmp_dir/{1,2,3}" manually but no differences.
        Hide
        gsbiju Biju Nair added a comment -

        Thanks YoungWoo Kim for the review and catching the issue. Looks like the additional tick got added when I created the patch in a fresh local git repo. Sorry about that.

        Show
        gsbiju Biju Nair added a comment - Thanks YoungWoo Kim for the review and catching the issue. Looks like the additional tick got added when I created the patch in a fresh local git repo. Sorry about that.
        Hide
        gsbiju Biju Nair added a comment -

        Updated patch for review comments.

        Show
        gsbiju Biju Nair added a comment - Updated patch for review comments.
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Thank you Biju Nair!

        +        HBASE_TMP_DIR=" -Djava.io.tmpdir=${JAVA_TMP_DIR}/${OFFSET}"`
        

        Looks like it is a unexpected backtick at the end of the line!?

        Show
        warwithin YoungWoo Kim added a comment - - edited Thank you Biju Nair ! + HBASE_TMP_DIR= " -Djava.io.tmpdir=${JAVA_TMP_DIR}/${OFFSET}" ` Looks like it is a unexpected backtick at the end of the line!?
        Hide
        gsbiju Biju Nair added a comment -

        YoungWoo Kim, please have a look at the updated patch and let me know if you find any issues.

        Show
        gsbiju Biju Nair added a comment - YoungWoo Kim , please have a look at the updated patch and let me know if you find any issues.
        Hide
        gsbiju Biju Nair added a comment - - edited

        Updated patch (#1) with changes from patch #3 of BIGTOP-1632. Since I had submitted patch #3 of BIGTOP-1632 after the ticket got marked as resolved, it didn't make it to the master branch.

        Show
        gsbiju Biju Nair added a comment - - edited Updated patch (#1) with changes from patch #3 of BIGTOP-1632 . Since I had submitted patch #3 of BIGTOP-1632 after the ticket got marked as resolved, it didn't make it to the master branch.
        Hide
        warwithin YoungWoo Kim added a comment -

        Got it, feel free to ping me when you are ready. Thanks!

        Show
        warwithin YoungWoo Kim added a comment - Got it, feel free to ping me when you are ready. Thanks!
        Hide
        gsbiju Biju Nair added a comment -

        Thanks YoungWoo Kim for taking a look at the patch. Can you please hold off your testing. Looks like there may be an issue with the original code which I would like to verify.

        Show
        gsbiju Biju Nair added a comment - Thanks YoungWoo Kim for taking a look at the patch. Can you please hold off your testing. Looks like there may be an issue with the original code which I would like to verify.
        Hide
        warwithin YoungWoo Kim added a comment -

        Thanks Biju Nair, looks good to me! Will test your patch at tonight.

        Show
        warwithin YoungWoo Kim added a comment - Thanks Biju Nair , looks good to me! Will test your patch at tonight.
        Hide
        gsbiju Biju Nair added a comment -

        Patch attached.

        Show
        gsbiju Biju Nair added a comment - Patch attached.
        Hide
        gsbiju Biju Nair added a comment -

        Attached patch for the issue. Please review and provide andy comments.

        Show
        gsbiju Biju Nair added a comment - Attached patch for the issue. Please review and provide andy comments.

          People

          • Assignee:
            gsbiju Biju Nair
            Reporter:
            gsbiju Biju Nair
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development