HBase
  1. HBase
  2. HBASE-4391

Add ability to start RS as root and call mlockall

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.95.0
    • Component/s: regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      A new profile, native, has been added to pom.xml Adding '-Pnative' will build libmlockall_agent.so HBase will look for the .so under ./lib/native/PLATFORM (See JAVA_LIBRARY_PATH in the bin/hbase starup script).

      mlockall_agent is a simple VM Agent that allows to lock the address space of
      the region server process. This avoids the process' memory eviction under pressure.

      You can load the agent by adding it as a jvm option:
       export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"

      To use this feature, execute the following command as root:
      hbase --mlock user=hbase regionserver start
      Show
      A new profile, native, has been added to pom.xml Adding '-Pnative' will build libmlockall_agent.so HBase will look for the .so under ./lib/native/PLATFORM (See JAVA_LIBRARY_PATH in the bin/hbase starup script). mlockall_agent is a simple VM Agent that allows to lock the address space of the region server process. This avoids the process' memory eviction under pressure. You can load the agent by adding it as a jvm option:  export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase" To use this feature, execute the following command as root: hbase --mlock user=hbase regionserver start
    • Tags:
      0.96notable

      Description

      A common issue we've seen in practice is that users oversubscribe their region servers with too many MR tasks, etc. As soon as the machine starts swapping, the RS grinds to a halt, loses ZK session, aborts, etc.

      This can be combatted by starting the RS as root, calling mlockall(), and then setuid down to the hbase user. We should not require this, but we should provide it as an option.

      1. 4391-v4.patch
        9 kB
        Ted Yu
      2. HBASE-4391-v3.patch
        9 kB
        Matteo Bertozzi
      3. HBASE-4391-v2.patch
        9 kB
        Matteo Bertozzi
      4. HBASE-4391-v1.patch
        24 kB
        Matteo Bertozzi
      5. HBASE-4391-v0.patch
        24 kB
        Matteo Bertozzi

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Assignee Todd Lipcon [ tlipcon ]
          Hide
          Todd Lipcon added a comment -

          I've put an initial implementation of a JVMTI agent that accomplishes this here:

          https://github.com/toddlipcon/mlockall_agent

          It can be compiled using the "compile.sh" script in the repo, and then configured like so in hbase-env.sh:

          export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -agentpath:/home/todd/git/libmlockall_agent/libmlockall_agent.so=user=todd"
          

          After which point you must run the regionserver as root:

          
          todd@todd-w510:~/git/hbase$ ./bin/hbase regionserver 
          mlockall_agent: Unable to boost memlock resource limit: Operation not permitted
          mlockall_agent: (this may be because java was not run as root!)
          Error occurred during initialization of VM
          agent library failed to init: /home/todd/git/libmlockall_agent/libmlockall_agent.so
          todd@todd-w510:~/git/hbase$ sudo ./bin/hbase regionserver 
          mlockall_agent: Successfully locked memory and setuid to todd
          

          Perhaps some other people can check this out, and if it looks generally useful we could either build it along with HBase or at least point to it from docs?

          Show
          Todd Lipcon added a comment - I've put an initial implementation of a JVMTI agent that accomplishes this here: https://github.com/toddlipcon/mlockall_agent It can be compiled using the "compile.sh" script in the repo, and then configured like so in hbase-env.sh: export HBASE_REGIONSERVER_OPTS= "$HBASE_REGIONSERVER_OPTS -agentpath:/home/todd/git/libmlockall_agent/libmlockall_agent.so=user=todd" After which point you must run the regionserver as root: todd@todd-w510:~/git/hbase$ ./bin/hbase regionserver mlockall_agent: Unable to boost memlock resource limit: Operation not permitted mlockall_agent: ( this may be because java was not run as root!) Error occurred during initialization of VM agent library failed to init: /home/todd/git/libmlockall_agent/libmlockall_agent.so todd@todd-w510:~/git/hbase$ sudo ./bin/hbase regionserver mlockall_agent: Successfully locked memory and setuid to todd Perhaps some other people can check this out, and if it looks generally useful we could either build it along with HBase or at least point to it from docs?
          Hide
          Lars Hofhansl added a comment -

          Moving out of 0.94, pull back if feel otherwise.

          Show
          Lars Hofhansl added a comment - Moving out of 0.94, pull back if feel otherwise.
          Lars Hofhansl made changes -
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Hide
          Matteo Bertozzi added a comment -

          todd's code looks good to me.

          One way to avoid using root to start the RS is adding hbase user to /etc/security/limits.d/

          echo "$USER - memlock $MAX_MEMLOCK_SIZE" >/etc/security/limits.d/$USER.conf

          If this will be integrated in hbase, do you want to force everyone to use root or add hbase user in limits.d to start RS? or we can add some --no-swap flag to region server start script?

          Show
          Matteo Bertozzi added a comment - todd's code looks good to me. One way to avoid using root to start the RS is adding hbase user to /etc/security/limits.d/ echo "$USER - memlock $MAX_MEMLOCK_SIZE" >/etc/security/limits.d/$USER.conf If this will be integrated in hbase, do you want to force everyone to use root or add hbase user in limits.d to start RS? or we can add some --no-swap flag to region server start script?
          Hide
          Matteo Bertozzi added a comment -

          Something like this? where you can start hbase scripts with --mlock user=hbase to enable the mlock agent?

          Show
          Matteo Bertozzi added a comment - Something like this? where you can start hbase scripts with --mlock user=hbase to enable the mlock agent?
          Matteo Bertozzi made changes -
          Attachment HBASE-4391-v0.patch [ 12531566 ]
          Hide
          Andrew Purtell added a comment -

          If we want to approximate the way Hadoop core includes and builds native bits, the source layout could be more like:

          hbase-server/src/main/native/
            configure scripts
            m4/
            src/
              mlockall_agent/
          
          Show
          Andrew Purtell added a comment - If we want to approximate the way Hadoop core includes and builds native bits, the source layout could be more like: hbase-server/src/main/native/ configure scripts m4/ src/ mlockall_agent/
          Show
          Enis Soztutar added a comment - I've seen smt similar in accumulo code base: http://svn.apache.org/viewvc/accumulo/trunk/server/src/main/c%2B%2B/mlock/ http://svn.apache.org/viewvc/accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/MLock.java?view=log
          Hide
          Matteo Bertozzi added a comment -

          Accumulo has a "tserver.memory.lock" property used to enable mlock in TabletServer. At startup the property is read and if true mlockall() is called.

          Different approach but the result is the same... the Todd proposed solution is external from the RegionServer code (and maybe can be integrated in hadoop-common to be used by datanode & co without changing the hdfs code), the accumulo one is integrated in the tablet code...

          Show
          Matteo Bertozzi added a comment - Accumulo has a "tserver.memory.lock" property used to enable mlock in TabletServer. At startup the property is read and if true mlockall() is called. Different approach but the result is the same... the Todd proposed solution is external from the RegionServer code (and maybe can be integrated in hadoop-common to be used by datanode & co without changing the hdfs code), the accumulo one is integrated in the tablet code...
          Hide
          stack added a comment -

          @Matteo What about Andrew's suggested location for the src and native bits? Whats it like when we add compile of mlockall to our compile goal in mvn? Is it fast? On this question:

          If this will be integrated in hbase, do you want to force everyone to use root or add hbase user in limits.d to start RS? or we can add some --no-swap flag to region server start script?

          We cannot require you be root to start hbase. It looks like this not necessary anyways? If you want the mlockall facility, then you can require folks be root (or do you security.conf trick).

          The Todd trick is nice as is your tidy up. Thanks.

          Show
          stack added a comment - @Matteo What about Andrew's suggested location for the src and native bits? Whats it like when we add compile of mlockall to our compile goal in mvn? Is it fast? On this question: If this will be integrated in hbase, do you want to force everyone to use root or add hbase user in limits.d to start RS? or we can add some --no-swap flag to region server start script? We cannot require you be root to start hbase. It looks like this not necessary anyways? If you want the mlockall facility, then you can require folks be root (or do you security.conf trick). The Todd trick is nice as is your tidy up. Thanks.
          Hide
          Matteo Bertozzi added a comment -

          Moved to src/main/native as Andrew suggested

          Show
          Matteo Bertozzi added a comment - Moved to src/main/native as Andrew suggested
          Matteo Bertozzi made changes -
          Attachment HBASE-4391-v1.patch [ 12532246 ]
          Hide
          Matteo Bertozzi added a comment -

          Added profile "native" (-Pnative) to turn on and off the compilation of the mlock agent.

          hbase --mlock user=hbase regionserver start

          Show
          Matteo Bertozzi added a comment - Added profile "native" (-Pnative) to turn on and off the compilation of the mlock agent. hbase --mlock user=hbase regionserver start
          Matteo Bertozzi made changes -
          Attachment HBASE-4391-v2.patch [ 12533217 ]
          Matteo Bertozzi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Todd Lipcon [ tlipcon ] Matteo Bertozzi [ mbertozzi ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12533217/HBASE-4391-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestSplitLogManager

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2239//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2239//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2239//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2239//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533217/HBASE-4391-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2239//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2239//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2239//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2239//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Can you add some comment near the beginning of mlockall_agent.c for what it does ?

          +    LOG("Unable to setuid: specify a target username as the agent option user=<username>\n");
          +    return(1);
          

          Should different exit code be used to distinguish the error scenarios ?

          compile.sh isn't in repo. Please update release notes on how this feature is compiled and used.

          Show
          Ted Yu added a comment - Can you add some comment near the beginning of mlockall_agent.c for what it does ? + LOG( "Unable to setuid: specify a target username as the agent option user=<username>\n" ); + return (1); Should different exit code be used to distinguish the error scenarios ? compile.sh isn't in repo. Please update release notes on how this feature is compiled and used.
          Hide
          Matteo Bertozzi added a comment -

          compile.sh isn't in repo. Please update release notes on how this feature is compiled and used.

          Do you mean in the patch? or todd's repo?
          The attached patch has a new "native" profile added, that allows to compile the mlock_agent, and if you in your HBASE_HOME/ you've a native/ folder that contains the .so and if you specify the --mlock <ARG> options it checks if native/libmlock_agent.so is available or not.

          Show
          Matteo Bertozzi added a comment - compile.sh isn't in repo. Please update release notes on how this feature is compiled and used. Do you mean in the patch? or todd's repo? The attached patch has a new "native" profile added, that allows to compile the mlock_agent, and if you in your HBASE_HOME/ you've a native/ folder that contains the .so and if you specify the --mlock <ARG> options it checks if native/libmlock_agent.so is available or not.
          Hide
          Ted Yu added a comment -

          I was able to get the following with 'mvn -Pnative package -DskipTests':

          ./target/native/CMakeFiles/mlockall_agent.dir
          ./target/native/CMakeFiles/mlockall_agent.dir/src/mlockall_agent
          ./target/native/CMakeFiles/mlockall_agent.dir/src/mlockall_agent/mlockall_agent.c.o
          

          How do I build the .so ?

          Thanks

          Show
          Ted Yu added a comment - I was able to get the following with 'mvn -Pnative package -DskipTests': ./target/ native /CMakeFiles/mlockall_agent.dir ./target/ native /CMakeFiles/mlockall_agent.dir/src/mlockall_agent ./target/ native /CMakeFiles/mlockall_agent.dir/src/mlockall_agent/mlockall_agent.c.o How do I build the .so ? Thanks
          Hide
          Matteo Bertozzi added a comment -
          $ mvn package -DskipTests -Pnative
          $ ls ./hbase-server/target/native/libmlockall_agent.so
          

          or if you use assembly:assembly
          in ./target/hbase-0.95-SNAPSHOT.tar.gz
          under hbase-0.95-SNAPSHOT/native folder there's the libmlockall_agent.so

          Show
          Matteo Bertozzi added a comment - $ mvn package -DskipTests -Pnative $ ls ./hbase-server/target/ native /libmlockall_agent.so or if you use assembly:assembly in ./target/hbase-0.95-SNAPSHOT.tar.gz under hbase-0.95-SNAPSHOT/native folder there's the libmlockall_agent.so
          Matteo Bertozzi made changes -
          Attachment HBASE-4391-v3.patch [ 12533220 ]
          Hide
          Ted Yu added a comment -

          I built on MacBook. I guess the following is the equivalent:

          -rwxr-xr-x   1 zhihyu  110088321  13320 Jun 24 07:41 libmlockall_agent.dylib
          
          Show
          Ted Yu added a comment - I built on MacBook. I guess the following is the equivalent: -rwxr-xr-x 1 zhihyu 110088321 13320 Jun 24 07:41 libmlockall_agent.dylib
          Hide
          Matteo Bertozzi added a comment -

          Yeah .dylib is the equivalent to .so
          but I've hardcoded .so in the bin/ and assembly script
          any idea to pick the right one?

          (Do we need to support this on mac? you can always load by it hand as explained by todd)

          Show
          Matteo Bertozzi added a comment - Yeah .dylib is the equivalent to .so but I've hardcoded .so in the bin/ and assembly script any idea to pick the right one? (Do we need to support this on mac? you can always load by it hand as explained by todd)
          Hide
          Ted Yu added a comment -

          I don't think we need to support Mac.

          Show
          Ted Yu added a comment - I don't think we need to support Mac.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12533220/HBASE-4391-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2241//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2241//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533220/HBASE-4391-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2241//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2241//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2241//console This message is automatically generated.
          Ted Yu made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Release Note A new profile, native, has been added to pom.xml
          It would build libmlockall_agent.so

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"
          Ted Yu made changes -
          Release Note A new profile, native, has been added to pom.xml
          It would build libmlockall_agent.so

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"
          A new profile, native, has been added to pom.xml
          It would build libmlockall_agent.so

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"

          To use this feature, execute the following command as root:
          hbase --mlock user=hbase regionserver start
          Ted Yu made changes -
          Release Note A new profile, native, has been added to pom.xml
          It would build libmlockall_agent.so

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"

          To use this feature, execute the following command as root:
          hbase --mlock user=hbase regionserver start
          A new profile, native, has been added to pom.xml
          It would build libmlockall_agent.so

          mlockall_agent is a simple VM Agent that allows to lock the address space of
          the region server process. This avoids the process' memory eviction under pressure.

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"

          To use this feature, execute the following command as root:
          hbase --mlock user=hbase regionserver start
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Ted Yu added a comment -

          Patch v4 corrects some grammatical errors.

          Show
          Ted Yu added a comment - Patch v4 corrects some grammatical errors.
          Ted Yu made changes -
          Attachment 4391-v4.patch [ 12533224 ]
          Hide
          Ted Yu added a comment -

          Patch v4 integrated to trunk.

          Thanks for the initiative, Todd.

          Thanks for the patch, Matteo.

          Show
          Ted Yu added a comment - Patch v4 integrated to trunk. Thanks for the initiative, Todd. Thanks for the patch, Matteo.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3064 (See https://builds.apache.org/job/HBase-TRUNK/3064/)
          HBASE-4391 Add ability to start RS as root and call mlockall (Matteo Bertozzi) (Revision 1353289)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/bin/hbase-config.sh
          • /hbase/trunk/hbase-server/pom.xml
          • /hbase/trunk/hbase-server/src/main/native
          • /hbase/trunk/hbase-server/src/main/native/CMakeLists.txt
          • /hbase/trunk/hbase-server/src/main/native/src
          • /hbase/trunk/hbase-server/src/main/native/src/mlockall_agent
          • /hbase/trunk/hbase-server/src/main/native/src/mlockall_agent/mlockall_agent.c
          • /hbase/trunk/src/assembly/all.xml
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3064 (See https://builds.apache.org/job/HBase-TRUNK/3064/ ) HBASE-4391 Add ability to start RS as root and call mlockall (Matteo Bertozzi) (Revision 1353289) Result = FAILURE tedyu : Files : /hbase/trunk/bin/hbase-config.sh /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/native /hbase/trunk/hbase-server/src/main/native/CMakeLists.txt /hbase/trunk/hbase-server/src/main/native/src /hbase/trunk/hbase-server/src/main/native/src/mlockall_agent /hbase/trunk/hbase-server/src/main/native/src/mlockall_agent/mlockall_agent.c /hbase/trunk/src/assembly/all.xml
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #67 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/67/)
          HBASE-4391 Add ability to start RS as root and call mlockall (Matteo Bertozzi) (Revision 1353289)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/bin/hbase-config.sh
          • /hbase/trunk/hbase-server/pom.xml
          • /hbase/trunk/hbase-server/src/main/native
          • /hbase/trunk/hbase-server/src/main/native/CMakeLists.txt
          • /hbase/trunk/hbase-server/src/main/native/src
          • /hbase/trunk/hbase-server/src/main/native/src/mlockall_agent
          • /hbase/trunk/hbase-server/src/main/native/src/mlockall_agent/mlockall_agent.c
          • /hbase/trunk/src/assembly/all.xml
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #67 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/67/ ) HBASE-4391 Add ability to start RS as root and call mlockall (Matteo Bertozzi) (Revision 1353289) Result = FAILURE tedyu : Files : /hbase/trunk/bin/hbase-config.sh /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/native /hbase/trunk/hbase-server/src/main/native/CMakeLists.txt /hbase/trunk/hbase-server/src/main/native/src /hbase/trunk/hbase-server/src/main/native/src/mlockall_agent /hbase/trunk/hbase-server/src/main/native/src/mlockall_agent/mlockall_agent.c /hbase/trunk/src/assembly/all.xml
          Matteo Bertozzi made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.0 [ 12316419 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.0 [ 12316419 ]
          stack made changes -
          Release Note A new profile, native, has been added to pom.xml
          It would build libmlockall_agent.so

          mlockall_agent is a simple VM Agent that allows to lock the address space of
          the region server process. This avoids the process' memory eviction under pressure.

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"

          To use this feature, execute the following command as root:
          hbase --mlock user=hbase regionserver start
          A new profile, native, has been added to pom.xml Adding '-Pnative' will build libmlockall_agent.so

          mlockall_agent is a simple VM Agent that allows to lock the address space of
          the region server process. This avoids the process' memory eviction under pressure.

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"

          To use this feature, execute the following command as root:
          hbase --mlock user=hbase regionserver start
          Hide
          stack added a comment -

          I added a note under the hbase tools section on this facility.

          Show
          stack added a comment - I added a note under the hbase tools section on this facility.
          stack made changes -
          Release Note A new profile, native, has been added to pom.xml Adding '-Pnative' will build libmlockall_agent.so

          mlockall_agent is a simple VM Agent that allows to lock the address space of
          the region server process. This avoids the process' memory eviction under pressure.

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"

          To use this feature, execute the following command as root:
          hbase --mlock user=hbase regionserver start
          A new profile, native, has been added to pom.xml Adding '-Pnative' will build libmlockall_agent.so HBase will look for the .so under ./lib/native/PLATFORM (See JAVA_LIBRARY_PATH in the bin/hbase starup script).

          mlockall_agent is a simple VM Agent that allows to lock the address space of
          the region server process. This avoids the process' memory eviction under pressure.

          You can load the agent by adding it as a jvm option:
           export HBASE_REGIONSERVER_OPTS="-agentpath:./libmlockall_agent.so=user=hbase"

          To use this feature, execute the following command as root:
          hbase --mlock user=hbase regionserver start
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          Hide
          stack added a comment -

          Marking closed.

          Show
          stack added a comment - Marking closed.
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          stack made changes -
          Tags 0.96notable
          Harsh J made changes -
          Link This issue is superceded by HBASE-6567 [ HBASE-6567 ]

            People

            • Assignee:
              Matteo Bertozzi
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development