Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8582

HDFS Datanodes fail to start with USE_CDP_HIVE=true on Centos 6

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: Impala 3.3.0
    • Fix Version/s: None
    • Component/s: Infrastructure
    • Labels:

      Description

      On Centos 6, the HDFS Datanode won't start up with this error:

      2019-05-22 22:35:49,852 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      ...
      2019-05-22 22:35:52,497 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
      java.lang.RuntimeException: Cannot start datanode because the configured max locked memory size (dfs.datanode.max.locked.memory) is greater than zero and native code is not available.
      at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1379)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:500)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2782)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2690)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2732)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2876)
      at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2900)
      2019-05-22 22:35:52,506 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: Cannot start datanode because the configured max locked memory size (dfs.datanode.max.locked.memory) is greater than zero and native code is not available.

      There must be something about the CDP version of Hadoop binaries that impacts this. As far as I know, the CDP Hadoop binaries are built on Centos 7. This is likely to be fixed by getting appropriate binaries. Anecdotally, this seems fine on Ubuntu 16.04 and Centos 7. 

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              joemcdonnell Joe McDonnell
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: