Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18177

Error creating temporary staging folder on HDFS when creating Hive table

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.3.1
    • Fix Version/s: None
    • Component/s: Parser
    • Labels:
      None
    • Release Note:
      Configuration problem. The aux lib dir under $HIVE_HOME lacks packages.

      Description

      When creating a table in hive using statement:

      create database if not exists ${DB};
      use ${DB};
      
      drop table if exists date_dim;
      
      create table date_dim
      stored as ${FILE}
      as select * from ${SOURCE}.date_dim;
      

      The statement execution failed as:

      FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db. Error encountered near token 'TOK_TMP_FILE'
      FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db. Error encountered near token 'TOK_TMP_FILE'
      

      We got this exception stackļ¼š

      2017-11-29T17:32:47,646  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner: Completed phase 1 of Semantic Analysis
      2017-11-29T17:32:47,646  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner: Get metadata for source tables
      2017-11-29T17:32:47,646  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] metastore.HiveMetaStore: 0: get_table : db=tpcds_text_2 tbl=date_dim
      2017-11-29T17:32:47,647  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] HiveMetaStore.audit: ugi=hadoop	ip=unknown-ip-addr	cmd=get_table : db=tpcds_text_2 tbl=date_dim
      2017-11-29T17:32:47,748  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner: Get metadata for subqueries
      2017-11-29T17:32:47,748  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner: Get metadata for destination tables
      2017-11-29T17:32:47,748  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] metastore.HiveMetaStore: 0: get_database: tpcds_bin_partitioned_orc_2
      2017-11-29T17:32:47,748  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] HiveMetaStore.audit: ugi=hadoop	ip=unknown-ip-addr	cmd=get_database: tpcds_bin_partitioned_orc_2
      2017-11-29T17:32:48,308  INFO [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] common.FileUtils: Creating directory if it doesn't exist: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db/.hive-staging_hive_2017-11-29_17-32-47_541_2322222506518783479-1
      2017-11-29T17:32:48,330 ERROR [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] parse.CalcitePlanner: org.apache.hadoop.hive.ql.parse.SemanticException: 0:0 Error creating temporary folder on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db. Error encountered near token 'TOK_TMP_FILE'
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2211)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1934)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11080)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11133)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
      	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
      	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
      	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
      	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
      Caused by: java.lang.RuntimeException: Cannot create staging directory 'hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db/.hive-staging_hive_2017-11-29_17-32-47_541_2322222506518783479-1': Invalid host name: local host is: (unknown); destination host is: "ns-offline":8020; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
      	at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:374)
      	at org.apache.hadoop.hive.ql.Context.getExtTmpPathRelTo(Context.java:632)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2208)
      	... 25 more
      Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "ns-offline":8020; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
      	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:744)
      	at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:445)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1373)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1337)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
      	at com.sun.proxy.$Proxy36.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:787)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
      	at com.sun.proxy.$Proxy37.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1700)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1436)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1433)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1433)
      	at org.apache.hadoop.hive.common.FileUtils.mkdir(FileUtils.java:528)
      	at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:366)
      	... 27 more
      Caused by: java.net.UnknownHostException
      	... 52 more
      
      2017-11-29T17:32:48,331 ERROR [4d9462cf-43b0-4fea-b0c2-c1a9969d9763 main] ql.Driver: FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db. Error encountered near token 'TOK_TMP_FILE'
      org.apache.hadoop.hive.ql.parse.SemanticException: 0:0 Error creating temporary folder on: hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db. Error encountered near token 'TOK_TMP_FILE'
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2211)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1934)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11080)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11133)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
      	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
      	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
      	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
      	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
      Caused by: java.lang.RuntimeException: Cannot create staging directory 'hdfs://ns-offline/user/hive2/warehouse/tpcds_bin_partitioned_orc_2.db/.hive-staging_hive_2017-11-29_17-32-47_541_2322222506518783479-1': Invalid host name: local host is: (unknown); destination host is: "ns-offline":8020; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
      	at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:374)
      	at org.apache.hadoop.hive.ql.Context.getExtTmpPathRelTo(Context.java:632)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2208)
      	... 25 more
      Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "ns-offline":8020; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
      	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:744)
      	at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:445)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1373)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1337)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
      	at com.sun.proxy.$Proxy36.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:787)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
      	at com.sun.proxy.$Proxy37.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1700)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1436)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1433)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1433)
      	at org.apache.hadoop.hive.common.FileUtils.mkdir(FileUtils.java:528)
      	at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:366)
      	... 27 more
      Caused by: java.net.UnknownHostException
      	... 52 more
      

      In HADOOP core-site.xml:

      <property>
        <name>fs.default.name</name>
        <value>hdfs://ns-offline</value>
      </property>
      

      The Exception is thrown at Hive QL:
      https://github.com/apache/hive/blob/07fe7e210cb444aec43cb5adda37f8f7cd26f243/ql/src/java/org/apache/hadoop/hive/ql/Context.java#L396
      And called HDFS mkdir operation at:
      https://github.com/apache/hive/blob/07fe7e210cb444aec43cb5adda37f8f7cd26f243/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L579
      The call above took a conf parameter but not used.

      And also, in PathInfo class where we got the FileSystem instanceļ¼š
      https://github.com/apache/hive/blob/32e854ef1c25f21d53f7932723cfc76bf75a71cd/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/PathInfo.java#L69

      FileSystem fileSystem = inputPath.getFileSystem(hiveConf);
      

      We passed a hive configuration instead of HDFS config.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Aegeaner Aegeaner
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: