[HIVE-13721] HPL/SQL COPY FROM FTP Statement: lack of DIR option leads to NPE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: hpl/sql
• Assignee: Dmitry Tolpeko
• Reporter: Carter Shanklin
• Created: 09/May/16 19:52
• Updated: 21/Jul/17 17:45
• Resolved: 08/Jun/16 13:59

Description

The docs (http://www.hplsql.org/copy-from-ftp) suggest DIR is optional. When I left it out in:

copy from ftp hdp250.example.com user 'vagrant' pwd 'vagrant'  files 'sampledata.csv' to /tmp overwrite

I got:

Ln:2 Connected to ftp: hdp250.example.com (29 ms)
Ln:2 Retrieving directory listing
  Listing the current working FTP directory
Ln:2 Files to copy: 45 bytes, 1 file, 0 subdirectories scanned (27 ms)
Exception in thread "main" java.lang.NullPointerException
	at org.apache.hive.hplsql.Ftp.getTargetFileName(Ftp.java:342)
	at org.apache.hive.hplsql.Ftp.run(Ftp.java:149)
	at org.apache.hive.hplsql.Ftp.copyFiles(Ftp.java:121)
	at org.apache.hive.hplsql.Ftp.run(Ftp.java:91)
	at org.apache.hive.hplsql.Exec.visitCopy_from_ftp_stmt(Exec.java:1292)
	at org.apache.hive.hplsql.Exec.visitCopy_from_ftp_stmt(Exec.java:52)
	at org.apache.hive.hplsql.HplsqlParser$Copy_from_ftp_stmtContext.accept(HplsqlParser.java:11956)
	at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
	at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
	at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
	at org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
	at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
	at org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
	at org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
	at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
	at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
	at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
	at org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
	at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
	at org.apache.hive.hplsql.Exec.run(Exec.java:760)
	at org.apache.hive.hplsql.Exec.run(Exec.java:736)
	at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Traceback leads to:

  /**
   * Get the target file relative path and name
   */
  String getTargetFileName(String file) {
    int len = dir.length();
    return targetDir + file.substring(len);
  }

in Ftp.java

When I added DIR '/' this worked.