[HIVE-6546] WebHCat job submission for pig with -useHCatalog argument fails on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0, 0.12.0, 0.13.0, 0.14.0
• Fix Version/s: 0.13.0
• Component/s: WebHCat

Environment

HDInsight deploying HDP 1.3: c:\apps\dist\pig-0.11.0.1.3.2.0-05
Also on Windows HDP 1.3 one-box configuration.

• Assignee: Eric N. Hanson
• Reporter: Eric N. Hanson
• Created: 04/Mar/14 20:05
• Updated: 27/Mar/14 20:49
• Resolved: 27/Mar/14 19:13

Description

On a one-box windows setup, do the following from a powershell prompt:

cmd /c curl.exe -s `
-d user.name=hadoop `
-d arg=-useHCatalog `
-d execute="emp = load '/data/emp/emp_0.dat'; dump emp;" `
-d statusdir="/tmp/webhcat.output01" `
'http://localhost:50111/templeton/v1/pig' -v

The job fails with error code 7, but it should run.

I traced this down to the following. In the job configuration for the TempletonJobController, we have templeton.args set to

cmd,/c,call,C:\\hadoop\\\\pig-0.11.0.1.3.0.0-0846/bin/pig.cmd,-D_WEBHCAT_TOKEN_FILE_LOCATION_="-useHCatalog",-execute,"emp = load '/data/emp/emp_0.dat'; dump emp;"

Notice the = sign before "-useHCatalog". I think this should be a comma.

The bad string D_WEBHCAT_TOKEN_FILE_LOCATION_="-useHCatalog" gets created in org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows().

It happens at line 434:

      } else {
          if (i < args.length - 1) {
            prop += "=" + args[++i];   // RIGHT HERE! at iterations i = 37, 38
          }
        }

Bug is here:

      if (prop != null) {
        if (prop.contains("=")) {  // -D__WEBHCAT_TOKEN_FILE_LOCATION__ does not contain equal, so else branch is run and appends ="-useHCatalog",
          // everything good
        } else {
          if (i < args.length - 1) {
            prop += "=" + args[++i];
          }
        }
        newArgs.add(prop);
      }

One possible fix is to change the string constant org.apache.hcatalog.templeton.tool.TempletonControllerJob.TOKEN_FILE_ARG_PLACEHOLDER to have an "=" sign in it. Or, preProcessForWindows() itself could be changed.