[PIG-3904] Pig support windows i18n

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.2, 0.9.3
• Fix Version/s: 0.9.3
• Component/s: impl

Environment

Windows 7(de_DE/fr_FR/zh_CN)

• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Lizhao.Du
• Created: 18/Apr/14 10:41
• Updated: 25/Apr/14 00:09

Description

Utilize Pig run a pig script in Windows (de_DE), it is failed.
The error message showed "Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput", but /tmp/测试/pwInput is exist
in fact. Because of encoding of hadoop adoption is UTF-8. When encoding of client OS pig situated is different with it, hadoop will unrecognize these characters.

Log message as below:
==============================================
ERROR Spring Shell org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Backend error message during job submission
org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:282)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:270)
... 14 more

I have added a patch, PIG-3904.patch to fix it. It works.