Pig
  1. Pig
  2. PIG-3825

Stats collection needs to be changed for hadoop2 (with auto local mode)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      With auto local mode, hadoop 2 getJob, getMapTaskReports, getReduceTaskReports api cause NumberFormatException with stack similar to following-
      (I have also noticed that getMapTaskReports causes memory leak and OOMs for long running jobs as hadoop 2 map task reports are quite bulky, but that is a separate issue)

      Caused by: java.lang.NumberFormatException: For input string: "local154006779"
              at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
              at java.lang.Long.parseLong(Long.java:410)
              at java.lang.Long.parseLong(Long.java:468)
              at org.apache.hadoop.mapreduce.JobID.toClusterTimeStamp(JobID.java:172)
              at org.apache.hadoop.mapreduce.JobID.getAppId(JobID.java:167)
              at org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:79)
              at org.apache.hadoop.mapred.ClientServiceDelegate.<init>(ClientServiceDelegate.java:114)
              at org.apache.hadoop.mapred.ClientCache.getClient(ClientCache.java:68)
              at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:550)
              at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:182)
              at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:586)
              at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:584)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
              at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:584)
              at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:638)
              at org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:632)
              at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:318)
              at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:308)
              at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:240)
              at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:363)
              at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:374)
              ... 16 more
      
      1. PIG-3825.patch
        7 kB
        Aniket Mokashi

        Issue Links

          Activity

          Hide
          Cheolsoo Park added a comment -

          Aniket Mokashi, I am not sure it's a good idea to always ignore exceptions. Can't we explicitly use "local<number>" in the JobID to identify the local mode and handle it differently? I am worried that with your approach, not only NumberFormatException but also any other exceptions are swallowed. That might cover up some real problems.

          Can you also tell me which Hadoop version you're using? I tried EMR 2.2, but auto local mode doesn't seem to work at all. Basically, jobs never store anything. I am going to test it with EMR 2.3 as soon as it's released.

          Show
          Cheolsoo Park added a comment - Aniket Mokashi , I am not sure it's a good idea to always ignore exceptions. Can't we explicitly use "local<number>" in the JobID to identify the local mode and handle it differently? I am worried that with your approach, not only NumberFormatException but also any other exceptions are swallowed. That might cover up some real problems. Can you also tell me which Hadoop version you're using? I tried EMR 2.2, but auto local mode doesn't seem to work at all. Basically, jobs never store anything. I am going to test it with EMR 2.3 as soon as it's released.
          Hide
          Aniket Mokashi added a comment -

          Cheolsoo Park, I didn't like this approach myself a lot. But, it seems there is no non-hacky way of solving this. I will give it another try to make it the way you suggested.

          We are using hadoop-1.0.4 and 2.0.5 with pig_13. Auto local mode works for both the cases well. If jobs are not storing anything (when you are writing to s3 for example), you should check if your local job is getting required configuration or not.

          Show
          Aniket Mokashi added a comment - Cheolsoo Park , I didn't like this approach myself a lot. But, it seems there is no non-hacky way of solving this. I will give it another try to make it the way you suggested. We are using hadoop-1.0.4 and 2.0.5 with pig_13. Auto local mode works for both the cases well. If jobs are not storing anything (when you are writing to s3 for example), you should check if your local job is getting required configuration or not.
          Hide
          Cheolsoo Park added a comment -

          Aniket Mokashi, thanks for letting me know the version you're using. It doesn't work with 2.2. I am not storing to s3 but to hdfs. The job succeeds but the output dir only has _temporary with an empty file. Let me investigate it more.

          Show
          Cheolsoo Park added a comment - Aniket Mokashi , thanks for letting me know the version you're using. It doesn't work with 2.2. I am not storing to s3 but to hdfs. The job succeeds but the output dir only has _temporary with an empty file. Let me investigate it more.
          Hide
          Aniket Mokashi added a comment -

          I've tried to solve this with PIG-3913 that build's a shim for hadoop 2 to avoid above exception.

          Show
          Aniket Mokashi added a comment - I've tried to solve this with PIG-3913 that build's a shim for hadoop 2 to avoid above exception.
          Hide
          Aniket Mokashi added a comment -

          Fixed in PIG-3913.

          Show
          Aniket Mokashi added a comment - Fixed in PIG-3913 .

            People

            • Assignee:
              Aniket Mokashi
              Reporter:
              Aniket Mokashi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development