Pig
  1. Pig
  2. PIG-3825

Stats collection needs to be changed for hadoop2 (with auto local mode)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      With auto local mode, hadoop 2 getJob, getMapTaskReports, getReduceTaskReports api cause NumberFormatException with stack similar to following-
      (I have also noticed that getMapTaskReports causes memory leak and OOMs for long running jobs as hadoop 2 map task reports are quite bulky, but that is a separate issue)

      Caused by: java.lang.NumberFormatException: For input string: "local154006779"
              at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
              at java.lang.Long.parseLong(Long.java:410)
              at java.lang.Long.parseLong(Long.java:468)
              at org.apache.hadoop.mapreduce.JobID.toClusterTimeStamp(JobID.java:172)
              at org.apache.hadoop.mapreduce.JobID.getAppId(JobID.java:167)
              at org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:79)
              at org.apache.hadoop.mapred.ClientServiceDelegate.<init>(ClientServiceDelegate.java:114)
              at org.apache.hadoop.mapred.ClientCache.getClient(ClientCache.java:68)
              at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:550)
              at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:182)
              at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:586)
              at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:584)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
              at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:584)
              at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:638)
              at org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:632)
              at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:318)
              at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:308)
              at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:240)
              at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:363)
              at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:374)
              ... 16 more
      
      1. PIG-3825.patch
        7 kB
        Aniket Mokashi

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          4m 18s 1 Aniket Mokashi 21/Mar/14 02:11
          Patch Available Patch Available Open Open
          53d 16h 24m 1 Aniket Mokashi 13/May/14 19:36
          Open Open Resolved Resolved
          14d 30m 1 Aniket Mokashi 27/May/14 20:06
          Resolved Resolved Closed Closed
          40d 23h 1m 1 Daniel Dai 07/Jul/14 19:08
          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Aniket Mokashi made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Aniket Mokashi added a comment -

          Fixed in PIG-3913.

          Show
          Aniket Mokashi added a comment - Fixed in PIG-3913 .
          Hide
          Aniket Mokashi added a comment -

          I've tried to solve this with PIG-3913 that build's a shim for hadoop 2 to avoid above exception.

          Show
          Aniket Mokashi added a comment - I've tried to solve this with PIG-3913 that build's a shim for hadoop 2 to avoid above exception.
          Aniket Mokashi made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Aniket Mokashi made changes -
          Link This issue relates to PIG-3913 [ PIG-3913 ]
          Hide
          Cheolsoo Park added a comment -

          Aniket Mokashi, thanks for letting me know the version you're using. It doesn't work with 2.2. I am not storing to s3 but to hdfs. The job succeeds but the output dir only has _temporary with an empty file. Let me investigate it more.

          Show
          Cheolsoo Park added a comment - Aniket Mokashi , thanks for letting me know the version you're using. It doesn't work with 2.2. I am not storing to s3 but to hdfs. The job succeeds but the output dir only has _temporary with an empty file. Let me investigate it more.
          Hide
          Aniket Mokashi added a comment -

          Cheolsoo Park, I didn't like this approach myself a lot. But, it seems there is no non-hacky way of solving this. I will give it another try to make it the way you suggested.

          We are using hadoop-1.0.4 and 2.0.5 with pig_13. Auto local mode works for both the cases well. If jobs are not storing anything (when you are writing to s3 for example), you should check if your local job is getting required configuration or not.

          Show
          Aniket Mokashi added a comment - Cheolsoo Park , I didn't like this approach myself a lot. But, it seems there is no non-hacky way of solving this. I will give it another try to make it the way you suggested. We are using hadoop-1.0.4 and 2.0.5 with pig_13. Auto local mode works for both the cases well. If jobs are not storing anything (when you are writing to s3 for example), you should check if your local job is getting required configuration or not.
          Hide
          Cheolsoo Park added a comment -

          Aniket Mokashi, I am not sure it's a good idea to always ignore exceptions. Can't we explicitly use "local<number>" in the JobID to identify the local mode and handle it differently? I am worried that with your approach, not only NumberFormatException but also any other exceptions are swallowed. That might cover up some real problems.

          Can you also tell me which Hadoop version you're using? I tried EMR 2.2, but auto local mode doesn't seem to work at all. Basically, jobs never store anything. I am going to test it with EMR 2.3 as soon as it's released.

          Show
          Cheolsoo Park added a comment - Aniket Mokashi , I am not sure it's a good idea to always ignore exceptions. Can't we explicitly use "local<number>" in the JobID to identify the local mode and handle it differently? I am worried that with your approach, not only NumberFormatException but also any other exceptions are swallowed. That might cover up some real problems. Can you also tell me which Hadoop version you're using? I tried EMR 2.2, but auto local mode doesn't seem to work at all. Basically, jobs never store anything. I am going to test it with EMR 2.3 as soon as it's released.
          Aniket Mokashi made changes -
          Affects Version/s 0.13.0 [ 12324971 ]
          Aniket Mokashi made changes -
          Fix Version/s 0.13.0 [ 12324971 ]
          Aniket Mokashi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Aniket Mokashi made changes -
          Field Original Value New Value
          Attachment PIG-3825.patch [ 12635939 ]
          Aniket Mokashi created issue -

            People

            • Assignee:
              Aniket Mokashi
              Reporter:
              Aniket Mokashi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development