Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None

      Description

      To reproduce the error, please run:

      ant clean test -Dhadoopversion=23 -Dtestcase=TestRank1
      

      This fails with the following error:

      Caused by: java.lang.RuntimeException: Error to read counters into Rank operation counterSize 0
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:330)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:370)
          at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
      Caused by: java.lang.NullPointerException
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:359)
      

      I see the failures with hadoop-2.0.x only.

      1. PIG-2985.patch
        4 kB
        Rohini Palaniswamy

        Activity

        Hide
        Gianmarco De Francisci Morales added a comment -

        Did hadoop-2.0 change the way to access counters?

        Show
        Gianmarco De Francisci Morales added a comment - Did hadoop-2.0 change the way to access counters?
        Hide
        Rohini Palaniswamy added a comment -

        PigStatsUtil.java:

        jobClient = job.getJobClient();
        counters = jobClient.getJob(job.getAssignedJobID()).getCounters();
        

        should be

        counters = new Counters(job.getJob().getCounters());
        

        for H23. Each Job and JobClient has its own instance of LocalJobRunner. To access the job information, need to use the same Job/JobClient that the job was submitted with. In H20, job is submitted using JobClient, while in H23 job is submitted using Job.

        Show
        Rohini Palaniswamy added a comment - PigStatsUtil.java: jobClient = job.getJobClient(); counters = jobClient.getJob(job.getAssignedJobID()).getCounters(); should be counters = new Counters(job.getJob().getCounters()); for H23. Each Job and JobClient has its own instance of LocalJobRunner. To access the job information, need to use the same Job/JobClient that the job was submitted with. In H20, job is submitted using JobClient, while in H23 job is submitted using Job.
        Hide
        Gianmarco De Francisci Morales added a comment -

        This is a simple bug fix, should go to both 0.11 and trunk.

        Show
        Gianmarco De Francisci Morales added a comment - This is a simple bug fix, should go to both 0.11 and trunk.
        Hide
        Gianmarco De Francisci Morales added a comment -

        +1, committed to both trunk and 0.11.
        Thanks Rohini!

        Interestingly, tests with hadoop-2.0 take 1/3 of the time compared to hadoop-1.0

        Show
        Gianmarco De Francisci Morales added a comment - +1, committed to both trunk and 0.11. Thanks Rohini! Interestingly, tests with hadoop-2.0 take 1/3 of the time compared to hadoop-1.0

          People

          • Assignee:
            Rohini Palaniswamy
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development