Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4033

MiniMRClientClusterFactory is not setting the temp dir correctly in the conf used to init MiniMRYarnCluster

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Invalid
    • Affects Version/s: 0.23.3
    • Fix Version/s: 2.0.0-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      Oozie testcases are failing randomly because MR2 reports the job as unknown.

      This seems to happen when Oozie queries via JobClient.getJob(<JOBID>) for a <JOBID> that just finished.

      org.apache.oozie.action.ActionExecutorException: JA017: Unknown hadoop job [job_1332176678205_0011] associated with action [0000000-120319101023910-oozie-tucu-W@pig-action].  Failing this action!
      

      Oozie reports this error when JobClient.getJob(<JOBID>) returns NULL.

      Looking at the mini cluster logs the job definitely run.

       find . -name "*1332176678205_0011*"
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011/container_1332176678205_0011_01_000002
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011/container_1332176678205_0011_01_000001
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011/container_1332176678205_0011_01_000002
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011/container_1332176678205_0011_01_000001
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011/container_1332176678205_0011_01_000002
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011/container_1332176678205_0011_01_000001
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011/container_1332176678205_0011_01_000002
      ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011/container_1332176678205_0011_01_000001
      

      It seems there is a gap until the the job is avail in the JH server.

      If this gap is unavoidable we need to ensure Oozie always waits at least the gap time before querying for a job.

        Attachments

        1. minicluster-oozie-pig.txt
          604 kB
          Alejandro Abdelnur

          Activity

            People

            • Assignee:
              tucu00 Alejandro Abdelnur
              Reporter:
              tucu00 Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: