Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2943

Test-failures when using local-job-runner.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: distcp, mrv2
    • Labels:
      None

      Description

      Here's an example of a failure when a test attempts to run using the local-job-runner:

      [ERROR] TestIntegration - Exception encountered <java.lang.NullPointerException>java.lang.NullPointerException
      at org.apache.hadoop.mapreduce.Cluster.getStagingAreaDir(Cluster.java:263)
      at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:95)
      at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:337)
      at org.apache.hadoop.tools.DistCp.<init>(DistCp.java:77)
      at org.apache.hadoop.tools.TestIntegration.runTest(TestIntegration.java:449)
      at org.apache.hadoop.tools.TestIntegration.caseSingleFileMissingTarget(TestIntegration.java:79)
      at org.apache.hadoop.tools.TestIntegration.testSingleFileMissingTarget(TestIntegration.java:69)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
      ...

      This is a DistCp(v2) test, built and run using Maven. DistCp's tests depend on the following maven artifacts:
      1. hadoop-mapreduce-client-jobclient
      2. hadoop-hdfs
      3. hadoop-common

      The failure is in spite of doing a " conf.set(MRConfig.FRAMEWORK_NAME, "local");". Vinod (KV)'s in the loop on this. It appears that the root-cause lies in org/apache/hadoop/mapred/LocalClientProtocolProvider.class not being available in class-path. This might have to do with MR1 not having been maven-ized, or some such.

      I haven't a clean way around this for the moment, so I'll temporarily disable this distcp-test from running. It would be nice, however, to have maven-tests that use local-job-runner to run through.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mithun Mithun Radhakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: