Avro
  1. Avro
  2. AVRO-1336

Tests TestKeyValueWordCount and TestKeyWordCount fail due to missing files (Trevni mapreduce module)

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.4
    • Fix Version/s: None
    • Component/s: trevni
    • Labels:
      None
    • Environment:

      Red Hat Server 6.4
      Avro 1.7.4
      Maven 3.0.5
      Java 1.6 (IBM or Oracle)
      Hadoop1 = 1.1.2
      Hadoop2 = 2.0.4-alpha

    • Release Note:
      Fix Trevni build with Hadoop 2 in Avro 1.7.4

      Description

      I tried to build Avro 1.7.4 with Hadoop 2 and get the following errors in Trevni mapreduce module tests:

      ==================================================
      ...
      Running org.apache.trevni.avro.mapreduce.TestKeyValueWordCount
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.536 sec <<< FAILURE!
      Running org.apache.trevni.avro.mapreduce.TestKeyWordCount
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.505 sec <<< FAILURE!
      Running org.apache.trevni.avro.TestCases
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.671 sec
      Running org.apache.trevni.avro.TestWordCount
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.651 sec

      Results :

      Tests in error:
      testIOFormat(org.apache.trevni.avro.mapreduce.TestKeyValueWordCount): target/wc/trevniMapReduceKeyValueTest/out/part-r-00000/part-0.trv (No such file or directory)
      testIOFormat(org.apache.trevni.avro.mapreduce.TestKeyWordCount): target/wc/trevniMapReduceKeyTest/out/part-r-00000/part-0.trv (No such file or directory)

      Tests run: 23, Failures: 0, Errors: 2, Skipped: 0

      ...
      ==================================================

      StackTraces:

      -------------------------------------------------------------------------------
      Test set: org.apache.trevni.avro.mapreduce.TestKeyValueWordCount
      -------------------------------------------------------------------------------
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.537 sec <<< FAILURE!
      testIOFormat(org.apache.trevni.avro.mapreduce.TestKeyValueWordCount) Time elapsed: 2.029 sec <<< ERROR!
      java.io.FileNotFoundException: target/wc/trevniMapReduceKeyValueTest/out/part-r-00000/part-0.trv (No such file or directory)
      at java.io.FileInputStream.<init>(FileInputStream.java:124)
      at org.apache.trevni.InputFile.<init>(InputFile.java:33)
      at org.apache.trevni.avro.AvroColumnReader$Params.<init>(AvroColumnReader.java:69)
      at org.apache.trevni.avro.WordCountUtil.validateCountsFileGenericRecord(WordCountUtil.java:122)
      at org.apache.trevni.avro.mapreduce.TestKeyValueWordCount.checkOutputFormat(TestKeyValueWordCount.java:138)
      at org.apache.trevni.avro.mapreduce.TestKeyValueWordCount.testIOFormat(TestKeyValueWordCount.java:108)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
      at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
      at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
      at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
      at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
      ./trevni/avro/target/surefire-reports/org.apache.trevni.avro.mapreduce.TestKeyValueWordCount.txt (END)

      -------------------------------------------------------------------------------
      Test set: org.apache.trevni.avro.mapreduce.TestKeyWordCount
      -------------------------------------------------------------------------------
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.505 sec <<< FAILURE!
      testIOFormat(org.apache.trevni.avro.mapreduce.TestKeyWordCount) Time elapsed: 2.039 sec <<< ERROR!
      java.io.FileNotFoundException: target/wc/trevniMapReduceKeyTest/out/part-r-00000/part-0.trv (No such file or directory)
      at java.io.FileInputStream.<init>(FileInputStream.java:124)
      at org.apache.trevni.InputFile.<init>(InputFile.java:33)
      at org.apache.trevni.avro.AvroColumnReader$Params.<init>(AvroColumnReader.java:69)
      at org.apache.trevni.avro.WordCountUtil.validateCountsFile(WordCountUtil.java:109)
      at org.apache.trevni.avro.mapreduce.TestKeyWordCount.checkOutputFormat(TestKeyWordCount.java:149)
      at org.apache.trevni.avro.mapreduce.TestKeyWordCount.testIOFormat(TestKeyWordCount.java:120)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
      at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
      at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
      at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
      at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
      ./trevni/avro/target/surefire-reports/org.apache.trevni.avro.mapreduce.TestKeyWordCount.txt (END)

        Activity

        Hide
        Doug Cutting added a comment -

        I'm confused. Isn't this a duplicate of AVRO-1322, which was included in Avro 1.7.5?

        Show
        Doug Cutting added a comment - I'm confused. Isn't this a duplicate of AVRO-1322 , which was included in Avro 1.7.5?
        Hide
        Rodrigo Trujillo added a comment -

        Yes, it is an issue in 1.7.5

        Show
        Rodrigo Trujillo added a comment - Yes, it is an issue in 1.7.5
        Hide
        Doug Cutting added a comment -

        Is this still an issue in 1.7.5? If so, perhaps we can fix it in 1.7.6.

        Show
        Doug Cutting added a comment - Is this still an issue in 1.7.5? If so, perhaps we can fix it in 1.7.6.
        Hide
        Rodrigo Trujillo added a comment -

        — lang/java/trevni/avro/pom.xml.orig 2013-06-24 15:54:00.329422972 -0300
        +++ lang/java/trevni/avro/pom.xml 2013-06-24 15:55:08.984489901 -0300
        @@ -50,18 +50,6 @@
        <version>$

        {junit.version}

        </version>
        <scope>test</scope>
        </dependency>

        • <dependency>
        • <groupId>org.apache.avro</groupId>
        • <artifactId>avro-mapred</artifactId>
        • <version>$ {project.version}</version>
          - <scope>compile</scope>
          - </dependency>
          - <dependency>
          - <groupId>org.apache.hadoop</groupId>
          - <artifactId>hadoop-core</artifactId>
          - <version>${hadoop1.version}</version>
          - <scope>compile</scope>
          - </dependency>
          </dependencies>

          <build>
          @@ -70,10 +58,104 @@
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-checkstyle-plugin</artifactId>
          </plugin>
          + <plugin>
          + <groupId>org.apache.maven.plugins</groupId>
          + <artifactId>maven-jar-plugin</artifactId>
          + <executions>
          + <execution>
          + <id>main</id>
          + <goals><goal>jar</goal></goals>
          + <phase>package</phase>
          + </execution>
          + <execution>
          + <id>with-classifier</id>
          + <goals><goal>jar</goal></goals>
          + <phase>package</phase>
          + <configuration>
          + <classifier>${envClassifier}</classifier>
          + </configuration>
          + </execution>
          + </executions>
          + </plugin>
          </plugins>
          </build>

          <profiles>
          + <profile>
          + <id>hadoop1</id>
          + <activation>
          + <property>
          + <name>Unable to render embedded object: File (hadoop.version</name> <) not found.-- if no hadoop.version is set -->
          + </property>
          + </activation>
          + <properties>
          + <envClassifier>hadoop1</envClassifier>
          + </properties>
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-core</artifactId>
          + <version>${hadoop1.version}</version>
          + <!-- hadoop's execution environment provides its own jars, usurping any others.
          + So we should not include it here -->
          + <scope>provided</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.avro</groupId>
          + <artifactId>avro-mapred</artifactId>
          + <version>${project.version}

          </version>
          + <classifier>hadoop1</classifier>
          + <scope>compile</scope>
          + </dependency>
          + </dependencies>
          + </profile>
          + <profile>
          + <id>hadoop2</id>
          + <activation>
          + <property>
          + <name>hadoop.version</name>
          + <value>2</value>
          + </property>
          + </activation>
          + <properties>
          + <envClassifier>hadoop2</envClassifier>
          + </properties>
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-client</artifactId>
          + <version>$

          {hadoop2.version}</version>
          + <!-- hadoop's execution environment provides its own jars, usurping any others.
          + So we should not include it here -->
          + <scope>provided</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-mapreduce-client-common</artifactId>
          + <version>${hadoop2.version}

          </version>
          + <scope>test</scope> <!-- for LocalJobRunner -->
          + </dependency>
          + <dependency>
          + <groupId>commons-httpclient</groupId>
          + <artifactId>commons-httpclient</artifactId>
          + <version>$

          {commons-httpclient.version}

          </version>
          + <scope>test</scope> <!-- for LocalJobRunner -->
          + </dependency>
          + <dependency>
          + <groupId>org.apache.avro</groupId>
          + <artifactId>avro-mapred</artifactId>
          + <version>$

          {project.version}</version>
          + <classifier>hadoop2</classifier>
          + <scope>compile</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.avro</groupId>
          + <artifactId>avro</artifactId>
          + <version>${project.version}

          </version>
          + <scope>compile</scope>
          + </dependency>
          + </dependencies>
          + </profile>
          </profiles>

        </project>

        Show
        Rodrigo Trujillo added a comment - — lang/java/trevni/avro/pom.xml.orig 2013-06-24 15:54:00.329422972 -0300 +++ lang/java/trevni/avro/pom.xml 2013-06-24 15:55:08.984489901 -0300 @@ -50,18 +50,6 @@ <version>$ {junit.version} </version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro-mapred</artifactId> <version>$ {project.version}</version> - <scope>compile</scope> - </dependency> - <dependency> - <groupId>org.apache.hadoop</groupId> - <artifactId>hadoop-core</artifactId> - <version>${hadoop1.version}</version> - <scope>compile</scope> - </dependency> </dependencies> <build> @@ -70,10 +58,104 @@ <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-checkstyle-plugin</artifactId> </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-jar-plugin</artifactId> + <executions> + <execution> + <id>main</id> + <goals><goal>jar</goal></goals> + <phase>package</phase> + </execution> + <execution> + <id>with-classifier</id> + <goals><goal>jar</goal></goals> + <phase>package</phase> + <configuration> + <classifier>${envClassifier}</classifier> + </configuration> + </execution> + </executions> + </plugin> </plugins> </build> <profiles> + <profile> + <id>hadoop1</id> + <activation> + <property> + <name> Unable to render embedded object: File (hadoop.version</name> <) not found. -- if no hadoop.version is set --> + </property> + </activation> + <properties> + <envClassifier>hadoop1</envClassifier> + </properties> + <dependencies> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-core</artifactId> + <version>${hadoop1.version}</version> + <!-- hadoop's execution environment provides its own jars, usurping any others. + So we should not include it here --> + <scope>provided</scope> + </dependency> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro-mapred</artifactId> + <version>${project.version} </version> + <classifier>hadoop1</classifier> + <scope>compile</scope> + </dependency> + </dependencies> + </profile> + <profile> + <id>hadoop2</id> + <activation> + <property> + <name>hadoop.version</name> + <value>2</value> + </property> + </activation> + <properties> + <envClassifier>hadoop2</envClassifier> + </properties> + <dependencies> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-client</artifactId> + <version>$ {hadoop2.version}</version> + <!-- hadoop's execution environment provides its own jars, usurping any others. + So we should not include it here --> + <scope>provided</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-common</artifactId> + <version>${hadoop2.version} </version> + <scope>test</scope> <!-- for LocalJobRunner --> + </dependency> + <dependency> + <groupId>commons-httpclient</groupId> + <artifactId>commons-httpclient</artifactId> + <version>$ {commons-httpclient.version} </version> + <scope>test</scope> <!-- for LocalJobRunner --> + </dependency> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro-mapred</artifactId> + <version>$ {project.version}</version> + <classifier>hadoop2</classifier> + <scope>compile</scope> + </dependency> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> + <version>${project.version} </version> + <scope>compile</scope> + </dependency> + </dependencies> + </profile> </profiles> </project>
        Hide
        Rodrigo Trujillo added a comment -

        This patch backports lang/java/trevni/avro/pom.xml 1.7.5-snapshot version

        Show
        Rodrigo Trujillo added a comment - This patch backports lang/java/trevni/avro/pom.xml 1.7.5-snapshot version
        Hide
        Rodrigo Trujillo added a comment -

        The error is caused by:
        lang/java/trevni/avro/pom.xml

        It fixes Hadoop in version 1

        Using pom.xml from trunk fixes the problem

        Show
        Rodrigo Trujillo added a comment - The error is caused by: lang/java/trevni/avro/pom.xml It fixes Hadoop in version 1 Using pom.xml from trunk fixes the problem

          People

          • Assignee:
            Rodrigo Trujillo
            Reporter:
            Rodrigo Trujillo
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development