Avro
  1. Avro
  2. AVRO-878

TestWordCount.testProjection is broken

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.6.0
    • Component/s: java
    • Labels:
      None
    • Tags:
      unittest, mapred, hadoop

      Description

      TestWordCount.testProjection in avro/mapred/TestWordCount.java is broken. It appears to be using the wrong schema to read the output of the map reduce job.

      1. lewi-ipc-reports.tar.gz
        45 kB
        Jeremy Lewi
      2. TEST-org.apache.avro.mapred.TestWordCount.xml
        15 kB
        Jeremy Lewi
      3. TEST-org.apache.avro.mapred.TestWordCount.xml
        15 kB
        Jeremy Lewi
      4. AVRO-878.patch
        5 kB
        Jeremy Lewi

        Issue Links

          Activity

          Hide
          Jeremy Lewi added a comment -

          Here's a patch. I changed the schema and updated the test testProjection

          Show
          Jeremy Lewi added a comment - Here's a patch. I changed the schema and updated the test testProjection
          Hide
          Doug Cutting added a comment -

          The intent of this test is to use a different schema and show that things still work. The key field is absent in the new schema and is ignored in the data read, and the rank field is not present so its default value is used. 'svn blame' shows that this test was added in AVRO-534 to check that a different schema may be used. How is this causing problems for AVRO-570?

          Show
          Doug Cutting added a comment - The intent of this test is to use a different schema and show that things still work. The key field is absent in the new schema and is ignored in the data read, and the rank field is not present so its default value is used. 'svn blame' shows that this test was added in AVRO-534 to check that a different schema may be used. How is this causing problems for AVRO-570 ?
          Hide
          Jeremy Lewi added a comment -

          Thanks for explaining the test; I was clearly mistaken about what it was supposed to do.

          This test is failing on trunk for me. Am I the only one?

          It relates to 570 only in that I was getting test failures in mapred when checking my patch for 570; since I'm getting this test failure on trunk I don't think its due to the code in the patch for AVRO-570 and I wanted to make note of this.

          Show
          Jeremy Lewi added a comment - Thanks for explaining the test; I was clearly mistaken about what it was supposed to do. This test is failing on trunk for me. Am I the only one? It relates to 570 only in that I was getting test failures in mapred when checking my patch for 570; since I'm getting this test failure on trunk I don't think its due to the code in the patch for AVRO-570 and I wanted to make note of this.
          Hide
          Doug Cutting added a comment -

          This test passes for me on Linux. Can you provide more details of how it fails for you?

          Show
          Doug Cutting added a comment - This test passes for me on Linux. Can you provide more details of how it fails for you?
          Hide
          Jeremy Lewi added a comment -

          Here's the surefire output for TestWordCount.

          Perhaps, I'm misusing maven and therefore pulling in the wrong dependencies so here's what I did.

          1. cd AVRO/lang/java
          2. mvn install -D skipTests
          3. cd mapred
          4 mvn test -X

          I ran "mvn test" from the mapred subdirectory because if I tried to run from the "java" directory, I got test failures in the "ipc" package and this caused the tests for subsequent packages to be skipped.

          Show
          Jeremy Lewi added a comment - Here's the surefire output for TestWordCount. Perhaps, I'm misusing maven and therefore pulling in the wrong dependencies so here's what I did. 1. cd AVRO/lang/java 2. mvn install -D skipTests 3. cd mapred 4 mvn test -X I ran "mvn test" from the mapred subdirectory because if I tried to run from the "java" directory, I got test failures in the "ipc" package and this caused the tests for subsequent packages to be skipped.
          Hide
          Jeremy Lewi added a comment -

          Here's the surefire output for TestWordCount.

          Perhaps, I'm misusing maven and therefore pulling in the wrong dependencies so here's what I did.

          1. cd AVRO/lang/java
          2. mvn install -D skipTests
          3. cd mapred
          4 mvn test -X

          I ran "mvn test" from the mapred subdirectory because if I tried to run from the "java" directory, I got test failures in the "ipc" package and this caused the tests for subsequent packages to be skipped.

          Show
          Jeremy Lewi added a comment - Here's the surefire output for TestWordCount. Perhaps, I'm misusing maven and therefore pulling in the wrong dependencies so here's what I did. 1. cd AVRO/lang/java 2. mvn install -D skipTests 3. cd mapred 4 mvn test -X I ran "mvn test" from the mapred subdirectory because if I tried to run from the "java" directory, I got test failures in the "ipc" package and this caused the tests for subsequent packages to be skipped.
          Hide
          Doug Cutting added a comment -

          What tests fail in the IPC package? What OS & Java are you using?

          Show
          Doug Cutting added a comment - What tests fail in the IPC package? What OS & Java are you using?
          Hide
          Jeremy Lewi added a comment -

          Here are the surefire reports for ipc.

          Show
          Jeremy Lewi added a comment - Here are the surefire reports for ipc.
          Hide
          Jeremy Lewi added a comment -

          I'm not that familiar with surefire reports so I just ziped up all the surefire reports for the ipc module and attached them to this ticket.

          I'm running Fedora 15.

          java -version
          java version "1.7.0"
          Java(TM) SE Runtime Environment (build 1.7.0-b147)
          Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)

          javac -version
          javac 1.7.0

          Show
          Jeremy Lewi added a comment - I'm not that familiar with surefire reports so I just ziped up all the surefire reports for the ipc module and attached them to this ticket. I'm running Fedora 15. java -version java version "1.7.0" Java(TM) SE Runtime Environment (build 1.7.0-b147) Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode) javac -version javac 1.7.0
          Hide
          Doug Cutting added a comment -

          On trunk it should work to use 'mvn clean test' from the top-level of the project. I don't recommend installing snapshot versions as it can create a lot of confusion while developing.

          The failures in the logs you sent were in TestProtocolHttp, TestProtocolHttps and TestProtocolNetty. In all cases the underlying exception was an inability of a test client to connect to a test server running on localhost. I don't know what causes this in your case.

          Can you please try this with Java 1.6? Are there any other Fedora uses who can reproduce this?

          Show
          Doug Cutting added a comment - On trunk it should work to use 'mvn clean test' from the top-level of the project. I don't recommend installing snapshot versions as it can create a lot of confusion while developing. The failures in the logs you sent were in TestProtocolHttp, TestProtocolHttps and TestProtocolNetty. In all cases the underlying exception was an inability of a test client to connect to a test server running on localhost. I don't know what causes this in your case. Can you please try this with Java 1.6? Are there any other Fedora uses who can reproduce this?
          Hide
          Doug Cutting added a comment -

          This patch is no longer valid.

          Show
          Doug Cutting added a comment - This patch is no longer valid.
          Hide
          Jeremy Lewi added a comment -

          I tried it with Java 1.6 and all tests passed.

          I also removed the snapshot from my local repository.

          Show
          Jeremy Lewi added a comment - I tried it with Java 1.6 and all tests passed. I also removed the snapshot from my local repository.
          Hide
          Doug Cutting added a comment -

          So should we resolve this as "Not a problem" or should we keep it open as a problem when running under Java 7? Is Java 7 a platform that Avro needs to support at this point?

          Show
          Doug Cutting added a comment - So should we resolve this as "Not a problem" or should we keep it open as a problem when running under Java 7? Is Java 7 a platform that Avro needs to support at this point?
          Hide
          Scott Carey added a comment -

          We should at some point test and be confident Java 7 works. But not until 7 update 2. The loop predication bugs in the original Java 7 release have a chance of hitting some of the optimized loops we have in the binary decoder or encoder. With update 2 due out shortly, there is little reason to test much until then.

          I expect that I will be using Avro in a Java 7 environment at the beginning of 2012.

          I think a ticket specific to Java 7 issues is preferable to morphing this one.

          Show
          Scott Carey added a comment - We should at some point test and be confident Java 7 works. But not until 7 update 2. The loop predication bugs in the original Java 7 release have a chance of hitting some of the optimized loops we have in the binary decoder or encoder. With update 2 due out shortly, there is little reason to test much until then. I expect that I will be using Avro in a Java 7 environment at the beginning of 2012. I think a ticket specific to Java 7 issues is preferable to morphing this one.
          Hide
          Doug Cutting added a comment -

          Closing this as "Not a problem" for now, since we don't yet expect things to work on Java 7. Filed AVRO-926 to address Java 7 issues at some point in the future.

          Show
          Doug Cutting added a comment - Closing this as "Not a problem" for now, since we don't yet expect things to work on Java 7. Filed AVRO-926 to address Java 7 issues at some point in the future.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jeremy Lewi
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development