Hadoop Common
  1. Hadoop Common
  2. HADOOP-7646

Make hadoop-common use same version of avro as HBase

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: io, ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
      When building HBase on top of hadoop, this should be consistent.

      1. HADOOP-7646-branch-22-shv.patch
        10 kB
        Konstantin Shvachko
      2. HADOOP-7646-branch-22-shv.patch
        10 kB
        Konstantin Shvachko
      3. HADOOP-7646-branch-22.patch
        4 kB
        Joep Rottinghuis
      4. HADOOP-7646-branch-22.patch
        10 kB
        Joep Rottinghuis
      5. HADOOP-7646-branch-22.patch
        10 kB
        Joep Rottinghuis
      6. HADOOP-7646-branch-22.patch
        10 kB
        Joep Rottinghuis

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          I just committed this. Thank you Joep.

          Show
          Konstantin Shvachko added a comment - I just committed this. Thank you Joep.
          Hide
          Konstantin Shvachko added a comment -

          Removed white space change.

          Show
          Konstantin Shvachko added a comment - Removed white space change.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #86 (See https://builds.apache.org/job/Hadoop-Common-22-branch/86/)
          HADOOP-7646. Upgrade to Avro 1.5.3. Contributed by Joep Rottinghuis.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176688
          Files :

          • /hadoop/common/branches/branch-0.22/common/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/common/build.xml
          • /hadoop/common/branches/branch-0.22/common/ivy.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-instrumented-template.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-template.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/libraries.properties
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroRpcEngine.java
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroSpecificRpcEngine.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/io/AvroTestUtil.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/AvroTestProtocol.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/TestAvroRpc.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #86 (See https://builds.apache.org/job/Hadoop-Common-22-branch/86/ ) HADOOP-7646 . Upgrade to Avro 1.5.3. Contributed by Joep Rottinghuis. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176688 Files : /hadoop/common/branches/branch-0.22/common/CHANGES.txt /hadoop/common/branches/branch-0.22/common/build.xml /hadoop/common/branches/branch-0.22/common/ivy.xml /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-instrumented-template.xml /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-template.xml /hadoop/common/branches/branch-0.22/common/ivy/libraries.properties /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroRpcEngine.java /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroSpecificRpcEngine.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/io/AvroTestUtil.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/AvroTestProtocol.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/TestAvroRpc.java
          Hide
          Konstantin Shvachko added a comment -

          Here is a modification. The point is that BinaryEncoder needs to flush before written data will be available for the reader. Looks like a difference in semantics with the old avro.

          Show
          Konstantin Shvachko added a comment - Here is a modification. The point is that BinaryEncoder needs to flush before written data will be available for the reader. Looks like a difference in semantics with the old avro.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #85 (See https://builds.apache.org/job/Hadoop-Common-22-branch/85/)
          HADOOP-7646.Revert due to test failures.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176616
          Files :

          • /hadoop/common/branches/branch-0.22/common/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/common/build.xml
          • /hadoop/common/branches/branch-0.22/common/ivy.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-instrumented-template.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-template.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/libraries.properties
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroRpcEngine.java
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroSpecificRpcEngine.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/io/AvroTestUtil.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/AvroTestProtocol.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/TestAvroRpc.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #85 (See https://builds.apache.org/job/Hadoop-Common-22-branch/85/ ) HADOOP-7646 .Revert due to test failures. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176616 Files : /hadoop/common/branches/branch-0.22/common/CHANGES.txt /hadoop/common/branches/branch-0.22/common/build.xml /hadoop/common/branches/branch-0.22/common/ivy.xml /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-instrumented-template.xml /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-template.xml /hadoop/common/branches/branch-0.22/common/ivy/libraries.properties /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroRpcEngine.java /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroSpecificRpcEngine.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/io/AvroTestUtil.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/AvroTestProtocol.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/TestAvroRpc.java
          Hide
          Konstantin Shvachko added a comment -

          I think I found out what is going on with the tests. Reverting the patch.

          Show
          Konstantin Shvachko added a comment - I think I found out what is going on with the tests. Reverting the patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #84 (See https://builds.apache.org/job/Hadoop-Common-22-branch/84/)
          HADOOP-7646. Upgrade to Avro 1.5.3. Contributed by Joep Rottinghuis.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176485
          Files :

          • /hadoop/common/branches/branch-0.22/common/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/common/build.xml
          • /hadoop/common/branches/branch-0.22/common/ivy.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-instrumented-template.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-template.xml
          • /hadoop/common/branches/branch-0.22/common/ivy/libraries.properties
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroRpcEngine.java
          • /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroSpecificRpcEngine.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/io/AvroTestUtil.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/AvroTestProtocol.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/TestAvroRpc.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #84 (See https://builds.apache.org/job/Hadoop-Common-22-branch/84/ ) HADOOP-7646 . Upgrade to Avro 1.5.3. Contributed by Joep Rottinghuis. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176485 Files : /hadoop/common/branches/branch-0.22/common/CHANGES.txt /hadoop/common/branches/branch-0.22/common/build.xml /hadoop/common/branches/branch-0.22/common/ivy.xml /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-instrumented-template.xml /hadoop/common/branches/branch-0.22/common/ivy/hadoop-common-template.xml /hadoop/common/branches/branch-0.22/common/ivy/libraries.properties /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/io/serializer/avro/AvroSerialization.java /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroRpcEngine.java /hadoop/common/branches/branch-0.22/common/src/java/org/apache/hadoop/ipc/AvroSpecificRpcEngine.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/io/AvroTestUtil.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/AvroTestProtocol.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/ipc/TestAvroRpc.java
          Hide
          Konstantin Shvachko added a comment -

          The last patch passes the tests.

          Show
          Konstantin Shvachko added a comment - The last patch passes the tests.
          Hide
          Konstantin Shvachko added a comment -

          3 tests are failing with this patch:
          [junit] Test org.apache.hadoop.fs.TestPath FAILED
          [junit] Test org.apache.hadoop.io.TestEnumSetWritable FAILED
          [junit] Test org.apache.hadoop.io.TestText FAILED
          Will commit as long as those are fixed.

          Show
          Konstantin Shvachko added a comment - 3 tests are failing with this patch: [junit] Test org.apache.hadoop.fs.TestPath FAILED [junit] Test org.apache.hadoop.io.TestEnumSetWritable FAILED [junit] Test org.apache.hadoop.io.TestText FAILED Will commit as long as those are fixed.
          Hide
          Konstantin Shvachko added a comment -

          +1

          Show
          Konstantin Shvachko added a comment - +1
          Hide
          Joep Rottinghuis added a comment -

          A patch is available for each of the sub-projects: common, hdfs, and mapreduce. The latter one needs review as I think more work is required there.

          Show
          Joep Rottinghuis added a comment - A patch is available for each of the sub-projects: common, hdfs, and mapreduce. The latter one needs review as I think more work is required there.
          Hide
          Joep Rottinghuis added a comment -

          Ivy seems to cache some of the dependencies. I've found that cleaning out the local cache (~/.ivy2 and ~/.m2/cache if needed) can help with this.
          After that I build common first with target mvn:install, then hdfs, then mapreduce.

          Show
          Joep Rottinghuis added a comment - Ivy seems to cache some of the dependencies. I've found that cleaning out the local cache (~/.ivy2 and ~/.m2/cache if needed) can help with this. After that I build common first with target mvn:install, then hdfs, then mapreduce.
          Hide
          Joep Rottinghuis added a comment -

          Let me file a Jira for hdfs and mapreduce (the latter seems to be the most work).
          Then I'll run all three through our Jenkins builds together and do a quick test on it before we commit them together.

          Show
          Joep Rottinghuis added a comment - Let me file a Jira for hdfs and mapreduce (the latter seems to be the most work). Then I'll run all three through our Jenkins builds together and do a quick test on it before we commit them together.
          Hide
          Joep Rottinghuis added a comment -

          I had used the fully qualified java.lang.CharSequence to match the generated AvroSpecificTestProtocol, but adjusted according to Doug's suggestion.

          Show
          Joep Rottinghuis added a comment - I had used the fully qualified java.lang.CharSequence to match the generated AvroSpecificTestProtocol, but adjusted according to Doug's suggestion.
          Hide
          Joep Rottinghuis added a comment -

          Eli: yes makes sense to have all branches on the same version.
          Doug: good point, will make suggested change.

          Show
          Joep Rottinghuis added a comment - Eli: yes makes sense to have all branches on the same version. Doug: good point, will make suggested change.
          Hide
          Doug Cutting added a comment -

          Patch looks good to me.

          One minor improvement would to be replace java.lang.CharSequence with just CharSequence.

          Show
          Doug Cutting added a comment - Patch looks good to me. One minor improvement would to be replace java.lang.CharSequence with just CharSequence.
          Hide
          Eli Collins added a comment -

          Btw trunk is currently on 1.5.3 in some places (HADOOP-7264, see /hadoop-project/pom.xml) and 1.5.2 in others (/hadoop-mapreduce-project/ivy/libraries.properties). How about we bring all branches/projects up to 1.5.3?

          Show
          Eli Collins added a comment - Btw trunk is currently on 1.5.3 in some places ( HADOOP-7264 , see /hadoop-project/pom.xml) and 1.5.2 in others (/hadoop-mapreduce-project/ivy/libraries.properties). How about we bring all branches/projects up to 1.5.3?
          Hide
          Joep Rottinghuis added a comment -

          Resolved compilation issues. Further testing to follow...

          Show
          Joep Rottinghuis added a comment - Resolved compilation issues. Further testing to follow...
          Hide
          Joep Rottinghuis added a comment -

          Resolved all compilation issues. Further testing will follow.

          Show
          Joep Rottinghuis added a comment - Resolved all compilation issues. Further testing will follow.
          Hide
          Joep Rottinghuis added a comment -

          Still get compilation error, but it's not immediately obvious why. Should probably call it quits for today and look at this again tomorrow.

          Show
          Joep Rottinghuis added a comment - Still get compilation error, but it's not immediately obvious why. Should probably call it quits for today and look at this again tomorrow.

            People

            • Assignee:
              Joep Rottinghuis
              Reporter:
              Joep Rottinghuis
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development