Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Avro 1.3.0 has now been released. HADOOP-6486 and HDFS-892 require it, and the version of Avro used by MapReduce should be synchronized with these projects.

      1. MAPREDUCE-1556.patch
        5 kB
        Doug Cutting
      2. MAPREDUCE-1556.patch
        4 kB
        Doug Cutting
      3. MAPREDUCE-1556.patch
        3 kB
        Doug Cutting

        Issue Links

          Activity

          Hide
          Doug Cutting added a comment -

          Here's a patch. It:

          • updates Avro in library.propertiesAvro to 1.3.0, and removes Avro-only dependencies, since these are now pulled in through Avro's POM.
          • modifies ivysettings.xml to permit development with local versions of modules from organizations besides org.apache.hadoop.
          • updates jobhistory code for changes in Avro APIs
          Show
          Doug Cutting added a comment - Here's a patch. It: updates Avro in library.propertiesAvro to 1.3.0, and removes Avro-only dependencies, since these are now pulled in through Avro's POM. modifies ivysettings.xml to permit development with local versions of modules from organizations besides org.apache.hadoop. updates jobhistory code for changes in Avro APIs
          Hide
          Giridharan Kesavan added a comment -

          patch test on hudson is stuck for 17 hrs, I 've to kill this patch test job.
          https://issues.apache.org/jira/browse/MAPREDUCE-1561

          Show
          Giridharan Kesavan added a comment - patch test on hudson is stuck for 17 hrs, I 've to kill this patch test job. https://issues.apache.org/jira/browse/MAPREDUCE-1561
          Hide
          Doug Cutting added a comment -

          Try again now that HADOOP-6486 has been committed.

          Show
          Doug Cutting added a comment - Try again now that HADOOP-6486 has been committed.
          Hide
          Doug Cutting added a comment -

          Try cleaning Ivy's cache of old Avro stuff...

          Show
          Doug Cutting added a comment - Try cleaning Ivy's cache of old Avro stuff...
          Hide
          Doug Cutting added a comment -

          Update eclipse classpath and pom template to Avro 1.3.0 too.

          Show
          Doug Cutting added a comment - Update eclipse classpath and pom template to Avro 1.3.0 too.
          Hide
          Tom White added a comment -

          The MapReduce build is currently broken - this change fixes it. Hudson seems to be stuck. I ran TestJobHistory locally which exercises the Avro dependencies, and it passes, so I suggest we commit this now.

          Show
          Tom White added a comment - The MapReduce build is currently broken - this change fixes it. Hudson seems to be stuck. I ran TestJobHistory locally which exercises the Avro dependencies, and it passes, so I suggest we commit this now.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Doug!

          Show
          Tom White added a comment - I've just committed this. Thanks Doug!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #274 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/274/)
          . upgrade to Avro 1.3.0. Contributed by cutting.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #274 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/274/ ) . upgrade to Avro 1.3.0. Contributed by cutting.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #258 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/258/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #258 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/258/ )
          Hide
          Chris Douglas added a comment -

          With HADOOP-6486, the following warnings show up on trunk using a binary tarball:

          SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
          SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
          2010-03-18 01:11:32.988::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
          2010-03-18 01:11:33.056::INFO:  jetty-6.1.14
          

          The slf4j-simple jar is no longer included.

          Show
          Chris Douglas added a comment - With HADOOP-6486 , the following warnings show up on trunk using a binary tarball: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 2010-03-18 01:11:32.988::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2010-03-18 01:11:33.056::INFO: jetty-6.1.14 The slf4j-simple jar is no longer included.
          Hide
          Doug Cutting added a comment -

          > The slf4j-simple jar is no longer included.

          It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts?

          That error message is triggered when no slf4j implementation is on the classpath. When is this printed?

          Show
          Doug Cutting added a comment - > The slf4j-simple jar is no longer included. It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts? That error message is triggered when no slf4j implementation is on the classpath. When is this printed?
          Hide
          Chris Douglas added a comment -

          It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts?

          I think Avro marking it as a dependency masked MR/HDFS/Common's dependence on it, which isn't explicit? It's printed whenever I try to start a MR *Tracker from a binary tarball.

          Show
          Chris Douglas added a comment - It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts? I think Avro marking it as a dependency masked MR/HDFS/Common's dependence on it, which isn't explicit? It's printed whenever I try to start a MR *Tracker from a binary tarball.
          Hide
          Chris Douglas added a comment -

          Should it be in common->master instead of test->master ?

          Show
          Chris Douglas added a comment - Should it be in common->master instead of test->master ?
          Hide
          Chris Douglas added a comment -

          Yea, that worked. Filed MAPREDUCE-1622

          Show
          Chris Douglas added a comment - Yea, that worked. Filed MAPREDUCE-1622

            People

            • Assignee:
              Doug Cutting
              Reporter:
              Doug Cutting
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development