Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Avro 1.3.0 has now been released. HADOOP-6486 and HDFS-892 require it, and the version of Avro used by MapReduce should be synchronized with these projects.

      1. MAPREDUCE-1556.patch
        5 kB
        Doug Cutting
      2. MAPREDUCE-1556.patch
        4 kB
        Doug Cutting
      3. MAPREDUCE-1556.patch
        3 kB
        Doug Cutting

        Issue Links

          Activity

          Doug Cutting created issue -
          Doug Cutting made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-6486 [ HADOOP-6486 ]
          Hide
          Doug Cutting added a comment -

          Here's a patch. It:

          • updates Avro in library.propertiesAvro to 1.3.0, and removes Avro-only dependencies, since these are now pulled in through Avro's POM.
          • modifies ivysettings.xml to permit development with local versions of modules from organizations besides org.apache.hadoop.
          • updates jobhistory code for changes in Avro APIs
          Show
          Doug Cutting added a comment - Here's a patch. It: updates Avro in library.propertiesAvro to 1.3.0, and removes Avro-only dependencies, since these are now pulled in through Avro's POM. modifies ivysettings.xml to permit development with local versions of modules from organizations besides org.apache.hadoop. updates jobhistory code for changes in Avro APIs
          Doug Cutting made changes -
          Attachment MAPREDUCE-1556.patch [ 12437781 ]
          Doug Cutting made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Giridharan Kesavan added a comment -

          patch test on hudson is stuck for 17 hrs, I 've to kill this patch test job.
          https://issues.apache.org/jira/browse/MAPREDUCE-1561

          Show
          Giridharan Kesavan added a comment - patch test on hudson is stuck for 17 hrs, I 've to kill this patch test job. https://issues.apache.org/jira/browse/MAPREDUCE-1561
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Doug Cutting added a comment -

          Try again now that HADOOP-6486 has been committed.

          Show
          Doug Cutting added a comment - Try again now that HADOOP-6486 has been committed.
          Doug Cutting made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Doug Cutting added a comment -

          Try cleaning Ivy's cache of old Avro stuff...

          Show
          Doug Cutting added a comment - Try cleaning Ivy's cache of old Avro stuff...
          Doug Cutting made changes -
          Attachment MAPREDUCE-1556.patch [ 12438463 ]
          Doug Cutting made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Doug Cutting added a comment -

          Update eclipse classpath and pom template to Avro 1.3.0 too.

          Show
          Doug Cutting added a comment - Update eclipse classpath and pom template to Avro 1.3.0 too.
          Doug Cutting made changes -
          Attachment MAPREDUCE-1556.patch [ 12438527 ]
          Doug Cutting made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tom White made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Tom White made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Tom White added a comment -

          The MapReduce build is currently broken - this change fixes it. Hudson seems to be stuck. I ran TestJobHistory locally which exercises the Avro dependencies, and it passes, so I suggest we commit this now.

          Show
          Tom White added a comment - The MapReduce build is currently broken - this change fixes it. Hudson seems to be stuck. I ran TestJobHistory locally which exercises the Avro dependencies, and it passes, so I suggest we commit this now.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Doug!

          Show
          Tom White added a comment - I've just committed this. Thanks Doug!
          Tom White made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #274 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/274/)
          . upgrade to Avro 1.3.0. Contributed by cutting.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #274 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/274/ ) . upgrade to Avro 1.3.0. Contributed by cutting.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #258 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/258/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #258 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/258/ )
          Hide
          Chris Douglas added a comment -

          With HADOOP-6486, the following warnings show up on trunk using a binary tarball:

          SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
          SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
          2010-03-18 01:11:32.988::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
          2010-03-18 01:11:33.056::INFO:  jetty-6.1.14
          

          The slf4j-simple jar is no longer included.

          Show
          Chris Douglas added a comment - With HADOOP-6486 , the following warnings show up on trunk using a binary tarball: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 2010-03-18 01:11:32.988::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2010-03-18 01:11:33.056::INFO: jetty-6.1.14 The slf4j-simple jar is no longer included.
          Hide
          Doug Cutting added a comment -

          > The slf4j-simple jar is no longer included.

          It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts?

          That error message is triggered when no slf4j implementation is on the classpath. When is this printed?

          Show
          Doug Cutting added a comment - > The slf4j-simple jar is no longer included. It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts? That error message is triggered when no slf4j implementation is on the classpath. When is this printed?
          Hide
          Chris Douglas added a comment -

          It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts?

          I think Avro marking it as a dependency masked MR/HDFS/Common's dependence on it, which isn't explicit? It's printed whenever I try to start a MR *Tracker from a binary tarball.

          Show
          Chris Douglas added a comment - It shouldn't be, should it? Avro 1.2 used to pull it in as a required dependency, which was a bug, and Common explicitly had to block it. Avro 1.3 only requires the slf4j API, not any of its implementations. Hadoop uses the log4j implementation of slf4j, which both common and mapreduce's ivy.xml explicitly require, but perhaps not in all the right contexts? I think Avro marking it as a dependency masked MR/HDFS/Common's dependence on it, which isn't explicit? It's printed whenever I try to start a MR *Tracker from a binary tarball.
          Hide
          Chris Douglas added a comment -

          Should it be in common->master instead of test->master ?

          Show
          Chris Douglas added a comment - Should it be in common->master instead of test->master ?
          Hide
          Chris Douglas added a comment -

          Yea, that worked. Filed MAPREDUCE-1622

          Show
          Chris Douglas added a comment - Yea, that worked. Filed MAPREDUCE-1622
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Doug Cutting
              Reporter:
              Doug Cutting
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development