Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: metrics
    • Labels:
      None

      Description

      Write a metrics2 sink plugin for Hadoop to send metrics directly to Apache Kafka in addition to the current, Graphite (Hadoop-9704), Ganglia and File sinks.

      1. HADOOP-10949.patch
        7 kB
        Babak Behzad
      2. HADOOP-10949.patch
        7 kB
        Abin Shahab
      3. HADOOP-10949.patch
        14 kB
        Abin Shahab
      4. HADOOP-10949.patch
        14 kB
        Abin Shahab
      5. HADOOP-10949.patch
        13 kB
        Abin Shahab
      6. HADOOP-10949.patch
        13 kB
        Abin Shahab
      7. HADOOP-10949.patch
        13 kB
        Abin Shahab
      8. HADOOP-10949.patch
        13 kB
        Abin Shahab
      9. HADOOP-10949.patch
        13 kB
        Abin Shahab
      10. HADOOP-10949.patch
        13 kB
        Abin Shahab
      11. HADOOP-10949.patch
        13 kB
        Abin Shahab
      12. HADOOP-10949-1.patch
        14 kB
        Abin Shahab
      13. HADOOP-10949-2.patch
        14 kB
        Abin Shahab
      14. HADOOP-10949-4.patch
        16 kB
        Babak Behzad
      15. HADOOP-10949-5.patch
        16 kB
        Babak Behzad
      16. HADOOP-10949-6.patch
        20 kB
        Babak Behzad
      17. HADOOP-10949-6-1.patch
        0.6 kB
        Babak Behzad

        Issue Links

          Activity

          Hide
          Hongyuan Li Hongyuan Li added a comment -

          i filed HADOOP-14623 to update this module.

          Show
          Hongyuan Li Hongyuan Li added a comment - i filed HADOOP-14623 to update this module.
          Hide
          babakbehzad Babak Behzad added a comment -

          Since this ticket is already fixed, we created a new JIRA for the above fix. Here's the link: https://issues.apache.org/jira/browse/HADOOP-12556

          Show
          babakbehzad Babak Behzad added a comment - Since this ticket is already fixed, we created a new JIRA for the above fix. Here's the link: https://issues.apache.org/jira/browse/HADOOP-12556
          Hide
          babakbehzad Babak Behzad added a comment -

          There was a hadoop-kafka artifact missing from hadoop-tools-dist's pom.xml which was causing the compiled Kafka jar files to be copied to the target dist directory. The new patch adds this in order to complete this fix.

          Show
          babakbehzad Babak Behzad added a comment - There was a hadoop-kafka artifact missing from hadoop-tools-dist's pom.xml which was causing the compiled Kafka jar files to be copied to the target dist directory. The new patch adds this in order to complete this fix.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2397 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2397/)
          HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8)

          • hadoop-tools/hadoop-kafka/pom.xml
          • hadoop-project/pom.xml
          • hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          • hadoop-tools/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2397 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2397/ ) HADOOP-10949 . metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8) hadoop-tools/hadoop-kafka/pom.xml hadoop-project/pom.xml hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java hadoop-tools/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #457 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/457/)
          HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8)

          • hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          • hadoop-project/pom.xml
          • hadoop-tools/pom.xml
          • hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          • hadoop-tools/hadoop-kafka/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #457 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/457/ ) HADOOP-10949 . metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8) hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java hadoop-project/pom.xml hadoop-tools/pom.xml hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java hadoop-tools/hadoop-kafka/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2427 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2427/)
          HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8)

          • hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-kafka/pom.xml
          • hadoop-tools/pom.xml
          • hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          • hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2427 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2427/ ) HADOOP-10949 . metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8) hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-kafka/pom.xml hadoop-tools/pom.xml hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java hadoop-project/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #483 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/483/)
          HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8)

          • hadoop-tools/pom.xml
          • hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          • hadoop-tools/hadoop-kafka/pom.xml
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #483 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/483/ ) HADOOP-10949 . metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8) hadoop-tools/pom.xml hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java hadoop-tools/hadoop-kafka/pom.xml hadoop-project/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #491 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/491/)
          HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/pom.xml
          • hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          • hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          • hadoop-project/pom.xml
          • hadoop-tools/hadoop-kafka/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #491 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/491/ ) HADOOP-10949 . metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/pom.xml hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java hadoop-project/pom.xml hadoop-tools/hadoop-kafka/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1221 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1221/)
          HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8)

          • hadoop-tools/hadoop-kafka/pom.xml
          • hadoop-project/pom.xml
          • hadoop-tools/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          • hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1221 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1221/ ) HADOOP-10949 . metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8) hadoop-tools/hadoop-kafka/pom.xml hadoop-project/pom.xml hadoop-tools/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8572 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8572/)
          HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/pom.xml
          • hadoop-tools/hadoop-kafka/pom.xml
          • hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java
          • hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java
          • hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8572 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8572/ ) HADOOP-10949 . metrics2 sink plugin for Apache Kafka (Babak Behzad via (aw: rev b59e434116a54464efa91c6e81dcd2eb94fdf1f8) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/pom.xml hadoop-tools/hadoop-kafka/pom.xml hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java hadoop-project/pom.xml
          Hide
          aw Allen Wittenauer added a comment -

          +1 committed to trunk.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - +1 committed to trunk. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 21s There were no new javac warning messages.
          +1 javadoc 10m 0s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 0s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 0m 0s The patch does not introduce any new Findbugs (version ) warnings.
              36m 24s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754704/HADOOP-10949-6.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 970daaa
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7617/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7617/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 21s There were no new javac warning messages. +1 javadoc 10m 0s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 0s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 0m 0s The patch does not introduce any new Findbugs (version ) warnings.     36m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754704/HADOOP-10949-6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 970daaa Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7617/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7617/console This message was automatically generated.
          Hide
          babakbehzad Babak Behzad added a comment -

          Ok, based on comments from Ravi Prakash, Allen Wittenauer, Steve Loughran we moved KafkaSink code and its test file into a new directory under /hadoop-tools/hadoop-kafka. This will avoid the problem of having hadoop-common depend on any Kafka libraries. The new patch file uploaded (HADOOP-10949-6.patch) has all the corresponding updated version of the pom.xml files and code.

          Show
          babakbehzad Babak Behzad added a comment - Ok, based on comments from Ravi Prakash , Allen Wittenauer , Steve Loughran we moved KafkaSink code and its test file into a new directory under /hadoop-tools/hadoop-kafka . This will avoid the problem of having hadoop-common depend on any Kafka libraries. The new patch file uploaded ( HADOOP-10949 -6.patch ) has all the corresponding updated version of the pom.xml files and code.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 27s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 8s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 47s Tests passed in hadoop-common.
              63m 55s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754116/HADOOP-10949-5.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c83d13c
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7599/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7599/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7599/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 27s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 8s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 47s Tests passed in hadoop-common.     63m 55s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754116/HADOOP-10949-5.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c83d13c hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7599/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7599/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7599/console This message was automatically generated.
          Hide
          babakbehzad Babak Behzad added a comment -

          I have attached the patch with Steve Loughran's comment of using SLF4J instead of commons-logging.

          The only decision to make now seems to be where to put this code so that everyone is happy. I am going to discuss it more and will put a comment here soon.

          Show
          babakbehzad Babak Behzad added a comment - I have attached the patch with Steve Loughran 's comment of using SLF4J instead of commons-logging. The only decision to make now seems to be where to put this code so that everyone is happy. I am going to discuss it more and will put a comment here soon.
          Hide
          aw Allen Wittenauer added a comment -

          Are you talking about the how the tarball packaging.

          Yup. Pushing this to tools, other than moving kafka out of the common deps, has other, almost invisible impacts.

          Show
          aw Allen Wittenauer added a comment - Are you talking about the how the tarball packaging. Yup. Pushing this to tools, other than moving kafka out of the common deps, has other, almost invisible impacts.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm confused. I'm thinking of /hadoop-tools/* projects which all generate JAR files off the core. Are you talking about the how the tarball packaging.

          I'd be reluctant to move existing metrics, though its the kind of stuff that would really belong in a hadoop-common-server package, rather than the "one jar for everyone" view.

          Show
          stevel@apache.org Steve Loughran added a comment - I'm confused. I'm thinking of /hadoop-tools/* projects which all generate JAR files off the core. Are you talking about the how the tarball packaging. I'd be reluctant to move existing metrics, though its the kind of stuff that would really belong in a hadoop-common-server package, rather than the "one jar for everyone" view.
          Hide
          aw Allen Wittenauer added a comment - - edited

          hadoop-tools still suffers from the same sorts of problems though, just on a smaller scale, made worse by how it is laid out vs. the rest of Hadoop. To be more specific, most of the sub-projects do:

          share/hadoop/(project)        <- hadoop bits
          share/hadoop/(project)/lib   <- dependencies
          

          tools, on the other hand, just does share/hadoop/tools/lib, with both the hadoop bits and the dependencies intermingled. Every time someone issues a command that needs tools support (e.g., distcp), all of this stuff gets pulled in. This is a historical issue, and probably one that should get broken in 3.x.

          If we move all of the metrics sinks so their own universe, the file layout should be changed appropriately.

          Show
          aw Allen Wittenauer added a comment - - edited hadoop-tools still suffers from the same sorts of problems though, just on a smaller scale, made worse by how it is laid out vs. the rest of Hadoop. To be more specific, most of the sub-projects do: share/hadoop/(project) <- hadoop bits share/hadoop/(project)/lib <- dependencies tools, on the other hand, just does share/hadoop/tools/lib, with both the hadoop bits and the dependencies intermingled. Every time someone issues a command that needs tools support (e.g., distcp), all of this stuff gets pulled in. This is a historical issue, and probably one that should get broken in 3.x. If we move all of the metrics sinks so their own universe, the file layout should be changed appropriately.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1 to adding kafka on the hadoop-common classpath; too many people will hate us.

          that means it either goes into a (new) hadoop-tools/hadoop-sinks or hadoop-tools/hadoop-kafka; with the latter allowing us to add more kafka features (logging, ...)

          Personally, I'd prefer a hadoop-kafka one

          Code-wise, I think the obvious thing to me is that you can/should use SLF4J instead of commons-logging; you get to avoid all the if-clauses around logs and have the strings built for you

          Show
          stevel@apache.org Steve Loughran added a comment - -1 to adding kafka on the hadoop-common classpath; too many people will hate us. that means it either goes into a (new) hadoop-tools/hadoop-sinks or hadoop-tools/hadoop-kafka ; with the latter allowing us to add more kafka features (logging, ...) Personally, I'd prefer a hadoop-kafka one Code-wise, I think the obvious thing to me is that you can/should use SLF4J instead of commons-logging; you get to avoid all the if-clauses around logs and have the strings built for you
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 27s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 9s There were no new javac warning messages.
          +1 javadoc 10m 12s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 10s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 54s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 59s Tests passed in hadoop-common.
              65m 29s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752824/HADOOP-10949-4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 90fe7bc
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7542/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7542/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7542/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 27s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 9s There were no new javac warning messages. +1 javadoc 10m 12s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 10s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 54s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 23m 59s Tests passed in hadoop-common.     65m 29s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752824/HADOOP-10949-4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 90fe7bc hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7542/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7542/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7542/console This message was automatically generated.
          Hide
          babakbehzad Babak Behzad added a comment -

          The patch that I just uploaded (HADOOP-10949-4.patch) should pass everything, as I just checked on my own machine. Ravi Prakash and I are working on the logistics of putting these sinks into a separate module as Haohui Mai suggested.

          Show
          babakbehzad Babak Behzad added a comment - The patch that I just uploaded ( HADOOP-10949 -4.patch) should pass everything, as I just checked on my own machine. Ravi Prakash and I are working on the logistics of putting these sinks into a separate module as Haohui Mai suggested.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 14s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 6s There were no new javac warning messages.
          +1 javadoc 10m 6s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 11s The applied patch generated 3 new checkstyle issues (total was 0, now 3).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 1m 58s The patch appears to introduce 5 new Findbugs (version 3.0.0) warnings.
          +1 common tests 23m 21s Tests passed in hadoop-common.
              64m 28s  



          Reason Tests
          FindBugs module:hadoop-common



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752617/HADOOP-10949-3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f44b599
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 14s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 6s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 11s The applied patch generated 3 new checkstyle issues (total was 0, now 3). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 1m 58s The patch appears to introduce 5 new Findbugs (version 3.0.0) warnings. +1 common tests 23m 21s Tests passed in hadoop-common.     64m 28s   Reason Tests FindBugs module:hadoop-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752617/HADOOP-10949-3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f44b599 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7535/console This message was automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          I'm concerned on putting Kafka as a dependency in hadoop-common.

          It might make sense to put these sinks into a separate module.

          Show
          wheat9 Haohui Mai added a comment - I'm concerned on putting Kafka as a dependency in hadoop-common. It might make sense to put these sinks into a separate module.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 43s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 44s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 9s The applied patch generated 4 new checkstyle issues (total was 0, now 4).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
          -1 findbugs 1m 55s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 29s Tests passed in hadoop-common.
              62m 24s  



          Reason Tests
          FindBugs module:hadoop-common



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751864/HADOOP-10949-2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b5ce87f
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 9s The applied patch generated 4 new checkstyle issues (total was 0, now 4). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. -1 findbugs 1m 55s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. +1 common tests 22m 29s Tests passed in hadoop-common.     62m 24s   Reason Tests FindBugs module:hadoop-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751864/HADOOP-10949-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b5ce87f checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/console This message was automatically generated.
          Hide
          raviprak Ravi Prakash added a comment -

          1. dateFormat and timeFormat can be static final members of KafkaSink.
          2. Any reason you want to initialize hostname on every putMetrics()?
          3. Please conform to Hadoop coding style and fix indentation:

              ProducerRecord<Integer, byte[]> data
                        = new ProducerRecord<Integer, byte[]>(
                        topic,
                        jsonLines.toString().getBytes());
          

          4. Please document the expected behavior of the KafkaProducer in putMetrics(), of putMetrics and also an example record json.
          5. flush() could be a synchronous send() of an empty record or it should throw a MetricsException too IMHO.
          6. Please document the test. Please follow the coding style guide for the test as well.

          Show
          raviprak Ravi Prakash added a comment - 1. dateFormat and timeFormat can be static final members of KafkaSink. 2. Any reason you want to initialize hostname on every putMetrics() ? 3. Please conform to Hadoop coding style and fix indentation: ProducerRecord< Integer , byte []> data = new ProducerRecord< Integer , byte []>( topic, jsonLines.toString().getBytes()); 4. Please document the expected behavior of the KafkaProducer in putMetrics(), of putMetrics and also an example record json. 5. flush() could be a synchronous send() of an empty record or it should throw a MetricsException too IMHO. 6. Please document the test. Please follow the coding style guide for the test as well.
          Hide
          ashahab Abin Shahab added a comment -

          ravi Added all the changes you've requested. Please review.

          Show
          ashahab Abin Shahab added a comment - ravi Added all the changes you've requested. Please review.
          Hide
          raviprak Ravi Prakash added a comment -

          Please also document test code. What is it even testing?

          Show
          raviprak Ravi Prakash added a comment - Please also document test code. What is it even testing?
          Hide
          raviprak Ravi Prakash added a comment -

          Also, please version your patch files.

          Show
          raviprak Ravi Prakash added a comment - Also, please version your patch files.
          Hide
          raviprak Ravi Prakash added a comment -

          1. You're sending the same message to multiple topics? Why? Do you have a usecase in mind? I thought constraining to 1 topic was perfectly acceptable.
          2. producer = null; should be in a finally block in close()
          3. Please follow 2-space indentation.
          4. Rather than multiple calls to cal, you should use DateFormat.
          5. I am still not sure about the synchronous send. https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example says

          ** PLEASE NOTE ** The recommended producer is from latest stable release using the new Java producer http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html

          Show
          raviprak Ravi Prakash added a comment - 1. You're sending the same message to multiple topics? Why? Do you have a usecase in mind? I thought constraining to 1 topic was perfectly acceptable. 2. producer = null; should be in a finally block in close() 3. Please follow 2-space indentation. 4. Rather than multiple calls to cal , you should use DateFormat. 5. I am still not sure about the synchronous send. https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example says ** PLEASE NOTE ** The recommended producer is from latest stable release using the new Java producer http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 1s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 1s There were no new javac warning messages.
          +1 javadoc 9m 48s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 9s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
          -1 findbugs 1m 56s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 59s Tests failed in hadoop-common.
              63m 39s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.fs.sftp.TestSFTPFileSystem



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751507/HADOOP-10949.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0bc15cb
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7516/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7516/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7516/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7516/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 9s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. -1 findbugs 1m 56s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 22m 59s Tests failed in hadoop-common.     63m 39s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.fs.sftp.TestSFTPFileSystem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751507/HADOOP-10949.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0bc15cb Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7516/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7516/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7516/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7516/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 26s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 8m 38s The applied patch generated 1 additional warning messages.
          +1 javadoc 10m 34s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 18s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse.
          -1 findbugs 2m 0s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 25s Tests failed in hadoop-common.
              68m 13s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.net.TestNetUtils
            hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751330/HADOOP-10949.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3aac475
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/artifact/patchprocess/diffJavacWarnings.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 8m 38s The applied patch generated 1 additional warning messages. +1 javadoc 10m 34s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 18s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse. -1 findbugs 2m 0s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 22m 25s Tests failed in hadoop-common.     68m 13s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.net.TestNetUtils   hadoop.ha.TestZKFailoverController Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751330/HADOOP-10949.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3aac475 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/artifact/patchprocess/diffJavacWarnings.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7514/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 5m 42s trunk compilation may be broken.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 1m 31s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751286/HADOOP-10949.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f61120d
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7512/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 5m 42s trunk compilation may be broken. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 1m 31s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751286/HADOOP-10949.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f61120d Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7512/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 10s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 57s The applied patch generated 2 additional warning messages.
          +1 javadoc 9m 53s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 5s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 1m 59s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 21m 57s Tests failed in hadoop-common.
              62m 29s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.net.TestNetUtils
          Timed out tests org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751203/HADOOP-10949.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 22dc5fc
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/artifact/patchprocess/diffJavacWarnings.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 10s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 57s The applied patch generated 2 additional warning messages. +1 javadoc 9m 53s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 5s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 1m 59s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 21m 57s Tests failed in hadoop-common.     62m 29s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestNetUtils Timed out tests org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751203/HADOOP-10949.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 22dc5fc javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/artifact/patchprocess/diffJavacWarnings.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7508/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 4s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 8m 1s The applied patch generated 2 additional warning messages.
          +1 javadoc 9m 59s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 7s The applied patch generated 1 new checkstyle issues (total was 0, now 1).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 0s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 23m 0s Tests failed in hadoop-common.
              63m 42s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.security.token.delegation.web.TestWebDelegationToken
            hadoop.ha.TestZKFailoverController
            hadoop.net.TestNetUtils



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751115/HADOOP-10949.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 71aedfa
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 8m 1s The applied patch generated 2 additional warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 7s The applied patch generated 1 new checkstyle issues (total was 0, now 1). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 0s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 23m 0s Tests failed in hadoop-common.     63m 42s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.security.token.delegation.web.TestWebDelegationToken   hadoop.ha.TestZKFailoverController   hadoop.net.TestNetUtils Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751115/HADOOP-10949.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71aedfa javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7503/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 3s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 8m 7s The applied patch generated 3 additional warning messages.
          +1 javadoc 9m 55s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 5s The applied patch generated 120 new checkstyle issues (total was 0, now 120).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 25s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          -1 findbugs 1m 59s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 23m 14s Tests failed in hadoop-common.
              64m 53s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.net.TestNetUtils



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751060/HADOOP-10949.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 14215c8
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 3s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 8m 7s The applied patch generated 3 additional warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 5s The applied patch generated 120 new checkstyle issues (total was 0, now 120). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 25s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. -1 findbugs 1m 59s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 23m 14s Tests failed in hadoop-common.     64m 53s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestNetUtils Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751060/HADOOP-10949.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 14215c8 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7500/console This message was automatically generated.
          Hide
          ashahab Abin Shahab added a comment -

          Fixed checkstyle errors.

          Show
          ashahab Abin Shahab added a comment - Fixed checkstyle errors.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 58s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 8m 3s The applied patch generated 3 additional warning messages.
          +1 javadoc 9m 47s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 5s The applied patch generated 120 new checkstyle issues (total was 0, now 120).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          -1 findbugs 1m 57s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 23m 0s Tests failed in hadoop-common.
              63m 22s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.net.TestNetUtils
            hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751060/HADOOP-10949.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3a76a01
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 58s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 8m 3s The applied patch generated 3 additional warning messages. +1 javadoc 9m 47s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 5s The applied patch generated 120 new checkstyle issues (total was 0, now 120). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. -1 findbugs 1m 57s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 23m 0s Tests failed in hadoop-common.     63m 22s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.net.TestNetUtils   hadoop.ha.TestZKFailoverController Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751060/HADOOP-10949.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3a76a01 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7498/console This message was automatically generated.
          Hide
          ashahab Abin Shahab added a comment -

          Ravi Prakash Incorporated the changes you've requested and added the test.

          Show
          ashahab Abin Shahab added a comment - Ravi Prakash Incorporated the changes you've requested and added the test.
          Hide
          ashahab Abin Shahab added a comment -

          Send in this class does not return any futures:
          kafka.javaapi.producer.Producer.
          This is the old producer.
          Version 0.8.2(the latest) introduced a new producer that has futures:
          http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html

          So that explains why we have the old one.

          Regarding your other comment, KafkaSink IS being inited using a
          SubsetConfiguration.

          On Mon, Aug 17, 2015 at 4:09 PM, Ravi Prakash (JIRA) <jira@apache.org>

          Show
          ashahab Abin Shahab added a comment - Send in this class does not return any futures: kafka.javaapi.producer.Producer. This is the old producer. Version 0.8.2(the latest) introduced a new producer that has futures: http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html So that explains why we have the old one. Regarding your other comment, KafkaSink IS being inited using a SubsetConfiguration . On Mon, Aug 17, 2015 at 4:09 PM, Ravi Prakash (JIRA) <jira@apache.org>
          Hide
          raviprak Ravi Prakash added a comment -

          Also init should probably match the way GraphiteSink gets inited (using a SubsetConfiguration).
          Also it seems we aren't doing anything with the futures that send returns. That seems wrong.

          Show
          raviprak Ravi Prakash added a comment - Also init should probably match the way GraphiteSink gets inited (using a SubsetConfiguration ). Also it seems we aren't doing anything with the futures that send returns. That seems wrong.
          Hide
          raviprak Ravi Prakash added a comment -

          1. Please try using exclusions in the pom file. Please refer to https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html . Can you try a clean rebuild?
          2. Also if(producer != null){ should probably be the first thing you check in putMetrics() so that you don't do a lot of work when the producer failed to init.
          3. close should probably set producer to null. You can also use IOUtils.closeStream()
          4 the finally doesn't seem to be required in close()
          5. Please don't catch Throwable in close()
          6. In flush, do you really need the try-catch?
          7. Please use 2 spaces for indentation.
          8. Please conform to the 80-char limit for lines
          9. Please add more documentation to the class (including the constraint that only 1 topic can be written to)
          10. Should the scope of kafka be provided. I don't know if we should start packaging kafka along with hadoop.
          11. In init, is producer = new Producer<Integer, byte[]>(config); ever going to throw an exception?

          Show
          raviprak Ravi Prakash added a comment - 1. Please try using exclusions in the pom file. Please refer to https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html . Can you try a clean rebuild? 2. Also if(producer != null){ should probably be the first thing you check in putMetrics() so that you don't do a lot of work when the producer failed to init. 3. close should probably set producer to null. You can also use IOUtils.closeStream() 4 the finally doesn't seem to be required in close() 5. Please don't catch Throwable in close() 6. In flush, do you really need the try-catch? 7. Please use 2 spaces for indentation. 8. Please conform to the 80-char limit for lines 9. Please add more documentation to the class (including the constraint that only 1 topic can be written to) 10. Should the scope of kafka be provided . I don't know if we should start packaging kafka along with hadoop. 11. In init, is producer = new Producer<Integer, byte[]>(config); ever going to throw an exception?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 54s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 9m 45s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 9s The applied patch generated 113 new checkstyle issues (total was 0, now 113).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          -1 install 0m 37s The patch causes mvn install to fail.
          +1 eclipse:eclipse 1m 12s The patch built with eclipse:eclipse.
          -1 findbugs 1m 57s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 20s Tests failed in hadoop-common.
              62m 12s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.net.TestNetUtils



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750881/HADOOP-10949.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c77bd6a
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 9m 45s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 9s The applied patch generated 113 new checkstyle issues (total was 0, now 113). +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 install 0m 37s The patch causes mvn install to fail. +1 eclipse:eclipse 1m 12s The patch built with eclipse:eclipse. -1 findbugs 1m 57s The patch appears to introduce 2 new Findbugs (version 3.0.0) warnings. -1 common tests 22m 20s Tests failed in hadoop-common.     62m 12s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestNetUtils Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750881/HADOOP-10949.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c77bd6a checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7491/console This message was automatically generated.
          Hide
          ashahab Abin Shahab added a comment -

          ravi, should I specify a branch of snappy? It builds despite the warning, and at this point I'm not sure how else to resolve the conflict.

          Show
          ashahab Abin Shahab added a comment - ravi , should I specify a branch of snappy? It builds despite the warning, and at this point I'm not sure how else to resolve the conflict.
          Hide
          raviprak Ravi Prakash added a comment -

          This new dependency that you want to add seems to be causing a conflict in the version of snappy.

          [WARNING] 
          Dependency convergence error for org.xerial.snappy:snappy-java:1.1.1.6 paths to dependency are:
          +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
            +-org.apache.kafka:kafka_2.10:0.8.2.1
              +-org.apache.kafka:kafka-clients:0.8.2.1
                +-org.xerial.snappy:snappy-java:1.1.1.6
          and
          +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
            +-org.apache.avro:avro:1.7.4
              +-org.xerial.snappy:snappy-java:1.0.4.1
          
          [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message:
          Failed while enforcing releasability the error(s) are [
          Dependency convergence error for org.xerial.snappy:snappy-java:1.1.1.6 paths to dependency are:
          +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
            +-org.apache.kafka:kafka_2.10:0.8.2.1
              +-org.apache.kafka:kafka-clients:0.8.2.1
                +-org.xerial.snappy:snappy-java:1.1.1.6
          and
          +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
            +-org.apache.avro:avro:1.7.4
              +-org.xerial.snappy:snappy-java:1.0.4.1
          ]
          
          Show
          raviprak Ravi Prakash added a comment - This new dependency that you want to add seems to be causing a conflict in the version of snappy. [WARNING] Dependency convergence error for org.xerial.snappy:snappy-java:1.1.1.6 paths to dependency are: +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT +-org.apache.kafka:kafka_2.10:0.8.2.1 +-org.apache.kafka:kafka-clients:0.8.2.1 +-org.xerial.snappy:snappy-java:1.1.1.6 and +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT +-org.apache.avro:avro:1.7.4 +-org.xerial.snappy:snappy-java:1.0.4.1 [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message: Failed while enforcing releasability the error(s) are [ Dependency convergence error for org.xerial.snappy:snappy-java:1.1.1.6 paths to dependency are: +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT +-org.apache.kafka:kafka_2.10:0.8.2.1 +-org.apache.kafka:kafka-clients:0.8.2.1 +-org.xerial.snappy:snappy-java:1.1.1.6 and +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT +-org.apache.avro:avro:1.7.4 +-org.xerial.snappy:snappy-java:1.0.4.1 ]
          Hide
          aw Allen Wittenauer added a comment -
          • Is there a reason the version isn't stored and taken from hadoop-project/pom.xml?
          • This also needs tests.
          Show
          aw Allen Wittenauer added a comment - Is there a reason the version isn't stored and taken from hadoop-project/pom.xml? This also needs tests.
          Hide
          ashahab Abin Shahab added a comment -

          Patch with pom.xml changes.

          Show
          ashahab Abin Shahab added a comment - Patch with pom.xml changes.
          Hide
          ashahab Abin Shahab added a comment -

          Submitting patch with pom.xml changes

          Show
          ashahab Abin Shahab added a comment - Submitting patch with pom.xml changes
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12673027/HADOOP-10949.patch
          against trunk revision 9112f09.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The applied patch generated 1157 javac compiler warnings (more than the trunk's current 1154 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
          org.apache.hadoop.hdfs.TestDatanodeDeath

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5595//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5595//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5595//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12673027/HADOOP-10949.patch against trunk revision 9112f09. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 javac . The applied patch generated 1157 javac compiler warnings (more than the trunk's current 1154 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery org.apache.hadoop.hdfs.TestDatanodeDeath Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5595//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5595//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5595//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          Yup, patch the pom.xml as well.

          Show
          aw Allen Wittenauer added a comment - Yup, patch the pom.xml as well.
          Hide
          babakbehzad Babak Behzad added a comment -

          Allen Wittenauer: The reason that it does not compile is that it needs Kafka jar libraries! How can we add those? Should I also upload pom.xml patch here?

          Show
          babakbehzad Babak Behzad added a comment - Allen Wittenauer : The reason that it does not compile is that it needs Kafka jar libraries! How can we add those? Should I also upload pom.xml patch here?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12673027/HADOOP-10949.patch
          against trunk revision 16333b4.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4863//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12673027/HADOOP-10949.patch against trunk revision 16333b4. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4863//console This message is automatically generated.
          Hide
          babakbehzad Babak Behzad added a comment -

          I attached my first patch available for this. It has been tested, but needs reviews and also a test. I will start working on the test for it soon.

          Show
          babakbehzad Babak Behzad added a comment - I attached my first patch available for this. It has been tested, but needs reviews and also a test. I will start working on the test for it soon.

            People

            • Assignee:
              babakbehzad Babak Behzad
              Reporter:
              babakbehzad Babak Behzad
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development