Nutch
  1. Nutch
  2. NUTCH-1078

Upgrade all instances of commons logging to slf4j (with log4j backend)

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.4
    • Fix Version/s: 1.4
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      Whilst working on another issue, I noticed that some classes still import and use commons logging for example HttpBase.java

      import java.util.*;
      
      // Commons Logging imports
      import org.apache.commons.logging.Log;
      import org.apache.commons.logging.LogFactory;
      
      // Nutch imports
      import org.apache.nutch.crawl.CrawlDatum;
      

      At this stage I am unsure how many (if any others) still import and reply upon commons logging, however they should be upgraded to slf4j for branch-1.4.

      1. NUTCH-1078-branch-1.4-20110816.patch
        133 kB
        Lewis John McGibbney
      2. NUTCH-1078-branch-1.4-20110824-v2.patch
        142 kB
        Lewis John McGibbney
      3. NUTCH-1078-branch-1.4-20110911-v3.patch
        147 kB
        Lewis John McGibbney
      4. NUTCH-1078-branch-1.4-20110916-v4.patch
        147 kB
        Lewis John McGibbney
      5. NUTCH-1078-trunk-20110929.patch
        2 kB
        Lewis John McGibbney

        Issue Links

          Activity

          Hide
          Lewis John McGibbney added a comment -

          This issue is inherently large to deal with and it would need to be phased in my opinion. Almost every class we maintain contains commons logging imports and usage. Ultimately it would mean that we could drop a couple project dependencies from runtime if implemented fully across the whole Nutch code base.

          I know that this is pretty vague but does anyone have ideas for progressing... it was not my intention to unearth quite this large a task. Thanks

          Show
          Lewis John McGibbney added a comment - This issue is inherently large to deal with and it would need to be phased in my opinion. Almost every class we maintain contains commons logging imports and usage. Ultimately it would mean that we could drop a couple project dependencies from runtime if implemented fully across the whole Nutch code base. I know that this is pretty vague but does anyone have ideas for progressing... it was not my intention to unearth quite this large a task. Thanks
          Hide
          Markus Jelsma added a comment -

          hmm, you could try to find/replace the imports in the whole project tree. I don't really expect problems (ahum).

          Show
          Markus Jelsma added a comment - hmm, you could try to find/replace the imports in the whole project tree. I don't really expect problems (ahum).
          Hide
          Julien Nioche added a comment -

          slf4j has a library that can do the conversion automatically. That's what I used for the trunk

          Show
          Julien Nioche added a comment - slf4j has a library that can do the conversion automatically. That's what I used for the trunk
          Hide
          Lewis John McGibbney added a comment -

          patch attachment for branch-1.4. This upgrades all instances of Jakarta Commons Logging to slf4j logging. For reference information can be found on the migration tool here [1].

          Some limitation which I need to check is if any logging constructs of non-string type have been missed, as well as build scripts.

          [1] http://www.slf4j.org/migrator.html

          Show
          Lewis John McGibbney added a comment - patch attachment for branch-1.4. This upgrades all instances of Jakarta Commons Logging to slf4j logging. For reference information can be found on the migration tool here [1] . Some limitation which I need to check is if any logging constructs of non-string type have been missed, as well as build scripts. [1] http://www.slf4j.org/migrator.html
          Hide
          Markus Jelsma added a comment - - edited

          the FATAL level is not supported.

          You have to convert them manually. This is limitation is not deemed very serious because there are usually very few log statements bearing the FATAL level.

              [javac] /home/markus/projects/apache/nutch/branches/branch-1.4/src/java/org/apache/nutch/fetcher/Fetcher.java:531: cannot find symbol
              [javac] symbol  : method fatal(java.lang.String,java.io.IOException)
              [javac] location: interface org.slf4j.Logger
              [javac]               LOG.fatal("QueueFeeder error reading input, record " + cnt, e);
              [javac]                  ^
              [javac] /home/markus/projects/apache/nutch/branches/branch-1.4/src/java/org/apache/nutch/fetcher/Fetcher.java:788: cannot find symbol
              [javac] symbol  : method isFatalEnabled()
              [javac] location: interface org.slf4j.Logger
              [javac]         if (LOG.isFatalEnabled()) {
          

          A lot of errors about these two fatal methods.

          Show
          Markus Jelsma added a comment - - edited the FATAL level is not supported. You have to convert them manually. This is limitation is not deemed very serious because there are usually very few log statements bearing the FATAL level. [javac] /home/markus/projects/apache/nutch/branches/branch-1.4/src/java/org/apache/nutch/fetcher/Fetcher.java:531: cannot find symbol [javac] symbol : method fatal(java.lang. String ,java.io.IOException) [javac] location: interface org.slf4j.Logger [javac] LOG.fatal( "QueueFeeder error reading input, record " + cnt, e); [javac] ^ [javac] /home/markus/projects/apache/nutch/branches/branch-1.4/src/java/org/apache/nutch/fetcher/Fetcher.java:788: cannot find symbol [javac] symbol : method isFatalEnabled() [javac] location: interface org.slf4j.Logger [javac] if (LOG.isFatalEnabled()) { A lot of errors about these two fatal methods.
          Hide
          Lewis John McGibbney added a comment -

          Thanks Markus. I will get this sorted out and submit a new patch in due course. In addition I'll ensure that the two points mentioned above are also addressed.

          Show
          Lewis John McGibbney added a comment - Thanks Markus. I will get this sorted out and submit a new patch in due course. In addition I'll ensure that the two points mentioned above are also addressed.
          Hide
          Lewis John McGibbney added a comment -

          The attached patch replaces all FATAL logging errors with the slightly less severe ERROR logging. The patch fails with one javac error as follows

              [javac] /home/lewis/ASF/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java:95: cannot find symbol
              [javac] symbol  : method error(java.lang.Exception)
              [javac] location: interface org.slf4j.Logger
              [javac]       LOG.error(e);
              [javac]          ^
              [javac] Note: Some input files use or override a deprecated API.
              [javac] Note: Recompile with -Xlint:deprecation for details.
              [javac] Note: Some input files use unchecked or unsafe operations.
              [javac] Note: Recompile with -Xlint:unchecked for details.
              [javac] 1 error
          
          BUILD FAILED
          /home/lewis/ASF/branch-1.4/build.xml:96: Compile failed; see the compiler error output for details.
          

          Can anyone give a clue to as what is wrong and hwat I need to substitute the LOG.error(e); call in SolrWriter.java with. Thanks

          Show
          Lewis John McGibbney added a comment - The attached patch replaces all FATAL logging errors with the slightly less severe ERROR logging. The patch fails with one javac error as follows [javac] /home/lewis/ASF/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java:95: cannot find symbol [javac] symbol : method error(java.lang.Exception) [javac] location: interface org.slf4j.Logger [javac] LOG.error(e); [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 1 error BUILD FAILED /home/lewis/ASF/branch-1.4/build.xml:96: Compile failed; see the compiler error output for details. Can anyone give a clue to as what is wrong and hwat I need to substitute the LOG.error(e); call in SolrWriter.java with. Thanks
          Hide
          Lewis John McGibbney added a comment -

          patch attachment compiles and passes all tests successfully. During the creation of the patch I noticed some irregular behaviour with the org.apache.nutch.parse.feed.FeedParser class.

          The org.apache.nutch.indexer.anchor.AnchorIndexingFilter import broke the build...

          As you can see I've removed this particular import from the class as per

          // APACHE imports
          import org.slf4j.Logger;
          import org.slf4j.LoggerFactory;
          import org.apache.hadoop.conf.Configuration;
          import org.apache.hadoop.io.Text;
          import org.apache.hadoop.util.StringUtils;
          // import org.apache.nutch.indexer.anchor.AnchorIndexingFilter; removed as per NUTCH-1078
          import org.apache.nutch.metadata.Feed;
          import org.apache.nutch.metadata.Metadata;
          import org.apache.nutch.net.URLFilters;
          import org.apache.nutch.net.URLNormalizers;
          import org.apache.nutch.net.protocols.Response;
          

          This is ready to be tested. Thanks

          Show
          Lewis John McGibbney added a comment - patch attachment compiles and passes all tests successfully. During the creation of the patch I noticed some irregular behaviour with the org.apache.nutch.parse.feed.FeedParser class. The org.apache.nutch.indexer.anchor.AnchorIndexingFilter import broke the build... As you can see I've removed this particular import from the class as per // APACHE imports import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.Text; import org.apache.hadoop.util.StringUtils; // import org.apache.nutch.indexer.anchor.AnchorIndexingFilter; removed as per NUTCH-1078 import org.apache.nutch.metadata.Feed; import org.apache.nutch.metadata.Metadata; import org.apache.nutch.net.URLFilters; import org.apache.nutch.net.URLNormalizers; import org.apache.nutch.net.protocols.Response; This is ready to be tested. Thanks
          Hide
          Lewis John McGibbney added a comment -

          any comments please?

          Show
          Lewis John McGibbney added a comment - any comments please?
          Hide
          Markus Jelsma added a comment -

          Ah, the org.apache.nutch.tools.Benchmark fails. All others patches seem to work, even with my local changes.

          Show
          Markus Jelsma added a comment - Ah, the org.apache.nutch.tools.Benchmark fails. All others patches seem to work, even with my local changes.
          Hide
          Lewis John McGibbney added a comment - - edited

          thank you Markus. I noticed that previously, before submitting the v3 patch, some classes failed between my eclipse and when testing locally. I'll get another patch submitted and sort this out.

          Show
          Lewis John McGibbney added a comment - - edited thank you Markus. I noticed that previously, before submitting the v3 patch, some classes failed between my eclipse and when testing locally. I'll get another patch submitted and sort this out.
          Hide
          Lewis John McGibbney added a comment -

          patch fixes the error logging you highlighted Markus, thanks for that.

          This passes tests, compile and builds fine for me on my local copy with no modifications.

          Show
          Lewis John McGibbney added a comment - patch fixes the error logging you highlighted Markus, thanks for that. This passes tests, compile and builds fine for me on my local copy with no modifications.
          Hide
          Lewis John McGibbney added a comment -

          Any further comments on this one please.

          Show
          Lewis John McGibbney added a comment - Any further comments on this one please.
          Hide
          Sami Siren added a comment -

          Lewis, after applying this patch I still see some references to commons-logging in ivy/ivy.xml and some pom.xml files too. I think those should also be removed, no?

          Show
          Sami Siren added a comment - Lewis, after applying this patch I still see some references to commons-logging in ivy/ivy.xml and some pom.xml files too. I think those should also be removed, no?
          Hide
          Lewis John McGibbney added a comment -

          Hi Sami, please see NUTCH-1091, which I opened. I think this is the next move to drop it completely, however I think there might be couple of issue here and there.

          Show
          Lewis John McGibbney added a comment - Hi Sami, please see NUTCH-1091 , which I opened. I think this is the next move to drop it completely, however I think there might be couple of issue here and there.
          Hide
          Sami Siren added a comment -

          Oh, ok. I didn't realize there was another issue open about removing those.

          Show
          Sami Siren added a comment - Oh, ok. I didn't realize there was another issue open about removing those.
          Hide
          Markus Jelsma added a comment -

          Push it in Lewis! I'll fix whatever breaks here
          cheers

          Show
          Markus Jelsma added a comment - Push it in Lewis! I'll fix whatever breaks here cheers
          Hide
          Lewis John McGibbney added a comment -

          Committed @ revision 1174191.

          Would like to say a big thanks to everyone for keeping me right on this one. Ta

          Show
          Lewis John McGibbney added a comment - Committed @ revision 1174191. Would like to say a big thanks to everyone for keeping me right on this one. Ta
          Hide
          Markus Jelsma added a comment -

          Cheers! Only had to recommit the CHANGELOG entry for NUTCH-1115 which was committed a few houts ago.

          Show
          Markus Jelsma added a comment - Cheers! Only had to recommit the CHANGELOG entry for NUTCH-1115 which was committed a few houts ago.
          Hide
          Markus Jelsma added a comment -

          Too bad, the fetcher's logging is not partially broken, something i didn't see happening in a controlled environment since there were no exceptions. This is what's happening in a production environment:

          2011-09-22 19:36:02,046 ERROR org.apache.nutch.util.LogUtil: Cannot log with method [null]
          java.lang.NullPointerException
          	at org.apache.nutch.util.LogUtil$1.flush(LogUtil.java:103)
          	at java.io.PrintStream.write(PrintStream.java:432)
          	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
          	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
          	at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
          	at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
          	at java.io.PrintStream.newLine(PrintStream.java:496)
          	at java.io.PrintStream.println(PrintStream.java:774)
          	at java.lang.Throwable.printStackTrace(Throwable.java:461)
          	at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:197)
          	at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:665)
          

          Can you check it out?

          Show
          Markus Jelsma added a comment - Too bad, the fetcher's logging is not partially broken, something i didn't see happening in a controlled environment since there were no exceptions. This is what's happening in a production environment: 2011-09-22 19:36:02,046 ERROR org.apache.nutch.util.LogUtil: Cannot log with method [ null ] java.lang.NullPointerException at org.apache.nutch.util.LogUtil$1.flush(LogUtil.java:103) at java.io.PrintStream.write(PrintStream.java:432) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85) at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168) at java.io.PrintStream.newLine(PrintStream.java:496) at java.io.PrintStream.println(PrintStream.java:774) at java.lang.Throwable.printStackTrace(Throwable.java:461) at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:197) at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:665) Can you check it out?
          Hide
          Markus Jelsma added a comment -

          This one also pops up in the fetcher:

          2011-09-22 19:44:26,929 ERROR org.apache.nutch.util.LogUtil: Cannot init log methods
          java.lang.NoSuchMethodException: org.slf4j.Logger.trace(java.lang.Object)
          	at java.lang.Class.getMethod(Class.java:1605)
          	at org.apache.nutch.util.LogUtil.<clinit>(LogUtil.java:48)
          	at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:197)
          	at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:665)
          
          Show
          Markus Jelsma added a comment - This one also pops up in the fetcher: 2011-09-22 19:44:26,929 ERROR org.apache.nutch.util.LogUtil: Cannot init log methods java.lang.NoSuchMethodException: org.slf4j.Logger.trace(java.lang. Object ) at java.lang. Class .getMethod( Class .java:1605) at org.apache.nutch.util.LogUtil.<clinit>(LogUtil.java:48) at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:197) at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:665)
          Hide
          Hudson added a comment -

          Integrated in Nutch-branch-1.4 #14 (See https://builds.apache.org/job/Nutch-branch-1.4/14/)
          Recommitted CHANGELOG entry for NUTCH-1115. Was overwritten by NUTCH-1078 commit
          commit to address NUTCH-1078 and update of changes.txt

          markus : http://svn.apache.org/viewvc/nutch/branches/branch-1.4/viewvc/?view=rev&root=&revision=1174222
          Files :

          • /nutch/branches/branch-1.4/CHANGES.txt

          lewismc : http://svn.apache.org/viewvc/nutch/branches/branch-1.4/viewvc/?view=rev&root=&revision=1174191
          Files :

          • /nutch/branches/branch-1.4/CHANGES.txt
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/AbstractFetchSchedule.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/Crawl.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDb.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDbFilter.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDbMerger.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDbReader.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDbReducer.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/FetchScheduleFactory.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/Generator.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/Injector.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/LinkDb.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/LinkDbFilter.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/LinkDbMerger.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/LinkDbReader.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/MapWritable.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/SignatureFactory.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/URLPartitioner.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/fetcher/Fetcher.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/fetcher/OldFetcher.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/IndexerMapReduce.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/IndexingFilters.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/IndexingFiltersChecker.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrClean.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrDeleteDuplicates.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrMappingReader.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrUtils.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrWriter.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/net/URLNormalizers.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/OutlinkExtractor.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParseOutputFormat.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParsePluginsReader.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParseResult.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParseSegment.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParseUtil.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParserChecker.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParserFactory.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/plugin/PluginDescriptor.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/plugin/PluginManifestParser.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/plugin/PluginRepository.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/protocol/ProtocolFactory.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/LinkDumper.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/LinkRank.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/Loops.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/NodeDumper.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/ScoreUpdater.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/WebGraph.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/segment/SegmentMergeFilters.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/segment/SegmentMerger.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/segment/SegmentReader.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/CrawlDBScanner.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/DmozParser.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/FreeGenerator.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/ResolveUrls.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/arc/ArcRecordReader.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/arc/ArcSegmentCreator.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/proxy/LogDebugHandler.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/proxy/SegmentHandler.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/proxy/TestbedProxy.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/DeflateUtils.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/DomUtil.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/EncodingDetector.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/GZIPUtils.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/LogUtil.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/ObjectCache.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/domain/DomainStatistics.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/domain/DomainSuffixes.java
          • /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/domain/DomainSuffixesReader.java
          • /nutch/branches/branch-1.4/src/plugin/creativecommons/src/java/org/creativecommons/nutch/CCIndexingFilter.java
          • /nutch/branches/branch-1.4/src/plugin/creativecommons/src/java/org/creativecommons/nutch/CCParseFilter.java
          • /nutch/branches/branch-1.4/src/plugin/feed/src/java/org/apache/nutch/parse/feed/FeedParser.java
          • /nutch/branches/branch-1.4/src/plugin/feed/src/test/org/apache/nutch/parse/feed/TestFeedParser.java
          • /nutch/branches/branch-1.4/src/plugin/index-anchor/src/java/org/apache/nutch/indexer/anchor/AnchorIndexingFilter.java
          • /nutch/branches/branch-1.4/src/plugin/index-basic/src/java/org/apache/nutch/indexer/basic/BasicIndexingFilter.java
          • /nutch/branches/branch-1.4/src/plugin/index-more/src/java/org/apache/nutch/indexer/more/MoreIndexingFilter.java
          • /nutch/branches/branch-1.4/src/plugin/index-static/src/java/org/apache/nutch/indexer/staticfield/StaticFieldIndexer.java
          • /nutch/branches/branch-1.4/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java
          • /nutch/branches/branch-1.4/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/RobotRulesParser.java
          • /nutch/branches/branch-1.4/src/plugin/lib-regex-filter/src/java/org/apache/nutch/urlfilter/api/RegexURLFilterBase.java
          • /nutch/branches/branch-1.4/src/plugin/lib-regex-filter/src/test/org/apache/nutch/urlfilter/api/RegexURLFilterBaseTest.java
          • /nutch/branches/branch-1.4/src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag/RelTagParser.java
          • /nutch/branches/branch-1.4/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext/ExtParser.java
          • /nutch/branches/branch-1.4/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java
          • /nutch/branches/branch-1.4/src/plugin/parse-js/src/java/org/apache/nutch/parse/js/JSParseFilter.java
          • /nutch/branches/branch-1.4/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf/SWFParser.java
          • /nutch/branches/branch-1.4/src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/TikaParser.java
          • /nutch/branches/branch-1.4/src/plugin/parse-tika/src/test/org/apache/nutch/tika/TestFeedParser.java
          • /nutch/branches/branch-1.4/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip/ZipParser.java
          • /nutch/branches/branch-1.4/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip/ZipTextExtractor.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-file/src/java/org/apache/nutch/protocol/file/File.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Ftp.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/PrintCommandListener.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/Http.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/DummySSLProtocolSocketFactory.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/DummyX509TrustManager.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/Http.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/HttpAuthenticationFactory.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/HttpBasicAuthentication.java
          • /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/HttpResponse.java
          • /nutch/branches/branch-1.4/src/plugin/scoring-opic/src/java/org/apache/nutch/scoring/opic/OPICScoringFilter.java
          • /nutch/branches/branch-1.4/src/plugin/subcollection/src/java/org/apache/nutch/collection/CollectionManager.java
          • /nutch/branches/branch-1.4/src/plugin/subcollection/src/java/org/apache/nutch/indexer/subcollection/SubcollectionIndexingFilter.java
          • /nutch/branches/branch-1.4/src/plugin/tld/src/java/org/apache/nutch/indexer/tld/TLDIndexingFilter.java
          • /nutch/branches/branch-1.4/src/plugin/urlfilter-domain/src/java/org/apache/nutch/urlfilter/domain/DomainURLFilter.java
          • /nutch/branches/branch-1.4/src/plugin/urlfilter-domain/src/test/org/apache/nutch/urlfilter/domain/TestDomainURLFilter.java
          • /nutch/branches/branch-1.4/src/plugin/urlfilter-prefix/src/java/org/apache/nutch/urlfilter/prefix/PrefixURLFilter.java
          • /nutch/branches/branch-1.4/src/plugin/urlfilter-suffix/src/java/org/apache/nutch/urlfilter/suffix/SuffixURLFilter.java
          • /nutch/branches/branch-1.4/src/plugin/urlmeta/src/java/org/apache/nutch/indexer/urlmeta/URLMetaIndexingFilter.java
          • /nutch/branches/branch-1.4/src/plugin/urlmeta/src/java/org/apache/nutch/scoring/urlmeta/URLMetaScoringFilter.java
          • /nutch/branches/branch-1.4/src/plugin/urlnormalizer-basic/src/java/org/apache/nutch/net/urlnormalizer/basic/BasicURLNormalizer.java
          • /nutch/branches/branch-1.4/src/plugin/urlnormalizer-regex/src/java/org/apache/nutch/net/urlnormalizer/regex/RegexURLNormalizer.java
          • /nutch/branches/branch-1.4/src/plugin/urlnormalizer-regex/src/test/org/apache/nutch/net/urlnormalizer/regex/TestRegexURLNormalizer.java
          • /nutch/branches/branch-1.4/src/test/org/apache/nutch/crawl/CrawlDBTestUtil.java
          Show
          Hudson added a comment - Integrated in Nutch-branch-1.4 #14 (See https://builds.apache.org/job/Nutch-branch-1.4/14/ ) Recommitted CHANGELOG entry for NUTCH-1115 . Was overwritten by NUTCH-1078 commit commit to address NUTCH-1078 and update of changes.txt markus : http://svn.apache.org/viewvc/nutch/branches/branch-1.4/viewvc/?view=rev&root=&revision=1174222 Files : /nutch/branches/branch-1.4/CHANGES.txt lewismc : http://svn.apache.org/viewvc/nutch/branches/branch-1.4/viewvc/?view=rev&root=&revision=1174191 Files : /nutch/branches/branch-1.4/CHANGES.txt /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/AbstractFetchSchedule.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/Crawl.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDb.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDbFilter.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDbMerger.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDbReader.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/CrawlDbReducer.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/FetchScheduleFactory.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/Generator.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/Injector.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/LinkDb.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/LinkDbFilter.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/LinkDbMerger.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/LinkDbReader.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/MapWritable.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/SignatureFactory.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/crawl/URLPartitioner.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/fetcher/Fetcher.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/fetcher/OldFetcher.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/IndexerMapReduce.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/IndexingFilters.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/IndexingFiltersChecker.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrClean.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrDeleteDuplicates.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrIndexer.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrMappingReader.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrUtils.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/indexer/solr/SolrWriter.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/net/URLNormalizers.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/OutlinkExtractor.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParseOutputFormat.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParsePluginsReader.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParseResult.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParseSegment.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParseUtil.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParserChecker.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/parse/ParserFactory.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/plugin/PluginDescriptor.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/plugin/PluginManifestParser.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/plugin/PluginRepository.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/protocol/ProtocolFactory.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/LinkDumper.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/LinkRank.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/Loops.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/NodeDumper.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/ScoreUpdater.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/scoring/webgraph/WebGraph.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/segment/SegmentMergeFilters.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/segment/SegmentMerger.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/segment/SegmentReader.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/CrawlDBScanner.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/DmozParser.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/FreeGenerator.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/ResolveUrls.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/arc/ArcRecordReader.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/arc/ArcSegmentCreator.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/proxy/LogDebugHandler.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/proxy/SegmentHandler.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/tools/proxy/TestbedProxy.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/DeflateUtils.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/DomUtil.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/EncodingDetector.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/GZIPUtils.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/LogUtil.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/ObjectCache.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/domain/DomainStatistics.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/domain/DomainSuffixes.java /nutch/branches/branch-1.4/src/java/org/apache/nutch/util/domain/DomainSuffixesReader.java /nutch/branches/branch-1.4/src/plugin/creativecommons/src/java/org/creativecommons/nutch/CCIndexingFilter.java /nutch/branches/branch-1.4/src/plugin/creativecommons/src/java/org/creativecommons/nutch/CCParseFilter.java /nutch/branches/branch-1.4/src/plugin/feed/src/java/org/apache/nutch/parse/feed/FeedParser.java /nutch/branches/branch-1.4/src/plugin/feed/src/test/org/apache/nutch/parse/feed/TestFeedParser.java /nutch/branches/branch-1.4/src/plugin/index-anchor/src/java/org/apache/nutch/indexer/anchor/AnchorIndexingFilter.java /nutch/branches/branch-1.4/src/plugin/index-basic/src/java/org/apache/nutch/indexer/basic/BasicIndexingFilter.java /nutch/branches/branch-1.4/src/plugin/index-more/src/java/org/apache/nutch/indexer/more/MoreIndexingFilter.java /nutch/branches/branch-1.4/src/plugin/index-static/src/java/org/apache/nutch/indexer/staticfield/StaticFieldIndexer.java /nutch/branches/branch-1.4/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java /nutch/branches/branch-1.4/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/RobotRulesParser.java /nutch/branches/branch-1.4/src/plugin/lib-regex-filter/src/java/org/apache/nutch/urlfilter/api/RegexURLFilterBase.java /nutch/branches/branch-1.4/src/plugin/lib-regex-filter/src/test/org/apache/nutch/urlfilter/api/RegexURLFilterBaseTest.java /nutch/branches/branch-1.4/src/plugin/microformats-reltag/src/java/org/apache/nutch/microformats/reltag/RelTagParser.java /nutch/branches/branch-1.4/src/plugin/parse-ext/src/java/org/apache/nutch/parse/ext/ExtParser.java /nutch/branches/branch-1.4/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java /nutch/branches/branch-1.4/src/plugin/parse-js/src/java/org/apache/nutch/parse/js/JSParseFilter.java /nutch/branches/branch-1.4/src/plugin/parse-swf/src/java/org/apache/nutch/parse/swf/SWFParser.java /nutch/branches/branch-1.4/src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/TikaParser.java /nutch/branches/branch-1.4/src/plugin/parse-tika/src/test/org/apache/nutch/tika/TestFeedParser.java /nutch/branches/branch-1.4/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip/ZipParser.java /nutch/branches/branch-1.4/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip/ZipTextExtractor.java /nutch/branches/branch-1.4/src/plugin/protocol-file/src/java/org/apache/nutch/protocol/file/File.java /nutch/branches/branch-1.4/src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Ftp.java /nutch/branches/branch-1.4/src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/PrintCommandListener.java /nutch/branches/branch-1.4/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/Http.java /nutch/branches/branch-1.4/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/DummySSLProtocolSocketFactory.java /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/DummyX509TrustManager.java /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/Http.java /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/HttpAuthenticationFactory.java /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/HttpBasicAuthentication.java /nutch/branches/branch-1.4/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/HttpResponse.java /nutch/branches/branch-1.4/src/plugin/scoring-opic/src/java/org/apache/nutch/scoring/opic/OPICScoringFilter.java /nutch/branches/branch-1.4/src/plugin/subcollection/src/java/org/apache/nutch/collection/CollectionManager.java /nutch/branches/branch-1.4/src/plugin/subcollection/src/java/org/apache/nutch/indexer/subcollection/SubcollectionIndexingFilter.java /nutch/branches/branch-1.4/src/plugin/tld/src/java/org/apache/nutch/indexer/tld/TLDIndexingFilter.java /nutch/branches/branch-1.4/src/plugin/urlfilter-domain/src/java/org/apache/nutch/urlfilter/domain/DomainURLFilter.java /nutch/branches/branch-1.4/src/plugin/urlfilter-domain/src/test/org/apache/nutch/urlfilter/domain/TestDomainURLFilter.java /nutch/branches/branch-1.4/src/plugin/urlfilter-prefix/src/java/org/apache/nutch/urlfilter/prefix/PrefixURLFilter.java /nutch/branches/branch-1.4/src/plugin/urlfilter-suffix/src/java/org/apache/nutch/urlfilter/suffix/SuffixURLFilter.java /nutch/branches/branch-1.4/src/plugin/urlmeta/src/java/org/apache/nutch/indexer/urlmeta/URLMetaIndexingFilter.java /nutch/branches/branch-1.4/src/plugin/urlmeta/src/java/org/apache/nutch/scoring/urlmeta/URLMetaScoringFilter.java /nutch/branches/branch-1.4/src/plugin/urlnormalizer-basic/src/java/org/apache/nutch/net/urlnormalizer/basic/BasicURLNormalizer.java /nutch/branches/branch-1.4/src/plugin/urlnormalizer-regex/src/java/org/apache/nutch/net/urlnormalizer/regex/RegexURLNormalizer.java /nutch/branches/branch-1.4/src/plugin/urlnormalizer-regex/src/test/org/apache/nutch/net/urlnormalizer/regex/TestRegexURLNormalizer.java /nutch/branches/branch-1.4/src/test/org/apache/nutch/crawl/CrawlDBTestUtil.java
          Hide
          Lewis John McGibbney added a comment -

          yes Markus I'll address this ASAP, thanks.

          Show
          Lewis John McGibbney added a comment - yes Markus I'll address this ASAP, thanks.
          Hide
          Lewis John McGibbney added a comment -

          This is a pretty tricky case of the following

          only messages of type String are supported
          
          If one of your log statements contains a non-string object as its sole parameter, you will have to manually add a toString() method call on the object.
          
          For example,
          
          logger.debug(new Object());
          
          has to be manually re-written as
          
          logger.debug(new Object().toString()); 
          

          There are quite a few examples of this in fetcher.java I'll start working my way through them and get a patch submitted later today Markus.

          Show
          Lewis John McGibbney added a comment - This is a pretty tricky case of the following only messages of type String are supported If one of your log statements contains a non-string object as its sole parameter, you will have to manually add a toString() method call on the object. For example, logger.debug( new Object ()); has to be manually re-written as logger.debug( new Object ().toString()); There are quite a few examples of this in fetcher.java I'll start working my way through them and get a patch submitted later today Markus.
          Hide
          Markus Jelsma added a comment -

          Mmm, when i look at the stack trace it seems it's a problem with lib-http's getProtocolOutput call's to LogUtil.getErrorStream()

          Show
          Markus Jelsma added a comment - Mmm, when i look at the stack trace it seems it's a problem with lib-http's getProtocolOutput call's to LogUtil.getErrorStream()
          Hide
          Lewis John McGibbney added a comment -

          OK so this kinda opened up a couple of avenues. I am interested to see how we review and act on this one. Having discussed our problem with some slf4j users, the following seems to sum up the options.
          ----------------------------------
          The problem has to do with how LogUtil is implemented.
          Here are two possible solutions.

          1) Get rid of the LogUtil class. For example, in HttpBase, instead of

           } catch (Throwable e) {
            e.printStackTrace(LogUtil.getErrorStream(logger));
            return new ProtocolOutput(null, new ProtocolStatus(e));
           }
          

          write instead

           } catch (Throwable e) {
            logger.error("Failed to get protocol output", e);
          
            return new ProtocolOutput(null, new ProtocolStatus(e));
           }
          

          This latter form is the standard way of logging exceptions and this is the format some other Nutch code uses. I noticed this when I was creating the various patches.
          We would need to repeat the above operation in all places where LogUtil is used. There are about 60 such places located in about a dozen classes.

          2) In LogUtil, replace

          static {
           try {
            TRACE=Logger.class.getMethod("trace", new Class[] { Object.class });
            DEBUG=Logger.class.getMethod("debug", new Class[] { Object.class });
            INFO=Logger.class.getMethod("info",  new Class[] { Object.class });
           } catch ...
           }
          }
          

          with

          static {
           try {
            TRACE=Logger.class.getMethod("trace", new Class[] {String.class});
            DEBUG=Logger.class.getMethod("debug", new Class[] {String.class});
            INFO=Logger.class.getMethod("info",  new Class[] {String.class});
           } catch ...
           }
          }
          

          On line 104 change

           method.invoke(logger, new Object[] {toString().trim() });
          

          to

           method.invoke(logger, new String[] {toString().trim() });
          

          Further to this, a suggestion was raised that unless there is a good reason not to, it might be an idea to get rid of LogUtil in from the Nutch code base altogether as it seems like a very convoluted way for logging exceptions. I (and the kind folks from slf4j who I spoke with) by no means understand every aspect of why we currently use LogUtil, therefore I really need some opinions from you guys to sort this one out. Thanks

          Show
          Lewis John McGibbney added a comment - OK so this kinda opened up a couple of avenues. I am interested to see how we review and act on this one. Having discussed our problem with some slf4j users, the following seems to sum up the options. ---------------------------------- The problem has to do with how LogUtil is implemented. Here are two possible solutions. 1) Get rid of the LogUtil class. For example, in HttpBase, instead of } catch (Throwable e) { e.printStackTrace(LogUtil.getErrorStream(logger)); return new ProtocolOutput( null , new ProtocolStatus(e)); } write instead } catch (Throwable e) { logger.error( "Failed to get protocol output" , e); return new ProtocolOutput( null , new ProtocolStatus(e)); } This latter form is the standard way of logging exceptions and this is the format some other Nutch code uses. I noticed this when I was creating the various patches. We would need to repeat the above operation in all places where LogUtil is used. There are about 60 such places located in about a dozen classes. 2) In LogUtil, replace static { try { TRACE=Logger.class.getMethod( "trace" , new Class [] { Object .class }); DEBUG=Logger.class.getMethod( "debug" , new Class [] { Object .class }); INFO=Logger.class.getMethod( "info" , new Class [] { Object .class }); } catch ... } } with static { try { TRACE=Logger.class.getMethod( "trace" , new Class [] { String .class}); DEBUG=Logger.class.getMethod( "debug" , new Class [] { String .class}); INFO=Logger.class.getMethod( "info" , new Class [] { String .class}); } catch ... } } On line 104 change method.invoke(logger, new Object [] {toString().trim() }); to method.invoke(logger, new String [] {toString().trim() }); Further to this, a suggestion was raised that unless there is a good reason not to, it might be an idea to get rid of LogUtil in from the Nutch code base altogether as it seems like a very convoluted way for logging exceptions. I (and the kind folks from slf4j who I spoke with) by no means understand every aspect of why we currently use LogUtil, therefore I really need some opinions from you guys to sort this one out. Thanks
          Hide
          Lewis John McGibbney added a comment -

          is there any preference to how we are to address the recent issues as highlighted above? If we can get some ideas out then I can get cracking with a patch. Thanks

          Show
          Lewis John McGibbney added a comment - is there any preference to how we are to address the recent issues as highlighted above? If we can get some ideas out then I can get cracking with a patch. Thanks
          Hide
          Markus Jelsma added a comment -

          Marked as blocker due to described issues in fetcher.

          Show
          Markus Jelsma added a comment - Marked as blocker due to described issues in fetcher.
          Hide
          Julien Nioche added a comment -

          I had modified LogUtil in 2.0 (see http://svn.apache.org/viewvc/nutch/branches/nutchgora/src/java/org/apache/nutch/util/LogUtil.java?view=markup). A simple option would be to copy the version from 2.0 to trunk but getting rid of it altogether is fine too. Let's go for whatever is quickest, we can't have the trunk broken like this

          Show
          Julien Nioche added a comment - I had modified LogUtil in 2.0 (see http://svn.apache.org/viewvc/nutch/branches/nutchgora/src/java/org/apache/nutch/util/LogUtil.java?view=markup ). A simple option would be to copy the version from 2.0 to trunk but getting rid of it altogether is fine too. Let's go for whatever is quickest, we can't have the trunk broken like this
          Hide
          Lewis John McGibbney added a comment -

          OK I will be working on this later today and will hopefully have a patch sorted for tonight. Thanks

          Show
          Lewis John McGibbney added a comment - OK I will be working on this later today and will hopefully have a patch sorted for tonight. Thanks
          Hide
          Lewis John McGibbney added a comment -

          The attached patch changes the LogUtil class to accommodate the requirements of the slf4j framework.

          I ask if this could be tested and if necessary I will update other classes as per my earlier comments.

          At this stage in time my thinking is that we should separate the removal of LogUtil from this issue as they fundamentally, they mean different things for the Nutch codebase.

          Show
          Lewis John McGibbney added a comment - The attached patch changes the LogUtil class to accommodate the requirements of the slf4j framework. I ask if this could be tested and if necessary I will update other classes as per my earlier comments. At this stage in time my thinking is that we should separate the removal of LogUtil from this issue as they fundamentally, they mean different things for the Nutch codebase.
          Hide
          Markus Jelsma added a comment -

          This patch makes me happy, i can see the exception in the fetcher again. Bring it in!

          Show
          Markus Jelsma added a comment - This patch makes me happy, i can see the exception in the fetcher again. Bring it in!
          Hide
          Lewis John McGibbney added a comment -

          Committed @ revision 1177290 in trunk.

          This second commit for this issue fixes the exceptions error we were getting. As before, big thanks for the help and guidance.

          Show
          Lewis John McGibbney added a comment - Committed @ revision 1177290 in trunk. This second commit for this issue fixes the exceptions error we were getting. As before, big thanks for the help and guidance.
          Hide
          Hudson added a comment -

          Integrated in Nutch-trunk #1619 (See https://builds.apache.org/job/Nutch-trunk/1619/)
          commit to address NUTCH-1078 and update to changes.txt

          lewismc : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177290
          Files :

          • /nutch/trunk/CHANGES.txt
          • /nutch/trunk/src/java/org/apache/nutch/util/LogUtil.java
          Show
          Hudson added a comment - Integrated in Nutch-trunk #1619 (See https://builds.apache.org/job/Nutch-trunk/1619/ ) commit to address NUTCH-1078 and update to changes.txt lewismc : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177290 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/src/java/org/apache/nutch/util/LogUtil.java

            People

            • Assignee:
              Lewis John McGibbney
              Reporter:
              Lewis John McGibbney
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development