Nutch
  1. Nutch
  2. NUTCH-1153

LinkRank must not log all hyperlinks

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      The LinkRank reducer log's all hyperlinks with the associated statistics at INFO level. This results in much increased I/O-wait. Level should be reduced to DEBUG.

      1. NUTCH-1153-1.5-2.patch
        2 kB
        Markus Jelsma
      2. NUTCH-1153-1.5-1.patch
        0.6 kB
        Markus Jelsma

        Activity

        Hide
        Markus Jelsma added a comment -

        Patch for trunk.

        Show
        Markus Jelsma added a comment - Patch for trunk.
        Hide
        Markus Jelsma added a comment -

        Final patch also disabled writing of _SUCCESS files by recent Hadoop versions.

        Show
        Markus Jelsma added a comment - Final patch also disabled writing of _SUCCESS files by recent Hadoop versions.
        Hide
        Markus Jelsma added a comment -

        Committed for 1.5 in rev. 1200344.

        Show
        Markus Jelsma added a comment - Committed for 1.5 in rev. 1200344.
        Hide
        Hudson added a comment -

        Integrated in nutch-trunk-maven #16 (See https://builds.apache.org/job/nutch-trunk-maven/16/)
        NUTCH-1153 LinkRank not to log all keys and not to write Hadoop _SUCCESS file

        markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200344
        Files :

        • /nutch/trunk/CHANGES.txt
        • /nutch/trunk/src/java/org/apache/nutch/scoring/webgraph/LinkRank.java
        Show
        Hudson added a comment - Integrated in nutch-trunk-maven #16 (See https://builds.apache.org/job/nutch-trunk-maven/16/ ) NUTCH-1153 LinkRank not to log all keys and not to write Hadoop _SUCCESS file markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200344 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/src/java/org/apache/nutch/scoring/webgraph/LinkRank.java
        Hide
        Hudson added a comment -

        Integrated in Nutch-trunk #1659 (See https://builds.apache.org/job/Nutch-trunk/1659/)
        NUTCH-1153 LinkRank not to log all keys and not to write Hadoop _SUCCESS file

        markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200344
        Files :

        • /nutch/trunk/CHANGES.txt
        • /nutch/trunk/src/java/org/apache/nutch/scoring/webgraph/LinkRank.java
        Show
        Hudson added a comment - Integrated in Nutch-trunk #1659 (See https://builds.apache.org/job/Nutch-trunk/1659/ ) NUTCH-1153 LinkRank not to log all keys and not to write Hadoop _SUCCESS file markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1200344 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/src/java/org/apache/nutch/scoring/webgraph/LinkRank.java
        Hide
        Hudson added a comment -

        Integrated in Nutch-trunk-ant #76 (See https://builds.apache.org/job/Nutch-trunk-ant/76/)
        NUTCH-1153 LinkRank not to log all keys and not to write Hadoop _SUCCESS file

        markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1200344
        Files :

        • /nutch/trunk/CHANGES.txt
        • /nutch/trunk/src/java/org/apache/nutch/scoring/webgraph/LinkRank.java
        Show
        Hudson added a comment - Integrated in Nutch-trunk-ant #76 (See https://builds.apache.org/job/Nutch-trunk-ant/76/ ) NUTCH-1153 LinkRank not to log all keys and not to write Hadoop _SUCCESS file markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1200344 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/src/java/org/apache/nutch/scoring/webgraph/LinkRank.java

          People

          • Assignee:
            Markus Jelsma
            Reporter:
            Markus Jelsma
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development