Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2369

Create a new GraphGenerator Tool for writing Nutch Records as a Full Web Graph

    Details

    • Flags:
      Important

      Description

      I've been thinking for quite some time now that a new Tool which writes Nutch data out as full graph data would be an excellent addition to the codebase.

      My thoughts involves writing data using Tinkerpop's ScriptInputFormat and ScriptOutputFormat's to create Vertex objects representing Nutch Crawl Records.

      http://tinkerpop.apache.org/javadocs/current/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptInputFormat.html
      http://tinkerpop.apache.org/javadocs/current/full/index.html?org/apache/tinkerpop/gremlin/hadoop/structure/io/script/ScriptOutputFormat.html

      I envisage that each Vertex object would require the CrawlDB, LinkDB a Segment and possibly the HostDB in order to be fully populated. Graph characteristics e.g. Edge's would comes from those existing data structures as well.

      It is my intention to propose this as a GSoC project for 2017 and I have already talked offline with a potential student Omkar Reddy about him participating as the student.

      Essentially, if we were able to create a Graph enabling true traversal, this could be a game changer for how Nutch Crawl data is interpreted. It is my feeling that this issue most likely also involved an entire upgrade of the Hadoop API's from mapred to mapreduce for the master codebase.

        Activity

        Hide
        omkar20895 Omkar Reddy added a comment -

        Branch 1.x Lewis John McGibbney. Thanks.

        Show
        omkar20895 Omkar Reddy added a comment - Branch 1.x Lewis John McGibbney . Thanks.
        Hide
        lewismc Lewis John McGibbney added a comment -

        Omkar Reddy would you rather work on branch 2.x or 1.x?

        Show
        lewismc Lewis John McGibbney added a comment - Omkar Reddy would you rather work on branch 2.x or 1.x?

          People

          • Assignee:
            lewismc Lewis John McGibbney
            Reporter:
            lewismc Lewis John McGibbney
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development