Uploaded image for project: 'Giraph'
  1. Giraph
  2. GIRAPH-153

HBase/Accumulo Input and Output formats

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.1.0
    • Fix Version/s: None
    • Component/s: bsp
    • Labels:
      None
    • Environment:

      Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB

      Description

      Four abstract classes that wrap their respective delegate input/output formats for
      easy hooks into vertex input format subclasses. I've included some sample programs that show two very simple graph
      algorithms. I have a graph generator that builds out a very simple directed structure, starting with a few 'root' nodes.

      Root nodes are defined as nodes which are not listed as a child anywhere in the graph.

      Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. Every vertex starts thinking it's a root. At superstep 0, send a message down to each
      child as a non-root notification. After superstep 1, only root nodes will have never been messaged.

      Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by bundling the notification logic followed by root node propagation. Once we've marked the appropriate nodes as roots, tell every child which roots it can be traced back to via one or more spanning trees. This will take N + 2 supersteps where N is the maximum number of hops from any root to any leaf, plus 2 supersteps for the initial root flagging.

      I've included all relevant code plus DistributedCacheHelper.java for recursive cache file and archive searches. It is more hadoop centric than giraph, but these jobs use it so I figured why not commit here.

      These have been tested through local JobRunner, pseudo-distributed on the aforementioned hardware, and full distributed on EC2. More details in the comments.

        Attachments

        1. GIRAPH-153.patch
          80 kB
          Brian Femiano
        2. GIRAPH-153.3.patch
          77 kB
          Brian Femiano
        3. GIRAPH-153.2.patch
          244 kB
          Brian Femiano
        4. GIRAPH-153.1.patch
          85 kB
          Brian Femiano

          Issue Links

            Activity

              People

              • Assignee:
                bfem Brian Femiano
                Reporter:
                bfem Brian Femiano
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: