Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0
    • Component/s: replication
    • Labels:
      None

      Description

      The use case here is where people have multiple data centers and need to replicate the data in between them. Accumulo can model this replication after the way that HBase currently handles the replication as detailed here (http://hbase.apache.org/replication.html).

      There will be one master Cluster and multiple slave clusters. Accumulo will use the Master-Push model to replicate the statements from the master clusters WAL to the various slaves WALs.

        Issue Links

        1.
        Define replication state machines for data Sub-task Resolved Josh Elser  
         
        2.
        Define ReplicaSystem interface Sub-task Resolved Josh Elser  
         
        3.
        Define Thrift IDL for data transfer Sub-task Resolved Josh Elser  
         
        4.
        Define storage data structure for data that needs replication Sub-task Resolved Josh Elser  
         
        5.
        Update GC to not remove files before replication has happened Sub-task Resolved Josh Elser  
         
        6.
        Modify "live ingest" code path to create replication entries Sub-task Resolved Josh Elser  
         
        7.
        Create service in Master to assign replication work Sub-task Resolved Josh Elser  
         
        8.
        Create insight to replication in the Monitor Sub-task Resolved Josh Elser  
         
        9.
        Create services that handles replication "receives" Sub-task Resolved Josh Elser  
         
        10.
        Provide authentication criteria between a master and slave Sub-task Resolved Josh Elser  
         
        11.
        Enable wire encryption for replication Sub-task Resolved Unassigned  
         
        12.
        Property additions for configuration of replication Sub-task Resolved Josh Elser  
         
        13.
        Don't try to assign permissions to !SYSTEM user when creating a table Sub-task Resolved Josh Elser  
         
        14.
        Create service in Master to manage replication records Sub-task Resolved Josh Elser  
         
        15.
        Create ReplicationOperations Sub-task Resolved Josh Elser  
         
        16.
        Provide WorkAssigner which is order-aware Sub-task Resolved Josh Elser  
         
        17.
        Create and configure a shell formatter for Status messages Sub-task Resolved Josh Elser  
         
        18.
        Immediately re-queue files which have more data to replicate Sub-task Resolved Josh Elser  
         
        19.
        Need to re-use DataInputStream for reading files that need replication Sub-task Resolved Josh Elser  
         
        20.
        Update user documentation Sub-task Resolved Josh Elser  
         
        21.
        Replication table should be user-controlled or live in accumulo namespace Sub-task Resolved Christopher Tubbs

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
         
        22.
        Clean up early draft versions of thrift replication classes Sub-task Resolved Josh Elser

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 10m
         

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          857d 11h 38m 1 Josh Elser 14/Jun/14 05:55
          Christopher Tubbs made changes -
          Link This issue is related to ACCUMULO-3737 [ ACCUMULO-3737 ]
          Hide
          ASF subversion and git services added a comment -

          Commit bc7b5ff3719ecda36c462dfbd75a9e0852d674fc in accumulo's branch refs/heads/master from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bc7b5ff ]

          ACCUMULO-378 Remove improper javadocs (no descriptions with tags)

          Show
          ASF subversion and git services added a comment - Commit bc7b5ff3719ecda36c462dfbd75a9e0852d674fc in accumulo's branch refs/heads/master from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bc7b5ff ] ACCUMULO-378 Remove improper javadocs (no descriptions with tags)
          Hide
          ASF subversion and git services added a comment -

          Commit ab3ac0a1c836e7322247175c6e1ce5dfda264db0 in accumulo's branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ab3ac0a ]

          ACCUMULO-378 remove unused import

          Show
          ASF subversion and git services added a comment - Commit ab3ac0a1c836e7322247175c6e1ce5dfda264db0 in accumulo's branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ab3ac0a ] ACCUMULO-378 remove unused import
          Josh Elser made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Josh Elser added a comment -

          a2aabbc merges this into upstream.

          Show
          Josh Elser added a comment - a2aabbc merges this into upstream.
          Hide
          ASF subversion and git services added a comment -

          Commit a2aabbcbbcb17e2ecccb71e58b531d3135b6ada0 in accumulo's branch refs/heads/master from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=a2aabbc ]

          ACCUMULO-378 Merge replication work into upstream.

          Show
          ASF subversion and git services added a comment - Commit a2aabbcbbcb17e2ecccb71e58b531d3135b6ada0 in accumulo's branch refs/heads/master from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=a2aabbc ] ACCUMULO-378 Merge replication work into upstream.
          Hide
          ASF subversion and git services added a comment -

          Commit 19bdff332fdb11431a8324450de7d8a23d79f545 in accumulo's branch refs/heads/master from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=19bdff3 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Show
          ASF subversion and git services added a comment - Commit 19bdff332fdb11431a8324450de7d8a23d79f545 in accumulo's branch refs/heads/master from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=19bdff3 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378
          Hide
          ASF subversion and git services added a comment -

          Commit f8831026a2a0a8e4be8c0ee6fcb9b296a78d2bee in accumulo's branch refs/heads/master from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f883102 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Conflicts:
          server/tserver/pom.xml

          Show
          ASF subversion and git services added a comment - Commit f8831026a2a0a8e4be8c0ee6fcb9b296a78d2bee in accumulo's branch refs/heads/master from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f883102 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378 Conflicts: server/tserver/pom.xml
          Hide
          ASF subversion and git services added a comment -

          Commit 264fad8eb304246214bb3eee4df981f92de9a2e3 in accumulo's branch refs/heads/master from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=264fad8 ]

          ACCUMULO-378 More test stabilization

          Show
          ASF subversion and git services added a comment - Commit 264fad8eb304246214bb3eee4df981f92de9a2e3 in accumulo's branch refs/heads/master from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=264fad8 ] ACCUMULO-378 More test stabilization
          Hide
          ASF subversion and git services added a comment -

          Commit f8831026a2a0a8e4be8c0ee6fcb9b296a78d2bee in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f883102 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Conflicts:
          server/tserver/pom.xml

          Show
          ASF subversion and git services added a comment - Commit f8831026a2a0a8e4be8c0ee6fcb9b296a78d2bee in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f883102 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378 Conflicts: server/tserver/pom.xml
          Hide
          ASF subversion and git services added a comment -

          Commit 264fad8eb304246214bb3eee4df981f92de9a2e3 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=264fad8 ]

          ACCUMULO-378 More test stabilization

          Show
          ASF subversion and git services added a comment - Commit 264fad8eb304246214bb3eee4df981f92de9a2e3 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=264fad8 ] ACCUMULO-378 More test stabilization
          Hide
          ASF subversion and git services added a comment -

          Commit 3a4cb95d82bfd52d8354f049d27368091421ab9d in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3a4cb95 ]

          ACCUMULO-378 More reviewboard changes.

          Make Combiner members internal again, removal of ReplicationTable
          in core (put table name in ReplicationConstants),

          Show
          ASF subversion and git services added a comment - Commit 3a4cb95d82bfd52d8354f049d27368091421ab9d in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3a4cb95 ] ACCUMULO-378 More reviewboard changes. Make Combiner members internal again, removal of ReplicationTable in core (put table name in ReplicationConstants),
          Hide
          ASF subversion and git services added a comment -

          Commit 8f53141334822045495e37a62b9a7f4685bbd4dc in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=8f53141 ]

          ACCUMULO-378 Test stabilization

          Show
          ASF subversion and git services added a comment - Commit 8f53141334822045495e37a62b9a7f4685bbd4dc in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=8f53141 ] ACCUMULO-378 Test stabilization
          Hide
          ASF subversion and git services added a comment -

          Commit 5365b550120224920deebb08e15dca879de95885 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5365b55 ]

          ACCUMULO-378 Test stabilization

          Show
          ASF subversion and git services added a comment - Commit 5365b550120224920deebb08e15dca879de95885 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5365b55 ] ACCUMULO-378 Test stabilization
          Hide
          ASF subversion and git services added a comment -

          Commit 97044d749890e8d08d015ef121c1a4b04f396735 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=97044d7 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Show
          ASF subversion and git services added a comment - Commit 97044d749890e8d08d015ef121c1a4b04f396735 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=97044d7 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378
          Hide
          ASF subversion and git services added a comment -

          Commit 028b78000a23409070e7c973731597ea69d846bc in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=028b780 ]

          ACCUMULO-378 Change plural property to match other convention

          Show
          ASF subversion and git services added a comment - Commit 028b78000a23409070e7c973731597ea69d846bc in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=028b780 ] ACCUMULO-378 Change plural property to match other convention
          Hide
          ASF subversion and git services added a comment -

          Commit 5257b80abb34d3843d9a6958e41f06979ea30139 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5257b80 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Show
          ASF subversion and git services added a comment - Commit 5257b80abb34d3843d9a6958e41f06979ea30139 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5257b80 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378
          Hide
          ASF subversion and git services added a comment -

          Commit 856f2359004f3b4ff209fc6ff61db76d4ab4c192 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=856f235 ]

          ACCUMULO-378 More test stabilization

          Show
          ASF subversion and git services added a comment - Commit 856f2359004f3b4ff209fc6ff61db76d4ab4c192 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=856f235 ] ACCUMULO-378 More test stabilization
          Hide
          ASF subversion and git services added a comment -

          Commit ada6ce464b1e9d818c06655369b30a45afa840c0 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ada6ce4 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Conflicts:
          server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java

          Show
          ASF subversion and git services added a comment - Commit ada6ce464b1e9d818c06655369b30a45afa840c0 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ada6ce4 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378 Conflicts: server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
          Hide
          ASF subversion and git services added a comment -

          Commit 49fc9855f996ae0f5b3cc20e03e77ea8f707d640 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=49fc985 ]

          ACCUMULO-378 Can't use '_' as the row separator for Order records.

          The ULongLexicoder might create bytes that actually equal the '_', which
          will mess up the splitting logic of the row key. Switch it to a \x00 instead
          and find the last instance of it.

          Show
          ASF subversion and git services added a comment - Commit 49fc9855f996ae0f5b3cc20e03e77ea8f707d640 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=49fc985 ] ACCUMULO-378 Can't use '_' as the row separator for Order records. The ULongLexicoder might create bytes that actually equal the '_', which will mess up the splitting logic of the row key. Switch it to a \x00 instead and find the last instance of it.
          Hide
          ASF subversion and git services added a comment -

          Commit 0ff0e021d3fc95794137dfeb3f6e1335b61b0a16 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=0ff0e02 ]

          ACCUMULO-378 Resize the threadpool used for sending data by checking the configuration periodically

          Use the SimpleTimer to just schedule a check of the configuration
          to see what the value is for the maximum size of the threadpool which
          is used with the DistributedWorkQueue and ReplicationProcessor.

          Show
          ASF subversion and git services added a comment - Commit 0ff0e021d3fc95794137dfeb3f6e1335b61b0a16 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=0ff0e02 ] ACCUMULO-378 Resize the threadpool used for sending data by checking the configuration periodically Use the SimpleTimer to just schedule a check of the configuration to see what the value is for the maximum size of the threadpool which is used with the DistributedWorkQueue and ReplicationProcessor.
          Hide
          ASF subversion and git services added a comment -

          Commit db10cfe26823f5a31b516070521f54e5b890fb7b in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=db10cfe ]

          ACCUMULO-378 Allow for dynamic reconfiguration of the WorkAssigner impl

          Show
          ASF subversion and git services added a comment - Commit db10cfe26823f5a31b516070521f54e5b890fb7b in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=db10cfe ] ACCUMULO-378 Allow for dynamic reconfiguration of the WorkAssigner impl
          Hide
          ASF subversion and git services added a comment -

          Commit 58fbf1438d2cd82133ca0d559dfb22d8b8f55d7f in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=58fbf14 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Conflicts:
          core/src/main/java/org/apache/accumulo/core/conf/Property.java

          Show
          ASF subversion and git services added a comment - Commit 58fbf1438d2cd82133ca0d559dfb22d8b8f55d7f in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=58fbf14 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378 Conflicts: core/src/main/java/org/apache/accumulo/core/conf/Property.java
          Hide
          ASF subversion and git services added a comment -

          Commit 1f0ee9c5027714997a04c5cacf4ef92fff27ecc1 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1f0ee9c ]

          ACCUMULO-378 Set a more realistic maximum size on the distributed work queue for work assignment.

          Show
          ASF subversion and git services added a comment - Commit 1f0ee9c5027714997a04c5cacf4ef92fff27ecc1 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1f0ee9c ] ACCUMULO-378 Set a more realistic maximum size on the distributed work queue for work assignment.
          Hide
          ASF subversion and git services added a comment -

          Commit 27905426f5a0a3f5c140d1ae1b0c00497e6ad0bb in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2790542 ]

          ACCUMULO-378 Try to reduce the spam in the logs when the peer is unavailable.

          Increase the sleep time at a linear scale when we fail to talk to the replication
          coordinator. Eventually, bail out completely and let the replication process
          take over again.

          Show
          ASF subversion and git services added a comment - Commit 27905426f5a0a3f5c140d1ae1b0c00497e6ad0bb in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2790542 ] ACCUMULO-378 Try to reduce the spam in the logs when the peer is unavailable. Increase the sleep time at a linear scale when we fail to talk to the replication coordinator. Eventually, bail out completely and let the replication process take over again.
          Hide
          ASF subversion and git services added a comment -

          Commit 5e8d6d2c7fdfd41a99ab812500eaf92661ba8481 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5e8d6d2 ]

          ACCUMULO-378 Removing experimental annotation.

          In talking to Christopher Tubbs on the subject, he stated that the intent of Experimental
          was for features that are incomplete or not expected to fully work. Replication is
          not one of those features (it is tested and expected to work with known functionality),
          therefore I'm removing the experimental annotation.

          Show
          ASF subversion and git services added a comment - Commit 5e8d6d2c7fdfd41a99ab812500eaf92661ba8481 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5e8d6d2 ] ACCUMULO-378 Removing experimental annotation. In talking to Christopher Tubbs on the subject, he stated that the intent of Experimental was for features that are incomplete or not expected to fully work. Replication is not one of those features (it is tested and expected to work with known functionality), therefore I'm removing the experimental annotation.
          Hide
          ASF subversion and git services added a comment -

          Commit 03d57520a9668fb0d82b62708096ce6b595b0cdc in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=03d5752 ]

          ACCUMULO-378 Jenkins fix – retry if we get a security exception because the grant on the replication table didn't happen yet

          Show
          ASF subversion and git services added a comment - Commit 03d57520a9668fb0d82b62708096ce6b595b0cdc in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=03d5752 ] ACCUMULO-378 Jenkins fix – retry if we get a security exception because the grant on the replication table didn't happen yet
          Hide
          ASF subversion and git services added a comment -

          Commit 070ceb1dacd53ed169ee48af7b72b0788220a941 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=070ceb1 ]

          ACCUMULO-378 (re)Create the UnorderedWorkAssigner which doesn't care about replaying files in any order

          Increase the default size of the threadpool too.

          Show
          ASF subversion and git services added a comment - Commit 070ceb1dacd53ed169ee48af7b72b0788220a941 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=070ceb1 ] ACCUMULO-378 (re)Create the UnorderedWorkAssigner which doesn't care about replaying files in any order Increase the default size of the threadpool too.
          Hide
          ASF subversion and git services added a comment -

          Commit fa18d9dcf9d6cd052c8b17403e95fc36fd347f72 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=fa18d9d ]

          ACCUMULO-378 More consumable label on trace data

          Show
          ASF subversion and git services added a comment - Commit fa18d9dcf9d6cd052c8b17403e95fc36fd347f72 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=fa18d9d ] ACCUMULO-378 More consumable label on trace data
          Hide
          ASF subversion and git services added a comment -

          Commit 092e22ea5509ab8961a80de682687da503bec30a in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=092e22e ]

          ACCUMULO-378 Try to get the tracing working for tservers sending data

          Show
          ASF subversion and git services added a comment - Commit 092e22ea5509ab8961a80de682687da503bec30a in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=092e22e ] ACCUMULO-378 Try to get the tracing working for tservers sending data
          Hide
          ASF subversion and git services added a comment -

          Commit 3a619ffe08d8d90432218a3138faa572017b1f06 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3a619ff ]

          ACCUMULO-378 Add tracing into the AccumuloReplicaSystem.

          Show
          ASF subversion and git services added a comment - Commit 3a619ffe08d8d90432218a3138faa572017b1f06 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3a619ff ] ACCUMULO-378 Add tracing into the AccumuloReplicaSystem.
          Hide
          ASF subversion and git services added a comment -

          Commit 9d9b5ed24f3e425459108a993ab2cea121d1b612 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9d9b5ed ]

          ACCUMULO-378 Test class consolidation

          Show
          ASF subversion and git services added a comment - Commit 9d9b5ed24f3e425459108a993ab2cea121d1b612 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9d9b5ed ] ACCUMULO-378 Test class consolidation
          Hide
          ASF subversion and git services added a comment -

          Commit 3b727cf94f4341d39adf07dcbd62361b7d4a0de3 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3b727cf ]

          ACCUMULO-378 Make the MockReplicaSystem a bit more honest and only set full replication when closed and inf length

          Show
          ASF subversion and git services added a comment - Commit 3b727cf94f4341d39adf07dcbd62361b7d4a0de3 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3b727cf ] ACCUMULO-378 Make the MockReplicaSystem a bit more honest and only set full replication when closed and inf length
          Hide
          ASF subversion and git services added a comment -

          Commit db3818b180f8611a80005a750b534a477ffb442d in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=db3818b ]

          ACCUMULO-378 Raise the limit back up to 50M which is the default.

          I didn't realized when I was having heap issues that I was running
          with a 1G heap.

          Show
          ASF subversion and git services added a comment - Commit db3818b180f8611a80005a750b534a477ffb442d in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=db3818b ] ACCUMULO-378 Raise the limit back up to 50M which is the default. I didn't realized when I was having heap issues that I was running with a 1G heap.
          Hide
          ASF subversion and git services added a comment -

          Commit 842c632f992faf6080b0f4a466e63c43308c05cf in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=842c632 ]

          ACCUMULO-378 Ensure that the peer MAC is shutdown

          Show
          ASF subversion and git services added a comment - Commit 842c632f992faf6080b0f4a466e63c43308c05cf in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=842c632 ] ACCUMULO-378 Ensure that the peer MAC is shutdown
          Hide
          ASF subversion and git services added a comment -

          Commit 327b0abc0dfcdf17422effd2f797fcfc519b6373 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=327b0ab ]

          ACCUMULO-378 Better logging to whom the master is connecting

          Show
          ASF subversion and git services added a comment - Commit 327b0abc0dfcdf17422effd2f797fcfc519b6373 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=327b0ab ] ACCUMULO-378 Better logging to whom the master is connecting
          Hide
          ASF subversion and git services added a comment -

          Commit 4288bbe442394af8512cd0d801dbb98d76d10b46 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4288bbe ]

          ACCUMULO-378 Need to turn on Trace for traces to be logged.

          Show
          ASF subversion and git services added a comment - Commit 4288bbe442394af8512cd0d801dbb98d76d10b46 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4288bbe ] ACCUMULO-378 Need to turn on Trace for traces to be logged.
          Josh Elser made changes -
          Component/s replication [ 12322902 ]
          Hide
          ASF subversion and git services added a comment -

          Commit 59177233fd903d5c69592c67e603c58bc2a0ed2a in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5917723 ]

          ACCUMULO-378 When re-syncing to where we left off on reads, we need to track all tids for our table

          Fixes an issue where when the DEFINE_TABLET wasn't contained in the batch
          of log entries that we were reading, we ignored all of the mutations.When
          we read past all of the old data, we still need to track the tids
          for the table which we're replicating.

          Show
          ASF subversion and git services added a comment - Commit 59177233fd903d5c69592c67e603c58bc2a0ed2a in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5917723 ] ACCUMULO-378 When re-syncing to where we left off on reads, we need to track all tids for our table Fixes an issue where when the DEFINE_TABLET wasn't contained in the batch of log entries that we were reading, we ignored all of the mutations.When we read past all of the old data, we still need to track the tids for the table which we're replicating.
          Hide
          ASF subversion and git services added a comment -

          Commit abea3c6afb06a7abaaba8bc92698dc25fdf12276 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=abea3c6 ]

          ACCUMULO-378 Increase the timeout as jenkins failed here writing the data

          Show
          ASF subversion and git services added a comment - Commit abea3c6afb06a7abaaba8bc92698dc25fdf12276 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=abea3c6 ] ACCUMULO-378 Increase the timeout as jenkins failed here writing the data
          Hide
          ASF subversion and git services added a comment -

          Commit 3243d2ff9209246e7b03453460dfd4a3f231b190 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3243d2f ]

          ACCUMULO-378 Lower the batchwriter "batch" size, and make it configurable.

          Show
          ASF subversion and git services added a comment - Commit 3243d2ff9209246e7b03453460dfd4a3f231b190 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3243d2f ] ACCUMULO-378 Lower the batchwriter "batch" size, and make it configurable.
          Hide
          ASF subversion and git services added a comment -

          Commit da0a228d861952979b1dcc3bb561e2860719203a in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=da0a228 ]

          ACCUMULO-378 Use the system configuration, not site configuration so we pull from zk too

          Show
          ASF subversion and git services added a comment - Commit da0a228d861952979b1dcc3bb561e2860719203a in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=da0a228 ] ACCUMULO-378 Use the system configuration, not site configuration so we pull from zk too
          Hide
          Josh Elser added a comment -

          Made a review with the current progress. "Big" things still missing are: bulk import replication, authorization/authentication to peers, encryption of data sent to peers, and monitor integration.

          I'll be closing out the sub-tasks that I believe are done or sufficient for a first implementation.

          Show
          Josh Elser added a comment - Made a review with the current progress. "Big" things still missing are: bulk import replication, authorization/authentication to peers, encryption of data sent to peers, and monitor integration. I'll be closing out the sub-tasks that I believe are done or sufficient for a first implementation.
          Josh Elser made changes -
          Remote Link This issue links to "Implementation ReviewBoard (Web Link)" [ 15127 ]
          Hide
          ASF subversion and git services added a comment -

          Commit 4ac04b95d7f1919a35af2d9e60d80a2c9b94074c in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ac04b9 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Show
          ASF subversion and git services added a comment - Commit 4ac04b95d7f1919a35af2d9e60d80a2c9b94074c in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4ac04b9 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378
          Hide
          ASF subversion and git services added a comment -

          Commit 178ffe977027c62629fb46337eee02312114a048 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=178ffe9 ]

          ACCUMULO-378 Remove compiler warnings and unused classes, resolve formatting issues

          Show
          ASF subversion and git services added a comment - Commit 178ffe977027c62629fb46337eee02312114a048 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=178ffe9 ] ACCUMULO-378 Remove compiler warnings and unused classes, resolve formatting issues
          Hide
          ASF subversion and git services added a comment -

          Commit 417b0b3328f7ad81bcde36a00e15091822f32b22 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=417b0b3 ]

          ACCUMULO-378 Remove service port configuration in favor of random

          Show
          ASF subversion and git services added a comment - Commit 417b0b3328f7ad81bcde36a00e15091822f32b22 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=417b0b3 ] ACCUMULO-378 Remove service port configuration in favor of random
          Hide
          ASF subversion and git services added a comment -

          Commit 53e06a2590d7cc4bb4da373f3154a5cfbbc6a9bd in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=53e06a2 ]

          ACCUMULO-378 Metadata table has the versioning iterator set at 10.

          Someone is a terrible person and set the versioning iterator on the metadata
          table to 10 instead of 20 like it is for literally every other table.

          Show
          ASF subversion and git services added a comment - Commit 53e06a2590d7cc4bb4da373f3154a5cfbbc6a9bd in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=53e06a2 ] ACCUMULO-378 Metadata table has the versioning iterator set at 10. Someone is a terrible person and set the versioning iterator on the metadata table to 10 instead of 20 like it is for literally every other table.
          Hide
          ASF subversion and git services added a comment -

          Commit 2817f42d4d2758b8fe8b9789d992a2ae06ec5486 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2817f42 ]

          ACCUMULO-378 Utility to just print the replication records repeatedly

          Show
          ASF subversion and git services added a comment - Commit 2817f42d4d2758b8fe8b9789d992a2ae06ec5486 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2817f42 ] ACCUMULO-378 Utility to just print the replication records repeatedly
          Hide
          ASF subversion and git services added a comment -

          Commit f2c0c16db3f37d630f0616aabe3a18f7deee64a5 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f2c0c16 ]

          Merge remote-tracking branch 'origin/master' into ACCUMULO-378

          Show
          ASF subversion and git services added a comment - Commit f2c0c16db3f37d630f0616aabe3a18f7deee64a5 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f2c0c16 ] Merge remote-tracking branch 'origin/master' into ACCUMULO-378
          Hide
          ASF subversion and git services added a comment -

          Commit 440d7fcd498eb48a5e665dcb73ad795f3ba547cc in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=440d7fc ]

          ACCUMULO-378 Tune down some aggressive logging

          Show
          ASF subversion and git services added a comment - Commit 440d7fcd498eb48a5e665dcb73ad795f3ba547cc in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=440d7fc ] ACCUMULO-378 Tune down some aggressive logging
          Hide
          Josh Elser added a comment -

          For those who care or want to watch progress, to keep from spamming everyone on the commits@a.a.o list with duplicate commits, I've been pushing my work to my github. When I get to something cohesive, I'll work on merging it back into the asf branch in preparation for a merge into the main tree.

          Show
          Josh Elser added a comment - For those who care or want to watch progress, to keep from spamming everyone on the commits@a.a.o list with duplicate commits, I've been pushing my work to my github . When I get to something cohesive, I'll work on merging it back into the asf branch in preparation for a merge into the main tree.
          Hide
          ASF subversion and git services added a comment -

          Commit 5fd07ec03059daa21758404de0c059a2dd5c395a in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5fd07ec ]

          Merge branch 'master' into ACCUMULO-378

          Show
          ASF subversion and git services added a comment - Commit 5fd07ec03059daa21758404de0c059a2dd5c395a in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=5fd07ec ] Merge branch 'master' into ACCUMULO-378
          Hide
          ASF subversion and git services added a comment -

          Commit de7f591ab6f818e1967f0d4d0e266a803b6f086d in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=de7f591 ]

          ACCUMULO-378 Add details on replication "bookkeeping" on the master cluster.

          Show
          ASF subversion and git services added a comment - Commit de7f591ab6f818e1967f0d4d0e266a803b6f086d in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=de7f591 ] ACCUMULO-378 Add details on replication "bookkeeping" on the master cluster.
          Hide
          ASF subversion and git services added a comment -

          Commit 13561ebbb7480c18df3538c1eed04e8f218cfca2 in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=13561eb ]

          ACCUMULO-378 Design document with first round of changes.

          Show
          ASF subversion and git services added a comment - Commit 13561ebbb7480c18df3538c1eed04e8f218cfca2 in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=13561eb ] ACCUMULO-378 Design document with first round of changes.
          Josh Elser made changes -
          Remote Link This issue links to "Original design doc link (Web Link)" [ 14768 ]
          Josh Elser made changes -
          Remote Link This issue links to "Design document review (Web Link)" [ 14742 ]
          Hide
          Josh Elser added a comment -

          New reviewboard that I own (instead of Keith) which will let me administer it better. We can leave the other reviewboard up to continue discussion there, but please start new discussion on this link.

          Show
          Josh Elser added a comment - New reviewboard that I own (instead of Keith) which will let me administer it better. We can leave the other reviewboard up to continue discussion there, but please start new discussion on this link.
          Josh Elser made changes -
          Remote Link This issue links to "Active Design Doc Review (Web Link)" [ 14767 ]
          Keith Turner made changes -
          Remote Link This issue links to "Design document review (Web Link)" [ 14742 ]
          Hide
          Josh Elser added a comment -

          FYI, I plan on starting to break down things into sub-tasks that can be worked on that (hopefully) are disjoint. Meanwhile, any feedback is welcome – although starting a thread on dev@a.a.o is likely better than doing it here.

          Show
          Josh Elser added a comment - FYI, I plan on starting to break down things into sub-tasks that can be worked on that (hopefully) are disjoint. Meanwhile, any feedback is welcome – although starting a thread on dev@a.a.o is likely better than doing it here.
          Hide
          Josh Elser added a comment -

          Design document that I've been working out that outlines some implementation details.

          Show
          Josh Elser added a comment - Design document that I've been working out that outlines some implementation details.
          Josh Elser made changes -
          Remote Link This issue links to "Design Document (Web Link)" [ 14726 ]
          Josh Elser made changes -
          Fix Version/s 1.7.0 [ 12324607 ]
          Josh Elser made changes -
          Assignee Josh Elser [ elserj ]
          Hide
          Josh Elser added a comment -

          Keith Turner, Sapan Shah, did you guys ever come up with any sort of design document? Looking back at the last chatter, we were still in a localfs WAL capability which is a bit out of date considering current Accumulo support

          Given Ravi Mutyala's question on dev@a.a.o about this, any interest in thinking about this as a major 1.7 feature? It will give something to think about while testing 1.6.0

          Show
          Josh Elser added a comment - Keith Turner , Sapan Shah , did you guys ever come up with any sort of design document? Looking back at the last chatter, we were still in a localfs WAL capability which is a bit out of date considering current Accumulo support Given Ravi Mutyala 's question on dev@a.a.o about this, any interest in thinking about this as a major 1.7 feature? It will give something to think about while testing 1.6.0
          Sapan Shah made changes -
          Assignee Sapan Shah [ sapanbshah42 ]
          Hide
          Jeff Whiting added a comment -

          While thinking about replication master - master replication should also be considered as it can have a large implications on how the replication is implemented.

          Show
          Jeff Whiting added a comment - While thinking about replication master - master replication should also be considered as it can have a large implications on how the replication is implemented.
          Gavin made changes -
          Field Original Value New Value
          Workflow no-reopen-closed, patch-avail [ 12652377 ] patch-available, re-open possible [ 12671648 ]
          Hide
          Keith Turner added a comment -

          We were discussing generating secondary indexes. This feature may be useful for that in addition to replicating to a remote cluster. So instead of replicating data to a remote cluster, replicate to another table on the local cluster with a data transformation step. For example, data is inserted in table A, then the mutations from table A get pushed to table B with a transformation step. This could also push bulk imports to table B and through the transformation.

          Show
          Keith Turner added a comment - We were discussing generating secondary indexes. This feature may be useful for that in addition to replicating to a remote cluster. So instead of replicating data to a remote cluster, replicate to another table on the local cluster with a data transformation step. For example, data is inserted in table A, then the mutations from table A get pushed to table B with a transformation step. This could also push bulk imports to table B and through the transformation.
          Hide
          Keith Turner added a comment -

          Sapan and I were discussing this issue. We were considering the use case were a user wants to filter some data in a table. To do this they may add filter, force a compaction, and then remove the filter. It would be nice to have this action replicate to the backup cluster. This may be easier if the action were more atomic, see ACCUMULO-420.

          Show
          Keith Turner added a comment - Sapan and I were discussing this issue. We were considering the use case were a user wants to filter some data in a table. To do this they may add filter, force a compaction, and then remove the filter. It would be nice to have this action replicate to the backup cluster. This may be easier if the action were more atomic, see ACCUMULO-420 .
          Hide
          Keith Turner added a comment -

          Replicating all of zookeeper would not work well, would not want to replicate info related to the root tablet location, tablet servers, loggers, and FATE operations from the master cluster. ZOOKEEPER-892 mentions the ability to replicate a sub-tree.

          Show
          Keith Turner added a comment - Replicating all of zookeeper would not work well, would not want to replicate info related to the root tablet location, tablet servers, loggers, and FATE operations from the master cluster. ZOOKEEPER-892 mentions the ability to replicate a sub-tree.
          Hide
          Keith Turner added a comment -

          Replicating table configuration would be useful. For example if a user enables an age off iterator on the master cluster for major compaction, it would be nice to have that run on the slave cluster and throw old data away. Would want the same iterators configured for the master and slave table, compression, locality groups, etc. Wonder if we could leverage ZOOKEEPER-892.

          Show
          Keith Turner added a comment - Replicating table configuration would be useful. For example if a user enables an age off iterator on the master cluster for major compaction, it would be nice to have that run on the slave cluster and throw old data away. Would want the same iterators configured for the master and slave table, compression, locality groups, etc. Wonder if we could leverage ZOOKEEPER-892 .
          Hide
          Sapan Shah added a comment -

          John: I am currently adapting the WAL to append to a cloned copy in HDFS while still being performant.

          Keith:

          I think collaborating would be a great idea. I'll work on getting design document together. I will be at the meetup, so we can discuss there the various tasks to work on for this. I see there being quite a bit.

          For the questions you asked.
          1) To begin with I was thinking about maybe doing just select tables so that you did not have complete replicas. Then maybe working on a way to possibly do total replicas.
          2) I am still working out a good way to have ZooKeeper send the updates for the user information. I am not sure about the table metadata yet, as if all we are doing is calling the client API, I think that might be taken care of, shouldn't it? As the slave table will maintain its own metadata.
          3) What you described with cloning the table, copying the data, and replicating the logs was my current plan.
          4) I have not looked into FATE that much, but will check it out.
          5) I am not sure about replicating the splits unless the user defined the splits before hand.

          Let me check into FATE, but from the skimming it seems really useful for this.

          Show
          Sapan Shah added a comment - John: I am currently adapting the WAL to append to a cloned copy in HDFS while still being performant. Keith: I think collaborating would be a great idea. I'll work on getting design document together. I will be at the meetup, so we can discuss there the various tasks to work on for this. I see there being quite a bit. For the questions you asked. 1) To begin with I was thinking about maybe doing just select tables so that you did not have complete replicas. Then maybe working on a way to possibly do total replicas. 2) I am still working out a good way to have ZooKeeper send the updates for the user information. I am not sure about the table metadata yet, as if all we are doing is calling the client API, I think that might be taken care of, shouldn't it? As the slave table will maintain its own metadata. 3) What you described with cloning the table, copying the data, and replicating the logs was my current plan. 4) I have not looked into FATE that much, but will check it out. 5) I am not sure about replicating the splits unless the user defined the splits before hand. Let me check into FATE, but from the skimming it seems really useful for this.
          Hide
          Keith Turner added a comment -

          I would like to collaborate w/ you on this. It seems like a starting point might be a design doc. Would you mind putting together a design doc detailing your thoughts on this? Any other suggestions on how we could collaborate? We could also meet at the meetup (http://www.meetup.com/Accumulo-Users-DC/events/45491582/) if you are in this area.

          Show
          Keith Turner added a comment - I would like to collaborate w/ you on this. It seems like a starting point might be a design doc. Would you mind putting together a design doc detailing your thoughts on this? Any other suggestions on how we could collaborate? We could also meet at the meetup ( http://www.meetup.com/Accumulo-Users-DC/events/45491582/ ) if you are in this area.
          Hide
          jv added a comment -

          I need a bit of clarification- are you adapting the WAL to log to HDFS via appends or are you working on a mechanism to shove the logs into HDFS once they are complete?

          Show
          jv added a comment - I need a bit of clarification- are you adapting the WAL to log to HDFS via appends or are you working on a mechanism to shove the logs into HDFS once they are complete?
          Hide
          Keith Turner added a comment -

          This sounds really cool. I looked at the HBase doc, it seems like it replays the walogs on the slave cluster through the client API.

          Where you thinking of doing this for all tables, or just select tables?
          What are your thoughts on replicating user and table metadata in zookeeper?
          What are your thoughts on enabling replication for existing data? (we clould clone the table, copy its existing data, and replicate new walogs created after the clone operation).
          How are you thinking of handling bulk imported data? (could possible copy to slave and bulk import on their also, this could be a FATE operation initiated by the bulk import FATE operation).
          What are your thoughts on replicating split and merge operations on the master cluster?

          I am wondering how much we can leverage FATE to make this easier and more reliable.

          Show
          Keith Turner added a comment - This sounds really cool. I looked at the HBase doc, it seems like it replays the walogs on the slave cluster through the client API. Where you thinking of doing this for all tables, or just select tables? What are your thoughts on replicating user and table metadata in zookeeper? What are your thoughts on enabling replication for existing data? (we clould clone the table, copy its existing data, and replicate new walogs created after the clone operation). How are you thinking of handling bulk imported data? (could possible copy to slave and bulk import on their also, this could be a FATE operation initiated by the bulk import FATE operation). What are your thoughts on replicating split and merge operations on the master cluster? I am wondering how much we can leverage FATE to make this easier and more reliable.
          Hide
          Sapan Shah added a comment -

          I have started some basic work on this, such as working on trying to get the WAL working on HDFS.

          Show
          Sapan Shah added a comment - I have started some basic work on this, such as working on trying to get the WAL working on HDFS.
          Sapan Shah created issue -

            People

            • Assignee:
              Josh Elser
              Reporter:
              Sapan Shah
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h
                1h

                  Development