Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0
    • Component/s: replication
    • Labels:
      None

      Description

      The current WorkAssigner implementation, which uses the DistributedWorkQueue, is great because it allows the Master to be unaware of what tservers are available, and to allow any tserver to perform the replication.

      The downside of this is that it is possible to replicate data that was ingested later before the earlier ingested data. For example, say table1 uses wal1 to ingest some data. We record that wal1 has some replication to do, but, for whatever reason, we don't get to it. More data is ingested into table1, and it starts using wal2 after enough data was ingested. Now, we have wal1 and wal2 which both have data to be replicated for table1.

      Using the DistributedWorkQueue, we have no guarantee that wal1 will be replicated before wal2, which means we might replay a column update for the same row in the wrong order (update from wal2 and then update from wal1).

      While the DistributedWorkQueue is nice for the mentioned reason, in addition to the higher throughput, it has obvious deficiencies depending on the workload and table schema. We need to create a WorkAssigner that is order aware (what was the order in which the WALs for a table were minor compacted, and ensure that replication occurs in that same order.

        Activity

        Hide
        ASF subversion and git services added a comment -

        Commit 50117b35535f7701ebde58f089489dd8dbcd432d in accumulo's branch refs/heads/ACCUMULO-378 from Josh Elser
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=50117b3 ]

        ACCUMULO-2819 Add a test to ensure that an inprogress file that's fully replicated is removed.

        This is mainly an optimization as it would be cleaned up outside of the
        work assignment loop, but it should help to make a more responsive work assignment loop.

        Show
        ASF subversion and git services added a comment - Commit 50117b35535f7701ebde58f089489dd8dbcd432d in accumulo's branch refs/heads/ ACCUMULO-378 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=50117b3 ] ACCUMULO-2819 Add a test to ensure that an inprogress file that's fully replicated is removed. This is mainly an optimization as it would be cleaned up outside of the work assignment loop, but it should help to make a more responsive work assignment loop.

          People

          • Assignee:
            Josh Elser
            Reporter:
            Josh Elser
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development