Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-1696

deep copy in the compaction scope iterators can throw off the stats

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.6.0
    • Component/s: tserver
    • Labels:
      None

      Description

      When application-level iterators deep copy the source iterator in a major compaction, the stats can be significantly off. We count two things in a major compaction:
      1. Entries read. This is done using a counting iterator sitting just above the system iterators.
      2. Entries written. This is done by counting the entries that are written to the RFile.
      Here's an example of what we see in the Accumulo logs:

      2013-09-06 11:53:31,371 [tabletserver.Compactor] DEBUG: Compaction k;row11;row10 20 read | 382,629 written |      3 entries/sec |  5.337 secs
      

      In this case, we're only counting 20 entries read, presumably because the iterators have been deep copied and the counting iterator that is being polled does not get a complete view of how many entries were read. Instead of 3 entries/sec we should have registered close to 72k entries/sec.

      To fix this, should we be counting all reads coming from any of the deep copies of the source iterators? This could be done by using a CountingIterator that keeps one counter for all deep copies. Thread-level counters could be used for lock-free counts in case multiple threads are ever used.

        Activity

        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit de24f8322f1ef2d4817b335ae90b131f0a7b2c1c in branch refs/heads/master from [~keith_turner]
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=de24f83 ]

        ACCUMULO-1696 fixed compaction debug counts for deep copies

        Show
        jira-bot ASF subversion and git services added a comment - Commit de24f8322f1ef2d4817b335ae90b131f0a7b2c1c in branch refs/heads/master from [~keith_turner] [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=de24f83 ] ACCUMULO-1696 fixed compaction debug counts for deep copies

          People

          • Assignee:
            kturner Keith Turner
            Reporter:
            afuchs Adam Fuchs
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development