Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: tserver
    • Labels:
      None

      Description

      It would be nice to have extensible triggers for determining when to run compactions. This can facilitate better ways to age-off data, for example.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e00e395919198cc5f6478d553f5da3319defaa45 in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e00e395 ]

          ACCUMULO-1451 add functional test for configurable compactions

          Show
          jira-bot ASF subversion and git services added a comment - Commit e00e395919198cc5f6478d553f5da3319defaa45 in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e00e395 ] ACCUMULO-1451 add functional test for configurable compactions
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b698b4001165b8c4ff3cedc4005bcc4a38c8242a in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=b698b40 ]

          ACCUMULO-1451 fix inadvertent commit that leaves test folder around after the test

          Show
          jira-bot ASF subversion and git services added a comment - Commit b698b4001165b8c4ff3cedc4005bcc4a38c8242a in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=b698b40 ] ACCUMULO-1451 fix inadvertent commit that leaves test folder around after the test
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit dfbc8b8f5ad5e69b47f74782684ea114491e0b56 in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=dfbc8b8 ]

          ACCUMULO-1451 remove stack traces from message

          Show
          jira-bot ASF subversion and git services added a comment - Commit dfbc8b8f5ad5e69b47f74782684ea114491e0b56 in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=dfbc8b8 ] ACCUMULO-1451 remove stack traces from message
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a0e024451bdaea6f22706f3a38944778a94e539b in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=a0e0244 ]

          ACCUMULO-1451 fix endless false major compaction

          Show
          jira-bot ASF subversion and git services added a comment - Commit a0e024451bdaea6f22706f3a38944778a94e539b in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=a0e0244 ] ACCUMULO-1451 fix endless false major compaction
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f57ca2b9e1c032bb266adf9db2487189f690d855 in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f57ca2b ]

          ACCUMULO-1451 added strategy-specific configuration, documented the strategy, and added a specific call for shouldCompact

          Show
          jira-bot ASF subversion and git services added a comment - Commit f57ca2b9e1c032bb266adf9db2487189f690d855 in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f57ca2b ] ACCUMULO-1451 added strategy-specific configuration, documented the strategy, and added a specific call for shouldCompact
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b90710296fe5393c55bdcd1a8187971e477d405f in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=b907102 ]

          ACCUMULO-1451 cleanup imports

          Show
          jira-bot ASF subversion and git services added a comment - Commit b90710296fe5393c55bdcd1a8187971e477d405f in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=b907102 ] ACCUMULO-1451 cleanup imports
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit bcaefcd2ed4f8807ca6a042580ca17da69f4d60d in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bcaefcd ]

          ACCUMULO-1451 make output file parameters configurable; more test stabilization

          Show
          jira-bot ASF subversion and git services added a comment - Commit bcaefcd2ed4f8807ca6a042580ca17da69f4d60d in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bcaefcd ] ACCUMULO-1451 make output file parameters configurable; more test stabilization
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c16b6ae4fc7e607011c1ae7c01f3ea3bd0c182ba in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c16b6ae ]

          ACCUMULO-1451 from feedback from Keith Turner, removed concept of passes, multiple output files, made interface abstract class, moved rules for USER and CHOP into tablet

          Show
          jira-bot ASF subversion and git services added a comment - Commit c16b6ae4fc7e607011c1ae7c01f3ea3bd0c182ba in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c16b6ae ] ACCUMULO-1451 from feedback from Keith Turner , removed concept of passes, multiple output files, made interface abstract class, moved rules for USER and CHOP into tablet
          Hide
          kturner Keith Turner added a comment -

          Based on ACCUMULO-1787, we also discussed allowing the compaction strategy to override the configuration used to create a RFile (falling back to table config if compaction strategy does not set it) for 1.6.0

          Show
          kturner Keith Turner added a comment - Based on ACCUMULO-1787 , we also discussed allowing the compaction strategy to override the configuration used to create a RFile (falling back to table config if compaction strategy does not set it) for 1.6.0
          Hide
          kturner Keith Turner added a comment - - edited

          Eric Newton and I were just discussing this. We were talking about limiting the scope of this for 1.6 w/o precluding useful changes that we are aware of for later versions.

          Some changes we disccused for 1.6.

          • Move chop logic completely outside of compaction strategy (So user can not break merges). Also do not invoke compaction strategy for user initiated compactions.
          • Move delete propagation logic outside of compaction strategy
          • Gear APIs for 1.6 twoards 1 output file, but structure current API such that support multiple files can be added in future w/o breaking compaction strategies written for 1.6.
          • Return a single list of output files, dropping passes. The implementation may still do multiple passes, but the strategy will not have control over this.
          • Push allowing users to specify a compaction strategy (and config for the strategy) on a user initiated compaction to a later release.
          Show
          kturner Keith Turner added a comment - - edited Eric Newton and I were just discussing this. We were talking about limiting the scope of this for 1.6 w/o precluding useful changes that we are aware of for later versions. Some changes we disccused for 1.6. Move chop logic completely outside of compaction strategy (So user can not break merges). Also do not invoke compaction strategy for user initiated compactions. Move delete propagation logic outside of compaction strategy Gear APIs for 1.6 twoards 1 output file, but structure current API such that support multiple files can be added in future w/o breaking compaction strategies written for 1.6. Return a single list of output files, dropping passes. The implementation may still do multiple passes, but the strategy will not have control over this. Push allowing users to specify a compaction strategy (and config for the strategy) on a user initiated compaction to a later release.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9f9917ca4b8f6d1bcce96d586c38c74d37c4dfd2 in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9f9917c ]

          ACCUMULO-1451 add functional test of the configurability of the compaction strategy

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9f9917ca4b8f6d1bcce96d586c38c74d37c4dfd2 in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9f9917c ] ACCUMULO-1451 add functional test of the configurability of the compaction strategy
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 656a3614e0981462a0ee2c54b4754466d67fb72f in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=656a361 ]

          ACCUMULO-1451 make class selection dynamic, and add license headers

          Show
          jira-bot ASF subversion and git services added a comment - Commit 656a3614e0981462a0ee2c54b4754466d67fb72f in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=656a361 ] ACCUMULO-1451 make class selection dynamic, and add license headers
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 4c3da02bcdb14caec648716679317cac837789cf in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4c3da02 ]

          ACCUMULO-1451 making major compaction configurable

          Show
          jira-bot ASF subversion and git services added a comment - Commit 4c3da02bcdb14caec648716679317cac837789cf in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=4c3da02 ] ACCUMULO-1451 making major compaction configurable
          Hide
          kturner Keith Turner added a comment -

          In addition to deciding when to compact, the following may also be useful for this plugin :

          • Allow the plugin to decided what files to compact
          • Allow the plugin to be configured per table w/ options (like an iterator can have options)
          • Allow the plugin to be configured per compaction (like a scan time iterators and compaction time iterators) overriding the plugin set for the table.
          Show
          kturner Keith Turner added a comment - In addition to deciding when to compact, the following may also be useful for this plugin : Allow the plugin to decided what files to compact Allow the plugin to be configured per table w/ options (like an iterator can have options) Allow the plugin to be configured per compaction (like a scan time iterators and compaction time iterators) overriding the plugin set for the table.
          Hide
          kturner Keith Turner added a comment - - edited

          This would also be useful for implementing ACCUMULO-1266, although the relevant data needed to make a decision would still need to be obtained.

          Show
          kturner Keith Turner added a comment - - edited This would also be useful for implementing ACCUMULO-1266 , although the relevant data needed to make a decision would still need to be obtained.

            People

            • Assignee:
              ecn Eric Newton
              Reporter:
              mdrob Mike Drob
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development