Accumulo
  1. Accumulo
  2. ACCUMULO-875

add list compactions command to shell

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      The shell currently has a list scans command which is very useful for debugging. Another command that would be useful is list compactions.

      The things that would be useful to know about a running compaction are the following :

      • Tablet server
      • Age
      • State (if showing queued compactions)
      • Type (User, Chop, System, Idle)
      • Tablet
      • Iterator info (iterators could have changed since it started and there are per compaction iterators)
      • Input files
      • Output file
      • Rate
      • Key/values read so far (could do bytes also)
      • Key/values written so far (could do bytes also)
      • Locality group compacting

      Could consider using JMX to collect this info from tablet servers. List scans currently uses a thrift call to get its info from tablet servers. Making the info available via JMX would enable other tools to use the information.

        Activity

        Keith Turner created issue -
        Christopher Tubbs made changes -
        Field Original Value New Value
        Assignee Christopher Tubbs [ ctubbsii ]
        Keith Turner made changes -
        Assignee Christopher Tubbs [ ctubbsii ] Keith Turner [ kturner ]
        Keith Turner made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk-Hadoop-2.0 #7 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/7/)
        ACCUMULO-875 added API and shell command for listing active compactions on a tablet server (Revision 1435371)

        Result = SUCCESS
        kturner :
        Files :

        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/ActiveCompaction.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/InstanceOperations.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/InstanceOperationsImpl.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/mock/MockInstanceOperations.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ActiveCompaction.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/CompactionReason.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/CompactionType.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/IteratorConfig.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TIteratorSetting.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/Shell.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ActiveCompactionIterator.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ListCompactionsCommand.java
        • /accumulo/trunk/core/src/main/thrift/tabletserver.thrift
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/MinorCompactor.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Tablet.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServerResourceManager.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/test/performance/thrift/NullTserver.java
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk-Hadoop-2.0 #7 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/7/ ) ACCUMULO-875 added API and shell command for listing active compactions on a tablet server (Revision 1435371) Result = SUCCESS kturner : Files : /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/ActiveCompaction.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/InstanceOperations.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/InstanceOperationsImpl.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/mock/MockInstanceOperations.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ActiveCompaction.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/CompactionReason.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/CompactionType.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/IteratorConfig.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TIteratorSetting.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/Shell.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ActiveCompactionIterator.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ListCompactionsCommand.java /accumulo/trunk/core/src/main/thrift/tabletserver.thrift /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/MinorCompactor.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Tablet.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServerResourceManager.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/test/performance/thrift/NullTserver.java
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk #649 (See https://builds.apache.org/job/Accumulo-Trunk/649/)
        ACCUMULO-875 added API and shell command for listing active compactions on a tablet server (Revision 1435371)

        Result = SUCCESS
        kturner :
        Files :

        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/ActiveCompaction.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/InstanceOperations.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/InstanceOperationsImpl.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/mock/MockInstanceOperations.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ActiveCompaction.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/CompactionReason.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/CompactionType.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/IteratorConfig.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TIteratorSetting.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/Shell.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ActiveCompactionIterator.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ListCompactionsCommand.java
        • /accumulo/trunk/core/src/main/thrift/tabletserver.thrift
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/MinorCompactor.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Tablet.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServerResourceManager.java
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/test/performance/thrift/NullTserver.java
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk #649 (See https://builds.apache.org/job/Accumulo-Trunk/649/ ) ACCUMULO-875 added API and shell command for listing active compactions on a tablet server (Revision 1435371) Result = SUCCESS kturner : Files : /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/ActiveCompaction.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/InstanceOperations.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/InstanceOperationsImpl.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/mock/MockInstanceOperations.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ActiveCompaction.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/CompactionReason.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/CompactionType.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/IteratorConfig.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TIteratorSetting.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/Shell.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ActiveCompactionIterator.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ListCompactionsCommand.java /accumulo/trunk/core/src/main/thrift/tabletserver.thrift /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/MinorCompactor.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Tablet.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServerResourceManager.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/test/performance/thrift/NullTserver.java
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk #652 (See https://builds.apache.org/job/Accumulo-Trunk/652/)
        ACCUMULO-875 made list compactions API return files being compacted instead of a count. Shell still shows a count (Revision 1436995)

        Result = SUCCESS
        kturner :
        Files :

        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/ActiveCompaction.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ActiveCompaction.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/IteratorConfig.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TIteratorSetting.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ActiveCompactionIterator.java
        • /accumulo/trunk/core/src/main/thrift/tabletserver.thrift
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk #652 (See https://builds.apache.org/job/Accumulo-Trunk/652/ ) ACCUMULO-875 made list compactions API return files being compacted instead of a count. Shell still shows a count (Revision 1436995) Result = SUCCESS kturner : Files : /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/ActiveCompaction.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ActiveCompaction.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/IteratorConfig.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TIteratorSetting.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ActiveCompactionIterator.java /accumulo/trunk/core/src/main/thrift/tabletserver.thrift /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk-Hadoop-2.0 #10 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/10/)
        ACCUMULO-875 made list compactions API return files being compacted instead of a count. Shell still shows a count (Revision 1436995)

        Result = SUCCESS
        kturner :
        Files :

        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/ActiveCompaction.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ActiveCompaction.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/IteratorConfig.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TIteratorSetting.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java
        • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ActiveCompactionIterator.java
        • /accumulo/trunk/core/src/main/thrift/tabletserver.thrift
        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk-Hadoop-2.0 #10 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/10/ ) ACCUMULO-875 made list compactions API return files being compacted instead of a count. Shell still shows a count (Revision 1436995) Result = SUCCESS kturner : Files : /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/admin/ActiveCompaction.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ActiveCompaction.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/IteratorConfig.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TIteratorSetting.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/TabletClientService.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ActiveCompactionIterator.java /accumulo/trunk/core/src/main/thrift/tabletserver.thrift /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk #667 (See https://builds.apache.org/job/Accumulo-Trunk/667/)
        ACCUMULO-875 list compactions was not showing entries read unless something was written (Revision 1438319)

        Result = SUCCESS
        kturner :
        Files :

        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk #667 (See https://builds.apache.org/job/Accumulo-Trunk/667/ ) ACCUMULO-875 list compactions was not showing entries read unless something was written (Revision 1438319) Result = SUCCESS kturner : Files : /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        Hide
        Hudson added a comment -

        Integrated in Accumulo-Trunk-Hadoop-2.0 #25 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/25/)
        ACCUMULO-875 list compactions was not showing entries read unless something was written (Revision 1438319)

        Result = UNSTABLE
        kturner :
        Files :

        • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java
        Show
        Hudson added a comment - Integrated in Accumulo-Trunk-Hadoop-2.0 #25 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/25/ ) ACCUMULO-875 list compactions was not showing entries read unless something was written (Revision 1438319) Result = UNSTABLE kturner : Files : /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/Compactor.java

          People

          • Assignee:
            Keith Turner
            Reporter:
            Keith Turner
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development