Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15364

Support sort the output according to the number of occurrences of the opcode for StatisticsEditsVisitor

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Invalid
    • Affects Version/s: 3.2.1
    • Fix Version/s: None
    • Component/s: tools
    • Labels:
      None

      Description

            At present, when we execute `hdfs oev -p stats -i edits -o edits.stats`, the output format is as follows, and all the opcodes will be output once.

      VERSION : -65
      OP_ADD ( 0): 2
      OP_RENAME_OLD ( 1): 2
      OP_DELETE ( 2): 0
      OP_MKDIR ( 3): 5
      OP_SET_REPLICATION ( 4): 0
      OP_DATANODE_ADD ( 5): 0
      OP_DATANODE_REMOVE ( 6): 0
      OP_SET_PERMISSIONS ( 7): 4
      OP_SET_OWNER ( 8): 1
      OP_CLOSE ( 9): 2
      OP_SET_GENSTAMP_V1 ( 10): 0
      OP_SET_NS_QUOTA ( 11): 0
      OP_CLEAR_NS_QUOTA ( 12): 0
      OP_TIMES ( 13): 0
      OP_SET_QUOTA ( 14): 0
      OP_RENAME ( 15): 0
      OP_CONCAT_DELETE ( 16): 0
      OP_SYMLINK ( 17): 0
      OP_GET_DELEGATION_TOKEN ( 18): 0
      OP_RENEW_DELEGATION_TOKEN ( 19): 0
      OP_CANCEL_DELEGATION_TOKEN ( 20): 0
      OP_UPDATE_MASTER_KEY ( 21): 0
      OP_REASSIGN_LEASE ( 22): 0
      OP_END_LOG_SEGMENT ( 23): 1
      OP_START_LOG_SEGMENT ( 24): 1
      OP_UPDATE_BLOCKS ( 25): 0
      OP_CREATE_SNAPSHOT ( 26): 0
      OP_DELETE_SNAPSHOT ( 27): 0
      OP_RENAME_SNAPSHOT ( 28): 0
      OP_ALLOW_SNAPSHOT ( 29): 0
      OP_DISALLOW_SNAPSHOT ( 30): 0
      OP_SET_GENSTAMP_V2 ( 31): 2
      OP_ALLOCATE_BLOCK_ID ( 32): 2
      OP_ADD_BLOCK ( 33): 2
      OP_ADD_CACHE_DIRECTIVE ( 34): 0
      OP_REMOVE_CACHE_DIRECTIVE ( 35): 0
      OP_ADD_CACHE_POOL ( 36): 0
      OP_MODIFY_CACHE_POOL ( 37): 0
      OP_REMOVE_CACHE_POOL ( 38): 0
      OP_MODIFY_CACHE_DIRECTIVE ( 39): 0
      OP_SET_ACL ( 40): 0
      OP_ROLLING_UPGRADE_START ( 41): 0
      OP_ROLLING_UPGRADE_FINALIZE ( 42): 0
      OP_SET_XATTR ( 43): 0
      OP_REMOVE_XATTR ( 44): 0
      OP_SET_STORAGE_POLICY ( 45): 0
      OP_TRUNCATE ( 46): 0
      OP_APPEND ( 47): 0
      OP_SET_QUOTA_BY_STORAGETYPE ( 48): 0
      OP_ADD_ERASURE_CODING_POLICY ( 49): 0
      OP_ENABLE_ERASURE_CODING_POLIC ( 50): 0
      OP_DISABLE_ERASURE_CODING_POLI ( 51): 0
      OP_REMOVE_ERASURE_CODING_POLIC ( 52): 0
      OP_INVALID ( -1): 0

       But in general, the edits file we parse does not involve all the operation codes. If all the operation codes are output, it is unfriendly for the cluster administrator to view the output.
          we usually only care about what opcodes appear in the edits file.We can output the opcodes that appeared in the edits file and sort them.

      For example, we can execute the following command:

       hdfs oev -p stats -i edits_0000000000000001321-0000000000000001344 -sort -o edits.stats -v

      The output format is as follows:

      VERSION : -65
      OP_MKDIR ( 3): 5
      OP_SET_PERMISSIONS ( 7): 4
      OP_ADD ( 0): 2
      OP_RENAME_OLD ( 1): 2
      OP_CLOSE ( 9): 2
      OP_SET_GENSTAMP_V2 ( 31): 2
      OP_ALLOCATE_BLOCK_ID ( 32): 2
      OP_ADD_BLOCK ( 33): 2
      OP_SET_OWNER ( 8): 1
      OP_END_LOG_SEGMENT ( 23): 1
      OP_START_LOG_SEGMENT ( 24): 1

        Attachments

        1. HDFS-15364.002.patch
          11 kB
          bianqi
        2. HDFS-15364.001.patch
          11 kB
          bianqi

          Activity

            People

            • Assignee:
              bianqi bianqi
              Reporter:
              bianqi bianqi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: