Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-13453

S3Guard: Instrument new functionality with Hadoop metrics.

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Provide Hadoop metrics showing operational details of the S3Guard implementation.

      The metrics will be implemented in this ticket:

      ● S3GuardRechecksNthPercentileLatency (MutableQuantiles) ­​ Percentile time spent
      in rechecks attempting to achieve consistency. Repeated for multiple percentile values
      of N.  This metric is an indicator of the additional latency cost of running S3A with
      S3Guard.
      ● S3GuardRechecksNumOps (MutableQuantiles) ­​ Number of times a consistency
      recheck was required while attempting to achieve consistency.
      ● S3GuardStoreNthPercentileLatency (MutableQuantiles) ­​ Percentile time spent in
      operations against the consistent store, including both write operations during file system
      mutations and read operations during file system consistency checks. Repeated for
      multiple percentile values of N. This metric is an indicator of latency to the consistent
      store implementation.
      ● S3GuardConsistencyStoreNumOps (MutableQuantiles) ­​ Number of operations
      against the consistent store, including both write operations during file system mutations
      and read operations during file system consistency checks.
      ● S3GuardConsistencyStoreFailures (MutableCounterLong) ­​ Number of failures
      during operations against the consistent store implementation.
      ● S3GuardConsistencyStoreTimeouts (MutableCounterLong) ­​ Number of timeouts
      during operations against the consistent store implementation.
      ● S3GuardInconsistencies (MutableCounterLong) ­ C​ ount of times S3Guard failed to
      achieve consistency, even after exhausting all rechecks. A high count may indicate
      unexpected out­of­band modification of the S3 bucket contents, such as by an external
      tool that does not make corresponding updates to the consistent store.

        Attachments

        1. HADOOP-13453-HADOOP-13345-001.patch
          6 kB
          Ai Deng
        2. HADOOP-13453-HADOOP-13345-002.patch
          13 kB
          Ai Deng
        3. HADOOP-13453-HADOOP-13345-003.patch
          18 kB
          Ai Deng
        4. HADOOP-13453-HADOOP-13345-004.patch
          11 kB
          Steve Loughran
        5. HADOOP-13453-HADOOP-13345-005.patch
          12 kB
          Steve Loughran

          Activity

            People

            • Assignee:
              aid129 Ai Deng
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: