Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-492

Expose corrupt replica/block information

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New server web pages provide block information: corrupt_replicas_xml and block_info_xml.
    • Tags:
      block forensic blocks corrupt replica

      Description

      This adds two additional functions to FSNamesystem to provide more information about corrupt replicas. It also adds two servlets to the namenode that provide information (in JSON) about all blocks with corrupt replicas as well as information about a specific block. It also changes the file browsing servlet by adding a link from block ids to the above mentioned block information page.

      These JSON pages are designed to be used by client side tools which wish to analyze corrupt block/replicas. The only change to an existing (non-servlet) class is described below.

      Currently, CorruptReplicasMap stores a map of corrupt replica information and allows insertion and deletion. It also gives information about the corrupt replicas for a specific block. It does not allow iteration over all corrupt blocks. Two additional functions will be added to FSNamesystem (which will call BlockManager which will call CorruptReplicasMap). The first will return the size of the corrupt replicas map, which represents the number of blocks that have corrupt replicas (but less than the number of corrupt replicas if a block has multiple corrupt replicas). The second will allow "paging" through a list of block ids that contain corrupt replicas:

      public synchronized List<Long> getCorruptReplicaBlockIds(int n, Long startingBlockId)

      n is the number of block ids to return and startingBlockId is the block id offset. To prevent a large number of items being returned at one time, n is constrained to 0 <= n <= 100. If startingBlockId is null, up to n items are returned starting at the beginning of the list. Ordering is enforced through the internal use of TreeMap in CorruptReplicasMap.

      1. hdfs-492-4.patch
        22 kB
        Bill Zeller
      2. hdfs-492-5.patch
        22 kB
        Bill Zeller
      3. hdfs-492-8.patch
        22 kB
        Bill Zeller
      4. hdfs-492-9.patch
        25 kB
        Bill Zeller
      5. hdfs-492-10.patch
        25 kB
        Bill Zeller
      6. hdfs-492-11.patch
        25 kB
        Bill Zeller
      7. hdfs-492-13.patch
        26 kB
        Bill Zeller

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Bill Zeller
            Reporter:
            Bill Zeller
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Remaining Estimate - 48h
              48h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development