Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6581 Write to single replica in memory
  3. HDFS-6978

Directory scanner should correctly reconcile blocks on RAM disk

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HDFS-6581
    • Fix Version/s: 2.6.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It used to be very unlikely that the directory scanner encountered two replicas of the same block on different volumes.

      With memory storage, it is very likely to hit this with the following sequence of events:

      1. Block is written to RAM disk
      2. Lazy writer saves a copy on persistent volume
      3. DN attempts to evict the original replica from RAM disk, file deletion fails as the replica is in use.
      4. Directory scanner finds a replica on both RAM disk and persistent storage.

      The directory scanner should never delete the block on persistent storage.

        Attachments

        1. HDFS-6978.02.patch
          25 kB
          Arpit Agarwal
        2. HDFS-6978.01.patch
          23 kB
          Arpit Agarwal

          Activity

            People

            • Assignee:
              arp Arpit Agarwal
              Reporter:
              arp Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: