Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: HDFS-6581
    • Fix Version/s: 2.6.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It used to be very unlikely that the directory scanner encountered two replicas of the same block on different volumes.

      With memory storage, it is very likely to hit this with the following sequence of events:

      1. Block is written to RAM disk
      2. Lazy writer saves a copy on persistent volume
      3. DN attempts to evict the original replica from RAM disk, file deletion fails as the replica is in use.
      4. Directory scanner finds a replica on both RAM disk and persistent storage.

      The directory scanner should never delete the block on persistent storage.

      1. HDFS-6978.02.patch
        25 kB
        Arpit Agarwal
      2. HDFS-6978.01.patch
        23 kB
        Arpit Agarwal

        Activity

        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/)
        HDFS-6978. Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) (arp: rev 9f22fb8c9a10952225e15c7b67b5f77fa44b155d)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyWriteReplicaTracker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/ ) HDFS-6978 . Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) (arp: rev 9f22fb8c9a10952225e15c7b67b5f77fa44b155d) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyWriteReplicaTracker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-hdfs-project/hadoop-hdfs/CHANGES- HDFS-6581 .txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/)
        HDFS-6978. Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) (arp: rev 9f22fb8c9a10952225e15c7b67b5f77fa44b155d)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyWriteReplicaTracker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/ ) HDFS-6978 . Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) (arp: rev 9f22fb8c9a10952225e15c7b67b5f77fa44b155d) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyWriteReplicaTracker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-hdfs-project/hadoop-hdfs/CHANGES- HDFS-6581 .txt
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #698 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/698/)
        HDFS-6978. Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) (arp: rev 9f22fb8c9a10952225e15c7b67b5f77fa44b155d)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyWriteReplicaTracker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #698 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/698/ ) HDFS-6978 . Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) (arp: rev 9f22fb8c9a10952225e15c7b67b5f77fa44b155d) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyWriteReplicaTracker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/CHANGES- HDFS-6581 .txt
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6163 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6163/)
        HDFS-6978. Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) (arp: rev 9f22fb8c9a10952225e15c7b67b5f77fa44b155d)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyWriteReplicaTracker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6163 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6163/ ) HDFS-6978 . Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) (arp: rev 9f22fb8c9a10952225e15c7b67b5f77fa44b155d) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java hadoop-hdfs-project/hadoop-hdfs/CHANGES- HDFS-6581 .txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyWriteReplicaTracker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
        Hide
        Arpit Agarwal added a comment -

        Thanks Jitendra Nath Pandey! Committed to the feature branch.

        Show
        Arpit Agarwal added a comment - Thanks Jitendra Nath Pandey ! Committed to the feature branch.
        Hide
        Jitendra Nath Pandey added a comment -

        +1

        Show
        Jitendra Nath Pandey added a comment - +1
        Hide
        Arpit Agarwal added a comment -

        Thanks for the review Jitendra, agreed that the existing Math.min checks are redundant, removed them.

        Show
        Arpit Agarwal added a comment - Thanks for the review Jitendra, agreed that the existing Math.min checks are redundant, removed them.
        Hide
        Jitendra Nath Pandey added a comment -

        +1 for the patch.

        A minor comment:

        +          if (d < blockpoolReport.length) {
        +            // There may be multiple on-disk records for the same block, don't increment
        +            // the memory record pointer if so. 
        +            ScanInfo nextInfo = blockpoolReport[Math.min(d, blockpoolReport.length - 1)];
        

        Math.min(d, blockpoolReport.length - 1) will always be equal to d, unless we have race conditions. I think the same applies to the previous lines with Math.min as well.

        Show
        Jitendra Nath Pandey added a comment - +1 for the patch. A minor comment: + if (d < blockpoolReport.length) { + // There may be multiple on-disk records for the same block, don't increment + // the memory record pointer if so. + ScanInfo nextInfo = blockpoolReport[ Math .min(d, blockpoolReport.length - 1)]; Math.min(d, blockpoolReport.length - 1) will always be equal to d, unless we have race conditions. I think the same applies to the previous lines with Math.min as well.
        Hide
        Arpit Agarwal added a comment -

        Attaching patch:

        1. Fixes DirectoryScanner.scan to support multiple on-disk replicas for a given block.
        2. Fix FsDatasetImpl.checkAndUpdate to reconcile multiple replicas using BlockPoolSlice.resolveDuplicateReplicas.
        3. Also fix moveReplicaToNewVolume to perform the file copy outside the DataSet lock. Required minor fixes to copyBlockFiles.
        4. Add test cases.
        Show
        Arpit Agarwal added a comment - Attaching patch: Fixes DirectoryScanner.scan to support multiple on-disk replicas for a given block. Fix FsDatasetImpl.checkAndUpdate to reconcile multiple replicas using BlockPoolSlice.resolveDuplicateReplicas. Also fix moveReplicaToNewVolume to perform the file copy outside the DataSet lock. Required minor fixes to copyBlockFiles. Add test cases.

          People

          • Assignee:
            Arpit Agarwal
            Reporter:
            Arpit Agarwal
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development