Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6581 Write to single replica in memory
  3. HDFS-6929

NN periodically unlinks lazy persist files with missing replicas from namespace

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • datanode
    • None
    • Reviewed

    Description

      Occasional data loss is expected when using the lazy persist flag due to node restarts. The NN will optionally unlink lazy persist files from the namespace to avoid them from showing up as corrupt files.

      This behavior can be turned off with a global option. In the future this may be made a per-file option controllable by the client.

      Attachments

        1. HDFS-6929.01.patch
          9 kB
          Arpit Agarwal
        2. HDFS-6929.02.patch
          10 kB
          Arpit Agarwal

        Issue Links

          Activity

            Does this patch add the global config to disable the scrubbing of corrupted lazy-persist files or is it planned for a later patch?

            jnp Jitendra Nath Pandey added a comment - Does this patch add the global config to disable the scrubbing of corrupted lazy-persist files or is it planned for a later patch?
            arp Arpit Agarwal added a comment -

            No, but I think it is simple enough to change the meaning of the value so that zero means 'never scrub'. Let me post an updated patch.

            arp Arpit Agarwal added a comment - No, but I think it is simple enough to change the meaning of the value so that zero means 'never scrub'. Let me post an updated patch.
            arp Arpit Agarwal added a comment -

            Updated patch to allow turning off the scrubber, document the option.

            arp Arpit Agarwal added a comment - Updated patch to allow turning off the scrubber, document the option.
            arp Arpit Agarwal added a comment -

            Committed to the feature branch. Thanks jnp.

            arp Arpit Agarwal added a comment - Committed to the feature branch. Thanks jnp .
            hudson Hudson added a comment -

            FAILURE: Integrated in Hadoop-trunk-Commit #6163 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6163/)
            HDFS-6929. NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) (aagarwal: rev 2e987148e02d0087fc70ce5b1ce571d3324bf1dd)

            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
            • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
            • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
            • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6163 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6163/ ) HDFS-6929 . NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) (aagarwal: rev 2e987148e02d0087fc70ce5b1ce571d3324bf1dd) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java hadoop-hdfs-project/hadoop-hdfs/CHANGES- HDFS-6581 .txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            hudson Hudson added a comment -

            FAILURE: Integrated in Hadoop-Yarn-trunk #698 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/698/)
            HDFS-6929. NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) (aagarwal: rev 2e987148e02d0087fc70ce5b1ce571d3324bf1dd)

            • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
            • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
            • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
            hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #698 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/698/ ) HDFS-6929 . NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) (aagarwal: rev 2e987148e02d0087fc70ce5b1ce571d3324bf1dd) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES- HDFS-6581 .txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
            hudson Hudson added a comment -

            FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/)
            HDFS-6929. NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) (aagarwal: rev 2e987148e02d0087fc70ce5b1ce571d3324bf1dd)

            • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
            • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
            • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
            hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/ ) HDFS-6929 . NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) (aagarwal: rev 2e987148e02d0087fc70ce5b1ce571d3324bf1dd) hadoop-hdfs-project/hadoop-hdfs/CHANGES- HDFS-6581 .txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
            hudson Hudson added a comment -

            FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/)
            HDFS-6929. NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) (aagarwal: rev 2e987148e02d0087fc70ce5b1ce571d3324bf1dd)

            • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
            • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
            • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
            hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/ ) HDFS-6929 . NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) (aagarwal: rev 2e987148e02d0087fc70ce5b1ce571d3324bf1dd) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java hadoop-hdfs-project/hadoop-hdfs/CHANGES- HDFS-6581 .txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

            People

              arp Arpit Agarwal
              arp Arpit Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: