Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8219

setStoragePolicy with folder behavior is different after cluster restart

    Details

      Description

      Reproduce steps.
      1) mkdir named /temp
      2) put one file A under /temp
      3) change /temp storage policy to COLD
      4) use -getStoragePolicy to query file A's storage policy, it is same with /temp
      5) change /temp folder storage policy again, will see file A's storage policy keep same with parent folder.

      then restart the cluster.
      do 3) 4) again, will find file A's storage policy is not change while parent folder's storage policy changes. It behaves different.

      As i debugged, found the code:
      in INodeFile.getStoragePolicyID

        public byte getStoragePolicyID() {
          byte id = getLocalStoragePolicyID();
          if (id == BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
            return this.getParent() != null ?
                this.getParent().getStoragePolicyID() : id;
          }
          return id;
        }
      

      If the file do not have its storage policy, it will use parent's. But after cluster restart, the file turns to have its own storage policy.

      1. HDFS-8219.patch
        3 kB
        Surendra Singh Lilhore
      2. HDFS-8219.unittest-norepro.patch
        3 kB
        Xiaoyu Yao

        Activity

        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Peter Shi for reporting the issue. I added a unit test based on the repro steps but cannot repro it. I will take a look if you can post a unit test that reproduces the issue.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Peter Shi for reporting the issue. I added a unit test based on the repro steps but cannot repro it. I will take a look if you can post a unit test that reproduces the issue.
        Hide
        shihaoliang Peter Shi added a comment -

        Currently I can not reproduce this in UT, only in my cluster. I will check whether it can be produced in trunk .

        Show
        shihaoliang Peter Shi added a comment - Currently I can not reproduce this in UT, only in my cluster. I will check whether it can be produced in trunk .
        Hide
        shihaoliang Peter Shi added a comment -

        Currently I can not reproduce this in UT, only in my cluster. I will check whether it can be produced in trunk .

        Show
        shihaoliang Peter Shi added a comment - Currently I can not reproduce this in UT, only in my cluster. I will check whether it can be produced in trunk .
        Hide
        shihaoliang Peter Shi added a comment -

        Currently I can not reproduce this in UT, only in my cluster. I will check whether it can be produced in trunk .

        Show
        shihaoliang Peter Shi added a comment - Currently I can not reproduce this in UT, only in my cluster. I will check whether it can be produced in trunk .
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Can I work on this issue?. I am able to reproduce this and I got the root cause.

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Can I work on this issue?. I am able to reproduce this and I got the root cause.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Little change in the scenario, before putting the file in directory first set the storage policy for directory.

        Root Cause

        When we will add edit log for new file, we will get the storage policy from INodeFile by getStoragePolicyID().

        	 AddOp op = AddOp.getInstance(cache.get())
        	....................
        	....................
                .setStoragePolicyId(newNode.getStoragePolicyID());
        	

        but this API will return the parent policy If file storagepolicy is UNSPECIFIED.

                public byte getStoragePolicyID() {
                    byte id = getLocalStoragePolicyID();
                    if (id == BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
                       return this.getParent() != null ?this.getParent().getStoragePolicyID() : id;
                    }
                    return id;
                }
        	

        So in edits log create file will store parent storage policy.

        Fix
        We should use INodeFile.getLocalStoragePolicyID(), it will give the current INodeFile storage policy

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Little change in the scenario, before putting the file in directory first set the storage policy for directory. Root Cause When we will add edit log for new file, we will get the storage policy from INodeFile by getStoragePolicyID() . AddOp op = AddOp.getInstance(cache.get()) .................... .................... .setStoragePolicyId(newNode.getStoragePolicyID()); but this API will return the parent policy If file storagepolicy is UNSPECIFIED. public byte getStoragePolicyID() { byte id = getLocalStoragePolicyID(); if (id == BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) { return this .getParent() != null ? this .getParent().getStoragePolicyID() : id; } return id; } So in edits log create file will store parent storage policy. Fix We should use INodeFile.getLocalStoragePolicyID() , it will give the current INodeFile storage policy
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Attached patch with fix and test. Please review..

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Attached patch with fix and test. Please review..
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Changed Severity to Major because NN storing wrong edit info..

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Changed Severity to Major because NN storing wrong edit info..
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 26s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 27s There were no new javac warning messages.
        +1 javadoc 9m 34s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 2m 18s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        +1 hdfs tests 168m 49s Tests passed in hadoop-hdfs.
            211m 22s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730165/HDFS-8219.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 338e88a
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10805/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10805/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10805/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 27s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 18s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 168m 49s Tests passed in hadoop-hdfs.     211m 22s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730165/HDFS-8219.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 338e88a hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10805/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10805/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10805/console This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        The patch looks good to me. Thanks Surendra Singh Lilhore for contribution. I will commit it shortly.

        Show
        xyao Xiaoyu Yao added a comment - The patch looks good to me. Thanks Surendra Singh Lilhore for contribution. I will commit it shortly.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7740 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7740/)
        HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7740 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7740/ ) HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/186/)
        HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/186/ ) HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #919 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/919/)
        HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #919 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/919/ ) HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2117 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2117/)
        HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2117 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2117/ ) HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #176 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/176/)
        HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #176 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/176/ ) HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2135/)
        HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2135/ ) HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/186/)
        HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/186/ ) HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (xyao: rev 0100b155019496d077f958904de7d385697d65d9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Sangjin Lee backported this to 2.6.1, making minor changes to make it work with 2.6.1.

        I just pushed the commit to 2.6.1 after running compilation and TestBlockStoragePolicy which changed in the patch.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Sangjin Lee backported this to 2.6.1, making minor changes to make it work with 2.6.1. I just pushed the commit to 2.6.1 after running compilation and TestBlockStoragePolicy which changed in the patch.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Just pulled this into branch-2.7 (release 2.7.2) as it already exists in 2.6.1.

        branch-2 patch applies cleanly for the most part, had to fix one very minor merge conflict in the test-case. Ran compilation and TestBlockStoragePolicy before the push.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Just pulled this into branch-2.7 (release 2.7.2) as it already exists in 2.6.1. branch-2 patch applies cleanly for the most part, had to fix one very minor merge conflict in the test-case. Ran compilation and TestBlockStoragePolicy before the push.

          People

          • Assignee:
            surendrasingh Surendra Singh Lilhore
            Reporter:
            shihaoliang Peter Shi
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development