Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8470

fsimage loading progress should update inode, delegation token and cache pool count.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. HDFS-8470_1.patch
      6 kB
      Surendra Singh Lilhore
    2. HDFS-8470_2.patch
      6 kB
      Surendra Singh Lilhore
    3. HDFS-8470_3.patch
      7 kB
      Surendra Singh Lilhore
    4. HDFS-8470.patch
      6 kB
      Surendra Singh Lilhore
    5. screenshot_after_fix.png
      13 kB
      Surendra Singh Lilhore
    6. screenshot-1.png
      14 kB
      tongshiquan

      Activity

      Hide
      surendrasingh Surendra Singh Lilhore added a comment -

      Attached patch. Please review..

      Show
      surendrasingh Surendra Singh Lilhore added a comment - Attached patch. Please review..
      Hide
      vinayrpet Vinayakumar B added a comment -

      prog.setCount(Phase.LOADING_FSIMAGE, currentStep, i + 1);
      Instead of like this, you can use, prog,getCounter(..) and counter.increment() ?

      Show
      vinayrpet Vinayakumar B added a comment - prog.setCount(Phase.LOADING_FSIMAGE, currentStep, i + 1); Instead of like this, you can use, prog,getCounter(..) and counter.increment() ?
      Hide
      surendrasingh Surendra Singh Lilhore added a comment -

      Thanks Vinayakumar B for reviewing.
      I have updated the patch.
      Please review.

      Show
      surendrasingh Surendra Singh Lilhore added a comment - Thanks Vinayakumar B for reviewing. I have updated the patch. Please review.
      Hide
      hadoopqa Hadoop QA added a comment -



      -1 overall



      Vote Subsystem Runtime Comment
      0 pre-patch 18m 23s Pre-patch trunk compilation is healthy.
      +1 @author 0m 0s The patch does not contain any @author tags.
      -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
      +1 javac 7m 47s There were no new javac warning messages.
      +1 javadoc 9m 55s There were no new javadoc warning messages.
      +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
      -1 checkstyle 2m 15s The applied patch generated 1 new checkstyle issues (total was 47, now 46).
      +1 whitespace 0m 0s The patch has no lines that end in whitespace.
      +1 install 1m 35s mvn install still works.
      +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
      +1 findbugs 3m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
      +1 native 3m 22s Pre-build of native portion
      +1 hdfs tests 167m 52s Tests passed in hadoop-hdfs.
          215m 36s  



      Subsystem Report/Notes
      Patch URL http://issues.apache.org/jira/secure/attachment/12736844/HDFS-8470_1.patch
      Optional Tests javadoc javac unit findbugs checkstyle
      git revision trunk / a2bd621
      checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11206/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
      hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11206/artifact/patchprocess/testrun_hadoop-hdfs.txt
      Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11206/testReport/
      Java 1.7.0_55
      uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11206/console

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 15s The applied patch generated 1 new checkstyle issues (total was 47, now 46). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 22s Pre-build of native portion +1 hdfs tests 167m 52s Tests passed in hadoop-hdfs.     215m 36s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736844/HDFS-8470_1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a2bd621 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11206/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11206/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11206/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11206/console This message was automatically generated.
      Hide
      surendrasingh Surendra Singh Lilhore added a comment -

      Fixed checkstyle error and attached updated patch.
      Please review.

      Show
      surendrasingh Surendra Singh Lilhore added a comment - Fixed checkstyle error and attached updated patch. Please review.
      Hide
      vinayrpet Vinayakumar B added a comment -

      prog.getCounter(Phase.LOADING_FSIMAGE, currentStep).increment();
      Instead of getting counter for every INode being loaded, you can take reference of counter outside for-loop and just increment it inside for-loop ?
      This might give little better loading time since number of INodes being loaded will be very huge and there is no need to lookup for the step counter everytime,

      Show
      vinayrpet Vinayakumar B added a comment - prog.getCounter(Phase.LOADING_FSIMAGE, currentStep).increment(); Instead of getting counter for every INode being loaded, you can take reference of counter outside for-loop and just increment it inside for-loop ? This might give little better loading time since number of INodes being loaded will be very huge and there is no need to lookup for the step counter everytime,
      Hide
      hadoopqa Hadoop QA added a comment -



      -1 overall



      Vote Subsystem Runtime Comment
      0 pre-patch 17m 45s Pre-patch trunk compilation is healthy.
      +1 @author 0m 0s The patch does not contain any @author tags.
      -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
      +1 javac 7m 35s There were no new javac warning messages.
      +1 javadoc 9m 55s There were no new javadoc warning messages.
      +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
      +1 checkstyle 2m 16s There were no new checkstyle issues.
      +1 whitespace 0m 0s The patch has no lines that end in whitespace.
      +1 install 1m 39s mvn install still works.
      +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
      +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
      +1 native 3m 21s Pre-build of native portion
      +1 hdfs tests 165m 5s Tests passed in hadoop-hdfs.
          211m 51s  



      Subsystem Report/Notes
      Patch URL http://issues.apache.org/jira/secure/attachment/12737082/HDFS-8470_2.patch
      Optional Tests javadoc javac unit findbugs checkstyle
      git revision trunk / 806e407
      hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11216/artifact/patchprocess/testrun_hadoop-hdfs.txt
      Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11216/testReport/
      Java 1.7.0_55
      uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11216/console

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 16s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 21s Pre-build of native portion +1 hdfs tests 165m 5s Tests passed in hadoop-hdfs.     211m 51s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737082/HDFS-8470_2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 806e407 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11216/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11216/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11216/console This message was automatically generated.
      Hide
      surendrasingh Surendra Singh Lilhore added a comment -

      Thanks Vinayakumar B for review comments.
      Attached updated patch, Please review..

      Show
      surendrasingh Surendra Singh Lilhore added a comment - Thanks Vinayakumar B for review comments. Attached updated patch, Please review..
      Hide
      vinayrpet Vinayakumar B added a comment -

      +1 for latest patch.
      Regarding test, currently there is no end-to-end test for Namenode's startupprogress. TestStartupProgress are just unit tests, not cluster tests. So it would be good to have cluster tests for NameNode startupprogress. Can be done in follow-up jira.

      Will be committing the latest patch soon.

      Show
      vinayrpet Vinayakumar B added a comment - +1 for latest patch. Regarding test, currently there is no end-to-end test for Namenode's startupprogress. TestStartupProgress are just unit tests, not cluster tests. So it would be good to have cluster tests for NameNode startupprogress. Can be done in follow-up jira. Will be committing the latest patch soon.
      Hide
      vinayrpet Vinayakumar B added a comment -

      Committed to trunk and branch-2.
      Thanks for the contribution Surendra Singh Lilhore.

      Show
      vinayrpet Vinayakumar B added a comment - Committed to trunk and branch-2. Thanks for the contribution Surendra Singh Lilhore .
      Hide
      vinayrpet Vinayakumar B added a comment -

      Raised followup HDFS-8524 for adding tests.

      Show
      vinayrpet Vinayakumar B added a comment - Raised followup HDFS-8524 for adding tests.
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-trunk-Commit #7949 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7949/)
      HDFS-8470. fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604)

      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
      • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7949 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7949/ ) HDFS-8470 . fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #217 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/217/)
      HDFS-8470. fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604)

      • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #217 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/217/ ) HDFS-8470 . fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Yarn-trunk #947 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/947/)
      HDFS-8470. fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604)

      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #947 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/947/ ) HDFS-8470 . fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Mapreduce-trunk #2163 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2163/)
      HDFS-8470. fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604)

      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
      • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2163 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2163/ ) HDFS-8470 . fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #215 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/215/)
      HDFS-8470. fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604)

      • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #215 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/215/ ) HDFS-8470 . fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Hdfs-trunk #2145 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2145/)
      HDFS-8470. fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604)

      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2145 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2145/ ) HDFS-8470 . fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
      Hide
      hudson Hudson added a comment -

      FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #206 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/206/)
      HDFS-8470. fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604)

      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
      • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
      Show
      hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #206 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/206/ ) HDFS-8470 . fsimage loading progress should update inode, delegation token and cache pool count. (Contributed by surendra singh lilhore) (vinayakumarb: rev e965dcec378cb807856372425307598792977604) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java

        People

        • Assignee:
          surendrasingh Surendra Singh Lilhore
          Reporter:
          tongshiquan tongshiquan
        • Votes:
          0 Vote for this issue
          Watchers:
          7 Start watching this issue

          Dates

          • Due:
            Created:
            Updated:
            Resolved:

            Development