Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9221

HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
    • Component/s: performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When the BufferDecoder in BlockListAsLongs converts the stored value to a ReplicaState enum it calls ReplicaState.getState(int) unfortunately this method creates a ReplicaState[] for each call since it calls ReplicaState.values().

      This patch creates a cached version of the values and thus avoid all allocation when doing the conversion.

        Issue Links

          Activity

          Hide
          sfriberg Staffan Friberg added a comment -

          Small change so could hopefully go into 2.8.0

          Show
          sfriberg Staffan Friberg added a comment - Small change so could hopefully go into 2.8.0
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1 pending jenkins. Thanks, Staffan Friberg.

          Show
          cmccabe Colin P. McCabe added a comment - +1 pending jenkins. Thanks, Staffan Friberg .
          Hide
          sfriberg Staffan Friberg added a comment -

          Uploaded new version of the patch as the old one was accidentally using the SVN repository.

          Show
          sfriberg Staffan Friberg added a comment - Uploaded new version of the patch as the old one was accidentally using the SVN repository.
          Hide
          daryn Daryn Sharp added a comment -

          +1

          Show
          daryn Daryn Sharp added a comment - +1
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 32s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 9m 7s There were no new javac warning messages.
          +1 javadoc 11m 43s There were no new javadoc warning messages.
          -1 release audit 0m 21s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 34s The applied patch generated 1 new checkstyle issues (total was 23, now 24).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 2m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 33s Pre-build of native portion
          -1 hdfs tests 210m 10s Tests failed in hadoop-hdfs.
              261m 11s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.TestFSNamesystem
          Timed out tests org.apache.hadoop.hdfs.TestDatanodeDeath



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765897/HADOOP-9221.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4f6e842
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12897/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12897/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12897/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12897/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12897/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 9m 7s There were no new javac warning messages. +1 javadoc 11m 43s There were no new javadoc warning messages. -1 release audit 0m 21s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 34s The applied patch generated 1 new checkstyle issues (total was 23, now 24). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 2m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 33s Pre-build of native portion -1 hdfs tests 210m 10s Tests failed in hadoop-hdfs.     261m 11s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFSNamesystem Timed out tests org.apache.hadoop.hdfs.TestDatanodeDeath Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765897/HADOOP-9221.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4f6e842 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12897/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12897/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12897/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12897/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12897/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          Committed to trunk and branch-2, thanks Staffan for the contribution, and Colin, Daryn for the review.

          Show
          hitliuyi Yi Liu added a comment - Committed to trunk and branch-2, thanks Staffan for the contribution, and Colin, Daryn for the review.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8609 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8609/)
          HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8609 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8609/ ) HDFS-9221 . HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #511 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/511/)
          HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #511 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/511/ ) HDFS-9221 . HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2457 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2457/)
          HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2457 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2457/ ) HDFS-9221 . HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1248 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1248/)
          HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1248 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1248/ ) HDFS-9221 . HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #523 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/523/)
          HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #523 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/523/ ) HDFS-9221 . HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2421 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2421/)
          HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2421 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2421/ ) HDFS-9221 . HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #483 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/483/)
          HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #483 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/483/ ) HDFS-9221 . HdfsServerConstants#ReplicaState#getState should avoid (yliu: rev 0ff1216100d16cfa862854a89cd1be8969b0bd7e) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          Hide
          kihwal Kihwal Lee added a comment -

          This is not a bug fix, but still a low risk performance improvement. What do others think about pulling this into 2.7.2?

          Show
          kihwal Kihwal Lee added a comment - This is not a bug fix, but still a low risk performance improvement. What do others think about pulling this into 2.7.2?
          Hide
          daryn Daryn Sharp added a comment -

          +1 on 2.7.2. It's a zero risk change with plenty of benefit.

          Show
          daryn Daryn Sharp added a comment - +1 on 2.7.2. It's a zero risk change with plenty of benefit.
          Hide
          kihwal Kihwal Lee added a comment -

          Cherry-picked to branch-2.7.

          Show
          kihwal Kihwal Lee added a comment - Cherry-picked to branch-2.7.

            People

            • Assignee:
              sfriberg Staffan Friberg
              Reporter:
              sfriberg Staffan Friberg
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development