Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1443 Improve Datanode startup time
  3. HDFS-1445

Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.20.204.0, 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Batch hardlinking during "upgrade" snapshots, cutting time from aprx 8 minutes per volume to aprx 8 seconds. Validated in both Linux and Windows. Depends on prior integration with patch for HADOOP-7133.
      Show
      Batch hardlinking during "upgrade" snapshots, cutting time from aprx 8 minutes per volume to aprx 8 seconds. Validated in both Linux and Windows. Depends on prior integration with patch for HADOOP-7133 .
    • Tags:
      hard links, upgrade, snapshot

      Description

      It was a bit of a puzzle why we can do a full scan of a disk in about 30 seconds during FSDir() or getVolumeMap(), but the same disk took 11 minutes to do Upgrade replication via hardlinks. It turns out that the org.apache.hadoop.fs.FileUtil.createHardLink() method does an outcall to Runtime.getRuntime().exec(), to utilize native filesystem hardlink capability. So it is forking a full-weight external process, and we call it on each individual file to be replicated.

      As a simple check on the possible cost of this approach, I built a Perl test script (under Linux on a production-class datanode). Perl also uses a compiled and optimized p-code engine, and it has both native support for hardlinks and the ability to do "exec".

      • A simple script to create 256,000 files in a directory tree organized like the Datanode, took 10 seconds to run.
      • Replicating that directory tree using hardlinks, the same way as the Datanode, took 12 seconds using native hardlink support.
      • The same replication using outcalls to exec, one per file, took 256 seconds!
      • Batching the calls, and doing 'exec' once per directory instead of once per file, took 16 seconds.

      Obviously, your mileage will vary based on the number of blocks per volume. A volume with less than about 4000 blocks will have only 65 directories. A volume with more than 4K and less than about 250K blocks will have 4200 directories (more or less). And there are two files per block (the data file and the .meta file). So the average number of files per directory may vary from 2:1 to 500:1. A node with 50K blocks and four volumes will have 25K files per volume, or an average of about 6:1. So this change may be expected to take it down from, say, 12 minutes per volume to 2.

        Issue Links

          Activity

          Matt Foley created issue -
          Matt Foley made changes -
          Field Original Value New Value
          Assignee Matt Foley [ mattf ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Affects Version/s 0.20.2 [ 12314204 ]
          Component/s data-node [ 12312927 ]
          Hide
          Todd Lipcon added a comment -

          We could alternatively just add some calls to the new NativeIO library from HADOOP-6978 to call link(2) via JNI. Then we avoid the fork entirely.

          Show
          Todd Lipcon added a comment - We could alternatively just add some calls to the new NativeIO library from HADOOP-6978 to call link(2) via JNI. Then we avoid the fork entirely.
          Hide
          M. C. Srivas added a comment -

          If no one really uses hardlinks, why don't you get rid of this altogether?

          Show
          M. C. Srivas added a comment - If no one really uses hardlinks, why don't you get rid of this altogether?
          Nigel Daley made changes -
          Fix Version/s 0.22.0 [ 12314241 ]
          Hide
          Matt Foley added a comment -

          Requires coordinated change in both COMMON and HDFS.

          Show
          Matt Foley added a comment - Requires coordinated change in both COMMON and HDFS.
          Matt Foley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Batch hardlinking during "upgrade" snapshots, cutting time from aprx 8 minutes per volume to aprx 8 seconds. Validated in both Linux and Windows. Requires coordinated change in both COMMON and HDFS.
          Fix Version/s 0.22.0 [ 12314241 ]
          Hide
          Matt Foley added a comment -

          This patch requires coordinated change in both COMMON and HDFS.

          Show
          Matt Foley added a comment - This patch requires coordinated change in both COMMON and HDFS.
          Matt Foley made changes -
          Attachment HDFS-1445-trunk.v22_common_1-of-2.patch [ 12470695 ]
          Attachment HDFS-1445-trunk.v22_hdfs_2-of-2.patch [ 12470696 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12470696/HDFS-1445-trunk.v22_hdfs_2-of-2.patch
          against trunk revision 1068968.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:

          -1 contrib tests. The patch failed contrib unit tests.

          -1 system test framework. The patch failed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/157//testReport/
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/157//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12470696/HDFS-1445-trunk.v22_hdfs_2-of-2.patch against trunk revision 1068968. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/157//testReport/ Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/157//console This message is automatically generated.
          Matt Foley made changes -
          Link This issue is cloned as HDFS-1617 [ HDFS-1617 ]
          Matt Foley made changes -
          Link This issue is cloned as HDFS-1617 [ HDFS-1617 ]
          Matt Foley made changes -
          Link This issue is blocked by HADOOP-7133 [ HADOOP-7133 ]
          Hide
          Matt Foley added a comment -

          The COMMON portion of this patch is submitted under HADOOP-7133 to activate automated testing.

          Show
          Matt Foley added a comment - The COMMON portion of this patch is submitted under HADOOP-7133 to activate automated testing.
          Hide
          Matt Foley added a comment -

          A new version of part-1 of this patch has been attached to HADOOP-7133. I will resubmit part-2, attached to this issue, as soon as part-1 has been approved and committed.

          The new version of part-1 no longer breaks HDFS in the interim.

          Show
          Matt Foley added a comment - A new version of part-1 of this patch has been attached to HADOOP-7133 . I will resubmit part-2, attached to this issue, as soon as part-1 has been approved and committed. The new version of part-1 no longer breaks HDFS in the interim.
          Matt Foley made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Matt Foley made changes -
          Attachment HDFS-1445-trunk.v22_common_1-of-2.patch [ 12470695 ]
          Hide
          Matt Foley added a comment -

          HADOOP-7133 has been integrated with trunk, so proceed with submitting this second part of the fix.

          Show
          Matt Foley added a comment - HADOOP-7133 has been integrated with trunk, so proceed with submitting this second part of the fix.
          Matt Foley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Batch hardlinking during "upgrade" snapshots, cutting time from aprx 8 minutes per volume to aprx 8 seconds. Validated in both Linux and Windows. Requires coordinated change in both COMMON and HDFS. Batch hardlinking during "upgrade" snapshots, cutting time from aprx 8 minutes per volume to aprx 8 seconds. Validated in both Linux and Windows. Depends on prior integration with patch for HADOOP-7133.
          Fix Version/s 0.23.0 [ 12315571 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Matt Foley made changes -
          Link This issue blocks HADOOP-7182 [ HADOOP-7182 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12470696/HDFS-1445-trunk.v22_hdfs_2-of-2.patch
          against trunk revision 1080380.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12470696/HDFS-1445-trunk.v22_hdfs_2-of-2.patch against trunk revision 1080380. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//console This message is automatically generated.
          Hide
          Matt Foley added a comment -

          I examined the three failures noted:
          org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransfer
          org.apache.hadoop.hdfsproxy.TestAuthorizationFilter.testPathPermit
          org.apache.hadoop.hdfsproxy.TestAuthorizationFilter.testPathPermitQualified

          The two hdfsproxy issues don't seem to have anything to do with this patch nor the associated patch for HADOOP-7133.

          The TestFileConcurrentReader issue also doesn't, and the noted failure, "Too many open files", does not occur when I run it locally even on a very small box.

          Request approval of this patch despite these unit test failures, since they appear unrelated.

          Show
          Matt Foley added a comment - I examined the three failures noted: org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransfer org.apache.hadoop.hdfsproxy.TestAuthorizationFilter.testPathPermit org.apache.hadoop.hdfsproxy.TestAuthorizationFilter.testPathPermitQualified The two hdfsproxy issues don't seem to have anything to do with this patch nor the associated patch for HADOOP-7133 . The TestFileConcurrentReader issue also doesn't, and the noted failure, "Too many open files", does not occur when I run it locally even on a very small box. Request approval of this patch despite these unit test failures, since they appear unrelated.
          Hide
          Jakob Homan added a comment -

          Approval granted. +1.
          Will commit Monday. We should really consider if it's time to remove the extra code for ancient upgrades, but that's for another JIRA.

          Show
          Jakob Homan added a comment - Approval granted. +1. Will commit Monday. We should really consider if it's time to remove the extra code for ancient upgrades, but that's for another JIRA.
          Hide
          Jakob Homan added a comment -

          I've committed this to trunk. Resolving as fixed. Thanks, Matt!

          Show
          Jakob Homan added a comment - I've committed this to trunk. Resolving as fixed. Thanks, Matt!
          Jakob Homan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #563 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/563/)
          HDFS-1445. Batch the calls in DataStorage to FileUtil.createHardLink(). Contributed by Matt Foley.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #563 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/563/ ) HDFS-1445 . Batch the calls in DataStorage to FileUtil.createHardLink(). Contributed by Matt Foley.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Matt Foley made changes -
          Fix Version/s 0.20.204.0 [ 12316319 ]
          Matt Foley made changes -
          Link This issue blocks HDFS-2126 [ HDFS-2126 ]
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was released.
          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Harsh J added a comment -

          This isn't in 0.20.204. Not sure how that got into the Fix Version. Neither is HADOOP-7133, which is a co-patch.

          Removing from Fix Version.

          CHANGES.txt does not carry these names, so we're good release-wise. Must've been a JIRA field error.

          Show
          Harsh J added a comment - This isn't in 0.20.204. Not sure how that got into the Fix Version. Neither is HADOOP-7133 , which is a co-patch. Removing from Fix Version. CHANGES.txt does not carry these names, so we're good release-wise. Must've been a JIRA field error.
          Harsh J made changes -
          Fix Version/s 0.20.204.0 [ 12316319 ]
          Hide
          Harsh J added a comment -

          I was wrong. This one does seem to be present. Not present in CHANGES.txt though.

          Show
          Harsh J added a comment - I was wrong. This one does seem to be present. Not present in CHANGES.txt though.
          Harsh J made changes -
          Fix Version/s 0.20.204.0 [ 12316319 ]

            People

            • Assignee:
              Matt Foley
              Reporter:
              Matt Foley
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development