[HDFS-1795] Port 0.20-append changes onto 0.20-security-203

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.20.205.0
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 29/Mar/11 21:31
• Updated: 15/Mar/12 06:44
• Resolved: 15/Mar/12 06:44

Description

Port 0.20-append changes onto 0.20-security-203.

I started with a Git repository cloned from git://git.apache.org/hadoop-common.git . Branch 'branch-0.20-security-203' was used as the starting point for the work. I then enumerated over the 0.20-append specific patches in 'branch-0.20-append'. Each was applied if not already via cherry pick except for as noted below. This process in effect replayed the evolution of 0.20-append branch on top of 0.20-security-203.

The specific functional changes that HBase absolutely relies upon are specially mentioned.

Generally I ran the full test suite after each change. There were a couple of exceptions where pairs of adjacent change sets were strongly related, in which case I applied them in sequence, then ran the test suite. During this process I encountered no test failures except for one test in TestFileAppend4, a test brought in from the append branch, and I still need to dig in to see if this is a real problem or if the test needs to be changed to work on top of security-203.

  commit b9ad012eaf3915c2169a02a7130b54cbcc1d8a89
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Fri Jun 4 07:20:10 2010 +0000
    HDFS-200. Support append and sync for hadoop 0.20 branch.
 
    Required for HBase

  commit c968e11b5a60fc6f28e4e43fbbc8a99e7e49a659
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 9 23:09:07 2010 +0000
    HDFS-101. DFSClient correctly detects second datanode failure in write
    pipeline. (Nicolas Spiegelberg via dhruba)

  Excluded
  Already in 0.20-security-203 according to search of Git change log

  commit 9f7e5ed2ff47444a1dcd12ed34796929d5b9f7d5
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 9 23:12:21 2010 +0000
    HDFS-988. Fix bug where savenameSpace can corrupt edits log.
    (Nicolas Spiegelberg via dhruba)

  commit dfbbd6fbadaa95c54a1040b4fe8854b1b858d7a5
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Thu Jun 10 18:46:03 2010 +0000
    HDFS-826. Allow a mechanism for an application to detect that
    datanode(s) have died in the write pipeline. (dhruba)

  Required for HBase

  Commit be8d32503d30208a2d7772b3b4b2a270938a4004
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Thu Jun 10 22:25:39 2010 +0000
    HDFS-142. Blocks that are being written by a client are stored in the
    blocksBeingWritten directory.
    (Dhruba Borthakur, Nicolas Spiegelberg, Todd Lipcon via dhruba)

  commit 856efc2e95aaacc597d669c1b053634ff752dbec
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Fri Jun 11 00:48:41 2010 +0000
    HDFS-630. Client can exclude specific nodes in the write pipeline.
    (Nicolas Spiegelberg via dhruba)

  Required for HBase

  commit 2da1a05fc0cc0429229e87694977bae2ba370625
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Fri Jun 11 01:02:13 2010 +0000
    HDFS-457. Better handling of volume failure in DataNode Storage.
    (Nicolas Spiegelberg via dhruba)

  Excluded
  Already in 0.20-security-203 according to search of Git change log

  commit bd42393cd3a3a731ea98b25ddb528ad03a1ab4af
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Fri Jun 11 23:37:38 2010 +0000
    HDFS-1054. remove sleep before retry for allocating a block.
    (Todd Lipcon via dhruba)

  commit 120441b9e571a5703ac39b47608e87182f0f4972
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 16 20:53:12 2010 +0000
    HDFS-445. pread should refetch block locations when necessary.
    (Todd Lipcon via dhruba)

  Excluded
  Already in 0.20-security-203 according to search of Git change log

  commit 2004aa453ba6b7ee2045093ba313ef8551a7f8da
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 16 20:59:10 2010 +0000
    HDFS-561. Fix write pipeline

  commit 2a8227b0e6be8937fc4a654899be2a22c1f6efbe
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 16 21:13:24 2010 +0000
    HDFS-927. DFSInputStream retries too many times for new block
    locations. (Todd Lipcon via dhruba)

  Excluded
  Already in 0.20-security-203 according to search of Git change log

  commit b1e49dbf50a429cf01b636caa2666ff81ed2a016
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 16 21:21:45 2010 +0000
    HDFS-1215. Fix unti test TestNodeCount.
    (Todd Lipcon via dhruba)

  Excluded
  TestNodeCount not failing on current working branch

  commit b5078c57478deda9974ed40897045d38b100b9fc
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 16 21:25:38 2010 +0000
    HDFS-606. Fix ConcurrentModificationException in
    invalidateCorruptReplicas. (Todd Lipcon via dhruba)

  commit c3d35f8cdd761cbcd0a4ef851e37ccdb2d25d7e0
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 16 21:42:53 2010 +0000
    HDFS-1216. Upgrade to junit 4.5
    (Todd Lipcon via dhruba)

  Excluded
  Already using Junit 4.5 on current working branch

  commit c21b6834e982b0bf2edaf61dae8a0028076ccb60
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Thu Jun 17 01:34:56 2010 +0000
    HDFS-1207. FSNamesystem.stallReplicationWork should be volatile.
    (Todd Lipcon via dhruba)

  commit e8a1a09fec716bb013f94f1c14f4ae8f8409ee68
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Wed Jun 16 21:57:39 2010 +0000
    HDFS-1141. completeFile does not check lease ownership.
    (Todd Lipcon via dhruba)

  commit ca458a07a6d8cb5cda8c589b1a23dabffacef7fe
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Thu Jun 17 01:44:32 2010 +0000
    HDFS-1204. Lease expiration should recover single files,
    not entire lease holder (Sam Rash via dhruba)

  commit 62441fbd516ec9132619d448a1051554d29d2dba
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Thu Jun 17 01:52:50 2010 +0000
    HDFS-1210. DFSClient should log exception when block recovery fails.
    (Todd Lipcon via dhruba)

  Not a required change but a minor useful one

  commit 08b67cdf9ff7dada8aa86a2443a0ffc1a0a83c20
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Thu Jun 17 01:59:30 2010 +0000
    HDFS-1211. Block receiver should not log "rewind" packets at INFO
    level. (Todd Lipcon via dhruba)

  Not a required change but a minor useful one

  commit 0c9911a623891e82c40d4c1c3a523f3a85a85216
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Sun Jun 20 08:31:54 2010 +0000
    HDFS-1118. Fix socketleak on DFSClient.
    (Zheng Shao via dhruba)

  commit db9989c091b37446e4f4d468b24c4bd08b1ea337
  Author: Tsz-wo Sze <[e-mail]>
  Date:   Mon Jun 21 21:26:45 2010 +0000
    MAPREDUCE-1880. Fix BigDecimal.divide(..) in the pi example.

  Excluded
  Not an HDFS change

  commit 11fa325540526da57a9e178463017853058ba921
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Tue Jun 22 05:55:48 2010 +0000
    HDFS-1254. Support append/sync via the default configuration.
    (dhruba)

  commit 9e405ca177aba719b04cfd7d33d400d311bf8db7
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Thu Jul 1 08:37:30 2010 +0000
    HDFS-1057.  Concurrent readers hit ChecksumExceptions if following
    a writer to very end of file (Sam Rash via dhruba)

  commit 6512b527cda8efa0c622a8657af96ac785625773
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Tue Jul 6 06:52:20 2010 +0000
    HDFS-1202.  DataBlockScanner throws NPE when updated before
    initialized. (Todd Lipcon via dhruba)

  commit 116e71fea80c31042520f364625ba7b1647cdb09
  Author: Tsz-wo Sze <[e-mail]>
  Date:   Tue Jul 6 17:39:25 2010 +0000
    HDFS-1258. Clearing namespace quota on "/" corrupts fs image.  

  commit ca88978b319debe146e9988d882393033b1d2d62
  Author: Tsz-wo Sze <[e-mail]>
  Date:   Wed Sep 22 20:52:25 2010 +0000
    MAPREDUCE-1280. Update Eclipse plugin to the new eclipse.jdt API.

  Excluded
  Not an HDFS change

  commit a68b163fc31c3d82c63b69ae99e9954fef4df4af
  Author: Hairong Kuang <[e-mail]>
  Date:   Mon Sep 27 18:14:20 2010 +0000
    HDFS-1346. DFSClient receives out of order packet ack.

  commit 7083b46208f56776be9ada2d0e60d95cb4fdb67e
  Author: Hairong Kuang <[e-mail]>
  Date:   Mon Nov 1 06:17:40 2010 +0000
    HDFS-724.  Use a bidirectional heartbeat to detect stuck pipeline.

  commit eb2346ea7726c33715f3e651f4babcb6e6c3f0c2
  Author: Hairong Kuang <[e-mail]>
  Date:   Tue Nov 16 17:53:17 2010 +0000
    Fix incorrect heartbeat ack serialization in HDFS-724 for append 0.20.

  Combined with 7083b462

  commit 0e996bdba420fd0031490457e27dda67f58de18f
  Author: Eli Collins <[e-mail]>
  Date:   Tue Nov 9 01:59:27 2010 +0000
    HDFS-1240. svn merge -c 1032810 from branch-0.20.

  Excluded
  TestDFSShell not failing on working branch

  Commit eb2346ea7726c33715f3e651f4babcb6e6c3f0c2
  Author: Hairong Kuang <[e-mail]>
  Date:   Tue Nov 16 17:53:17 2010 +0000
    HDFS-895. Allow hflush/sync to occur in parallel with new writes to
    the file

  Required to mitigate HLog flush penalty
  Need to raise default heap for tests for HDFS-895 unit tests to pass

  commit 9a3094f3a085817d377ce69fb8eb002f966f992b
  Author: Dhruba Borthakur <[e-mail]>
  Date:   Fri Dec 10 05:38:06 2010 +0000
    HDFS-1520. Lightweight NameNode operation recoverLease to trigger
    lease recovery. (Hairong Kuang via dhruba)

  Required by HBase for master to seize ownership of and quickly split
  HLogs of failed regionservers

  commit f9436bf6bd2f5bc4f8cebe2781c5ce628db31239
  Author: Hairong Kuang <[e-mail]>
  Date:   Fri Jan 7 20:11:38 2011 +0000
    HDFS-1555. Disallow pipeline recovery if a file is already being lease

  Excluded
  Bad commit

  commit 966ef38cb13d75ada40de2b9951496ddcb6918a4
  Author: Hairong Kuang <[e-mail]>
  Date:   Fri Jan 7 20:19:40 2011 +0000
    Revert the change made to HDFS-1555:

  Excluded

  commit e499be8c74a871ef06aa8a74c621e68d50167d3d
  Author: Hairong Kuang <[e-mail]>
  Date:   Fri Jan 7 20:25:23 2011 +0000
    HDFS-1555. Disallow pipelien recovery if a file is already being
    lease recovered

  commit df0d79cc2b09438c079fdf10b913936492117917
  Author: Hairong Kuang <[e-mail]>
  Date:   Mon Jan 10 19:01:36 2011 +0000
    HDFS-1554. New semantics for recoverLease.

  Required by HBase for master to seize ownership of and quickly split
  HLogs of failed regionservers