Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13192

org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Critical
        • Resolution: Fixed
        • Affects Version/s: 2.6.2
        • Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
        • Component/s: util
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          org.apache.hadoop.util.LineReader.readCustomLine() has a bug,
          when line is aaaabccc, recordDelimiter is aaab, the result should be a,ccc,
          show the code on line 310:
          for (; bufferPosn < bufferLength; ++bufferPosn) {
          if (buffer[bufferPosn] == recordDelimiterBytes[delPosn]) {
          delPosn++;
          if (delPosn >= recordDelimiterBytes.length)

          { bufferPosn++; break; }
          } else if (delPosn != 0) { bufferPosn--; delPosn = 0; }
          }

          shoud be :
          for (; bufferPosn < bufferLength; ++bufferPosn) {
          if (buffer[bufferPosn] == recordDelimiterBytes[delPosn]) {
          delPosn++;
          if (delPosn >= recordDelimiterBytes.length) { bufferPosn++; break; }

          } else if (delPosn != 0)

          { // ------------- change here ------------- start ---- bufferPosn -= delPosn; // ------------- change here ------------- end ---- delPosn = 0; }

          }

            Issue Links

            relates to

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5948 org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well

            • Critical - Crashes, loss of data, severe memory leak.
            • Closed
            links to

            Web Link GitHub Pull Request #99

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              GitHub user zhudebin opened a pull request:

              https://github.com/apache/hadoop/pull/95

              fix bug HADOOP-13192

              fix bug HADOOP-13192

              You can merge this pull request into a Git repository by running:

              $ git pull https://github.com/zhudebin/hadoop branch-2.6-fixbug

              Alternatively you can review and apply these changes as the patch at:

              https://github.com/apache/hadoop/pull/95.patch

              To close this pull request, make a commit to your master/trunk branch
              with (at least) the following in the commit message:

              This closes #95

              commit 0b1e66f01dae1c5558e897e35b1cbe533d9c4542
              Author: Andrew Wang <wang@apache.org>
              Date: 2015-03-24T05:00:34Z

              HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu.

              (cherry picked from commit 50ee8f4e67a66aa77c5359182f61f3e951844db6)
              (cherry picked from commit 2f46ee50bd4efc82ba3d30bd36f7637ea9d9714e)

              Conflicts:
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
              hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
              hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
              hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java
              hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

              (cherry picked from commit 03d4af39e794dc03d764122077b434d658b6405e)

              commit bc8728cd27870e048fd90d1e07ea92e8c9ed310d
              Author: Kihwal Lee <kihwal@apache.org>
              Date: 2015-03-30T15:11:25Z

              HDFS-7742. Favoring decommissioning node for replication can cause a block to stay
              underreplicated for long periods. Contributed by Nathan Roberts.
              (cherry picked from commit 04ee18ed48ceef34598f954ff40940abc9fde1d2)

              (cherry picked from commit c4cedfc1d601127430c70ca8ca4d4e2ee2d1003d)
              (cherry picked from commit c6b68a82adea8de488b255594d35db8e01f5fc8f)

              commit 8a9665a586624cfe7f11ad9e21976465e0bb0e21
              Author: Junping Du <junping_du@apache.org>
              Date: 2015-04-02T19:13:03Z

              MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe.
              (cherry picked from commit eccb7d46efbf07abcc6a01bd5e7d682f6815b824)

              (cherry picked from commit cacadea632f7ab6fe4fdb1432e1a2c48e8ebd55f)
              (cherry picked from commit 2abd4f61075739514fb3e63b118448895be02a30)

              commit c3f5ea11eca30a617cab2a716dd08dff20db3791
              Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
              Date: 2015-04-06T15:54:46Z

              HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe)

              (cherry picked from commit 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)
              (cherry picked from commit a827089905524e10638c783ba908a895d621911d)

              Conflicts:
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

              (cherry picked from commit c3a3092c37926eca75ea149c4c061742f6599b40)

              commit 31d30e8111d7c4ef6400b5fe51cc67b17ab34908
              Author: Arpit Agarwal <arp@apache.org>
              Date: 2015-04-08T18:38:21Z

              HDFS-8072. Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal)

              (cherry picked from commit f0324738c9db4f45d2b1ec5cfb46c5f2b7669571)

              Conflicts:
              hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java

              (cherry picked from commit de21de7e2243ef8a89082121d838b88e3c10f05b)

              commit 619f7938466e907f335941bbbbd928c6272a0482
              Author: Kihwal Lee <kihwal@apache.org>
              Date: 2015-04-08T20:39:25Z

              HDFS-8046. Allow better control of getContentSummary. Contributed by Kihwal Lee.
              (cherry picked from commit 285b31e75e51ec8e3a796c2cb0208739368ca9b8)

              (cherry picked from commit 7e622076d41a85fc9a8600fb270564a085f5cd83)

              Conflicts:
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java

              (cherry picked from commit 1ef5e0b18066ca949adcf4c55a41f186c47e7264)

              commit 752e3da738080cf7259b9085b736b42956d0bf1e
              Author: Vinod Kumar Vavilapalli <vinodkv@apache.org>
              Date: 2015-04-09T20:08:53Z

              YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp.

              (cherry picked from commit 9c5911294e0ba71aefe4763731b0e780cde9d0ca)
              (cherry picked from commit 1ff3fd33ed6f2ac09c774cc42b0107c5dbd9c19d)
              (cherry picked from commit 82c722aae86669325672dd10840447434f15e7fd)

              commit dcc0d4658c9bc7f488cecdcbc05e9216234b36aa
              Author: Akira Ajisaka <aajisaka@apache.org>
              Date: 2015-04-10T16:29:08Z

              MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li.

              (cherry picked from commit 4cec9975c0e2c2fcca090888dc2c5e149c5be1e9)
              (cherry picked from commit 3de0bf9a35094f3c2ac216992d861729251b6a3d)
              (cherry picked from commit c3b2b377f19293797a88f4636f07e2ab8e18c78b)

              commit b7e7896410bdd8aac5a8b9f9327227542e730ece
              Author: Jing Zhao <jing9@apache.org>
              Date: 2015-04-14T22:05:23Z

              HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao.

              (cherry picked from commit fddd55279d0bdd08b3b40aba6fe2ded1d2e0d846)
              (cherry picked from commit 38b031d6bab8527698bd186887d301bd6a63cf01)

              Conflicts:
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
              hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java

              (cherry picked from commit b4e227e65d3a81150f46aac5e987d759defc1452)

              commit 61f2ddb125f6af1775ce5d219f22eb079071869c
              Author: Wangda Tan <wangda@apache.org>
              Date: 2015-04-18T00:11:22Z

              YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda)

              (cherry picked from commit f65eeb412d140a3808bcf99344a9f3a965918f70)
              (cherry picked from commit e7cbecddc3e7ca5386c71aa4deb67f133611415c)
              (cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd)

              commit e081593042de460b5930dadd0bd3aa313fe09873
              Author: Wangda Tan <wangda@apache.org>
              Date: 2015-04-18T00:22:27Z

              YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda)

              (cherry picked from commit f47a5763acd55cb0b3f16152c7f8df06ec0e09a9)
              (cherry picked from commit 3316cd4357ff6ccc4c76584813092adb1c2b4d43)
              (cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670)

              commit 2f66dff7cec001064332cfcf7a0e8873cd98482e
              Author: Karthik Kambatla <kasha@apache.org>
              Date: 2015-03-04T22:42:07Z

              MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha)

              (cherry picked from commit c66c3ac6bf9f63177279feec3f2917e4b882e2bc)
              (cherry picked from commit f4d6c5e337e76dc408c9c8f19e306c3f4ba80d8e)
              (cherry picked from commit 2b9cac26ec5457ad4e18ec418ff8249c37861eb8)

              commit c253ac68ebb2f07a20af89636ea611fb887739f2
              Author: Robert Kanter <rkanter@apache.org>
              Date: 2015-04-20T21:14:08Z

              MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter)

              (cherry picked from commit d50e8f09287deeb51012d08e326a2ed71a6da869)
              (cherry picked from commit 756c2542930756fef1cbff82056b418070f8d55f)
              (cherry picked from commit 4f0fd74880e58958d89ca1cd4b7f665520595b2c)

              commit 736ebeca3347243fa122f976532ed2536e51b6f5
              Author: Gera Shegalov <gera@apache.org>
              Date: 2015-04-21T18:57:42Z

              HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up split calculation (gera)

              (cherry picked from commit 6d2cf9fbbd02482315a091ab07af26e40cc5134f)
              (cherry picked from commit 1544c63602089b690e850e0e30af4589513a2371)

              commit e0e93e9f8d5aaf2eb37d4e91f29f7395109109a2
              Author: Steve Loughran <stevel@apache.org>
              Date: 2015-04-23T20:39:30Z

              HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel)

              (cherry picked from commit a6a5d1d6b5ee76c829ba7b54a4ad619f7b986681)

              commit 9df4b7c10e38d517fc19826657e0f3779d543f07
              Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
              Date: 2015-03-15T05:36:46Z

              HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)

              (cherry picked from commit bc9cb3e271b22069a15ca110cd60c860250aaab2)
              (cherry picked from commit c71b54fa32ef9149692269a95c918506e338e181)

              commit 181281cfcada2246af998b388e9f7e96d0228022
              Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
              Date: 2015-04-24T02:03:44Z

              HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe)

              (cherry picked from commit a8898445dc9b5cdb7230e2e23a57393c9f378ff0)
              (cherry picked from commit 3d0385c3c8fab72be02453a158c7692212ab3b5e)

              commit 8770f1f03952661f49aea65ae6498be676677812
              Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
              Date: 2015-04-24T01:59:52Z

              HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm (cmccabe)

              (cherry picked from commit a0e0a63209b5eb17dca5cc503be36aa52defeabd)
              (cherry picked from commit 788b76761d5dfadf688406d50169e95401fe5d33)

              commit 9af5b1dcd08c41df026bb86129abd42863baccef
              Author: Xuan <xgong@apache.org>
              Date: 2015-01-26T03:37:57Z

              YARN-3024. LocalizerRunner should give DIE action when all resources are
              localized. Contributed by Chengbing Liu

              (cherry picked from commit 0d6bd62102f94c55d59f7a0a86a684e99d746127)
              (cherry picked from commit a7696b3fbfacd98a892bbb3678663658c7b9d2bd)
              (cherry picked from commit 9e30232004ab7c3c3bfde3b8b27c37fa7065f6be)

              commit 6ade6b5051fc16472ada1d5c73792999e74aa67c
              Author: Karthik Kambatla <kasha@apache.org>
              Date: 2015-04-26T16:13:46Z

              YARN-3464. Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha)

              (cherry picked from commit 47279c3228185548ed09c36579b420225e4894f5)
              (cherry picked from commit 4045c41afe440b773d006e962bf8a5eae3fdc284)
              (cherry picked from commit 6f2cc0dfa8f21984ecdab59dc087ccf525934930)

              commit 262275d561a3d138e67942a35b7c15f1ef01bfc8
              Author: Vinod Kumar Vavilapalli <vinodkv@apache.org>
              Date: 2015-04-27T21:58:16Z

              MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe.

              (cherry picked from commit 9fc32c5c4d1d5f50c605bdb0e3b13f44c86660c8)
              (cherry picked from commit 32dc13d907a416049bdb7deff429725bd6dbcb49)
              (cherry picked from commit aad56fe3a2f29e73a013b9afa9b44b151a34e0f3)

              commit c90f025e903573b5e94c2d205cc1e3a6691d4294
              Author: Gera Shegalov <gera@apache.org>
              Date: 2015-05-01T22:44:36Z

              HADOOP-11491. HarFs incorrectly declared as requiring an authority. (Brahma Reddy Battula via gera)

              (cherry picked from commit f343f8657e2b01773a32c2c7d960dc368954b42e)
              (cherry picked from commit 58970d69de8a1662e4548cd6d4ca460dd70562f8)

              commit 41601eae6b266ce88e47a06a1dc765cfc296b640
              Author: Jason Lowe <jlowe@apache.org>
              Date: 2015-05-04T19:02:39Z

              MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov
              (cherry picked from commit 7dc3c1203d1ab14c09d0aaf0869a5bcdfafb0a5a)

              (cherry picked from commit 87c2d915f1cc799cb4020c945c04d3ecb82ee963)
              (cherry picked from commit 1da62ba736f5f161a18a52b7ca0d212786f3848c)

              commit 08006b84f8cb6119bd17f6cfc8da2598e24346ca
              Author: Xiaoyu Yao <xyao@apache.org>
              Date: 2015-05-05T20:41:14Z

              HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao)

              (cherry picked from commit 0100b155019496d077f958904de7d385697d65d9)
              (cherry picked from commit e68e8b3b5cff85bfd8bb5b00b9033f63577856d6)

              Conflicts:
              hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java

              (cherry picked from commit b054cb68fa0fc6d1e9e77ac84575731e7d1ec0c7)

              commit bbed4e67a99196a750e9cb5ff9f9595b9bc7a196
              Author: Kihwal Lee <kihwal@apache.org>
              Date: 2015-05-08T14:32:07Z

              HDFS-7894. Rolling upgrade readiness is not updated in jmx until query command is issued. Contributed by Brahma Reddy Battula.
              (cherry picked from commit 6f622672b62aa8d719060063ef0e47480cdc8655)

              (cherry picked from commit 802a5775f3522c57c60ae29ecb9533dbbfecfe76)
              (cherry picked from commit 995382c5234ad6c07f327e5d1f2a1c7e391a0b60)

              commit 51d5fc62c2eeda42f64652980693157dc9686812
              Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
              Date: 2014-10-27T16:53:16Z

              HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe)
              (cherry picked from commit baf794dc404ac54f4e8332654eadfac1bebacb8f)

              (cherry picked from commit 5f3d967aaefa0b20ef1586b4048b8fa5345d2618)

              Conflicts:
              hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm

              (cherry picked from commit a776ef5ad2876b9acf6cf89824c306783f7759f1)

              commit 6c0ad99b9d45fbad0b860944dda92e1a7a989dc5
              Author: Tsuyoshi Ozawa <ozawa@apache.org>
              Date: 2015-05-12T15:28:17Z

              MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du.

              (cherry picked from commit f4e2b3cc0b1f4e49c306bc09a9dddd0495225bb2)
              (cherry picked from commit fb5b0ebb459cc8812084090a7ce7ac29e2ad147c)
              (cherry picked from commit 3a3dcf0bb689d6aa8bc524a31799067702814674)

              commit 778da79e6f28f5528e107f9b3122925f4a0f05b2
              Author: Jason Lowe <jlowe@apache.org>
              Date: 2015-05-13T21:06:47Z

              YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du
              (cherry picked from commit 711d77cc54a64b2c3db70bdacc6bf2245c896a4b)

              (cherry picked from commit a81ad814610936a02e55964fbe08f7b33fe29b23)
              (cherry picked from commit aa82b0684554be8d09f6fcd88826f167922280cc)

              commit 7b1a71a7ad5602dbeb5758a650812413c277c21c
              Author: Xuan <xgong@apache.org>
              Date: 2015-05-16T05:39:49Z

              YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang

              (cherry picked from commit b0ad644083a0dfae3a39159ac88b6fc09d846371)
              (cherry picked from commit 802676e1be350785d8c0ad35f6676eeb85b2467b)
              (cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298)

              commit b3ca4dfaff627a04a1a07aa305e1d30651e66056
              Author: Kihwal Lee <kihwal@apache.org>
              Date: 2015-05-08T21:37:26Z

              HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah.
              (cherry picked from commit 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

              (cherry picked from commit f264a5aeede7e144af11f5357c7f901993de8e12)

              Conflicts:
              hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java

              (cherry picked from commit 470019e9b88e0fcede926442b91d102b595c7ace)

              Show
              githubbot ASF GitHub Bot added a comment - GitHub user zhudebin opened a pull request: https://github.com/apache/hadoop/pull/95 fix bug HADOOP-13192 fix bug HADOOP-13192 You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhudebin/hadoop branch-2.6-fixbug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/95.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #95 commit 0b1e66f01dae1c5558e897e35b1cbe533d9c4542 Author: Andrew Wang <wang@apache.org> Date: 2015-03-24T05:00:34Z HDFS-7960 . The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. (cherry picked from commit 50ee8f4e67a66aa77c5359182f61f3e951844db6) (cherry picked from commit 2f46ee50bd4efc82ba3d30bd36f7637ea9d9714e) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java (cherry picked from commit 03d4af39e794dc03d764122077b434d658b6405e) commit bc8728cd27870e048fd90d1e07ea92e8c9ed310d Author: Kihwal Lee <kihwal@apache.org> Date: 2015-03-30T15:11:25Z HDFS-7742 . Favoring decommissioning node for replication can cause a block to stay underreplicated for long periods. Contributed by Nathan Roberts. (cherry picked from commit 04ee18ed48ceef34598f954ff40940abc9fde1d2) (cherry picked from commit c4cedfc1d601127430c70ca8ca4d4e2ee2d1003d) (cherry picked from commit c6b68a82adea8de488b255594d35db8e01f5fc8f) commit 8a9665a586624cfe7f11ad9e21976465e0bb0e21 Author: Junping Du <junping_du@apache.org> Date: 2015-04-02T19:13:03Z MAPREDUCE-6303 . Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe. (cherry picked from commit eccb7d46efbf07abcc6a01bd5e7d682f6815b824) (cherry picked from commit cacadea632f7ab6fe4fdb1432e1a2c48e8ebd55f) (cherry picked from commit 2abd4f61075739514fb3e63b118448895be02a30) commit c3f5ea11eca30a617cab2a716dd08dff20db3791 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2015-04-06T15:54:46Z HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cherry picked from commit 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) (cherry picked from commit a827089905524e10638c783ba908a895d621911d) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java (cherry picked from commit c3a3092c37926eca75ea149c4c061742f6599b40) commit 31d30e8111d7c4ef6400b5fe51cc67b17ab34908 Author: Arpit Agarwal <arp@apache.org> Date: 2015-04-08T18:38:21Z HDFS-8072 . Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal) (cherry picked from commit f0324738c9db4f45d2b1ec5cfb46c5f2b7669571) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java (cherry picked from commit de21de7e2243ef8a89082121d838b88e3c10f05b) commit 619f7938466e907f335941bbbbd928c6272a0482 Author: Kihwal Lee <kihwal@apache.org> Date: 2015-04-08T20:39:25Z HDFS-8046 . Allow better control of getContentSummary. Contributed by Kihwal Lee. (cherry picked from commit 285b31e75e51ec8e3a796c2cb0208739368ca9b8) (cherry picked from commit 7e622076d41a85fc9a8600fb270564a085f5cd83) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java (cherry picked from commit 1ef5e0b18066ca949adcf4c55a41f186c47e7264) commit 752e3da738080cf7259b9085b736b42956d0bf1e Author: Vinod Kumar Vavilapalli <vinodkv@apache.org> Date: 2015-04-09T20:08:53Z YARN-3055 . Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. (cherry picked from commit 9c5911294e0ba71aefe4763731b0e780cde9d0ca) (cherry picked from commit 1ff3fd33ed6f2ac09c774cc42b0107c5dbd9c19d) (cherry picked from commit 82c722aae86669325672dd10840447434f15e7fd) commit dcc0d4658c9bc7f488cecdcbc05e9216234b36aa Author: Akira Ajisaka <aajisaka@apache.org> Date: 2015-04-10T16:29:08Z MAPREDUCE-6300 . Task list sort by task id broken. Contributed by Siqi Li. (cherry picked from commit 4cec9975c0e2c2fcca090888dc2c5e149c5be1e9) (cherry picked from commit 3de0bf9a35094f3c2ac216992d861729251b6a3d) (cherry picked from commit c3b2b377f19293797a88f4636f07e2ab8e18c78b) commit b7e7896410bdd8aac5a8b9f9327227542e730ece Author: Jing Zhao <jing9@apache.org> Date: 2015-04-14T22:05:23Z HDFS-8127 . NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao. (cherry picked from commit fddd55279d0bdd08b3b40aba6fe2ded1d2e0d846) (cherry picked from commit 38b031d6bab8527698bd186887d301bd6a63cf01) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java (cherry picked from commit b4e227e65d3a81150f46aac5e987d759defc1452) commit 61f2ddb125f6af1775ce5d219f22eb079071869c Author: Wangda Tan <wangda@apache.org> Date: 2015-04-18T00:11:22Z YARN-3493 . RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) (cherry picked from commit f65eeb412d140a3808bcf99344a9f3a965918f70) (cherry picked from commit e7cbecddc3e7ca5386c71aa4deb67f133611415c) (cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd) commit e081593042de460b5930dadd0bd3aa313fe09873 Author: Wangda Tan <wangda@apache.org> Date: 2015-04-18T00:22:27Z YARN-3487 . CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (cherry picked from commit f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) (cherry picked from commit 3316cd4357ff6ccc4c76584813092adb1c2b4d43) (cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670) commit 2f66dff7cec001064332cfcf7a0e8873cd98482e Author: Karthik Kambatla <kasha@apache.org> Date: 2015-03-04T22:42:07Z MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (cherry picked from commit c66c3ac6bf9f63177279feec3f2917e4b882e2bc) (cherry picked from commit f4d6c5e337e76dc408c9c8f19e306c3f4ba80d8e) (cherry picked from commit 2b9cac26ec5457ad4e18ec418ff8249c37861eb8) commit c253ac68ebb2f07a20af89636ea611fb887739f2 Author: Robert Kanter <rkanter@apache.org> Date: 2015-04-20T21:14:08Z MAPREDUCE-6238 . MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter) (cherry picked from commit d50e8f09287deeb51012d08e326a2ed71a6da869) (cherry picked from commit 756c2542930756fef1cbff82056b418070f8d55f) (cherry picked from commit 4f0fd74880e58958d89ca1cd4b7f665520595b2c) commit 736ebeca3347243fa122f976532ed2536e51b6f5 Author: Gera Shegalov <gera@apache.org> Date: 2015-04-21T18:57:42Z HADOOP-11812 . Implement listLocatedStatus for ViewFileSystem to speed up split calculation (gera) (cherry picked from commit 6d2cf9fbbd02482315a091ab07af26e40cc5134f) (cherry picked from commit 1544c63602089b690e850e0e30af4589513a2371) commit e0e93e9f8d5aaf2eb37d4e91f29f7395109109a2 Author: Steve Loughran <stevel@apache.org> Date: 2015-04-23T20:39:30Z HADOOP-11730 . Regression: s3n read failure recovery broken. (Takenori Sato via stevel) (cherry picked from commit a6a5d1d6b5ee76c829ba7b54a4ad619f7b986681) commit 9df4b7c10e38d517fc19826657e0f3779d543f07 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2015-03-15T05:36:46Z HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cherry picked from commit bc9cb3e271b22069a15ca110cd60c860250aaab2) (cherry picked from commit c71b54fa32ef9149692269a95c918506e338e181) commit 181281cfcada2246af998b388e9f7e96d0228022 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2015-04-24T02:03:44Z HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cherry picked from commit a8898445dc9b5cdb7230e2e23a57393c9f378ff0) (cherry picked from commit 3d0385c3c8fab72be02453a158c7692212ab3b5e) commit 8770f1f03952661f49aea65ae6498be676677812 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2015-04-24T01:59:52Z HADOOP-11802 . DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm (cmccabe) (cherry picked from commit a0e0a63209b5eb17dca5cc503be36aa52defeabd) (cherry picked from commit 788b76761d5dfadf688406d50169e95401fe5d33) commit 9af5b1dcd08c41df026bb86129abd42863baccef Author: Xuan <xgong@apache.org> Date: 2015-01-26T03:37:57Z YARN-3024 . LocalizerRunner should give DIE action when all resources are localized. Contributed by Chengbing Liu (cherry picked from commit 0d6bd62102f94c55d59f7a0a86a684e99d746127) (cherry picked from commit a7696b3fbfacd98a892bbb3678663658c7b9d2bd) (cherry picked from commit 9e30232004ab7c3c3bfde3b8b27c37fa7065f6be) commit 6ade6b5051fc16472ada1d5c73792999e74aa67c Author: Karthik Kambatla <kasha@apache.org> Date: 2015-04-26T16:13:46Z YARN-3464 . Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha) (cherry picked from commit 47279c3228185548ed09c36579b420225e4894f5) (cherry picked from commit 4045c41afe440b773d006e962bf8a5eae3fdc284) (cherry picked from commit 6f2cc0dfa8f21984ecdab59dc087ccf525934930) commit 262275d561a3d138e67942a35b7c15f1ef01bfc8 Author: Vinod Kumar Vavilapalli <vinodkv@apache.org> Date: 2015-04-27T21:58:16Z MAPREDUCE-6324 . Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe. (cherry picked from commit 9fc32c5c4d1d5f50c605bdb0e3b13f44c86660c8) (cherry picked from commit 32dc13d907a416049bdb7deff429725bd6dbcb49) (cherry picked from commit aad56fe3a2f29e73a013b9afa9b44b151a34e0f3) commit c90f025e903573b5e94c2d205cc1e3a6691d4294 Author: Gera Shegalov <gera@apache.org> Date: 2015-05-01T22:44:36Z HADOOP-11491 . HarFs incorrectly declared as requiring an authority. (Brahma Reddy Battula via gera) (cherry picked from commit f343f8657e2b01773a32c2c7d960dc368954b42e) (cherry picked from commit 58970d69de8a1662e4548cd6d4ca460dd70562f8) commit 41601eae6b266ce88e47a06a1dc765cfc296b640 Author: Jason Lowe <jlowe@apache.org> Date: 2015-05-04T19:02:39Z MAPREDUCE-5649 . Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov (cherry picked from commit 7dc3c1203d1ab14c09d0aaf0869a5bcdfafb0a5a) (cherry picked from commit 87c2d915f1cc799cb4020c945c04d3ecb82ee963) (cherry picked from commit 1da62ba736f5f161a18a52b7ca0d212786f3848c) commit 08006b84f8cb6119bd17f6cfc8da2598e24346ca Author: Xiaoyu Yao <xyao@apache.org> Date: 2015-05-05T20:41:14Z HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (cherry picked from commit 0100b155019496d077f958904de7d385697d65d9) (cherry picked from commit e68e8b3b5cff85bfd8bb5b00b9033f63577856d6) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java (cherry picked from commit b054cb68fa0fc6d1e9e77ac84575731e7d1ec0c7) commit bbed4e67a99196a750e9cb5ff9f9595b9bc7a196 Author: Kihwal Lee <kihwal@apache.org> Date: 2015-05-08T14:32:07Z HDFS-7894 . Rolling upgrade readiness is not updated in jmx until query command is issued. Contributed by Brahma Reddy Battula. (cherry picked from commit 6f622672b62aa8d719060063ef0e47480cdc8655) (cherry picked from commit 802a5775f3522c57c60ae29ecb9533dbbfecfe76) (cherry picked from commit 995382c5234ad6c07f327e5d1f2a1c7e391a0b60) commit 51d5fc62c2eeda42f64652980693157dc9686812 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2014-10-27T16:53:16Z HDFS-7278 . Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cherry picked from commit baf794dc404ac54f4e8332654eadfac1bebacb8f) (cherry picked from commit 5f3d967aaefa0b20ef1586b4048b8fa5345d2618) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm (cherry picked from commit a776ef5ad2876b9acf6cf89824c306783f7759f1) commit 6c0ad99b9d45fbad0b860944dda92e1a7a989dc5 Author: Tsuyoshi Ozawa <ozawa@apache.org> Date: 2015-05-12T15:28:17Z MAPREDUCE-6361 . NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du. (cherry picked from commit f4e2b3cc0b1f4e49c306bc09a9dddd0495225bb2) (cherry picked from commit fb5b0ebb459cc8812084090a7ce7ac29e2ad147c) (cherry picked from commit 3a3dcf0bb689d6aa8bc524a31799067702814674) commit 778da79e6f28f5528e107f9b3122925f4a0f05b2 Author: Jason Lowe <jlowe@apache.org> Date: 2015-05-13T21:06:47Z YARN-3641 . NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du (cherry picked from commit 711d77cc54a64b2c3db70bdacc6bf2245c896a4b) (cherry picked from commit a81ad814610936a02e55964fbe08f7b33fe29b23) (cherry picked from commit aa82b0684554be8d09f6fcd88826f167922280cc) commit 7b1a71a7ad5602dbeb5758a650812413c277c21c Author: Xuan <xgong@apache.org> Date: 2015-05-16T05:39:49Z YARN-3526 . ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang (cherry picked from commit b0ad644083a0dfae3a39159ac88b6fc09d846371) (cherry picked from commit 802676e1be350785d8c0ad35f6676eeb85b2467b) (cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298) commit b3ca4dfaff627a04a1a07aa305e1d30651e66056 Author: Kihwal Lee <kihwal@apache.org> Date: 2015-05-08T21:37:26Z HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (cherry picked from commit 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) (cherry picked from commit f264a5aeede7e144af11f5357c7f901993de8e12) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java (cherry picked from commit 470019e9b88e0fcede926442b91d102b595c7ace)
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user zhudebin closed the pull request at:

              https://github.com/apache/hadoop/pull/95

              Show
              githubbot ASF GitHub Bot added a comment - Github user zhudebin closed the pull request at: https://github.com/apache/hadoop/pull/95
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              GitHub user zhudebin opened a pull request:

              https://github.com/apache/hadoop/pull/96

              fix bug HADOOP-13192

              fix bug HADOOP-13192

              You can merge this pull request into a Git repository by running:

              $ git pull https://github.com/zhudebin/hadoop branch-2.6-fixbug

              Alternatively you can review and apply these changes as the patch at:

              https://github.com/apache/hadoop/pull/96.patch

              To close this pull request, make a commit to your master/trunk branch
              with (at least) the following in the commit message:

              This closes #96

              commit d6f4ab4fd1423910824ffee5365cba2dbbfcd081
              Author: zhudebin <zdb_hn@163.com>
              Date: 2016-05-23T11:38:52Z

              fix bug HADOOP-13192

              Show
              githubbot ASF GitHub Bot added a comment - GitHub user zhudebin opened a pull request: https://github.com/apache/hadoop/pull/96 fix bug HADOOP-13192 fix bug HADOOP-13192 You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhudebin/hadoop branch-2.6-fixbug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/96.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #96 commit d6f4ab4fd1423910824ffee5365cba2dbbfcd081 Author: zhudebin <zdb_hn@163.com> Date: 2016-05-23T11:38:52Z fix bug HADOOP-13192
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              Hi binde, please do not set the fix version. It is set by committer when the source code is changed in the issue.
              Also, would you create a pull request for trunk instead of branch-2.6?
              https://wiki.apache.org/hadoop/HowToContribute#Generating_a_patch

              Show
              ajisakaa Akira Ajisaka added a comment - Hi binde , please do not set the fix version. It is set by committer when the source code is changed in the issue. Also, would you create a pull request for trunk instead of branch-2.6? https://wiki.apache.org/hadoop/HowToContribute#Generating_a_patch
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              GitHub user zhudebin opened a pull request:

              https://github.com/apache/hadoop/pull/99

              HADOOP-13192. org.apache.hadoop.util.LineReader match recordDelimiter has a bug

              fix bug HADOOP-13192
              org.apache.hadoop.util.LineReader match recordDelimiter has a bug

              You can merge this pull request into a Git repository by running:

              $ git pull https://github.com/zhudebin/hadoop trunk

              Alternatively you can review and apply these changes as the patch at:

              https://github.com/apache/hadoop/pull/99.patch

              To close this pull request, make a commit to your master/trunk branch
              with (at least) the following in the commit message:

              This closes #99

              commit 9a2d0f197747c4f10ff89a8e2e7ca0712e94b608
              Author: zhudebin <zdb_hn@163.com>
              Date: 2016-06-13T11:15:00Z

              HADOOP-13192 org.apache.hadoop.util.LineReader match recordDelimiter has a bug

              Show
              githubbot ASF GitHub Bot added a comment - GitHub user zhudebin opened a pull request: https://github.com/apache/hadoop/pull/99 HADOOP-13192 . org.apache.hadoop.util.LineReader match recordDelimiter has a bug fix bug HADOOP-13192 org.apache.hadoop.util.LineReader match recordDelimiter has a bug You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhudebin/hadoop trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/99.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #99 commit 9a2d0f197747c4f10ff89a8e2e7ca0712e94b608 Author: zhudebin <zdb_hn@163.com> Date: 2016-06-13T11:15:00Z HADOOP-13192 org.apache.hadoop.util.LineReader match recordDelimiter has a bug
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user zhudebin closed the pull request at:

              https://github.com/apache/hadoop/pull/96

              Show
              githubbot ASF GitHub Bot added a comment - Github user zhudebin closed the pull request at: https://github.com/apache/hadoop/pull/96
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              Reopen this to change the resolution.

              Show
              ajisakaa Akira Ajisaka added a comment - Reopen this to change the resolution.
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              Applied your patch and ran the existing tests.

              $ mvn test -Dtest=TestLineReader,TestLineRecordReader
(snip)
Running org.apache.hadoop.util.TestLineReader
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.368 sec <<< FAILURE! - in org.apache.hadoop.util.TestLineReader
testCustomDelimiter(org.apache.hadoop.util.TestLineReader)  Time elapsed: 0.23 sec  <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException: -1
        at org.apache.hadoop.util.LineReader.readCustomLine(LineReader.java:314)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:172)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:377)
        at org.apache.hadoop.util.TestLineReader.testCustomDelimiter(TestLineReader.java:98)

              The failure looks related to the patch. Hi binde, would you fix it and add a regression test for this bug?

              Show
              ajisakaa Akira Ajisaka added a comment - Applied your patch and ran the existing tests. $ mvn test -Dtest=TestLineReader,TestLineRecordReader (snip) Running org.apache.hadoop.util.TestLineReader Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.368 sec <<< FAILURE! - in org.apache.hadoop.util.TestLineReader testCustomDelimiter(org.apache.hadoop.util.TestLineReader) Time elapsed: 0.23 sec <<< ERROR! java.lang.ArrayIndexOutOfBoundsException: -1 at org.apache.hadoop.util.LineReader.readCustomLine(LineReader.java:314) at org.apache.hadoop.util.LineReader.readLine(LineReader.java:172) at org.apache.hadoop.util.LineReader.readLine(LineReader.java:377) at org.apache.hadoop.util.TestLineReader.testCustomDelimiter(TestLineReader.java:98) The failure looks related to the patch. Hi binde , would you fix it and add a regression test for this bug?
              Hide
              Permalink
              zhudebin binde added a comment -

              OK

              Show
              zhudebin binde added a comment - OK
              Hide
              Permalink
              zhudebin binde added a comment -

              add a test case

              Show
              zhudebin binde added a comment - add a test case
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              Hi binde, I assigned you to this issue.
              Please hit "Submit Patch" instead of "Resolve Issue" after you updated the pull request.

              Show
              ajisakaa Akira Ajisaka added a comment - Hi binde , I assigned you to this issue. Please hit "Submit Patch" instead of "Resolve Issue" after you updated the pull request.
              Hide
              Permalink
              zhudebin binde added a comment -

              fix bug HADOOP-13192. two patchs

              Show
              zhudebin binde added a comment - fix bug HADOOP-13192 . two patchs
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              Thanks binde for attaching the patches in the jira, but actually we don't need to attach the patches when there is a corresponding GitHub pull request because the Jenkins precommit job runs on the pull request. However, we need to hit "Submit Patch" to change the status to "Patch Available" to run the precommit job.

              Show
              ajisakaa Akira Ajisaka added a comment - Thanks binde for attaching the patches in the jira, but actually we don't need to attach the patches when there is a corresponding GitHub pull request because the Jenkins precommit job runs on the pull request. However, we need to hit "Submit Patch" to change the status to "Patch Available" to run the precommit job.
              Hide
              Permalink
              zhudebin binde added a comment -

              Okay, I understand.

              Show
              zhudebin binde added a comment - Okay, I understand.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -
              -1 overall



              Vote Subsystem Runtime Comment
              0 reexec 0m 31s Docker mode activated.
              +1 @author 0m 0s The patch does not contain any @author tags.
              +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              +1 mvninstall 6m 48s trunk passed
              +1 compile 7m 12s trunk passed
              +1 checkstyle 0m 24s trunk passed
              +1 mvnsite 0m 58s trunk passed
              +1 mvneclipse 0m 12s trunk passed
              +1 findbugs 1m 24s trunk passed
              +1 javadoc 0m 48s trunk passed
              +1 mvninstall 0m 43s the patch passed
              +1 compile 7m 35s the patch passed
              +1 javac 7m 35s the patch passed
              -1 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 4 new + 29 unchanged - 0 fixed = 33 total (was 29)
              +1 mvnsite 0m 54s the patch passed
              +1 mvneclipse 0m 14s the patch passed
              +1 whitespace 0m 0s The patch has no whitespace issues.
              +1 findbugs 1m 28s the patch passed
              +1 javadoc 0m 47s the patch passed
              +1 unit 8m 20s hadoop-common in the patch passed.
              +1 asflicense 0m 24s The patch does not generate ASF License warnings.
              39m 57s



              Subsystem Report/Notes
              Docker Image:yetus/hadoop:e2f6409
              JIRA Issue HADOOP-13192
              GITHUB PR https://github.com/apache/hadoop/pull/99
              Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
              uname Linux 89017c85d9eb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
              Build tool maven
              Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
              git revision trunk / 51d497f
              Default Java 1.8.0_91
              findbugs v3.0.0
              checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
              Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/testReport/
              modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
              Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/console
              Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 48s trunk passed +1 compile 7m 12s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 43s the patch passed +1 compile 7m 35s the patch passed +1 javac 7m 35s the patch passed -1 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 4 new + 29 unchanged - 0 fixed = 33 total (was 29) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 28s the patch passed +1 javadoc 0m 47s the patch passed +1 unit 8m 20s hadoop-common in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 39m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Issue HADOOP-13192 GITHUB PR https://github.com/apache/hadoop/pull/99 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 89017c85d9eb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 51d497f Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment - - edited

              Thanks binde for updating the patch. The fix looks good to me. I ran mvn test -Dtest=*Reader* and all the tests succeeded. Would you fix the checkstyle warnings? I'm +1 if that is addressed.

              Show
              ajisakaa Akira Ajisaka added a comment - - edited Thanks binde for updating the patch. The fix looks good to me. I ran mvn test -Dtest=*Reader* and all the tests succeeded. Would you fix the checkstyle warnings? I'm +1 if that is addressed.
              Hide
              Permalink
              zhudebin binde added a comment -

              ok

              Show
              zhudebin binde added a comment - ok
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -
              -1 overall



              Vote Subsystem Runtime Comment
              0 reexec 0m 28s Docker mode activated.
              +1 @author 0m 0s The patch does not contain any @author tags.
              +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              +1 mvninstall 8m 20s trunk passed
              +1 compile 8m 51s trunk passed
              +1 checkstyle 0m 27s trunk passed
              +1 mvnsite 1m 8s trunk passed
              +1 mvneclipse 0m 16s trunk passed
              +1 findbugs 1m 35s trunk passed
              +1 javadoc 0m 54s trunk passed
              +1 mvninstall 0m 47s the patch passed
              +1 compile 8m 28s the patch passed
              +1 javac 8m 28s the patch passed
              +1 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 0 new + 17 unchanged - 12 fixed = 17 total (was 29)
              +1 mvnsite 0m 57s the patch passed
              +1 mvneclipse 0m 13s the patch passed
              +1 whitespace 0m 0s The patch has no whitespace issues.
              +1 findbugs 1m 42s the patch passed
              +1 javadoc 0m 46s the patch passed
              -1 unit 7m 45s hadoop-common in the patch failed.
              +1 asflicense 0m 25s The patch does not generate ASF License warnings.
              44m 16s



              Reason Tests
              Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



              Subsystem Report/Notes
              Docker Image:yetus/hadoop:e2f6409
              JIRA Issue HADOOP-13192
              GITHUB PR https://github.com/apache/hadoop/pull/99
              Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
              uname Linux cb8ca8844db9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
              Build tool maven
              Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
              git revision trunk / 0761379
              Default Java 1.8.0_91
              findbugs v3.0.0
              unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
              Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/testReport/
              modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
              Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/console
              Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 20s trunk passed +1 compile 8m 51s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 35s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 8m 28s the patch passed +1 javac 8m 28s the patch passed +1 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 0 new + 17 unchanged - 12 fixed = 17 total (was 29) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 42s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 7m 45s hadoop-common in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 44m 16s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Issue HADOOP-13192 GITHUB PR https://github.com/apache/hadoop/pull/99 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cb8ca8844db9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0761379 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
              Hide
              Permalink
              zhudebin binde added a comment -

              I run the test case has no problem, why?

              Show
              zhudebin binde added a comment - I run the test case has no problem, why?
              Hide
              Permalink
              liuml07 Mingliang Liu added a comment -

              Hi binde, the failing test is a known bug, see HADOOP-12588. This is an intermittent failing unit test (UT) so you may not reproduce it consistently on your local machine. I had a look at the patch and think it is an unrelated failure UT.

              So when contributing patches and review the pre-commit Jenkins report, you can generally check if the failing tests are related. If unrelated, you can search JIRA ticket that reports it and comment about it here. You can file another bug for tracking the failing UTs if no one has reported it yet.

              Thanks for the contribution.

              Show
              liuml07 Mingliang Liu added a comment - Hi binde , the failing test is a known bug, see HADOOP-12588 . This is an intermittent failing unit test (UT) so you may not reproduce it consistently on your local machine. I had a look at the patch and think it is an unrelated failure UT. So when contributing patches and review the pre-commit Jenkins report, you can generally check if the failing tests are related. If unrelated, you can search JIRA ticket that reports it and comment about it here. You can file another bug for tracking the failing UTs if no one has reported it yet. Thanks for the contribution.
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              +1, committing this.

              Show
              ajisakaa Akira Ajisaka added a comment - +1, committing this.
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              Committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.7.3. Thanks binde for the contribution, and thanks Mingliang Liu for the help.

              Show
              ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.7.3. Thanks binde for the contribution, and thanks Mingliang Liu for the help.
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Hadoop-trunk-Commit #9985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9985/)
              HADOOP-13192. org.apache.hadoop.util.LineReader cannot handle multibyte (aajisaka: rev fc6b50cc574e144fd314dea6c11987c6a384bfa6)

              • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
              • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9985/ ) HADOOP-13192 . org.apache.hadoop.util.LineReader cannot handle multibyte (aajisaka: rev fc6b50cc574e144fd314dea6c11987c6a384bfa6) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user aajisaka commented on the issue:

              https://github.com/apache/hadoop/pull/99

              I've committed this. Would you close this pull request?

              Show
              githubbot ASF GitHub Bot added a comment - Github user aajisaka commented on the issue: https://github.com/apache/hadoop/pull/99 I've committed this. Would you close this pull request?
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user zhudebin commented on the issue:

              https://github.com/apache/hadoop/pull/99

              ok

              Show
              githubbot ASF GitHub Bot added a comment - Github user zhudebin commented on the issue: https://github.com/apache/hadoop/pull/99 ok
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user zhudebin closed the pull request at:

              https://github.com/apache/hadoop/pull/99

              Show
              githubbot ASF GitHub Bot added a comment - Github user zhudebin closed the pull request at: https://github.com/apache/hadoop/pull/99
              Hide
              Permalink
              haitao-tony sunhaitao added a comment -

              hi the second patch:0002-fix-bug-hadoop-1392-add-test-case-for-LineReader.patch does not remove " bufferPosn--;" which leads to dead lock and ut failure, please check

              Show
              haitao-tony sunhaitao added a comment - hi the second patch:0002-fix-bug-hadoop-1392-add-test-case-for-LineReader.patch does not remove " bufferPosn--;" which leads to dead lock and ut failure, please check
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              Hi sunhaitao, I didn't actually review the 02 patch. I reviewed the pull request https://github.com/apache/hadoop/pull/99 and merged it. The pull request removes "bufferPosn--", so I'm thinking the problem does not occur.

              Show
              ajisakaa Akira Ajisaka added a comment - Hi sunhaitao , I didn't actually review the 02 patch. I reviewed the pull request https://github.com/apache/hadoop/pull/99 and merged it. The pull request removes "bufferPosn--", so I'm thinking the problem does not occur.
              Hide
              Permalink
              haitao-tony sunhaitao added a comment -

              i advise to change the 002 patch so as to remove "bufferPosn--", in case of causing misleading when some guys merge this patch.

              Show
              haitao-tony sunhaitao added a comment - i advise to change the 002 patch so as to remove "bufferPosn--", in case of causing misleading when some guys merge this patch.
              Hide
              Permalink
              ajisakaa Akira Ajisaka added a comment -

              Attaching a patch for the commit. (Generated by git diff d0162f2040a4d98fbac51527dfc9447888cb32ab fc6b50cc574e144fd314dea6c11987c6a384bfa6)
              Thanks sunhaitao!

              Show
              ajisakaa Akira Ajisaka added a comment - Attaching a patch for the commit. (Generated by git diff d0162f2040a4d98fbac51527dfc9447888cb32ab fc6b50cc574e144fd314dea6c11987c6a384bfa6 ) Thanks sunhaitao !
              Hide
              Permalink
              haitao-tony sunhaitao added a comment -

              welcome

              Show
              haitao-tony sunhaitao added a comment - welcome
              Hide
              Permalink
              vinodkv Vinod Kumar Vavilapalli added a comment -

              Closing the JIRA as part of 2.7.3 release.

              Show
              vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
              Hide
              Permalink
              sjlee0 Sangjin Lee added a comment -

              Cherry-picked it to 2.6.5 (trivial).

              Show
              sjlee0 Sangjin Lee added a comment - Cherry-picked it to 2.6.5 (trivial).

                People

                • Assignee:
                  zhudebin binde
                  Reporter:
                  zhudebin binde
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  9 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - 5m
                    5m
                    Remaining:
                    Remaining Estimate - 5m
                    5m
                    Logged:
                    Time Spent - Not Specified
                    Not Specified

                      Development