Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13192

org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.2
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      org.apache.hadoop.util.LineReader.readCustomLine() has a bug,
      when line is aaaabccc, recordDelimiter is aaab, the result should be a,ccc,
      show the code on line 310:
      for (; bufferPosn < bufferLength; ++bufferPosn) {
      if (buffer[bufferPosn] == recordDelimiterBytes[delPosn]) {
      delPosn++;
      if (delPosn >= recordDelimiterBytes.length)

      { bufferPosn++; break; }
      } else if (delPosn != 0) { bufferPosn--; delPosn = 0; }
      }

      shoud be :
      for (; bufferPosn < bufferLength; ++bufferPosn) {
      if (buffer[bufferPosn] == recordDelimiterBytes[delPosn]) {
      delPosn++;
      if (delPosn >= recordDelimiterBytes.length) { bufferPosn++; break; }

      } else if (delPosn != 0)

      { // ------------- change here ------------- start ---- bufferPosn -= delPosn; // ------------- change here ------------- end ---- delPosn = 0; }

      }

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zhudebin opened a pull request:

          https://github.com/apache/hadoop/pull/95

          fix bug HADOOP-13192

          fix bug HADOOP-13192

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zhudebin/hadoop branch-2.6-fixbug

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/95.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #95


          commit 0b1e66f01dae1c5558e897e35b1cbe533d9c4542
          Author: Andrew Wang <wang@apache.org>
          Date: 2015-03-24T05:00:34Z

          HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu.

          (cherry picked from commit 50ee8f4e67a66aa77c5359182f61f3e951844db6)
          (cherry picked from commit 2f46ee50bd4efc82ba3d30bd36f7637ea9d9714e)

          Conflicts:
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

          (cherry picked from commit 03d4af39e794dc03d764122077b434d658b6405e)

          commit bc8728cd27870e048fd90d1e07ea92e8c9ed310d
          Author: Kihwal Lee <kihwal@apache.org>
          Date: 2015-03-30T15:11:25Z

          HDFS-7742. Favoring decommissioning node for replication can cause a block to stay
          underreplicated for long periods. Contributed by Nathan Roberts.
          (cherry picked from commit 04ee18ed48ceef34598f954ff40940abc9fde1d2)

          (cherry picked from commit c4cedfc1d601127430c70ca8ca4d4e2ee2d1003d)
          (cherry picked from commit c6b68a82adea8de488b255594d35db8e01f5fc8f)

          commit 8a9665a586624cfe7f11ad9e21976465e0bb0e21
          Author: Junping Du <junping_du@apache.org>
          Date: 2015-04-02T19:13:03Z

          MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe.
          (cherry picked from commit eccb7d46efbf07abcc6a01bd5e7d682f6815b824)

          (cherry picked from commit cacadea632f7ab6fe4fdb1432e1a2c48e8ebd55f)
          (cherry picked from commit 2abd4f61075739514fb3e63b118448895be02a30)

          commit c3f5ea11eca30a617cab2a716dd08dff20db3791
          Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
          Date: 2015-04-06T15:54:46Z

          HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe)

          (cherry picked from commit 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)
          (cherry picked from commit a827089905524e10638c783ba908a895d621911d)

          Conflicts:
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

          (cherry picked from commit c3a3092c37926eca75ea149c4c061742f6599b40)

          commit 31d30e8111d7c4ef6400b5fe51cc67b17ab34908
          Author: Arpit Agarwal <arp@apache.org>
          Date: 2015-04-08T18:38:21Z

          HDFS-8072. Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal)

          (cherry picked from commit f0324738c9db4f45d2b1ec5cfb46c5f2b7669571)

          Conflicts:
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java

          (cherry picked from commit de21de7e2243ef8a89082121d838b88e3c10f05b)

          commit 619f7938466e907f335941bbbbd928c6272a0482
          Author: Kihwal Lee <kihwal@apache.org>
          Date: 2015-04-08T20:39:25Z

          HDFS-8046. Allow better control of getContentSummary. Contributed by Kihwal Lee.
          (cherry picked from commit 285b31e75e51ec8e3a796c2cb0208739368ca9b8)

          (cherry picked from commit 7e622076d41a85fc9a8600fb270564a085f5cd83)

          Conflicts:
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java

          (cherry picked from commit 1ef5e0b18066ca949adcf4c55a41f186c47e7264)

          commit 752e3da738080cf7259b9085b736b42956d0bf1e
          Author: Vinod Kumar Vavilapalli <vinodkv@apache.org>
          Date: 2015-04-09T20:08:53Z

          YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp.

          (cherry picked from commit 9c5911294e0ba71aefe4763731b0e780cde9d0ca)
          (cherry picked from commit 1ff3fd33ed6f2ac09c774cc42b0107c5dbd9c19d)
          (cherry picked from commit 82c722aae86669325672dd10840447434f15e7fd)

          commit dcc0d4658c9bc7f488cecdcbc05e9216234b36aa
          Author: Akira Ajisaka <aajisaka@apache.org>
          Date: 2015-04-10T16:29:08Z

          MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li.

          (cherry picked from commit 4cec9975c0e2c2fcca090888dc2c5e149c5be1e9)
          (cherry picked from commit 3de0bf9a35094f3c2ac216992d861729251b6a3d)
          (cherry picked from commit c3b2b377f19293797a88f4636f07e2ab8e18c78b)

          commit b7e7896410bdd8aac5a8b9f9327227542e730ece
          Author: Jing Zhao <jing9@apache.org>
          Date: 2015-04-14T22:05:23Z

          HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao.

          (cherry picked from commit fddd55279d0bdd08b3b40aba6fe2ded1d2e0d846)
          (cherry picked from commit 38b031d6bab8527698bd186887d301bd6a63cf01)

          Conflicts:
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java

          (cherry picked from commit b4e227e65d3a81150f46aac5e987d759defc1452)

          commit 61f2ddb125f6af1775ce5d219f22eb079071869c
          Author: Wangda Tan <wangda@apache.org>
          Date: 2015-04-18T00:11:22Z

          YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda)

          (cherry picked from commit f65eeb412d140a3808bcf99344a9f3a965918f70)
          (cherry picked from commit e7cbecddc3e7ca5386c71aa4deb67f133611415c)
          (cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd)

          commit e081593042de460b5930dadd0bd3aa313fe09873
          Author: Wangda Tan <wangda@apache.org>
          Date: 2015-04-18T00:22:27Z

          YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda)

          (cherry picked from commit f47a5763acd55cb0b3f16152c7f8df06ec0e09a9)
          (cherry picked from commit 3316cd4357ff6ccc4c76584813092adb1c2b4d43)
          (cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670)

          commit 2f66dff7cec001064332cfcf7a0e8873cd98482e
          Author: Karthik Kambatla <kasha@apache.org>
          Date: 2015-03-04T22:42:07Z

          MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha)

          (cherry picked from commit c66c3ac6bf9f63177279feec3f2917e4b882e2bc)
          (cherry picked from commit f4d6c5e337e76dc408c9c8f19e306c3f4ba80d8e)
          (cherry picked from commit 2b9cac26ec5457ad4e18ec418ff8249c37861eb8)

          commit c253ac68ebb2f07a20af89636ea611fb887739f2
          Author: Robert Kanter <rkanter@apache.org>
          Date: 2015-04-20T21:14:08Z

          MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter)

          (cherry picked from commit d50e8f09287deeb51012d08e326a2ed71a6da869)
          (cherry picked from commit 756c2542930756fef1cbff82056b418070f8d55f)
          (cherry picked from commit 4f0fd74880e58958d89ca1cd4b7f665520595b2c)

          commit 736ebeca3347243fa122f976532ed2536e51b6f5
          Author: Gera Shegalov <gera@apache.org>
          Date: 2015-04-21T18:57:42Z

          HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up split calculation (gera)

          (cherry picked from commit 6d2cf9fbbd02482315a091ab07af26e40cc5134f)
          (cherry picked from commit 1544c63602089b690e850e0e30af4589513a2371)

          commit e0e93e9f8d5aaf2eb37d4e91f29f7395109109a2
          Author: Steve Loughran <stevel@apache.org>
          Date: 2015-04-23T20:39:30Z

          HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel)

          (cherry picked from commit a6a5d1d6b5ee76c829ba7b54a4ad619f7b986681)

          commit 9df4b7c10e38d517fc19826657e0f3779d543f07
          Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
          Date: 2015-03-15T05:36:46Z

          HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)

          (cherry picked from commit bc9cb3e271b22069a15ca110cd60c860250aaab2)
          (cherry picked from commit c71b54fa32ef9149692269a95c918506e338e181)

          commit 181281cfcada2246af998b388e9f7e96d0228022
          Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
          Date: 2015-04-24T02:03:44Z

          HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe)

          (cherry picked from commit a8898445dc9b5cdb7230e2e23a57393c9f378ff0)
          (cherry picked from commit 3d0385c3c8fab72be02453a158c7692212ab3b5e)

          commit 8770f1f03952661f49aea65ae6498be676677812
          Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
          Date: 2015-04-24T01:59:52Z

          HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm (cmccabe)

          (cherry picked from commit a0e0a63209b5eb17dca5cc503be36aa52defeabd)
          (cherry picked from commit 788b76761d5dfadf688406d50169e95401fe5d33)

          commit 9af5b1dcd08c41df026bb86129abd42863baccef
          Author: Xuan <xgong@apache.org>
          Date: 2015-01-26T03:37:57Z

          YARN-3024. LocalizerRunner should give DIE action when all resources are
          localized. Contributed by Chengbing Liu

          (cherry picked from commit 0d6bd62102f94c55d59f7a0a86a684e99d746127)
          (cherry picked from commit a7696b3fbfacd98a892bbb3678663658c7b9d2bd)
          (cherry picked from commit 9e30232004ab7c3c3bfde3b8b27c37fa7065f6be)

          commit 6ade6b5051fc16472ada1d5c73792999e74aa67c
          Author: Karthik Kambatla <kasha@apache.org>
          Date: 2015-04-26T16:13:46Z

          YARN-3464. Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha)

          (cherry picked from commit 47279c3228185548ed09c36579b420225e4894f5)
          (cherry picked from commit 4045c41afe440b773d006e962bf8a5eae3fdc284)
          (cherry picked from commit 6f2cc0dfa8f21984ecdab59dc087ccf525934930)

          commit 262275d561a3d138e67942a35b7c15f1ef01bfc8
          Author: Vinod Kumar Vavilapalli <vinodkv@apache.org>
          Date: 2015-04-27T21:58:16Z

          MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe.

          (cherry picked from commit 9fc32c5c4d1d5f50c605bdb0e3b13f44c86660c8)
          (cherry picked from commit 32dc13d907a416049bdb7deff429725bd6dbcb49)
          (cherry picked from commit aad56fe3a2f29e73a013b9afa9b44b151a34e0f3)

          commit c90f025e903573b5e94c2d205cc1e3a6691d4294
          Author: Gera Shegalov <gera@apache.org>
          Date: 2015-05-01T22:44:36Z

          HADOOP-11491. HarFs incorrectly declared as requiring an authority. (Brahma Reddy Battula via gera)

          (cherry picked from commit f343f8657e2b01773a32c2c7d960dc368954b42e)
          (cherry picked from commit 58970d69de8a1662e4548cd6d4ca460dd70562f8)

          commit 41601eae6b266ce88e47a06a1dc765cfc296b640
          Author: Jason Lowe <jlowe@apache.org>
          Date: 2015-05-04T19:02:39Z

          MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov
          (cherry picked from commit 7dc3c1203d1ab14c09d0aaf0869a5bcdfafb0a5a)

          (cherry picked from commit 87c2d915f1cc799cb4020c945c04d3ecb82ee963)
          (cherry picked from commit 1da62ba736f5f161a18a52b7ca0d212786f3848c)

          commit 08006b84f8cb6119bd17f6cfc8da2598e24346ca
          Author: Xiaoyu Yao <xyao@apache.org>
          Date: 2015-05-05T20:41:14Z

          HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao)

          (cherry picked from commit 0100b155019496d077f958904de7d385697d65d9)
          (cherry picked from commit e68e8b3b5cff85bfd8bb5b00b9033f63577856d6)

          Conflicts:
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java

          (cherry picked from commit b054cb68fa0fc6d1e9e77ac84575731e7d1ec0c7)

          commit bbed4e67a99196a750e9cb5ff9f9595b9bc7a196
          Author: Kihwal Lee <kihwal@apache.org>
          Date: 2015-05-08T14:32:07Z

          HDFS-7894. Rolling upgrade readiness is not updated in jmx until query command is issued. Contributed by Brahma Reddy Battula.
          (cherry picked from commit 6f622672b62aa8d719060063ef0e47480cdc8655)

          (cherry picked from commit 802a5775f3522c57c60ae29ecb9533dbbfecfe76)
          (cherry picked from commit 995382c5234ad6c07f327e5d1f2a1c7e391a0b60)

          commit 51d5fc62c2eeda42f64652980693157dc9686812
          Author: Colin Patrick Mccabe <cmccabe@cloudera.com>
          Date: 2014-10-27T16:53:16Z

          HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe)
          (cherry picked from commit baf794dc404ac54f4e8332654eadfac1bebacb8f)

          (cherry picked from commit 5f3d967aaefa0b20ef1586b4048b8fa5345d2618)

          Conflicts:
          hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm

          (cherry picked from commit a776ef5ad2876b9acf6cf89824c306783f7759f1)

          commit 6c0ad99b9d45fbad0b860944dda92e1a7a989dc5
          Author: Tsuyoshi Ozawa <ozawa@apache.org>
          Date: 2015-05-12T15:28:17Z

          MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du.

          (cherry picked from commit f4e2b3cc0b1f4e49c306bc09a9dddd0495225bb2)
          (cherry picked from commit fb5b0ebb459cc8812084090a7ce7ac29e2ad147c)
          (cherry picked from commit 3a3dcf0bb689d6aa8bc524a31799067702814674)

          commit 778da79e6f28f5528e107f9b3122925f4a0f05b2
          Author: Jason Lowe <jlowe@apache.org>
          Date: 2015-05-13T21:06:47Z

          YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du
          (cherry picked from commit 711d77cc54a64b2c3db70bdacc6bf2245c896a4b)

          (cherry picked from commit a81ad814610936a02e55964fbe08f7b33fe29b23)
          (cherry picked from commit aa82b0684554be8d09f6fcd88826f167922280cc)

          commit 7b1a71a7ad5602dbeb5758a650812413c277c21c
          Author: Xuan <xgong@apache.org>
          Date: 2015-05-16T05:39:49Z

          YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang

          (cherry picked from commit b0ad644083a0dfae3a39159ac88b6fc09d846371)
          (cherry picked from commit 802676e1be350785d8c0ad35f6676eeb85b2467b)
          (cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298)

          commit b3ca4dfaff627a04a1a07aa305e1d30651e66056
          Author: Kihwal Lee <kihwal@apache.org>
          Date: 2015-05-08T21:37:26Z

          HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah.
          (cherry picked from commit 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

          (cherry picked from commit f264a5aeede7e144af11f5357c7f901993de8e12)

          Conflicts:
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java

          (cherry picked from commit 470019e9b88e0fcede926442b91d102b595c7ace)


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zhudebin opened a pull request: https://github.com/apache/hadoop/pull/95 fix bug HADOOP-13192 fix bug HADOOP-13192 You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhudebin/hadoop branch-2.6-fixbug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/95.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #95 commit 0b1e66f01dae1c5558e897e35b1cbe533d9c4542 Author: Andrew Wang <wang@apache.org> Date: 2015-03-24T05:00:34Z HDFS-7960 . The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. (cherry picked from commit 50ee8f4e67a66aa77c5359182f61f3e951844db6) (cherry picked from commit 2f46ee50bd4efc82ba3d30bd36f7637ea9d9714e) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java (cherry picked from commit 03d4af39e794dc03d764122077b434d658b6405e) commit bc8728cd27870e048fd90d1e07ea92e8c9ed310d Author: Kihwal Lee <kihwal@apache.org> Date: 2015-03-30T15:11:25Z HDFS-7742 . Favoring decommissioning node for replication can cause a block to stay underreplicated for long periods. Contributed by Nathan Roberts. (cherry picked from commit 04ee18ed48ceef34598f954ff40940abc9fde1d2) (cherry picked from commit c4cedfc1d601127430c70ca8ca4d4e2ee2d1003d) (cherry picked from commit c6b68a82adea8de488b255594d35db8e01f5fc8f) commit 8a9665a586624cfe7f11ad9e21976465e0bb0e21 Author: Junping Du <junping_du@apache.org> Date: 2015-04-02T19:13:03Z MAPREDUCE-6303 . Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe. (cherry picked from commit eccb7d46efbf07abcc6a01bd5e7d682f6815b824) (cherry picked from commit cacadea632f7ab6fe4fdb1432e1a2c48e8ebd55f) (cherry picked from commit 2abd4f61075739514fb3e63b118448895be02a30) commit c3f5ea11eca30a617cab2a716dd08dff20db3791 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2015-04-06T15:54:46Z HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cherry picked from commit 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) (cherry picked from commit a827089905524e10638c783ba908a895d621911d) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java (cherry picked from commit c3a3092c37926eca75ea149c4c061742f6599b40) commit 31d30e8111d7c4ef6400b5fe51cc67b17ab34908 Author: Arpit Agarwal <arp@apache.org> Date: 2015-04-08T18:38:21Z HDFS-8072 . Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal) (cherry picked from commit f0324738c9db4f45d2b1ec5cfb46c5f2b7669571) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java (cherry picked from commit de21de7e2243ef8a89082121d838b88e3c10f05b) commit 619f7938466e907f335941bbbbd928c6272a0482 Author: Kihwal Lee <kihwal@apache.org> Date: 2015-04-08T20:39:25Z HDFS-8046 . Allow better control of getContentSummary. Contributed by Kihwal Lee. (cherry picked from commit 285b31e75e51ec8e3a796c2cb0208739368ca9b8) (cherry picked from commit 7e622076d41a85fc9a8600fb270564a085f5cd83) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java (cherry picked from commit 1ef5e0b18066ca949adcf4c55a41f186c47e7264) commit 752e3da738080cf7259b9085b736b42956d0bf1e Author: Vinod Kumar Vavilapalli <vinodkv@apache.org> Date: 2015-04-09T20:08:53Z YARN-3055 . Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. (cherry picked from commit 9c5911294e0ba71aefe4763731b0e780cde9d0ca) (cherry picked from commit 1ff3fd33ed6f2ac09c774cc42b0107c5dbd9c19d) (cherry picked from commit 82c722aae86669325672dd10840447434f15e7fd) commit dcc0d4658c9bc7f488cecdcbc05e9216234b36aa Author: Akira Ajisaka <aajisaka@apache.org> Date: 2015-04-10T16:29:08Z MAPREDUCE-6300 . Task list sort by task id broken. Contributed by Siqi Li. (cherry picked from commit 4cec9975c0e2c2fcca090888dc2c5e149c5be1e9) (cherry picked from commit 3de0bf9a35094f3c2ac216992d861729251b6a3d) (cherry picked from commit c3b2b377f19293797a88f4636f07e2ab8e18c78b) commit b7e7896410bdd8aac5a8b9f9327227542e730ece Author: Jing Zhao <jing9@apache.org> Date: 2015-04-14T22:05:23Z HDFS-8127 . NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao. (cherry picked from commit fddd55279d0bdd08b3b40aba6fe2ded1d2e0d846) (cherry picked from commit 38b031d6bab8527698bd186887d301bd6a63cf01) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java (cherry picked from commit b4e227e65d3a81150f46aac5e987d759defc1452) commit 61f2ddb125f6af1775ce5d219f22eb079071869c Author: Wangda Tan <wangda@apache.org> Date: 2015-04-18T00:11:22Z YARN-3493 . RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) (cherry picked from commit f65eeb412d140a3808bcf99344a9f3a965918f70) (cherry picked from commit e7cbecddc3e7ca5386c71aa4deb67f133611415c) (cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd) commit e081593042de460b5930dadd0bd3aa313fe09873 Author: Wangda Tan <wangda@apache.org> Date: 2015-04-18T00:22:27Z YARN-3487 . CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (cherry picked from commit f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) (cherry picked from commit 3316cd4357ff6ccc4c76584813092adb1c2b4d43) (cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670) commit 2f66dff7cec001064332cfcf7a0e8873cd98482e Author: Karthik Kambatla <kasha@apache.org> Date: 2015-03-04T22:42:07Z MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (cherry picked from commit c66c3ac6bf9f63177279feec3f2917e4b882e2bc) (cherry picked from commit f4d6c5e337e76dc408c9c8f19e306c3f4ba80d8e) (cherry picked from commit 2b9cac26ec5457ad4e18ec418ff8249c37861eb8) commit c253ac68ebb2f07a20af89636ea611fb887739f2 Author: Robert Kanter <rkanter@apache.org> Date: 2015-04-20T21:14:08Z MAPREDUCE-6238 . MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter) (cherry picked from commit d50e8f09287deeb51012d08e326a2ed71a6da869) (cherry picked from commit 756c2542930756fef1cbff82056b418070f8d55f) (cherry picked from commit 4f0fd74880e58958d89ca1cd4b7f665520595b2c) commit 736ebeca3347243fa122f976532ed2536e51b6f5 Author: Gera Shegalov <gera@apache.org> Date: 2015-04-21T18:57:42Z HADOOP-11812 . Implement listLocatedStatus for ViewFileSystem to speed up split calculation (gera) (cherry picked from commit 6d2cf9fbbd02482315a091ab07af26e40cc5134f) (cherry picked from commit 1544c63602089b690e850e0e30af4589513a2371) commit e0e93e9f8d5aaf2eb37d4e91f29f7395109109a2 Author: Steve Loughran <stevel@apache.org> Date: 2015-04-23T20:39:30Z HADOOP-11730 . Regression: s3n read failure recovery broken. (Takenori Sato via stevel) (cherry picked from commit a6a5d1d6b5ee76c829ba7b54a4ad619f7b986681) commit 9df4b7c10e38d517fc19826657e0f3779d543f07 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2015-03-15T05:36:46Z HDFS-7915 . The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) (cherry picked from commit bc9cb3e271b22069a15ca110cd60c860250aaab2) (cherry picked from commit c71b54fa32ef9149692269a95c918506e338e181) commit 181281cfcada2246af998b388e9f7e96d0228022 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2015-04-24T02:03:44Z HDFS-8070 . Pre- HDFS-7915 DFSClient cannot use short circuit on post- HDFS-7915 DataNode (cmccabe) (cherry picked from commit a8898445dc9b5cdb7230e2e23a57393c9f378ff0) (cherry picked from commit 3d0385c3c8fab72be02453a158c7692212ab3b5e) commit 8770f1f03952661f49aea65ae6498be676677812 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2015-04-24T01:59:52Z HADOOP-11802 . DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm (cmccabe) (cherry picked from commit a0e0a63209b5eb17dca5cc503be36aa52defeabd) (cherry picked from commit 788b76761d5dfadf688406d50169e95401fe5d33) commit 9af5b1dcd08c41df026bb86129abd42863baccef Author: Xuan <xgong@apache.org> Date: 2015-01-26T03:37:57Z YARN-3024 . LocalizerRunner should give DIE action when all resources are localized. Contributed by Chengbing Liu (cherry picked from commit 0d6bd62102f94c55d59f7a0a86a684e99d746127) (cherry picked from commit a7696b3fbfacd98a892bbb3678663658c7b9d2bd) (cherry picked from commit 9e30232004ab7c3c3bfde3b8b27c37fa7065f6be) commit 6ade6b5051fc16472ada1d5c73792999e74aa67c Author: Karthik Kambatla <kasha@apache.org> Date: 2015-04-26T16:13:46Z YARN-3464 . Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha) (cherry picked from commit 47279c3228185548ed09c36579b420225e4894f5) (cherry picked from commit 4045c41afe440b773d006e962bf8a5eae3fdc284) (cherry picked from commit 6f2cc0dfa8f21984ecdab59dc087ccf525934930) commit 262275d561a3d138e67942a35b7c15f1ef01bfc8 Author: Vinod Kumar Vavilapalli <vinodkv@apache.org> Date: 2015-04-27T21:58:16Z MAPREDUCE-6324 . Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe. (cherry picked from commit 9fc32c5c4d1d5f50c605bdb0e3b13f44c86660c8) (cherry picked from commit 32dc13d907a416049bdb7deff429725bd6dbcb49) (cherry picked from commit aad56fe3a2f29e73a013b9afa9b44b151a34e0f3) commit c90f025e903573b5e94c2d205cc1e3a6691d4294 Author: Gera Shegalov <gera@apache.org> Date: 2015-05-01T22:44:36Z HADOOP-11491 . HarFs incorrectly declared as requiring an authority. (Brahma Reddy Battula via gera) (cherry picked from commit f343f8657e2b01773a32c2c7d960dc368954b42e) (cherry picked from commit 58970d69de8a1662e4548cd6d4ca460dd70562f8) commit 41601eae6b266ce88e47a06a1dc765cfc296b640 Author: Jason Lowe <jlowe@apache.org> Date: 2015-05-04T19:02:39Z MAPREDUCE-5649 . Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov (cherry picked from commit 7dc3c1203d1ab14c09d0aaf0869a5bcdfafb0a5a) (cherry picked from commit 87c2d915f1cc799cb4020c945c04d3ecb82ee963) (cherry picked from commit 1da62ba736f5f161a18a52b7ca0d212786f3848c) commit 08006b84f8cb6119bd17f6cfc8da2598e24346ca Author: Xiaoyu Yao <xyao@apache.org> Date: 2015-05-05T20:41:14Z HDFS-8219 . setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) (cherry picked from commit 0100b155019496d077f958904de7d385697d65d9) (cherry picked from commit e68e8b3b5cff85bfd8bb5b00b9033f63577856d6) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java (cherry picked from commit b054cb68fa0fc6d1e9e77ac84575731e7d1ec0c7) commit bbed4e67a99196a750e9cb5ff9f9595b9bc7a196 Author: Kihwal Lee <kihwal@apache.org> Date: 2015-05-08T14:32:07Z HDFS-7894 . Rolling upgrade readiness is not updated in jmx until query command is issued. Contributed by Brahma Reddy Battula. (cherry picked from commit 6f622672b62aa8d719060063ef0e47480cdc8655) (cherry picked from commit 802a5775f3522c57c60ae29ecb9533dbbfecfe76) (cherry picked from commit 995382c5234ad6c07f327e5d1f2a1c7e391a0b60) commit 51d5fc62c2eeda42f64652980693157dc9686812 Author: Colin Patrick Mccabe <cmccabe@cloudera.com> Date: 2014-10-27T16:53:16Z HDFS-7278 . Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cherry picked from commit baf794dc404ac54f4e8332654eadfac1bebacb8f) (cherry picked from commit 5f3d967aaefa0b20ef1586b4048b8fa5345d2618) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm (cherry picked from commit a776ef5ad2876b9acf6cf89824c306783f7759f1) commit 6c0ad99b9d45fbad0b860944dda92e1a7a989dc5 Author: Tsuyoshi Ozawa <ozawa@apache.org> Date: 2015-05-12T15:28:17Z MAPREDUCE-6361 . NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du. (cherry picked from commit f4e2b3cc0b1f4e49c306bc09a9dddd0495225bb2) (cherry picked from commit fb5b0ebb459cc8812084090a7ce7ac29e2ad147c) (cherry picked from commit 3a3dcf0bb689d6aa8bc524a31799067702814674) commit 778da79e6f28f5528e107f9b3122925f4a0f05b2 Author: Jason Lowe <jlowe@apache.org> Date: 2015-05-13T21:06:47Z YARN-3641 . NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du (cherry picked from commit 711d77cc54a64b2c3db70bdacc6bf2245c896a4b) (cherry picked from commit a81ad814610936a02e55964fbe08f7b33fe29b23) (cherry picked from commit aa82b0684554be8d09f6fcd88826f167922280cc) commit 7b1a71a7ad5602dbeb5758a650812413c277c21c Author: Xuan <xgong@apache.org> Date: 2015-05-16T05:39:49Z YARN-3526 . ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang (cherry picked from commit b0ad644083a0dfae3a39159ac88b6fc09d846371) (cherry picked from commit 802676e1be350785d8c0ad35f6676eeb85b2467b) (cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298) commit b3ca4dfaff627a04a1a07aa305e1d30651e66056 Author: Kihwal Lee <kihwal@apache.org> Date: 2015-05-08T21:37:26Z HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (cherry picked from commit 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) (cherry picked from commit f264a5aeede7e144af11f5357c7f901993de8e12) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java (cherry picked from commit 470019e9b88e0fcede926442b91d102b595c7ace)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zhudebin closed the pull request at:

          https://github.com/apache/hadoop/pull/95

          Show
          githubbot ASF GitHub Bot added a comment - Github user zhudebin closed the pull request at: https://github.com/apache/hadoop/pull/95
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zhudebin opened a pull request:

          https://github.com/apache/hadoop/pull/96

          fix bug HADOOP-13192

          fix bug HADOOP-13192

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zhudebin/hadoop branch-2.6-fixbug

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/96.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #96


          commit d6f4ab4fd1423910824ffee5365cba2dbbfcd081
          Author: zhudebin <zdb_hn@163.com>
          Date: 2016-05-23T11:38:52Z

          fix bug HADOOP-13192


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zhudebin opened a pull request: https://github.com/apache/hadoop/pull/96 fix bug HADOOP-13192 fix bug HADOOP-13192 You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhudebin/hadoop branch-2.6-fixbug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/96.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #96 commit d6f4ab4fd1423910824ffee5365cba2dbbfcd081 Author: zhudebin <zdb_hn@163.com> Date: 2016-05-23T11:38:52Z fix bug HADOOP-13192
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi binde, please do not set the fix version. It is set by committer when the source code is changed in the issue.
          Also, would you create a pull request for trunk instead of branch-2.6?
          https://wiki.apache.org/hadoop/HowToContribute#Generating_a_patch

          Show
          ajisakaa Akira Ajisaka added a comment - Hi binde , please do not set the fix version. It is set by committer when the source code is changed in the issue. Also, would you create a pull request for trunk instead of branch-2.6? https://wiki.apache.org/hadoop/HowToContribute#Generating_a_patch
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zhudebin opened a pull request:

          https://github.com/apache/hadoop/pull/99

          HADOOP-13192. org.apache.hadoop.util.LineReader match recordDelimiter has a bug

          fix bug HADOOP-13192
          org.apache.hadoop.util.LineReader match recordDelimiter has a bug

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zhudebin/hadoop trunk

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/99.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #99


          commit 9a2d0f197747c4f10ff89a8e2e7ca0712e94b608
          Author: zhudebin <zdb_hn@163.com>
          Date: 2016-06-13T11:15:00Z

          HADOOP-13192 org.apache.hadoop.util.LineReader match recordDelimiter has a bug


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zhudebin opened a pull request: https://github.com/apache/hadoop/pull/99 HADOOP-13192 . org.apache.hadoop.util.LineReader match recordDelimiter has a bug fix bug HADOOP-13192 org.apache.hadoop.util.LineReader match recordDelimiter has a bug You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhudebin/hadoop trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/99.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #99 commit 9a2d0f197747c4f10ff89a8e2e7ca0712e94b608 Author: zhudebin <zdb_hn@163.com> Date: 2016-06-13T11:15:00Z HADOOP-13192 org.apache.hadoop.util.LineReader match recordDelimiter has a bug
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zhudebin closed the pull request at:

          https://github.com/apache/hadoop/pull/96

          Show
          githubbot ASF GitHub Bot added a comment - Github user zhudebin closed the pull request at: https://github.com/apache/hadoop/pull/96
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Reopen this to change the resolution.

          Show
          ajisakaa Akira Ajisaka added a comment - Reopen this to change the resolution.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Applied your patch and ran the existing tests.

          $ mvn test -Dtest=TestLineReader,TestLineRecordReader
          (snip)
          Running org.apache.hadoop.util.TestLineReader
          Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.368 sec <<< FAILURE! - in org.apache.hadoop.util.TestLineReader
          testCustomDelimiter(org.apache.hadoop.util.TestLineReader)  Time elapsed: 0.23 sec  <<< ERROR!
          java.lang.ArrayIndexOutOfBoundsException: -1
                  at org.apache.hadoop.util.LineReader.readCustomLine(LineReader.java:314)
                  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:172)
                  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:377)
                  at org.apache.hadoop.util.TestLineReader.testCustomDelimiter(TestLineReader.java:98)
          

          The failure looks related to the patch. Hi binde, would you fix it and add a regression test for this bug?

          Show
          ajisakaa Akira Ajisaka added a comment - Applied your patch and ran the existing tests. $ mvn test -Dtest=TestLineReader,TestLineRecordReader (snip) Running org.apache.hadoop.util.TestLineReader Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.368 sec <<< FAILURE! - in org.apache.hadoop.util.TestLineReader testCustomDelimiter(org.apache.hadoop.util.TestLineReader) Time elapsed: 0.23 sec <<< ERROR! java.lang.ArrayIndexOutOfBoundsException: -1 at org.apache.hadoop.util.LineReader.readCustomLine(LineReader.java:314) at org.apache.hadoop.util.LineReader.readLine(LineReader.java:172) at org.apache.hadoop.util.LineReader.readLine(LineReader.java:377) at org.apache.hadoop.util.TestLineReader.testCustomDelimiter(TestLineReader.java:98) The failure looks related to the patch. Hi binde , would you fix it and add a regression test for this bug?
          Hide
          zhudebin binde added a comment -

          OK

          Show
          zhudebin binde added a comment - OK
          Hide
          zhudebin binde added a comment -

          add a test case

          Show
          zhudebin binde added a comment - add a test case
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi binde, I assigned you to this issue.
          Please hit "Submit Patch" instead of "Resolve Issue" after you updated the pull request.

          Show
          ajisakaa Akira Ajisaka added a comment - Hi binde , I assigned you to this issue. Please hit "Submit Patch" instead of "Resolve Issue" after you updated the pull request.
          Hide
          zhudebin binde added a comment -

          fix bug HADOOP-13192. two patchs

          Show
          zhudebin binde added a comment - fix bug HADOOP-13192 . two patchs
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks binde for attaching the patches in the jira, but actually we don't need to attach the patches when there is a corresponding GitHub pull request because the Jenkins precommit job runs on the pull request. However, we need to hit "Submit Patch" to change the status to "Patch Available" to run the precommit job.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks binde for attaching the patches in the jira, but actually we don't need to attach the patches when there is a corresponding GitHub pull request because the Jenkins precommit job runs on the pull request. However, we need to hit "Submit Patch" to change the status to "Patch Available" to run the precommit job.
          Hide
          zhudebin binde added a comment -

          Okay, I understand.

          Show
          zhudebin binde added a comment - Okay, I understand.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 48s trunk passed
          +1 compile 7m 12s trunk passed
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 24s trunk passed
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 0m 43s the patch passed
          +1 compile 7m 35s the patch passed
          +1 javac 7m 35s the patch passed
          -1 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 4 new + 29 unchanged - 0 fixed = 33 total (was 29)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 28s the patch passed
          +1 javadoc 0m 47s the patch passed
          +1 unit 8m 20s hadoop-common in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          39m 57s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Issue HADOOP-13192
          GITHUB PR https://github.com/apache/hadoop/pull/99
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 89017c85d9eb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 51d497f
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 48s trunk passed +1 compile 7m 12s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 43s the patch passed +1 compile 7m 35s the patch passed +1 javac 7m 35s the patch passed -1 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 4 new + 29 unchanged - 0 fixed = 33 total (was 29) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 28s the patch passed +1 javadoc 0m 47s the patch passed +1 unit 8m 20s hadoop-common in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 39m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Issue HADOOP-13192 GITHUB PR https://github.com/apache/hadoop/pull/99 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 89017c85d9eb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 51d497f Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9813/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment - - edited

          Thanks binde for updating the patch. The fix looks good to me. I ran mvn test -Dtest=*Reader* and all the tests succeeded. Would you fix the checkstyle warnings? I'm +1 if that is addressed.

          Show
          ajisakaa Akira Ajisaka added a comment - - edited Thanks binde for updating the patch. The fix looks good to me. I ran mvn test -Dtest=*Reader* and all the tests succeeded. Would you fix the checkstyle warnings? I'm +1 if that is addressed.
          Hide
          zhudebin binde added a comment -

          ok

          Show
          zhudebin binde added a comment - ok
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 20s trunk passed
          +1 compile 8m 51s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 35s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 8m 28s the patch passed
          +1 javac 8m 28s the patch passed
          +1 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 0 new + 17 unchanged - 12 fixed = 17 total (was 29)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 42s the patch passed
          +1 javadoc 0m 46s the patch passed
          -1 unit 7m 45s hadoop-common in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          44m 16s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Issue HADOOP-13192
          GITHUB PR https://github.com/apache/hadoop/pull/99
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cb8ca8844db9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0761379
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 20s trunk passed +1 compile 8m 51s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 35s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 8m 28s the patch passed +1 javac 8m 28s the patch passed +1 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 0 new + 17 unchanged - 12 fixed = 17 total (was 29) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 42s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 7m 45s hadoop-common in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 44m 16s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Issue HADOOP-13192 GITHUB PR https://github.com/apache/hadoop/pull/99 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cb8ca8844db9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0761379 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9823/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhudebin binde added a comment -

          I run the test case has no problem, why?

          Show
          zhudebin binde added a comment - I run the test case has no problem, why?
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi binde, the failing test is a known bug, see HADOOP-12588. This is an intermittent failing unit test (UT) so you may not reproduce it consistently on your local machine. I had a look at the patch and think it is an unrelated failure UT.

          So when contributing patches and review the pre-commit Jenkins report, you can generally check if the failing tests are related. If unrelated, you can search JIRA ticket that reports it and comment about it here. You can file another bug for tracking the failing UTs if no one has reported it yet.

          Thanks for the contribution.

          Show
          liuml07 Mingliang Liu added a comment - Hi binde , the failing test is a known bug, see HADOOP-12588 . This is an intermittent failing unit test (UT) so you may not reproduce it consistently on your local machine. I had a look at the patch and think it is an unrelated failure UT. So when contributing patches and review the pre-commit Jenkins report, you can generally check if the failing tests are related. If unrelated, you can search JIRA ticket that reports it and comment about it here. You can file another bug for tracking the failing UTs if no one has reported it yet. Thanks for the contribution.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, committing this.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, committing this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.7.3. Thanks binde for the contribution, and thanks Mingliang Liu for the help.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.7.3. Thanks binde for the contribution, and thanks Mingliang Liu for the help.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9985/)
          HADOOP-13192. org.apache.hadoop.util.LineReader cannot handle multibyte (aajisaka: rev fc6b50cc574e144fd314dea6c11987c6a384bfa6)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9985/ ) HADOOP-13192 . org.apache.hadoop.util.LineReader cannot handle multibyte (aajisaka: rev fc6b50cc574e144fd314dea6c11987c6a384bfa6) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aajisaka commented on the issue:

          https://github.com/apache/hadoop/pull/99

          I've committed this. Would you close this pull request?

          Show
          githubbot ASF GitHub Bot added a comment - Github user aajisaka commented on the issue: https://github.com/apache/hadoop/pull/99 I've committed this. Would you close this pull request?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zhudebin commented on the issue:

          https://github.com/apache/hadoop/pull/99

          ok

          Show
          githubbot ASF GitHub Bot added a comment - Github user zhudebin commented on the issue: https://github.com/apache/hadoop/pull/99 ok
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zhudebin closed the pull request at:

          https://github.com/apache/hadoop/pull/99

          Show
          githubbot ASF GitHub Bot added a comment - Github user zhudebin closed the pull request at: https://github.com/apache/hadoop/pull/99
          Hide
          haitao-tony sunhaitao added a comment -

          hi the second patch:0002-fix-bug-hadoop-1392-add-test-case-for-LineReader.patch does not remove " bufferPosn--;" which leads to dead lock and ut failure, please check

          Show
          haitao-tony sunhaitao added a comment - hi the second patch:0002-fix-bug-hadoop-1392-add-test-case-for-LineReader.patch does not remove " bufferPosn--;" which leads to dead lock and ut failure, please check
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi sunhaitao, I didn't actually review the 02 patch. I reviewed the pull request https://github.com/apache/hadoop/pull/99 and merged it. The pull request removes "bufferPosn--", so I'm thinking the problem does not occur.

          Show
          ajisakaa Akira Ajisaka added a comment - Hi sunhaitao , I didn't actually review the 02 patch. I reviewed the pull request https://github.com/apache/hadoop/pull/99 and merged it. The pull request removes "bufferPosn--", so I'm thinking the problem does not occur.
          Hide
          haitao-tony sunhaitao added a comment -

          i advise to change the 002 patch so as to remove "bufferPosn--", in case of causing misleading when some guys merge this patch.

          Show
          haitao-tony sunhaitao added a comment - i advise to change the 002 patch so as to remove "bufferPosn--", in case of causing misleading when some guys merge this patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch for the commit. (Generated by git diff d0162f2040a4d98fbac51527dfc9447888cb32ab fc6b50cc574e144fd314dea6c11987c6a384bfa6)
          Thanks sunhaitao!

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch for the commit. (Generated by git diff d0162f2040a4d98fbac51527dfc9447888cb32ab fc6b50cc574e144fd314dea6c11987c6a384bfa6 ) Thanks sunhaitao !
          Hide
          haitao-tony sunhaitao added a comment -

          welcome

          Show
          haitao-tony sunhaitao added a comment - welcome
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Cherry-picked it to 2.6.5 (trivial).

          Show
          sjlee0 Sangjin Lee added a comment - Cherry-picked it to 2.6.5 (trivial).
          Hide
          davidjou David Jou added a comment -

          I wanna to report test case to show multibyte delimiter between buffers still incorrect. If the ambiguous characters is longer than one, the match processing will only do once and send all ambiguous characters as data when not matched.

          Delimiter = "***|";
          String CurrentBufferTailToken
          = "**|data**";
          String NextBufferHeadToken
          = "*|";

          Show
          davidjou David Jou added a comment - I wanna to report test case to show multibyte delimiter between buffers still incorrect. If the ambiguous characters is longer than one, the match processing will only do once and send all ambiguous characters as data when not matched. Delimiter = "***|"; String CurrentBufferTailToken = "** |data **"; String NextBufferHeadToken = "*|";

            People

            • Assignee:
              zhudebin binde
              Reporter:
              zhudebin binde
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 5m
                5m
                Remaining:
                Remaining Estimate - 5m
                5m
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development