Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
-
None
-
Hadoop 2.6.0, ZK 3.4.5, Centos 6
Description
I ran CI to test 1.6.2RC3 on a 20 node EC2 cluster. After it ran for 24hr I stopped ingest and ran the M/R verify job. Based on running listscans in the shell I could see mappers were not running locally. I saw multiple error message like the following when the M/R job started.
15/01/29 22:14:42 WARN split.JobSplitWriter: Max block location exceeded for split: Range: [14b5%00; : [] 9223372036854775807 false,1696969696969698%00; : [] 9223372036854775807 false) Locations: [ip-10-1-2-21.ec2.internal, ip-10-1-2-21.ec2.internal, ip-10-1-2-15.ec2.internal, ip-10-1-2-13.ec2.internal, ip-10-1-2-16.ec2.internal, ip-10-1-2-18.ec2.internal, ip-10-1-2-18.ec2.internal, ip-10-1-2-18.ec2.internal, ip-10-1-2-18.ec2.internal, ip-10-1-2-18.ec2.internal, ip-10-1-2-18.ec2.internal, ip-10-1-2-18.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-27.ec2.internal, ip-10-1-2-28.ec2.internal, ip-10-1-2-28.ec2.internal, ip-10-1-2-20.ec2.internal, ip-10-1-2-17.ec2.internal, ip-10-1-2-25.ec2.internal, ip-10-1-2-25.ec2.internal, ip-10-1-2-25.ec2.internal, ip-10-1-2-25.ec2.internal, ip-10-1-2-25.ec2.internal, ip-10-1-2-25.ec2.internal] Table: ci TableID: 2 InstanceName: accumulo zooKeepers: 10.1.2.10,10.1.2.11,10.1.2.12 principal: root tokenSource: INLINE authenticationToken: org.apache.accumulo.core.client.security.tokens.PasswordToken@fee189f1 authenticationTokenFile: null Authorizations: offlineScan: false mockInstance: false isolatedScan: false localIterators: false fetchColumns: [] iterators: [] logLevel: INFO splitsize: 32 maxsize: 10