Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8966

Delta files created by hive hcatalog streaming cannot be compacted

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 1.0.0
    • Component/s: HCatalog
    • Labels:
      None
    • Environment:

      hive

    • Release Note:
      Don't do compaction on the current delta if it has a file in bucket pattern but not compactable

      Description

      hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where "n" is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction.

      Did a test, after removed the bucket_n_flush_length file, then the "alter table partition compact" finished successfully. If don't delete that file, nothing will be compacted.
      This is probably a very severity bug. Both 0.13 and 0.14 have this issue

      1. HIVE-8966.2.patch
        16 kB
        Alan Gates
      2. HIVE-8966.3.patch
        19 kB
        Alan Gates
      3. HIVE-8966.4.patch
        82 kB
        Alan Gates
      4. HIVE-8966.5.patch
        87 kB
        Alan Gates
      5. HIVE-8966.6.patch
        89 kB
        Alan Gates
      6. HIVE-8966.patch
        2 kB
        Jihong Liu
      7. HIVE-8966-branch-1.patch
        101 kB
        Alan Gates

        Activity

        Hide
        alangates Alan Gates added a comment -

        This flush length file should be removed when the batch is closed. Are you closing the transaction batch on a regular basis?

        Show
        alangates Alan Gates added a comment - This flush length file should be removed when the batch is closed. Are you closing the transaction batch on a regular basis?
        Hide
        jihongliu Jihong Liu added a comment -

        Yes. Closed the transaction batch. Suggest to do either the following two updates, or do both:

        1. if a file is non-bucket file, don't try to compact it. So update the following code:
        in org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.java
        Change the following code:

        private void addFileToMap(Matcher matcher, Path file, boolean sawBase,
        Map<Integer, BucketTracker> splitToBucketMap) {
        if (!matcher.find())

        { LOG.warn("Found a non-bucket file that we thought matched the bucket pattern! " + file.toString()); }

        .....
        to:
        private void addFileToMap(Matcher matcher, Path file, boolean sawBase,
        Map<Integer, BucketTracker> splitToBucketMap) {
        if (!matcher.find())

        { LOG.warn("Found a non-bucket file that we thought matched the bucket pattern! " + file.toString()); return; }

        ....

        2. don't use the bucket file pattern to name to "flush_length" file. So update the following code:
        in org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.java
        change the following code:
        static Path getSideFile(org.apache.tools.ant.types.Path main)

        { return new Path(main + "_flush_length"); }

        to:
        static Path getSideFile(org.apache.tools.ant.types.Path main) {
        if (main.toString().startsWith("bucket_"))

        { return new Path("bkt"+main.toString().substring(6)+ "_flush_length"); }

        else return new Path(main + "_flush_length");
        }

        after did the above updates and re-compiled the hive-exec.jar, the compaction works fine now

        Show
        jihongliu Jihong Liu added a comment - Yes. Closed the transaction batch. Suggest to do either the following two updates, or do both: 1. if a file is non-bucket file, don't try to compact it. So update the following code: in org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.java Change the following code: private void addFileToMap(Matcher matcher, Path file, boolean sawBase, Map<Integer, BucketTracker> splitToBucketMap) { if (!matcher.find()) { LOG.warn("Found a non-bucket file that we thought matched the bucket pattern! " + file.toString()); } ..... to: private void addFileToMap(Matcher matcher, Path file, boolean sawBase, Map<Integer, BucketTracker> splitToBucketMap) { if (!matcher.find()) { LOG.warn("Found a non-bucket file that we thought matched the bucket pattern! " + file.toString()); return; } .... 2. don't use the bucket file pattern to name to "flush_length" file. So update the following code: in org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.java change the following code: static Path getSideFile(org.apache.tools.ant.types.Path main) { return new Path(main + "_flush_length"); } to: static Path getSideFile(org.apache.tools.ant.types.Path main) { if (main.toString().startsWith("bucket_")) { return new Path("bkt"+main.toString().substring(6)+ "_flush_length"); } else return new Path(main + "_flush_length"); } after did the above updates and re-compiled the hive-exec.jar, the compaction works fine now
        Hide
        alangates Alan Gates added a comment -

        1 might be the right thing to do. 2 breaks backward compatibility. Before we do that though I'd like to understand why you still see the flush length files hanging around. In my tests I don't see this issue because the flush length file is properly cleaned up. I want to make sure that its existence doesn't mean something else is wrong.

        Do you see the flush length files in all delta directories or only the most recent?

        Show
        alangates Alan Gates added a comment - 1 might be the right thing to do. 2 breaks backward compatibility. Before we do that though I'd like to understand why you still see the flush length files hanging around. In my tests I don't see this issue because the flush length file is properly cleaned up. I want to make sure that its existence doesn't mean something else is wrong. Do you see the flush length files in all delta directories or only the most recent?
        Hide
        jihongliu Jihong Liu added a comment -

        That flush_length file is only in the most recent delta. By the way, for streaming loading, a transaction batch is probably always open since data keeps coming. Is it possible to do compaction in the streaming loading environment? Thanks

        Show
        jihongliu Jihong Liu added a comment - That flush_length file is only in the most recent delta. By the way, for streaming loading, a transaction batch is probably always open since data keeps coming. Is it possible to do compaction in the streaming loading environment? Thanks
        Hide
        alangates Alan Gates added a comment -

        Ok, that makes sense. You're current delta has the file because it's still open and being written to. It also explains why my tests don't see it, as they don't run long enough. The streaming is always done by the time the compactor kicks in. Why don't you post a patch to this JIRA with the change for 1, and I can get that committed.

        Gunther Hagleitner, I'd like to put this in 0.14.1 as well as trunk if you're ok with it, since it blocks compaction for users using the streaming interface.

        Show
        alangates Alan Gates added a comment - Ok, that makes sense. You're current delta has the file because it's still open and being written to. It also explains why my tests don't see it, as they don't run long enough. The streaming is always done by the time the compactor kicks in. Why don't you post a patch to this JIRA with the change for 1, and I can get that committed. Gunther Hagleitner , I'd like to put this in 0.14.1 as well as trunk if you're ok with it, since it blocks compaction for users using the streaming interface.
        Hide
        hagleitn Gunther Hagleitner added a comment -

        +1 for 0.14.1

        Show
        hagleitn Gunther Hagleitner added a comment - +1 for 0.14.1
        Hide
        jihongliu Jihong Liu added a comment -

        Thanks. So now the fix is in 0.14.1?

        Show
        jihongliu Jihong Liu added a comment - Thanks. So now the fix is in 0.14.1?
        Show
        jihongliu Jihong Liu added a comment - https://issues.apache.org/jira/i#browse/HIVE-8966
        Hide
        jihongliu Jihong Liu added a comment -

        Patch for HIVE08966

        Show
        jihongliu Jihong Liu added a comment - Patch for HIVE08966
        Hide
        jihongliu Jihong Liu added a comment -

        The patch is attached. Please review. Thanks

        Show
        jihongliu Jihong Liu added a comment - The patch is attached. Please review. Thanks
        Hide
        jihongliu Jihong Liu added a comment -

        I think we may have to withdraw this patch for now. It looks like currently hive must not support doing compaction and loading in the same time for a partition.
        Without this patch, if loading for a partition is not completely finished, compaction will always fail, so nothing happen. After apply this patch, compaction will go through and finish. However we may loss data! I did a test. Data could be lost if we do compaction meanwhile the loading is not finished yet.
        But if keep the current version, it must be a limitation for hive. If streaming load to a partition for a long period, performance will be affected if cannot do compaction on it.

        For completely solve this issue, my initial thinking is that the delta files with open transaction should not be compacted. Currently they must be inlcuded, and it is probably the reason for data lost. But other closed delta files should be able to compact. So we can do compaction and loading in the same time.

        Show
        jihongliu Jihong Liu added a comment - I think we may have to withdraw this patch for now. It looks like currently hive must not support doing compaction and loading in the same time for a partition. Without this patch, if loading for a partition is not completely finished, compaction will always fail, so nothing happen. After apply this patch, compaction will go through and finish. However we may loss data! I did a test. Data could be lost if we do compaction meanwhile the loading is not finished yet. But if keep the current version, it must be a limitation for hive. If streaming load to a partition for a long period, performance will be affected if cannot do compaction on it. For completely solve this issue, my initial thinking is that the delta files with open transaction should not be compacted. Currently they must be inlcuded, and it is probably the reason for data lost. But other closed delta files should be able to compact. So we can do compaction and loading in the same time.
        Hide
        alangates Alan Gates added a comment -

        Jihong, thanks for doing the testing on this.

        We could change this to not compact the current delta file, or we could change the cleaner to not remove the delta file that was still open during compaction. I'll try to look at this in the next couple of days. We need to get this fixed for 0.14.1.

        Show
        alangates Alan Gates added a comment - Jihong, thanks for doing the testing on this. We could change this to not compact the current delta file, or we could change the cleaner to not remove the delta file that was still open during compaction. I'll try to look at this in the next couple of days. We need to get this fixed for 0.14.1.
        Hide
        jihongliu Jihong Liu added a comment -

        Great. I am working on that now. Will update you after finished the testing.

        Show
        jihongliu Jihong Liu added a comment - Great. I am working on that now. Will update you after finished the testing.
        Hide
        jihongliu Jihong Liu added a comment -

        The scenario of data lost:
        Assume when start compaction there are two deltas, delta_00011_00020 and delta_00021_00030, where the transaction batch in the first one is closed, and the second one still has transaction batch open. After compaction is finished, the status in compaction_ queue will become “ready_for_clean”. Then clean process will be triggered. Cleaner will remove all deltas if its transaction id is less than the base which just created and if there is no lock on it. In the meantime, we still load data into the second delta. When finish loading and close the transaction batch, cleaner detects no lock on that, so delete it. So the new data added after compaction will be lost.

        Show
        jihongliu Jihong Liu added a comment - The scenario of data lost: Assume when start compaction there are two deltas, delta_00011_00020 and delta_00021_00030, where the transaction batch in the first one is closed, and the second one still has transaction batch open. After compaction is finished, the status in compaction_ queue will become “ready_for_clean”. Then clean process will be triggered. Cleaner will remove all deltas if its transaction id is less than the base which just created and if there is no lock on it. In the meantime, we still load data into the second delta. When finish loading and close the transaction batch, cleaner detects no lock on that, so delete it. So the new data added after compaction will be lost.
        Hide
        jihongliu Jihong Liu added a comment -

        Solution:
        if the last delta has any file which is in bucket file pattern, but actually is non bucket file, don’t compact this delta. When a transaction is not close, a delta will have a file like bucket_n_flash_length, which is non bucket file. Actually for any reason, if the last delta has a file with bucket file pattern but not compactable, we should ignore this delta. Since after compaction, the delta will be removed. So if the whole delta cannot be compacted, leave it as what it is. So in the above scenario, the second delta will not be compacted. And the cleaner will not remove it because it has higher transaction id than the new created compaction file(base or delta).
        The reason we only do the above for the last delta is to consider the case that two or more transaction batches may be created and the last one is close first. Then if the last delta gets compacted, the transaction id in the base will be big, so all deltas will be removed by cleaner. So data could be lost. In this case, in the list of deltas for compaction, at least one delta has that bucket_n_flash_length file inside. Since we do not ignore it, the compaction will be auto-fail, so nothing happen, no data lost. In this case, the compaction can only be done after all transaction batches are closed. Although it is not so good, at least no data lost.
        The patch is attached. It adds one method to test whether needs to remove the last delta from the delta list. And before process the delta list, run that method. After applying this patch, no data is lost. We can do either major or minor compaction meanwhile keeping loading data in the same time.

        Show
        jihongliu Jihong Liu added a comment - Solution: if the last delta has any file which is in bucket file pattern, but actually is non bucket file, don’t compact this delta. When a transaction is not close, a delta will have a file like bucket_n_flash_length, which is non bucket file. Actually for any reason, if the last delta has a file with bucket file pattern but not compactable, we should ignore this delta. Since after compaction, the delta will be removed. So if the whole delta cannot be compacted, leave it as what it is. So in the above scenario, the second delta will not be compacted. And the cleaner will not remove it because it has higher transaction id than the new created compaction file(base or delta). The reason we only do the above for the last delta is to consider the case that two or more transaction batches may be created and the last one is close first. Then if the last delta gets compacted, the transaction id in the base will be big, so all deltas will be removed by cleaner. So data could be lost. In this case, in the list of deltas for compaction, at least one delta has that bucket_n_flash_length file inside. Since we do not ignore it, the compaction will be auto-fail, so nothing happen, no data lost. In this case, the compaction can only be done after all transaction batches are closed. Although it is not so good, at least no data lost. The patch is attached. It adds one method to test whether needs to remove the last delta from the delta list. And before process the delta list, run that method. After applying this patch, no data is lost. We can do either major or minor compaction meanwhile keeping loading data in the same time.
        Hide
        jihongliu Jihong Liu added a comment -

        By the way, hive may need another cleaning process which auto removes the bucket_n_flash_length file if the connection is actually closed. A program may not be able to close a transaction batch, due to many reasons, for example, network disconnected, server shutdown, application killed, and etc. So if the connection which creates a batch has been closed, that bucket_n_flash_length file needs to be removed. Otherwise that delta and the deltas after it can never be compacted unless we remove that file manually.

        Show
        jihongliu Jihong Liu added a comment - By the way, hive may need another cleaning process which auto removes the bucket_n_flash_length file if the connection is actually closed. A program may not be able to close a transaction batch, due to many reasons, for example, network disconnected, server shutdown, application killed, and etc. So if the connection which creates a batch has been closed, that bucket_n_flash_length file needs to be removed. Otherwise that delta and the deltas after it can never be compacted unless we remove that file manually.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12685584/HIVE-8966.patch

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1985/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1985/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1985/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
        + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-1985/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ svn = \s\v\n ]]
        + [[ -n '' ]]
        + [[ -d apache-svn-trunk-source ]]
        + [[ ! -d apache-svn-trunk-source/.svn ]]
        + [[ ! -d apache-svn-trunk-source ]]
        + cd apache-svn-trunk-source
        + svn revert -R .
        Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java'
        Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java'
        ++ awk '{print $2}'
        ++ egrep -v '^X|^Performing status on external'
        ++ svn status --no-ignore
        + rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/scheduler/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target accumulo-handler/target hwi/target common/target common/src/gen common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java.orig ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java
        + svn update
        
        Fetching external item into 'hcatalog/src/test/e2e/harness'
        External at revision 1643648.
        
        At revision 1643648.
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12685584 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12685584/HIVE-8966.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1985/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1985/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1985/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-1985/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java' Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java' ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/scheduler/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target accumulo-handler/target hwi/target common/target common/src/gen common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java.orig ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1643648. At revision 1643648. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12685584 - PreCommit-HIVE-TRUNK-Build
        Hide
        jihongliu Jihong Liu added a comment -

        Hi Alan,I have created a new patch. It works fine. The patch is pasted in that jira, also added comment about the logic. Please have a look. Thanks and have a good dayJihong
        From: Alan Gates (JIRA) <jira@apache.org> 
        To: jhliu08@yahoo.com
        Sent: Friday, December 5, 2014 7:41 AM
        Subject: [jira] [Commented] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted

            [ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14235645#comment-14235645 ]

        Alan Gates commented on HIVE-8966:
        ----------------------------------

        Jihong, thanks for doing the testing on this. 

        We could change this to not compact the current delta file, or we could change the cleaner to not remove the delta file that was still open during compaction.  I'll try to look at this in the next couple of days.  We need to get this fixed for 0.14.1.


        This message was sent by Atlassian JIRA
        (v6.3.4#6332)

        Show
        jihongliu Jihong Liu added a comment - Hi Alan,I have created a new patch. It works fine. The patch is pasted in that jira, also added comment about the logic. Please have a look. Thanks and have a good dayJihong From: Alan Gates (JIRA) <jira@apache.org>  To: jhliu08@yahoo.com Sent: Friday, December 5, 2014 7:41 AM Subject: [jira] [Commented] ( HIVE-8966 ) Delta files created by hive hcatalog streaming cannot be compacted     [ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14235645#comment-14235645 ] Alan Gates commented on HIVE-8966 : ---------------------------------- Jihong, thanks for doing the testing on this.  We could change this to not compact the current delta file, or we could change the cleaner to not remove the delta file that was still open during compaction.  I'll try to look at this in the next couple of days.  We need to get this fixed for 0.14.1. – This message was sent by Atlassian JIRA (v6.3.4#6332)
        Hide
        jihongliu Jihong Liu added a comment -

        Alan,
        I created a wrong patch about 1 hour ago. Before I removed it. QA automatically did the above test. Please ignore and look the current attached patch. I think it really solves the issue.

        Show
        jihongliu Jihong Liu added a comment - Alan, I created a wrong patch about 1 hour ago. Before I removed it. QA automatically did the above test. Please ignore and look the current attached patch. I think it really solves the issue.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12685590/HIVE-8966.patch

        ERROR: -1 due to 2 failed/errored test(s), 6696 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate
        org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1986/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1986/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1986/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12685590 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12685590/HIVE-8966.patch ERROR: -1 due to 2 failed/errored test(s), 6696 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1986/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1986/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1986/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12685590 - PreCommit-HIVE-TRUNK-Build
        Hide
        jihongliu Jihong Liu added a comment -

        I am confused about the QA test. The error looks like not related to HIVE-8966.patch. First, was this patch really included in the build? Also this patch is for 0.14.1, not for trunk.

        Show
        jihongliu Jihong Liu added a comment - I am confused about the QA test. The error looks like not related to HIVE-8966 .patch. First, was this patch really included in the build? Also this patch is for 0.14.1, not for trunk.
        Hide
        alangates Alan Gates added a comment -

        Don't worry about the results from testing, those tests are flaky. I'll review the patch.

        Show
        alangates Alan Gates added a comment - Don't worry about the results from testing, those tests are flaky. I'll review the patch.
        Hide
        alangates Alan Gates added a comment -

        Rather than go remove these directories from the list of deltas I think it makes more sense to change Directory.getAcidState to not include these deltas. We obviously can't do that in all cases, as readers need to see these deltas. But we can change it to see that this is the compactor and therefore those should be excluded. I'll post a patch with this change.

        Show
        alangates Alan Gates added a comment - Rather than go remove these directories from the list of deltas I think it makes more sense to change Directory.getAcidState to not include these deltas. We obviously can't do that in all cases, as readers need to see these deltas. But we can change it to see that this is the compactor and therefore those should be excluded. I'll post a patch with this change.
        Hide
        jihongliu Jihong Liu added a comment -

        I see. Basically there are two solutions. One is that when get the delta list, we don't include the current delta if it has open tranaction. So uptate the AcidUtil.getAcidState() directly. The other is what I posted here. We first get the delta list, then when do compaction, we don't compact the last one if there is open transaction. The first solution is better as long as changing getAcidState() doesn't affact other existing code, since it is a public static method.
        By the way, we should only do that to the current delta (the delta with the largest transaction id), not to all deltas which have open transactions. If I am correct, the base file will be named based on the largest transaction id in the deltas. So if the latest delta is closed, but an early delta has an open transaction, we should not do anything. So simply let the compaction fail. Otherwise, the base will be named by the last transaction id, and all early deltas will be removed. That will cause data lost. This is my understanding, please correct me, it it is not correct. Thanks

        Show
        jihongliu Jihong Liu added a comment - I see. Basically there are two solutions. One is that when get the delta list, we don't include the current delta if it has open tranaction. So uptate the AcidUtil.getAcidState() directly. The other is what I posted here. We first get the delta list, then when do compaction, we don't compact the last one if there is open transaction. The first solution is better as long as changing getAcidState() doesn't affact other existing code, since it is a public static method. By the way, we should only do that to the current delta (the delta with the largest transaction id), not to all deltas which have open transactions. If I am correct, the base file will be named based on the largest transaction id in the deltas. So if the latest delta is closed, but an early delta has an open transaction, we should not do anything. So simply let the compaction fail. Otherwise, the base will be named by the last transaction id, and all early deltas will be removed. That will cause data lost. This is my understanding, please correct me, it it is not correct. Thanks
        Hide
        alangates Alan Gates added a comment -

        A new version of the patch that moves Jihong's code into AcidUtils.getAcidState so that delta directories with flush length files are not put into the list of files to compact.

        Jihong Liu, could you test this on your end to make sure it addresses your issues. I'll also do some long running tests to see that it allows compaction while streaming is ongoing.

        Show
        alangates Alan Gates added a comment - A new version of the patch that moves Jihong's code into AcidUtils.getAcidState so that delta directories with flush length files are not put into the list of files to compact. Jihong Liu , could you test this on your end to make sure it addresses your issues. I'll also do some long running tests to see that it allows compaction while streaming is ongoing.
        Hide
        owen.omalley Owen O'Malley added a comment -

        Alan, your patch looks good +1

        Show
        owen.omalley Owen O'Malley added a comment - Alan, your patch looks good +1
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12686124/HIVE-8966.2.patch

        ERROR: -1 due to 2 failed/errored test(s), 6704 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_1
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2013/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2013/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2013/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12686124 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12686124/HIVE-8966.2.patch ERROR: -1 due to 2 failed/errored test(s), 6704 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_1 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2013/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2013/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2013/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12686124 - PreCommit-HIVE-TRUNK-Build
        Hide
        jihongliu Jihong Liu added a comment -

        Alan,
        Your idea is very good. But there is an issue here – we should only do this "compacting" test for the most recent delta, not for all deltas. Following is an example for the reason:
        Assume there are two deltas:
        1 delta_00011_00020 this delta has open transaction batch
        2 delta_00021_00030 this delta has no open transaction batch. All closed.

        In the above, the first delta has open transaction batch, the second has not. And the second delta is the most recent delta. This case is possible, especially when multiple threads write to the same partition. If we ignore the first one, then the compaction will success and create a base, like base_00030. Then cleaner will delete all the two deltas since their transaction id are less or equal to the base transaction id. Thus the data in delta 2 will be lost. This is why we should only test the most recent delta, all other deltas will be automatically in the list. Thus in this case, the compaction will be fail, since the "flush_length" file is there. And for this case, the compaction will be success only when all transaction batchs are closed. Although it is not perfect, at least no data lost. Since each delta file and transaction id for compaction is not saved anywhere, probably this is the only solution for now.
        In my removeNotCompactableDeltas() method, we first sort the deltas, then only check the last one. But the name: "removeNotCompactableDeltas" is not good, easy makes confusion. It will be clear if named it as "removeLastDeltaIfNotCompactable".
        Thanks

        Show
        jihongliu Jihong Liu added a comment - Alan, Your idea is very good. But there is an issue here – we should only do this "compacting" test for the most recent delta, not for all deltas. Following is an example for the reason: Assume there are two deltas: 1 delta_00011_00020 this delta has open transaction batch 2 delta_00021_00030 this delta has no open transaction batch. All closed. In the above, the first delta has open transaction batch, the second has not. And the second delta is the most recent delta. This case is possible, especially when multiple threads write to the same partition. If we ignore the first one, then the compaction will success and create a base, like base_00030. Then cleaner will delete all the two deltas since their transaction id are less or equal to the base transaction id. Thus the data in delta 2 will be lost. This is why we should only test the most recent delta, all other deltas will be automatically in the list. Thus in this case, the compaction will be fail, since the "flush_length" file is there. And for this case, the compaction will be success only when all transaction batchs are closed. Although it is not perfect, at least no data lost. Since each delta file and transaction id for compaction is not saved anywhere, probably this is the only solution for now. In my removeNotCompactableDeltas() method, we first sort the deltas, then only check the last one. But the name: "removeNotCompactableDeltas" is not good, easy makes confusion. It will be clear if named it as "removeLastDeltaIfNotCompactable". Thanks
        Hide
        alangates Alan Gates added a comment -

        Right, makes sense. I need to think about whether it makes more sense to change AcidUtils.getAcidState to catch this as well or whether your approach of post processing it in the compactor makes more sense.

        Show
        alangates Alan Gates added a comment - Right, makes sense. I need to think about whether it makes more sense to change AcidUtils.getAcidState to catch this as well or whether your approach of post processing it in the compactor makes more sense.
        Hide
        alangates Alan Gates added a comment -

        A new version of the patch which properly handles not putting any deltas in the list once we see a delta with a flush length file. Unfortunately, Owen O'Malley who needs to review this is out for a couple of weeks. Jihong Liu, please take a look at this and test it in your environment.

        Show
        alangates Alan Gates added a comment - A new version of the patch which properly handles not putting any deltas in the list once we see a delta with a flush length file. Unfortunately, Owen O'Malley who needs to review this is out for a couple of weeks. Jihong Liu , please take a look at this and test it in your environment.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12688699/HIVE-8966.3.patch

        ERROR: -1 due to 2 failed/errored test(s), 6724 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_lvj_mapjoin
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2168/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2168/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2168/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12688699 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12688699/HIVE-8966.3.patch ERROR: -1 due to 2 failed/errored test(s), 6724 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_lvj_mapjoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2168/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2168/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2168/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12688699 - PreCommit-HIVE-TRUNK-Build
        Hide
        jihongliu Jihong Liu added a comment -

        Did a test. Generally the new version works as expected. But for the following case, the compaction will always fail:

        1. due to any reason, the writer exits without closing a batch. So the "length" file is still there. This could happen, for example the program is killed, hive/server restarts.
        2. restart the program, so a new writer and a new batch is created and continute to write into the same partition. The data will go to a new delta.
        3. Now we manually delete that "length" file in the previous delta. Then do compaction, but it fails. Even we totally exit the program so that no any open batch and no any "length" file, the compaction will never success for this partition.

        However the current hive 14.0 will work fine for the above case.

        Show
        jihongliu Jihong Liu added a comment - Did a test. Generally the new version works as expected. But for the following case, the compaction will always fail: 1. due to any reason, the writer exits without closing a batch. So the "length" file is still there. This could happen, for example the program is killed, hive/server restarts. 2. restart the program, so a new writer and a new batch is created and continute to write into the same partition. The data will go to a new delta. 3. Now we manually delete that "length" file in the previous delta. Then do compaction, but it fails. Even we totally exit the program so that no any open batch and no any "length" file, the compaction will never success for this partition. However the current hive 14.0 will work fine for the above case.
        Hide
        alangates Alan Gates added a comment -

        What error message does it give when it fails? I would expect this to work.

        Show
        alangates Alan Gates added a comment - What error message does it give when it fails? I would expect this to work.
        Hide
        jihongliu Jihong Liu added a comment -

        The error occur when doing the mapreduce job. Following is log in hivemetastore.log

        2015-01-06 16:42:22,506 INFO [sfdmgctmn003.gid.gap.com-32]: compactor.Worker (Worker.java:run(137)) - Starting MAJOR compaction for ds_infra.event_metrics.date=2014-12-24
        2015-01-06 16:42:22,564 INFO [sfdmgctmn003.gid.gap.com-32]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(285)) - Timeline service address: http://sfdmgctmn003.gid.gap.com:8188/ws/v1/timeline/
        2015-01-06 16:42:22,622 INFO [sfdmgctmn003.gid.gap.com-32]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(285)) - Timeline service address: http://sfdmgctmn003.gid.gap.com:8188/ws/v1/timeline/
        2015-01-06 16:42:22,628 WARN [sfdmgctmn003.gid.gap.com-32]: mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(153)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
        2015-01-06 16:42:22,753 WARN [sfdmgctmn003.gid.gap.com-32]: split.JobSplitWriter (JobSplitWriter.java:writeOldSplits(168)) - Max block location exceeded for split: CompactorInputSplit

        {base: hdfs://sfdmgct/apps/hive/warehouse/ds_infra/event_metrics/date=2014-12-24/base_0035304, bucket: 1, length: 292280, deltas: [delta_0035311_0035313, delta_0035479_0035481, delta_0035491_0035493, delta_0035515_0035517, delta_0035533_0035535, delta_0035548_0035550, delta_0035563_0035565, delta_0035578_0035580, delta_0035593_0035595, delta_0035599_0035601, delta_0035656_0035658, delta_0035671_0035673, delta_0035686_0035688, delta_0035701_0035703, delta_0035716_0035718, delta_0035731_0035733, delta_0035746_0035748, delta_0035761_0035763, delta_0035776_0035778, delta_0035791_0035793, delta_0035806_0035808, delta_0035821_0035823, delta_0035830_0035832, delta_0035842_0035844, delta_0035854_0035856, delta_0035866_0035868, delta_0035878_0035880]}

        splitsize: 27 maxsize: 10
        2015-01-06 16:42:22,753 WARN [sfdmgctmn003.gid.gap.com-32]: split.JobSplitWriter (JobSplitWriter.java:writeOldSplits(168)) - Max block location exceeded for split: CompactorInputSplit

        {base: null, bucket: 3, length: 199770, deltas: [delta_0035311_0035313, delta_0035479_0035481, delta_0035491_0035493, delta_0035515_0035517, delta_0035533_0035535, delta_0035548_0035550, delta_0035563_0035565, delta_0035578_0035580, delta_0035593_0035595, delta_0035599_0035601, delta_0035656_0035658, delta_0035671_0035673, delta_0035686_0035688, delta_0035701_0035703, delta_0035716_0035718, delta_0035731_0035733, delta_0035746_0035748, delta_0035761_0035763, delta_0035776_0035778, delta_0035791_0035793, delta_0035806_0035808, delta_0035821_0035823, delta_0035830_0035832, delta_0035842_0035844, delta_0035854_0035856, delta_0035866_0035868, delta_0035878_0035880]}

        splitsize: 21 maxsize: 10
        2015-01-06 16:42:22,753 WARN [sfdmgctmn003.gid.gap.com-32]: split.JobSplitWriter (JobSplitWriter.java:writeOldSplits(168)) - Max block location exceeded for split: CompactorInputSplit

        {base: hdfs://sfdmgct/apps/hive/warehouse/ds_infra/event_metrics/date=2014-12-24/base_0035304, bucket: 0, length: 172391, deltas: [delta_0035311_0035313, delta_0035479_0035481, delta_0035491_0035493, delta_0035515_0035517, delta_0035533_0035535, delta_0035548_0035550, delta_0035563_0035565, delta_0035578_0035580, delta_0035593_0035595, delta_0035599_0035601, delta_0035656_0035658, delta_0035671_0035673, delta_0035686_0035688, delta_0035701_0035703, delta_0035716_0035718, delta_0035731_0035733, delta_0035746_0035748, delta_0035761_0035763, delta_0035776_0035778, delta_0035791_0035793, delta_0035806_0035808, delta_0035821_0035823, delta_0035830_0035832, delta_0035842_0035844, delta_0035854_0035856, delta_0035866_0035868, delta_0035878_0035880]}

        splitsize: 30 maxsize: 10
        2015-01-06 16:42:22,777 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:4
        2015-01-06 16:42:22,793 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_1419291043936_1639
        2015-01-06 16:42:23,000 INFO [sfdmgctmn003.gid.gap.com-32]: impl.YarnClientImpl (YarnClientImpl.java:submitApplication(251)) - Submitted application application_1419291043936_1639
        2015-01-06 16:42:23,001 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://sfdmgctmn002.gid.gap.com:8088/proxy/application_1419291043936_1639/
        2015-01-06 16:42:23,001 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:monitorAndPrintJob(1345)) - Running job: job_1419291043936_1639
        2015-01-06 16:42:30,042 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:monitorAndPrintJob(1366)) - Job job_1419291043936_1639 running in uber mode : false
        2015-01-06 16:42:30,043 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:monitorAndPrintJob(1373)) - map 0% reduce 0%
        2015-01-06 16:42:35,066 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:printTaskEvents(1452)) - Task Id : attempt_1419291043936_1639_m_000002_0, Status : FAILED
        2015-01-06 16:42:37,078 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:monitorAndPrintJob(1373)) - map 75% reduce 0%
        2015-01-06 16:42:41,091 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:printTaskEvents(1452)) - Task Id : attempt_1419291043936_1639_m_000002_1, Status : FAILED
        2015-01-06 16:42:45,105 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:printTaskEvents(1452)) - Task Id : attempt_1419291043936_1639_m_000002_2, Status : FAILED
        2015-01-06 16:42:52,124 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:monitorAndPrintJob(1373)) - map 100% reduce 0%
        2015-01-06 16:42:52,130 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:monitorAndPrintJob(1386)) - Job job_1419291043936_1639 failed with state FAILED due to: Task failed task_1419291043936_1639_m_000002
        Job failed as tasks failed. failedMaps:1 failedReduces:0

        2015-01-06 16:42:52,149 INFO [sfdmgctmn003.gid.gap.com-32]: mapreduce.Job (Job.java:monitorAndPrintJob(1391)) - Counters: 32
        File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=668781
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=840325
        HDFS: Number of bytes written=405818
        HDFS: Number of read operations=243
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=3
        Job Counters
        Failed map tasks=4
        Launched map tasks=7
        Other local map tasks=3
        Data-local map tasks=4
        Total time spent by all maps in occupied slots (ms)=32359
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=32359
        Total vcore-seconds taken by all map tasks=32359
        Total megabyte-seconds taken by all map tasks=463898624
        Map-Reduce Framework
        Map input records=3
        Map output records=0
        Input split bytes=10663
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=153
        CPU time spent (ms)=7680
        Physical memory (bytes) snapshot=1065402368
        Virtual memory (bytes) snapshot=39759937536
        Total committed heap usage (bytes)=6714032128
        File Input Format Counters
        Bytes Read=0
        File Output Format Counters
        Bytes Written=0
        2015-01-06 16:42:52,150 ERROR [sfdmgctmn003.gid.gap.com-32]: compactor.Worker (Worker.java:run(159)) - Caught exception while trying to compact ds_infra.event_metrics.date=2014-12-24. Marking clean to avoid repeated failures, java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:184)
        at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:145)

        2015-01-06 16:42:52,150 ERROR [sfdmgctmn003.gid.gap.com-32]: txn.CompactionTxnHandler (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least one row from completed_txn_components when marking compaction entry as clean!

        --------------------------------------------------------------------------------------------------------------------------------
        following is mapreduce job log. Got the log from uri mentioned in the above log: (http://sfdmgctmn002.gid.gap.com:8088/proxy/application_1419291043936_1639/)

        2015-01-06 16:42:25,991 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1419291043936_1639_000001
        2015-01-06 16:42:26,304 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        2015-01-06 16:42:26,314 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
        2015-01-06 16:42:26,314 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id

        { id: 1639 cluster_timestamp: 1419291043936 }

        attemptId: 1 } keyId: -950898635)
        2015-01-06 16:42:26,842 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
        2015-01-06 16:42:26,919 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter
        2015-01-06 16:42:26,921 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter
        2015-01-06 16:42:26,951 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
        2015-01-06 16:42:26,951 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
        2015-01-06 16:42:26,952 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
        2015-01-06 16:42:26,952 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
        2015-01-06 16:42:26,952 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
        2015-01-06 16:42:26,956 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
        2015-01-06 16:42:26,956 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
        2015-01-06 16:42:26,957 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
        2015-01-06 16:42:26,975 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring
        2015-01-06 16:42:26,987 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring
        2015-01-06 16:42:26,999 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring
        2015-01-06 16:42:27,023 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
        2015-01-06 16:42:27,050 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
        2015-01-06 16:42:27,191 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties
        2015-01-06 16:42:27,232 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
        2015-01-06 16:42:27,232 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
        2015-01-06 16:42:27,239 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1419291043936_1639 to jobTokenSecretManager
        2015-01-06 16:42:27,327 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1419291043936_1639 because: not enabled;
        2015-01-06 16:42:27,340 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1419291043936_1639 = 845156. Number of splits = 4
        2015-01-06 16:42:27,341 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1419291043936_1639 = 0
        2015-01-06 16:42:27,341 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from NEW to INITED
        2015-01-06 16:42:27,341 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1419291043936_1639.
        2015-01-06 16:42:27,359 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
        2015-01-06 16:42:27,366 INFO Socket Reader #1 for port 57524 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 57524
        2015-01-06 16:42:27,379 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
        2015-01-06 16:42:27,380 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
        2015-01-06 16:42:27,381 INFO [IPC Server listener on 57524] org.apache.hadoop.ipc.Server: IPC Server listener on 57524: starting
        2015-01-06 16:42:27,381 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at sfdmgctsn004.gid.gap.com/10.9.21.134:57524
        2015-01-06 16:42:27,429 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
        2015-01-06 16:42:27,431 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
        2015-01-06 16:42:27,439 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
        2015-01-06 16:42:27,472 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
        2015-01-06 16:42:27,472 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
        2015-01-06 16:42:27,475 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
        2015-01-06 16:42:27,475 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
        2015-01-06 16:42:27,482 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 45674
        2015-01-06 16:42:27,482 INFO [main] org.mortbay.log: jetty-6.1.26.hwx
        2015-01-06 16:42:27,505 INFO [main] org.mortbay.log: Extract jar:file:/data/sfdmgct05/hadoop/yarn/local/filecache/12/mapreduce.tar.gz/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.2.2.0.0-2041.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_45674_mapreduce____.ycambd/webapp
        2015-01-06 16:42:27,805 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:45674
        2015-01-06 16:42:27,806 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 45674
        2015-01-06 16:42:28,026 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
        2015-01-06 16:42:28,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1419291043936_1639
        2015-01-06 16:42:28,030 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
        2015-01-06 16:42:28,030 INFO Socket Reader #1 for port 33406 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33406
        2015-01-06 16:42:28,034 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
        2015-01-06 16:42:28,034 INFO [IPC Server listener on 33406] org.apache.hadoop.ipc.Server: IPC Server listener on 33406: starting
        2015-01-06 16:42:28,074 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
        2015-01-06 16:42:28,075 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
        2015-01-06 16:42:28,075 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
        2015-01-06 16:42:28,158 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:186368, vCores:32>
        2015-01-06 16:42:28,158 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: default
        2015-01-06 16:42:28,160 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
        2015-01-06 16:42:28,161 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
        2015-01-06 16:42:28,165 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from INITED to SETUP
        2015-01-06 16:42:28,167 INFO CommitterEvent Processor #0 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
        2015-01-06 16:42:28,168 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from SETUP to RUNNING
        2015-01-06 16:42:28,182 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,186 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,188 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,191 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06
        2015-01-06 16:42:28,194 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000000 Task Transitioned from NEW to SCHEDULED
        2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06
        2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000001 Task Transitioned from NEW to SCHEDULED
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000002 Task Transitioned from NEW to SCHEDULED
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05
        2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000003 Task Transitioned from NEW to SCHEDULED
        2015-01-06 16:42:28,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
        2015-01-06 16:42:28,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED
        2015-01-06 16:42:28,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED
        2015-01-06 16:42:28,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED
        2015-01-06 16:42:28,201 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:14336, vCores:1>
        2015-01-06 16:42:28,226 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1419291043936_1639, File: hdfs://sfdmgct:8020/user/hive/.staging/job_1419291043936_1639/job_1419291043936_1639_1.jhist
        2015-01-06 16:42:29,160 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
        2015-01-06 16:42:29,181 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=8 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:458752, vCores:79> knownNMs=5
        2015-01-06 16:42:30,189 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
        2015-01-06 16:42:30,190 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000002 to attempt_1419291043936_1639_m_000000_0
        2015-01-06 16:42:30,191 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000003 to attempt_1419291043936_1639_m_000001_0
        2015-01-06 16:42:30,191 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000004 to attempt_1419291043936_1639_m_000002_0
        2015-01-06 16:42:30,191 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000005 to attempt_1419291043936_1639_m_000003_0
        2015-01-06 16:42:30,191 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:30,217 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:30,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://sfdmgct/user/hive/.staging/job_1419291043936_1639/job.jar
        2015-01-06 16:42:30,229 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hive/.staging/job_1419291043936_1639/job.xml
        2015-01-06 16:42:30,231 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
        2015-01-06 16:42:30,231 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
        2015-01-06 16:42:30,231 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
        2015-01-06 16:42:30,247 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
        2015-01-06 16:42:30,248 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:30,248 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
        2015-01-06 16:42:30,249 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:30,249 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
        2015-01-06 16:42:30,249 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:30,250 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
        2015-01-06 16:42:30,251 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000002 taskAttempt attempt_1419291043936_1639_m_000000_0
        2015-01-06 16:42:30,251 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000003 taskAttempt attempt_1419291043936_1639_m_000001_0
        2015-01-06 16:42:30,251 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000004 taskAttempt attempt_1419291043936_1639_m_000002_0
        2015-01-06 16:42:30,251 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000005 taskAttempt attempt_1419291043936_1639_m_000003_0
        2015-01-06 16:42:30,252 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000002_0
        2015-01-06 16:42:30,252 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000001_0
        2015-01-06 16:42:30,252 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000003_0
        2015-01-06 16:42:30,252 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000000_0
        2015-01-06 16:42:30,253 INFO ContainerLauncher #2 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454
        2015-01-06 16:42:30,266 INFO ContainerLauncher #0 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454
        2015-01-06 16:42:30,267 INFO ContainerLauncher #3 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454
        2015-01-06 16:42:30,267 INFO ContainerLauncher #1 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454
        2015-01-06 16:42:30,294 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000002_0 : 13562
        2015-01-06 16:42:30,294 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000003_0 : 13562
        2015-01-06 16:42:30,294 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000000_0 : 13562
        2015-01-06 16:42:30,294 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000001_0 : 13562
        2015-01-06 16:42:30,295 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000000_0] using containerId: [container_1419291043936_1639_01_000002 on NM: [sfdmgctsn004.gid.gap.com:45454]
        2015-01-06 16:42:30,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
        2015-01-06 16:42:30,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000003_0] using containerId: [container_1419291043936_1639_01_000005 on NM: [sfdmgctsn004.gid.gap.com:45454]
        2015-01-06 16:42:30,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
        2015-01-06 16:42:30,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000002_0] using containerId: [container_1419291043936_1639_01_000004 on NM: [sfdmgctsn004.gid.gap.com:45454]
        2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
        2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000001_0] using containerId: [container_1419291043936_1639_01_000003 on NM: [sfdmgctsn004.gid.gap.com:45454]
        2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
        2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000000 Task Transitioned from SCHEDULED to RUNNING
        2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000003 Task Transitioned from SCHEDULED to RUNNING
        2015-01-06 16:42:30,299 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000002 Task Transitioned from SCHEDULED to RUNNING
        2015-01-06 16:42:30,299 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000001 Task Transitioned from SCHEDULED to RUNNING
        2015-01-06 16:42:31,194 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=8 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:401408, vCores:75> knownNMs=5
        2015-01-06 16:42:33,555 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE)
        2015-01-06 16:42:33,572 INFO [IPC Server handler 0 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000002 asked for a task
        2015-01-06 16:42:33,572 INFO [IPC Server handler 0 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000002 given task: attempt_1419291043936_1639_m_000000_0
        2015-01-06 16:42:33,587 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE)
        2015-01-06 16:42:33,596 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE)
        2015-01-06 16:42:33,601 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE)
        2015-01-06 16:42:33,602 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000005 asked for a task
        2015-01-06 16:42:33,602 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000005 given task: attempt_1419291043936_1639_m_000003_0
        2015-01-06 16:42:33,607 INFO [IPC Server handler 2 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000003 asked for a task
        2015-01-06 16:42:33,607 INFO [IPC Server handler 2 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000003 given task: attempt_1419291043936_1639_m_000001_0
        2015-01-06 16:42:33,612 INFO [IPC Server handler 3 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000004 asked for a task
        2015-01-06 16:42:33,612 INFO [IPC Server handler 3 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000004 given task: attempt_1419291043936_1639_m_000002_0
        2015-01-06 16:42:35,010 INFO [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000002_0 is : 0.0
        2015-01-06 16:42:35,014 FATAL [IPC Server handler 5 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1419291043936_1639_m_000002_0 - exited : java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:35,014 INFO [IPC Server handler 5 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_0: Error: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:35,016 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_0: Error: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:35,016 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
        2015-01-06 16:42:35,017 INFO ContainerLauncher #4 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000004 taskAttempt attempt_1419291043936_1639_m_000002_0
        2015-01-06 16:42:35,017 INFO ContainerLauncher #4 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000002_0
        2015-01-06 16:42:35,017 INFO ContainerLauncher #4 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454
        2015-01-06 16:42:35,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
        2015-01-06 16:42:35,030 INFO CommitterEvent Processor #1 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
        2015-01-06 16:42:35,031 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
        2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05
        2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05
        2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06
        2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05
        2015-01-06 16:42:35,035 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sfdmgctsn004.gid.gap.com
        2015-01-06 16:42:35,036 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED
        2015-01-06 16:42:35,037 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1419291043936_1639_m_000002_1 to list of failed maps
        2015-01-06 16:42:35,198 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:35,199 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:401408, vCores:75> knownNMs=5
        2015-01-06 16:42:35,697 INFO [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000001_0 is : 0.0
        2015-01-06 16:42:35,734 INFO [IPC Server handler 5 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000003_0 is : 0.0
        2015-01-06 16:42:35,833 INFO [IPC Server handler 6 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000000_0 is : 0.0
        2015-01-06 16:42:35,896 INFO [IPC Server handler 7 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000001_0 is : 1.0
        2015-01-06 16:42:35,901 INFO [IPC Server handler 8 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1419291043936_1639_m_000001_0
        2015-01-06 16:42:35,902 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP
        2015-01-06 16:42:35,903 INFO ContainerLauncher #5 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000003 taskAttempt attempt_1419291043936_1639_m_000001_0
        2015-01-06 16:42:35,903 INFO ContainerLauncher #5 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000001_0
        2015-01-06 16:42:35,903 INFO ContainerLauncher #5 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454
        2015-01-06 16:42:35,911 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED
        2015-01-06 16:42:35,912 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1419291043936_1639_m_000001_0
        2015-01-06 16:42:35,913 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000001 Task Transitioned from RUNNING to SUCCEEDED
        2015-01-06 16:42:35,913 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
        2015-01-06 16:42:35,925 INFO [IPC Server handler 9 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000003_0 is : 1.0
        2015-01-06 16:42:35,927 INFO [IPC Server handler 10 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1419291043936_1639_m_000003_0
        2015-01-06 16:42:35,928 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP
        2015-01-06 16:42:35,929 INFO ContainerLauncher #6 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000005 taskAttempt attempt_1419291043936_1639_m_000003_0
        2015-01-06 16:42:35,929 INFO ContainerLauncher #6 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000003_0
        2015-01-06 16:42:35,929 INFO ContainerLauncher #6 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454
        2015-01-06 16:42:35,936 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED
        2015-01-06 16:42:35,936 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1419291043936_1639_m_000003_0
        2015-01-06 16:42:35,936 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000003 Task Transitioned from RUNNING to SUCCEEDED
        2015-01-06 16:42:35,937 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 2
        2015-01-06 16:42:36,035 INFO [IPC Server handler 11 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000000_0 is : 1.0
        2015-01-06 16:42:36,037 INFO [IPC Server handler 12 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1419291043936_1639_m_000000_0
        2015-01-06 16:42:36,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP
        2015-01-06 16:42:36,039 INFO ContainerLauncher #7 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000002 taskAttempt attempt_1419291043936_1639_m_000000_0
        2015-01-06 16:42:36,039 INFO ContainerLauncher #7 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000000_0
        2015-01-06 16:42:36,039 INFO ContainerLauncher #7 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454
        2015-01-06 16:42:36,045 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED
        2015-01-06 16:42:36,045 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1419291043936_1639_m_000000_0
        2015-01-06 16:42:36,045 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000000 Task Transitioned from RUNNING to SUCCEEDED
        2015-01-06 16:42:36,046 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3
        2015-01-06 16:42:36,200 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:36,207 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000004
        2015-01-06 16:42:36,207 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
        2015-01-06 16:42:36,208 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_0: Container killed by the ApplicationMaster.
        Container killed on request. Exit code is 143
        Container exited with a non-zero exit code 143

        2015-01-06 16:42:36,208 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1419291043936_1639_01_000006, NodeId: sfdmgctsn003.gid.gap.com:45454, NodeHttpAddress: sfdmgctsn003.gid.gap.com:8042, Resource: <memory:14336, vCores:1>, Priority: 5, Token: Token

        { kind: ContainerToken, service: 10.9.21.133:45454 }

        , ] to fast fail map
        2015-01-06 16:42:36,208 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
        2015-01-06 16:42:36,208 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000006 to attempt_1419291043936_1639_m_000002_1
        2015-01-06 16:42:36,208 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:5 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:36,209 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05
        2015-01-06 16:42:36,209 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
        2015-01-06 16:42:36,210 INFO ContainerLauncher #8 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000006 taskAttempt attempt_1419291043936_1639_m_000002_1
        2015-01-06 16:42:36,210 INFO ContainerLauncher #8 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000002_1
        2015-01-06 16:42:36,210 INFO ContainerLauncher #8 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn003.gid.gap.com:45454
        2015-01-06 16:42:36,218 INFO ContainerLauncher #8 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000002_1 : 13562
        2015-01-06 16:42:36,219 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000002_1] using containerId: [container_1419291043936_1639_01_000006 on NM: [sfdmgctsn003.gid.gap.com:45454]
        2015-01-06 16:42:36,219 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
        2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=3 resourcelimit=<memory:444416, vCores:78> knownNMs=5
        2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000002
        2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000003
        2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000005
        2015-01-06 16:42:37,210 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000000_0: Container killed by the ApplicationMaster.
        Container killed on request. Exit code is 143
        Container exited with a non-zero exit code 143

        2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:5 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:37,210 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000001_0: Container killed by the ApplicationMaster.
        Container killed on request. Exit code is 143
        Container exited with a non-zero exit code 143

        2015-01-06 16:42:37,211 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000003_0: Container killed by the ApplicationMaster.
        Container killed on request. Exit code is 143
        Container exited with a non-zero exit code 143

        2015-01-06 16:42:39,096 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE)
        2015-01-06 16:42:39,106 INFO [IPC Server handler 0 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000006 asked for a task
        2015-01-06 16:42:39,106 INFO [IPC Server handler 0 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000006 given task: attempt_1419291043936_1639_m_000002_1
        2015-01-06 16:42:40,344 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000002_1 is : 0.0
        2015-01-06 16:42:40,345 FATAL [IPC Server handler 2 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1419291043936_1639_m_000002_1 - exited : java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:40,345 INFO [IPC Server handler 2 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_1: Error: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:40,346 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_1: Error: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:40,346 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
        2015-01-06 16:42:40,347 INFO ContainerLauncher #9 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000006 taskAttempt attempt_1419291043936_1639_m_000002_1
        2015-01-06 16:42:40,347 INFO ContainerLauncher #9 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000002_1
        2015-01-06 16:42:40,347 INFO ContainerLauncher #9 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn003.gid.gap.com:45454
        2015-01-06 16:42:40,353 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
        2015-01-06 16:42:40,354 INFO CommitterEvent Processor #2 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
        2015-01-06 16:42:40,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
        2015-01-06 16:42:40,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:40,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05
        2015-01-06 16:42:40,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05
        2015-01-06 16:42:40,355 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06
        2015-01-06 16:42:40,355 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05
        2015-01-06 16:42:40,355 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sfdmgctsn003.gid.gap.com
        2015-01-06 16:42:40,355 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from NEW to UNASSIGNED
        2015-01-06 16:42:40,355 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1419291043936_1639_m_000002_2 to list of failed maps
        2015-01-06 16:42:41,215 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:5 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:41,216 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:444416, vCores:78> knownNMs=5
        2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000006
        2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
        2015-01-06 16:42:42,218 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_1: Container killed by the ApplicationMaster.
        Container killed on request. Exit code is 143
        Container exited with a non-zero exit code 143

        2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1419291043936_1639_01_000007, NodeId: sfdmgctsn003.gid.gap.com:45454, NodeHttpAddress: sfdmgctsn003.gid.gap.com:8042, Resource: <memory:14336, vCores:1>, Priority: 5, Token: Token

        { kind: ContainerToken, service: 10.9.21.133:45454 }

        , ] to fast fail map
        2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
        2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000007 to attempt_1419291043936_1639_m_000002_2
        2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:6 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:42,218 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05
        2015-01-06 16:42:42,219 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
        2015-01-06 16:42:42,219 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000007 taskAttempt attempt_1419291043936_1639_m_000002_2
        2015-01-06 16:42:42,219 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000002_2
        2015-01-06 16:42:42,219 INFO ContainerLauncher #3 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn003.gid.gap.com:45454
        2015-01-06 16:42:42,226 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000002_2 : 13562
        2015-01-06 16:42:42,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000002_2] using containerId: [container_1419291043936_1639_01_000007 on NM: [sfdmgctsn003.gid.gap.com:45454]
        2015-01-06 16:42:42,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
        2015-01-06 16:42:43,220 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:444416, vCores:78> knownNMs=5
        2015-01-06 16:42:43,270 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE)
        2015-01-06 16:42:43,279 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000007 asked for a task
        2015-01-06 16:42:43,279 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000007 given task: attempt_1419291043936_1639_m_000002_2
        2015-01-06 16:42:44,498 INFO [IPC Server handler 3 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000002_2 is : 0.0
        2015-01-06 16:42:44,500 FATAL [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1419291043936_1639_m_000002_2 - exited : java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:44,500 INFO [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_2: Error: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:44,500 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_2: Error: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:44,501 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
        2015-01-06 16:42:44,501 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000007 taskAttempt attempt_1419291043936_1639_m_000002_2
        2015-01-06 16:42:44,501 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000002_2
        2015-01-06 16:42:44,501 INFO ContainerLauncher #1 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn003.gid.gap.com:45454
        2015-01-06 16:42:44,507 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
        2015-01-06 16:42:44,508 INFO CommitterEvent Processor #3 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
        2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
        2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05
        2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05
        2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05
        2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06
        2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05
        2015-01-06 16:42:44,509 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node sfdmgctsn003.gid.gap.com
        2015-01-06 16:42:44,509 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from NEW to UNASSIGNED
        2015-01-06 16:42:44,509 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1419291043936_1639_m_000002_3 to list of failed maps
        2015-01-06 16:42:45,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:6 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:45,223 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:444416, vCores:78> knownNMs=5
        2015-01-06 16:42:46,225 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000007
        2015-01-06 16:42:46,225 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
        2015-01-06 16:42:46,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1419291043936_1639_01_000008, NodeId: sfdmgctsn005.gid.gap.com:45454, NodeHttpAddress: sfdmgctsn005.gid.gap.com:8042, Resource: <memory:14336, vCores:1>, Priority: 5, Token: Token

        { kind: ContainerToken, service: 10.9.21.135:45454 }

        , ] to fast fail map
        2015-01-06 16:42:46,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_2: Container killed by the ApplicationMaster.
        Container killed on request. Exit code is 143
        Container exited with a non-zero exit code 143

        2015-01-06 16:42:46,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
        2015-01-06 16:42:46,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000008 to attempt_1419291043936_1639_m_000002_3
        2015-01-06 16:42:46,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:7 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:46,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06
        2015-01-06 16:42:46,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
        2015-01-06 16:42:46,227 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000008 taskAttempt attempt_1419291043936_1639_m_000002_3
        2015-01-06 16:42:46,227 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000002_3
        2015-01-06 16:42:46,227 INFO ContainerLauncher #0 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn005.gid.gap.com:45454
        2015-01-06 16:42:46,235 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000002_3 : 13562
        2015-01-06 16:42:46,235 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000002_3] using containerId: [container_1419291043936_1639_01_000008 on NM: [sfdmgctsn005.gid.gap.com:45454]
        2015-01-06 16:42:46,235 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from ASSIGNED to RUNNING
        2015-01-06 16:42:47,227 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:444416, vCores:78> knownNMs=5
        2015-01-06 16:42:49,199 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE)
        2015-01-06 16:42:49,209 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000008 asked for a task
        2015-01-06 16:42:49,209 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000008 given task: attempt_1419291043936_1639_m_000002_3
        2015-01-06 16:42:50,432 INFO [IPC Server handler 3 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000002_3 is : 0.0
        2015-01-06 16:42:50,434 FATAL [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1419291043936_1639_m_000002_3 - exited : java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:50,434 INFO [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_3: Error: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:50,434 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_3: Error: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369)
        at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228)
        at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510)
        at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        2015-01-06 16:42:50,435 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
        2015-01-06 16:42:50,435 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000008 taskAttempt attempt_1419291043936_1639_m_000002_3
        2015-01-06 16:42:50,435 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000002_3
        2015-01-06 16:42:50,435 INFO ContainerLauncher #2 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn005.gid.gap.com:45454
        2015-01-06 16:42:50,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
        2015-01-06 16:42:50,441 INFO CommitterEvent Processor #4 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
        2015-01-06 16:42:50,442 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
        2015-01-06 16:42:50,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000002 Task Transitioned from RUNNING to FAILED
        2015-01-06 16:42:50,443 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sfdmgctsn005.gid.gap.com
        2015-01-06 16:42:50,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 4
        2015-01-06 16:42:50,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0
        2015-01-06 16:42:50,444 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from RUNNING to FAIL_ABORT
        2015-01-06 16:42:50,444 INFO CommitterEvent Processor #0 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
        2015-01-06 16:42:50,455 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from FAIL_ABORT to FAILED
        2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
        2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
        2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
        2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
        2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
        2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
        2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
        2015-01-06 16:42:50,485 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://sfdmgct:8020/user/hive/.staging/job_1419291043936_1639/job_1419291043936_1639_1.jhist to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639-1420580542797-hive-sfdmgctmn003.gid.gap.com%2D32%2Dcompactor%2Dds_infra.eve-1420580570443-3-0-FAILED-default-1420580548162.jhist_tmp
        2015-01-06 16:42:50,504 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639-1420580542797-hive-sfdmgctmn003.gid.gap.com%2D32%2Dcompactor%2Dds_infra.eve-1420580570443-3-0-FAILED-default-1420580548162.jhist_tmp
        2015-01-06 16:42:50,507 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://sfdmgct:8020/user/hive/.staging/job_1419291043936_1639/job_1419291043936_1639_1_conf.xml to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639_conf.xml_tmp
        2015-01-06 16:42:50,524 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639_conf.xml_tmp
        2015-01-06 16:42:50,530 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639.summary_tmp to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639.summary
        2015-01-06 16:42:50,531 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639_conf.xml_tmp to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639_conf.xml
        2015-01-06 16:42:50,533 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639-1420580542797-hive-sfdmgctmn003.gid.gap.com%2D32%2Dcompactor%2Dds_infra.eve-1420580570443-3-0-FAILED-default-1420580548162.jhist_tmp to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639-1420580542797-hive-sfdmgctmn003.gid.gap.com%2D32%2Dcompactor%2Dds_infra.eve-1420580570443-3-0-FAILED-default-1420580548162.jhist
        2015-01-06 16:42:50,533 INFO [Thread-79] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
        2015-01-06 16:42:50,535 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Task failed task_1419291043936_1639_m_000002
        Job failed as tasks failed. failedMaps:1 failedReduces:0

        2015-01-06 16:42:50,536 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://sfdmgctmn004.gid.gap.com:19888/jobhistory/job/job_1419291043936_1639
        2015-01-06 16:42:50,539 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
        2015-01-06 16:42:51,540 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:7 ContRel:0 HostLocal:4 RackLocal:0
        2015-01-06 16:42:51,541 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://sfdmgct /user/hive/.staging/job_1419291043936_1639
        2015-01-06 16:42:51,543 INFO [Thread-79] org.apache.hadoop.ipc.Server: Stopping server on 33406
        2015-01-06 16:42:51,544 INFO [IPC Server listener on 33406] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 33406
        2015-01-06 16:42:51,545 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
        2015-01-06 16:42:51,545 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder

        Show
        jihongliu Jihong Liu added a comment - The error occur when doing the mapreduce job. Following is log in hivemetastore.log 2015-01-06 16:42:22,506 INFO [sfdmgctmn003.gid.gap.com-32] : compactor.Worker (Worker.java:run(137)) - Starting MAJOR compaction for ds_infra.event_metrics.date=2014-12-24 2015-01-06 16:42:22,564 INFO [sfdmgctmn003.gid.gap.com-32] : impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(285)) - Timeline service address: http://sfdmgctmn003.gid.gap.com:8188/ws/v1/timeline/ 2015-01-06 16:42:22,622 INFO [sfdmgctmn003.gid.gap.com-32] : impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(285)) - Timeline service address: http://sfdmgctmn003.gid.gap.com:8188/ws/v1/timeline/ 2015-01-06 16:42:22,628 WARN [sfdmgctmn003.gid.gap.com-32] : mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(153)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2015-01-06 16:42:22,753 WARN [sfdmgctmn003.gid.gap.com-32] : split.JobSplitWriter (JobSplitWriter.java:writeOldSplits(168)) - Max block location exceeded for split: CompactorInputSplit {base: hdfs://sfdmgct/apps/hive/warehouse/ds_infra/event_metrics/date=2014-12-24/base_0035304, bucket: 1, length: 292280, deltas: [delta_0035311_0035313, delta_0035479_0035481, delta_0035491_0035493, delta_0035515_0035517, delta_0035533_0035535, delta_0035548_0035550, delta_0035563_0035565, delta_0035578_0035580, delta_0035593_0035595, delta_0035599_0035601, delta_0035656_0035658, delta_0035671_0035673, delta_0035686_0035688, delta_0035701_0035703, delta_0035716_0035718, delta_0035731_0035733, delta_0035746_0035748, delta_0035761_0035763, delta_0035776_0035778, delta_0035791_0035793, delta_0035806_0035808, delta_0035821_0035823, delta_0035830_0035832, delta_0035842_0035844, delta_0035854_0035856, delta_0035866_0035868, delta_0035878_0035880]} splitsize: 27 maxsize: 10 2015-01-06 16:42:22,753 WARN [sfdmgctmn003.gid.gap.com-32] : split.JobSplitWriter (JobSplitWriter.java:writeOldSplits(168)) - Max block location exceeded for split: CompactorInputSplit {base: null, bucket: 3, length: 199770, deltas: [delta_0035311_0035313, delta_0035479_0035481, delta_0035491_0035493, delta_0035515_0035517, delta_0035533_0035535, delta_0035548_0035550, delta_0035563_0035565, delta_0035578_0035580, delta_0035593_0035595, delta_0035599_0035601, delta_0035656_0035658, delta_0035671_0035673, delta_0035686_0035688, delta_0035701_0035703, delta_0035716_0035718, delta_0035731_0035733, delta_0035746_0035748, delta_0035761_0035763, delta_0035776_0035778, delta_0035791_0035793, delta_0035806_0035808, delta_0035821_0035823, delta_0035830_0035832, delta_0035842_0035844, delta_0035854_0035856, delta_0035866_0035868, delta_0035878_0035880]} splitsize: 21 maxsize: 10 2015-01-06 16:42:22,753 WARN [sfdmgctmn003.gid.gap.com-32] : split.JobSplitWriter (JobSplitWriter.java:writeOldSplits(168)) - Max block location exceeded for split: CompactorInputSplit {base: hdfs://sfdmgct/apps/hive/warehouse/ds_infra/event_metrics/date=2014-12-24/base_0035304, bucket: 0, length: 172391, deltas: [delta_0035311_0035313, delta_0035479_0035481, delta_0035491_0035493, delta_0035515_0035517, delta_0035533_0035535, delta_0035548_0035550, delta_0035563_0035565, delta_0035578_0035580, delta_0035593_0035595, delta_0035599_0035601, delta_0035656_0035658, delta_0035671_0035673, delta_0035686_0035688, delta_0035701_0035703, delta_0035716_0035718, delta_0035731_0035733, delta_0035746_0035748, delta_0035761_0035763, delta_0035776_0035778, delta_0035791_0035793, delta_0035806_0035808, delta_0035821_0035823, delta_0035830_0035832, delta_0035842_0035844, delta_0035854_0035856, delta_0035866_0035868, delta_0035878_0035880]} splitsize: 30 maxsize: 10 2015-01-06 16:42:22,777 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:4 2015-01-06 16:42:22,793 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_1419291043936_1639 2015-01-06 16:42:23,000 INFO [sfdmgctmn003.gid.gap.com-32] : impl.YarnClientImpl (YarnClientImpl.java:submitApplication(251)) - Submitted application application_1419291043936_1639 2015-01-06 16:42:23,001 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://sfdmgctmn002.gid.gap.com:8088/proxy/application_1419291043936_1639/ 2015-01-06 16:42:23,001 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:monitorAndPrintJob(1345)) - Running job: job_1419291043936_1639 2015-01-06 16:42:30,042 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:monitorAndPrintJob(1366)) - Job job_1419291043936_1639 running in uber mode : false 2015-01-06 16:42:30,043 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:monitorAndPrintJob(1373)) - map 0% reduce 0% 2015-01-06 16:42:35,066 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:printTaskEvents(1452)) - Task Id : attempt_1419291043936_1639_m_000002_0, Status : FAILED 2015-01-06 16:42:37,078 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:monitorAndPrintJob(1373)) - map 75% reduce 0% 2015-01-06 16:42:41,091 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:printTaskEvents(1452)) - Task Id : attempt_1419291043936_1639_m_000002_1, Status : FAILED 2015-01-06 16:42:45,105 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:printTaskEvents(1452)) - Task Id : attempt_1419291043936_1639_m_000002_2, Status : FAILED 2015-01-06 16:42:52,124 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:monitorAndPrintJob(1373)) - map 100% reduce 0% 2015-01-06 16:42:52,130 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:monitorAndPrintJob(1386)) - Job job_1419291043936_1639 failed with state FAILED due to: Task failed task_1419291043936_1639_m_000002 Job failed as tasks failed. failedMaps:1 failedReduces:0 2015-01-06 16:42:52,149 INFO [sfdmgctmn003.gid.gap.com-32] : mapreduce.Job (Job.java:monitorAndPrintJob(1391)) - Counters: 32 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=668781 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=840325 HDFS: Number of bytes written=405818 HDFS: Number of read operations=243 HDFS: Number of large read operations=0 HDFS: Number of write operations=3 Job Counters Failed map tasks=4 Launched map tasks=7 Other local map tasks=3 Data-local map tasks=4 Total time spent by all maps in occupied slots (ms)=32359 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=32359 Total vcore-seconds taken by all map tasks=32359 Total megabyte-seconds taken by all map tasks=463898624 Map-Reduce Framework Map input records=3 Map output records=0 Input split bytes=10663 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=153 CPU time spent (ms)=7680 Physical memory (bytes) snapshot=1065402368 Virtual memory (bytes) snapshot=39759937536 Total committed heap usage (bytes)=6714032128 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=0 2015-01-06 16:42:52,150 ERROR [sfdmgctmn003.gid.gap.com-32] : compactor.Worker (Worker.java:run(159)) - Caught exception while trying to compact ds_infra.event_metrics.date=2014-12-24. Marking clean to avoid repeated failures, java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:184) at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:145) 2015-01-06 16:42:52,150 ERROR [sfdmgctmn003.gid.gap.com-32] : txn.CompactionTxnHandler (CompactionTxnHandler.java:markCleaned(327)) - Expected to remove at least one row from completed_txn_components when marking compaction entry as clean! -------------------------------------------------------------------------------------------------------------------------------- following is mapreduce job log. Got the log from uri mentioned in the above log: ( http://sfdmgctmn002.gid.gap.com:8088/proxy/application_1419291043936_1639/ ) 2015-01-06 16:42:25,991 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1419291043936_1639_000001 2015-01-06 16:42:26,304 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2015-01-06 16:42:26,314 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2015-01-06 16:42:26,314 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1639 cluster_timestamp: 1419291043936 } attemptId: 1 } keyId: -950898635) 2015-01-06 16:42:26,842 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2015-01-06 16:42:26,919 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter 2015-01-06 16:42:26,921 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter 2015-01-06 16:42:26,951 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2015-01-06 16:42:26,951 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2015-01-06 16:42:26,952 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2015-01-06 16:42:26,952 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2015-01-06 16:42:26,952 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2015-01-06 16:42:26,956 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2015-01-06 16:42:26,956 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2015-01-06 16:42:26,957 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2015-01-06 16:42:26,975 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring 2015-01-06 16:42:26,987 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring 2015-01-06 16:42:26,999 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring 2015-01-06 16:42:27,023 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 2015-01-06 16:42:27,050 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2015-01-06 16:42:27,191 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties 2015-01-06 16:42:27,232 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-01-06 16:42:27,232 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 2015-01-06 16:42:27,239 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1419291043936_1639 to jobTokenSecretManager 2015-01-06 16:42:27,327 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1419291043936_1639 because: not enabled; 2015-01-06 16:42:27,340 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1419291043936_1639 = 845156. Number of splits = 4 2015-01-06 16:42:27,341 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1419291043936_1639 = 0 2015-01-06 16:42:27,341 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from NEW to INITED 2015-01-06 16:42:27,341 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1419291043936_1639. 2015-01-06 16:42:27,359 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2015-01-06 16:42:27,366 INFO Socket Reader #1 for port 57524 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 57524 2015-01-06 16:42:27,379 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 2015-01-06 16:42:27,380 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2015-01-06 16:42:27,381 INFO [IPC Server listener on 57524] org.apache.hadoop.ipc.Server: IPC Server listener on 57524: starting 2015-01-06 16:42:27,381 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at sfdmgctsn004.gid.gap.com/10.9.21.134:57524 2015-01-06 16:42:27,429 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2015-01-06 16:42:27,431 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2015-01-06 16:42:27,439 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2015-01-06 16:42:27,472 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2015-01-06 16:42:27,472 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2015-01-06 16:42:27,475 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2015-01-06 16:42:27,475 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2015-01-06 16:42:27,482 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 45674 2015-01-06 16:42:27,482 INFO [main] org.mortbay.log: jetty-6.1.26.hwx 2015-01-06 16:42:27,505 INFO [main] org.mortbay.log: Extract jar: file:/data/sfdmgct05/hadoop/yarn/local/filecache/12/mapreduce.tar.gz/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0.2.2.0.0-2041.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_45674_mapreduce____.ycambd/webapp 2015-01-06 16:42:27,805 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:45674 2015-01-06 16:42:27,806 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 45674 2015-01-06 16:42:28,026 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2015-01-06 16:42:28,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1419291043936_1639 2015-01-06 16:42:28,030 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2015-01-06 16:42:28,030 INFO Socket Reader #1 for port 33406 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33406 2015-01-06 16:42:28,034 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2015-01-06 16:42:28,034 INFO [IPC Server listener on 33406] org.apache.hadoop.ipc.Server: IPC Server listener on 33406: starting 2015-01-06 16:42:28,074 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2015-01-06 16:42:28,075 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2015-01-06 16:42:28,075 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2015-01-06 16:42:28,158 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:186368, vCores:32> 2015-01-06 16:42:28,158 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: default 2015-01-06 16:42:28,160 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2015-01-06 16:42:28,161 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 2015-01-06 16:42:28,165 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from INITED to SETUP 2015-01-06 16:42:28,167 INFO CommitterEvent Processor #0 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 2015-01-06 16:42:28,168 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from SETUP to RUNNING 2015-01-06 16:42:28,182 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,186 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,188 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,191 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06 2015-01-06 16:42:28,194 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000000 Task Transitioned from NEW to SCHEDULED 2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06 2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,198 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000001 Task Transitioned from NEW to SCHEDULED 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000002 Task Transitioned from NEW to SCHEDULED 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05 2015-01-06 16:42:28,199 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000003 Task Transitioned from NEW to SCHEDULED 2015-01-06 16:42:28,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2015-01-06 16:42:28,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2015-01-06 16:42:28,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2015-01-06 16:42:28,200 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2015-01-06 16:42:28,201 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:14336, vCores:1> 2015-01-06 16:42:28,226 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1419291043936_1639, File: hdfs://sfdmgct:8020/user/hive/.staging/job_1419291043936_1639/job_1419291043936_1639_1.jhist 2015-01-06 16:42:29,160 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2015-01-06 16:42:29,181 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=8 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:458752, vCores:79> knownNMs=5 2015-01-06 16:42:30,189 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4 2015-01-06 16:42:30,190 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000002 to attempt_1419291043936_1639_m_000000_0 2015-01-06 16:42:30,191 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000003 to attempt_1419291043936_1639_m_000001_0 2015-01-06 16:42:30,191 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000004 to attempt_1419291043936_1639_m_000002_0 2015-01-06 16:42:30,191 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000005 to attempt_1419291043936_1639_m_000003_0 2015-01-06 16:42:30,191 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:30,217 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:30,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://sfdmgct/user/hive/.staging/job_1419291043936_1639/job.jar 2015-01-06 16:42:30,229 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hive/.staging/job_1419291043936_1639/job.xml 2015-01-06 16:42:30,231 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container 2015-01-06 16:42:30,231 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1 2015-01-06 16:42:30,231 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2015-01-06 16:42:30,247 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2015-01-06 16:42:30,248 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:30,248 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2015-01-06 16:42:30,249 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:30,249 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2015-01-06 16:42:30,249 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:30,250 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2015-01-06 16:42:30,251 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000002 taskAttempt attempt_1419291043936_1639_m_000000_0 2015-01-06 16:42:30,251 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000003 taskAttempt attempt_1419291043936_1639_m_000001_0 2015-01-06 16:42:30,251 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000004 taskAttempt attempt_1419291043936_1639_m_000002_0 2015-01-06 16:42:30,251 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000005 taskAttempt attempt_1419291043936_1639_m_000003_0 2015-01-06 16:42:30,252 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000002_0 2015-01-06 16:42:30,252 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000001_0 2015-01-06 16:42:30,252 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000003_0 2015-01-06 16:42:30,252 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000000_0 2015-01-06 16:42:30,253 INFO ContainerLauncher #2 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454 2015-01-06 16:42:30,266 INFO ContainerLauncher #0 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454 2015-01-06 16:42:30,267 INFO ContainerLauncher #3 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454 2015-01-06 16:42:30,267 INFO ContainerLauncher #1 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454 2015-01-06 16:42:30,294 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000002_0 : 13562 2015-01-06 16:42:30,294 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000003_0 : 13562 2015-01-06 16:42:30,294 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000000_0 : 13562 2015-01-06 16:42:30,294 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000001_0 : 13562 2015-01-06 16:42:30,295 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000000_0] using containerId: [container_1419291043936_1639_01_000002 on NM: [sfdmgctsn004.gid.gap.com:45454] 2015-01-06 16:42:30,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2015-01-06 16:42:30,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000003_0] using containerId: [container_1419291043936_1639_01_000005 on NM: [sfdmgctsn004.gid.gap.com:45454] 2015-01-06 16:42:30,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2015-01-06 16:42:30,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000002_0] using containerId: [container_1419291043936_1639_01_000004 on NM: [sfdmgctsn004.gid.gap.com:45454] 2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000001_0] using containerId: [container_1419291043936_1639_01_000003 on NM: [sfdmgctsn004.gid.gap.com:45454] 2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000000 Task Transitioned from SCHEDULED to RUNNING 2015-01-06 16:42:30,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000003 Task Transitioned from SCHEDULED to RUNNING 2015-01-06 16:42:30,299 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000002 Task Transitioned from SCHEDULED to RUNNING 2015-01-06 16:42:30,299 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000001 Task Transitioned from SCHEDULED to RUNNING 2015-01-06 16:42:31,194 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=8 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:401408, vCores:75> knownNMs=5 2015-01-06 16:42:33,555 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE) 2015-01-06 16:42:33,572 INFO [IPC Server handler 0 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000002 asked for a task 2015-01-06 16:42:33,572 INFO [IPC Server handler 0 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000002 given task: attempt_1419291043936_1639_m_000000_0 2015-01-06 16:42:33,587 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE) 2015-01-06 16:42:33,596 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE) 2015-01-06 16:42:33,601 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE) 2015-01-06 16:42:33,602 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000005 asked for a task 2015-01-06 16:42:33,602 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000005 given task: attempt_1419291043936_1639_m_000003_0 2015-01-06 16:42:33,607 INFO [IPC Server handler 2 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000003 asked for a task 2015-01-06 16:42:33,607 INFO [IPC Server handler 2 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000003 given task: attempt_1419291043936_1639_m_000001_0 2015-01-06 16:42:33,612 INFO [IPC Server handler 3 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000004 asked for a task 2015-01-06 16:42:33,612 INFO [IPC Server handler 3 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000004 given task: attempt_1419291043936_1639_m_000002_0 2015-01-06 16:42:35,010 INFO [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000002_0 is : 0.0 2015-01-06 16:42:35,014 FATAL [IPC Server handler 5 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1419291043936_1639_m_000002_0 - exited : java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:35,014 INFO [IPC Server handler 5 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_0: Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:35,016 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_0: Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:35,016 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP 2015-01-06 16:42:35,017 INFO ContainerLauncher #4 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000004 taskAttempt attempt_1419291043936_1639_m_000002_0 2015-01-06 16:42:35,017 INFO ContainerLauncher #4 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000002_0 2015-01-06 16:42:35,017 INFO ContainerLauncher #4 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454 2015-01-06 16:42:35,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP 2015-01-06 16:42:35,030 INFO CommitterEvent Processor #1 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2015-01-06 16:42:35,031 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED 2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05 2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05 2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06 2015-01-06 16:42:35,035 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05 2015-01-06 16:42:35,035 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sfdmgctsn004.gid.gap.com 2015-01-06 16:42:35,036 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2015-01-06 16:42:35,037 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1419291043936_1639_m_000002_1 to list of failed maps 2015-01-06 16:42:35,198 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:35,199 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:401408, vCores:75> knownNMs=5 2015-01-06 16:42:35,697 INFO [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000001_0 is : 0.0 2015-01-06 16:42:35,734 INFO [IPC Server handler 5 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000003_0 is : 0.0 2015-01-06 16:42:35,833 INFO [IPC Server handler 6 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000000_0 is : 0.0 2015-01-06 16:42:35,896 INFO [IPC Server handler 7 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000001_0 is : 1.0 2015-01-06 16:42:35,901 INFO [IPC Server handler 8 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1419291043936_1639_m_000001_0 2015-01-06 16:42:35,902 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2015-01-06 16:42:35,903 INFO ContainerLauncher #5 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000003 taskAttempt attempt_1419291043936_1639_m_000001_0 2015-01-06 16:42:35,903 INFO ContainerLauncher #5 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000001_0 2015-01-06 16:42:35,903 INFO ContainerLauncher #5 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454 2015-01-06 16:42:35,911 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000001_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-01-06 16:42:35,912 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1419291043936_1639_m_000001_0 2015-01-06 16:42:35,913 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000001 Task Transitioned from RUNNING to SUCCEEDED 2015-01-06 16:42:35,913 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1 2015-01-06 16:42:35,925 INFO [IPC Server handler 9 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000003_0 is : 1.0 2015-01-06 16:42:35,927 INFO [IPC Server handler 10 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1419291043936_1639_m_000003_0 2015-01-06 16:42:35,928 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2015-01-06 16:42:35,929 INFO ContainerLauncher #6 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000005 taskAttempt attempt_1419291043936_1639_m_000003_0 2015-01-06 16:42:35,929 INFO ContainerLauncher #6 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000003_0 2015-01-06 16:42:35,929 INFO ContainerLauncher #6 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454 2015-01-06 16:42:35,936 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000003_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-01-06 16:42:35,936 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1419291043936_1639_m_000003_0 2015-01-06 16:42:35,936 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000003 Task Transitioned from RUNNING to SUCCEEDED 2015-01-06 16:42:35,937 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 2 2015-01-06 16:42:36,035 INFO [IPC Server handler 11 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000000_0 is : 1.0 2015-01-06 16:42:36,037 INFO [IPC Server handler 12 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1419291043936_1639_m_000000_0 2015-01-06 16:42:36,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_CONTAINER_CLEANUP 2015-01-06 16:42:36,039 INFO ContainerLauncher #7 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000002 taskAttempt attempt_1419291043936_1639_m_000000_0 2015-01-06 16:42:36,039 INFO ContainerLauncher #7 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000000_0 2015-01-06 16:42:36,039 INFO ContainerLauncher #7 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn004.gid.gap.com:45454 2015-01-06 16:42:36,045 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000000_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-01-06 16:42:36,045 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1419291043936_1639_m_000000_0 2015-01-06 16:42:36,045 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000000 Task Transitioned from RUNNING to SUCCEEDED 2015-01-06 16:42:36,046 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3 2015-01-06 16:42:36,200 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:36,207 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000004 2015-01-06 16:42:36,207 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2015-01-06 16:42:36,208 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-01-06 16:42:36,208 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1419291043936_1639_01_000006, NodeId: sfdmgctsn003.gid.gap.com:45454, NodeHttpAddress: sfdmgctsn003.gid.gap.com:8042, Resource: <memory:14336, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.9.21.133:45454 } , ] to fast fail map 2015-01-06 16:42:36,208 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 2015-01-06 16:42:36,208 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000006 to attempt_1419291043936_1639_m_000002_1 2015-01-06 16:42:36,208 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:5 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:36,209 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05 2015-01-06 16:42:36,209 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2015-01-06 16:42:36,210 INFO ContainerLauncher #8 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000006 taskAttempt attempt_1419291043936_1639_m_000002_1 2015-01-06 16:42:36,210 INFO ContainerLauncher #8 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000002_1 2015-01-06 16:42:36,210 INFO ContainerLauncher #8 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn003.gid.gap.com:45454 2015-01-06 16:42:36,218 INFO ContainerLauncher #8 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000002_1 : 13562 2015-01-06 16:42:36,219 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000002_1] using containerId: [container_1419291043936_1639_01_000006 on NM: [sfdmgctsn003.gid.gap.com:45454] 2015-01-06 16:42:36,219 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from ASSIGNED to RUNNING 2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=3 resourcelimit=<memory:444416, vCores:78> knownNMs=5 2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000002 2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000003 2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000005 2015-01-06 16:42:37,210 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000000_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-01-06 16:42:37,210 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:5 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:37,210 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000001_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-01-06 16:42:37,211 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000003_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-01-06 16:42:39,096 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE) 2015-01-06 16:42:39,106 INFO [IPC Server handler 0 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000006 asked for a task 2015-01-06 16:42:39,106 INFO [IPC Server handler 0 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000006 given task: attempt_1419291043936_1639_m_000002_1 2015-01-06 16:42:40,344 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000002_1 is : 0.0 2015-01-06 16:42:40,345 FATAL [IPC Server handler 2 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1419291043936_1639_m_000002_1 - exited : java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:40,345 INFO [IPC Server handler 2 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_1: Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:40,346 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_1: Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:40,346 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP 2015-01-06 16:42:40,347 INFO ContainerLauncher #9 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000006 taskAttempt attempt_1419291043936_1639_m_000002_1 2015-01-06 16:42:40,347 INFO ContainerLauncher #9 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000002_1 2015-01-06 16:42:40,347 INFO ContainerLauncher #9 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn003.gid.gap.com:45454 2015-01-06 16:42:40,353 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP 2015-01-06 16:42:40,354 INFO CommitterEvent Processor #2 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2015-01-06 16:42:40,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_1 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED 2015-01-06 16:42:40,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:40,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05 2015-01-06 16:42:40,354 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05 2015-01-06 16:42:40,355 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06 2015-01-06 16:42:40,355 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05 2015-01-06 16:42:40,355 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sfdmgctsn003.gid.gap.com 2015-01-06 16:42:40,355 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from NEW to UNASSIGNED 2015-01-06 16:42:40,355 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1419291043936_1639_m_000002_2 to list of failed maps 2015-01-06 16:42:41,215 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:5 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:41,216 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:444416, vCores:78> knownNMs=5 2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000006 2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2015-01-06 16:42:42,218 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_1: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1419291043936_1639_01_000007, NodeId: sfdmgctsn003.gid.gap.com:45454, NodeHttpAddress: sfdmgctsn003.gid.gap.com:8042, Resource: <memory:14336, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.9.21.133:45454 } , ] to fast fail map 2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000007 to attempt_1419291043936_1639_m_000002_2 2015-01-06 16:42:42,218 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:6 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:42,218 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05 2015-01-06 16:42:42,219 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2015-01-06 16:42:42,219 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000007 taskAttempt attempt_1419291043936_1639_m_000002_2 2015-01-06 16:42:42,219 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000002_2 2015-01-06 16:42:42,219 INFO ContainerLauncher #3 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn003.gid.gap.com:45454 2015-01-06 16:42:42,226 INFO ContainerLauncher #3 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000002_2 : 13562 2015-01-06 16:42:42,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000002_2] using containerId: [container_1419291043936_1639_01_000007 on NM: [sfdmgctsn003.gid.gap.com:45454] 2015-01-06 16:42:42,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from ASSIGNED to RUNNING 2015-01-06 16:42:43,220 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:444416, vCores:78> knownNMs=5 2015-01-06 16:42:43,270 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE) 2015-01-06 16:42:43,279 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000007 asked for a task 2015-01-06 16:42:43,279 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000007 given task: attempt_1419291043936_1639_m_000002_2 2015-01-06 16:42:44,498 INFO [IPC Server handler 3 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000002_2 is : 0.0 2015-01-06 16:42:44,500 FATAL [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1419291043936_1639_m_000002_2 - exited : java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:44,500 INFO [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_2: Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:44,500 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_2: Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:44,501 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP 2015-01-06 16:42:44,501 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000007 taskAttempt attempt_1419291043936_1639_m_000002_2 2015-01-06 16:42:44,501 INFO ContainerLauncher #1 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000002_2 2015-01-06 16:42:44,501 INFO ContainerLauncher #1 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn003.gid.gap.com:45454 2015-01-06 16:42:44,507 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP 2015-01-06 16:42:44,508 INFO CommitterEvent Processor #3 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_2 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED 2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn004.gid.gap.com to /default/rack_05 2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn003.gid.gap.com to /default/rack_05 2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn001.gid.gap.com to /default/rack_05 2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06 2015-01-06 16:42:44,508 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn002.gid.gap.com to /default/rack_05 2015-01-06 16:42:44,509 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node sfdmgctsn003.gid.gap.com 2015-01-06 16:42:44,509 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from NEW to UNASSIGNED 2015-01-06 16:42:44,509 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1419291043936_1639_m_000002_3 to list of failed maps 2015-01-06 16:42:45,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:6 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:45,223 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:444416, vCores:78> knownNMs=5 2015-01-06 16:42:46,225 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1419291043936_1639_01_000007 2015-01-06 16:42:46,225 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2015-01-06 16:42:46,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1419291043936_1639_01_000008, NodeId: sfdmgctsn005.gid.gap.com:45454, NodeHttpAddress: sfdmgctsn005.gid.gap.com:8042, Resource: <memory:14336, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.9.21.135:45454 } , ] to fast fail map 2015-01-06 16:42:46,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_2: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-01-06 16:42:46,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 2015-01-06 16:42:46,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1419291043936_1639_01_000008 to attempt_1419291043936_1639_m_000002_3 2015-01-06 16:42:46,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:7 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:46,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sfdmgctsn005.gid.gap.com to /default/rack_06 2015-01-06 16:42:46,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2015-01-06 16:42:46,227 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1419291043936_1639_01_000008 taskAttempt attempt_1419291043936_1639_m_000002_3 2015-01-06 16:42:46,227 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1419291043936_1639_m_000002_3 2015-01-06 16:42:46,227 INFO ContainerLauncher #0 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn005.gid.gap.com:45454 2015-01-06 16:42:46,235 INFO ContainerLauncher #0 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1419291043936_1639_m_000002_3 : 13562 2015-01-06 16:42:46,235 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1419291043936_1639_m_000002_3] using containerId: [container_1419291043936_1639_01_000008 on NM: [sfdmgctsn005.gid.gap.com:45454] 2015-01-06 16:42:46,235 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from ASSIGNED to RUNNING 2015-01-06 16:42:47,227 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1419291043936_1639: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:444416, vCores:78> knownNMs=5 2015-01-06 16:42:49,199 INFO Socket Reader #1 for port 33406 SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1419291043936_1639 (auth:SIMPLE) 2015-01-06 16:42:49,209 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1419291043936_1639_m_000008 asked for a task 2015-01-06 16:42:49,209 INFO [IPC Server handler 1 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1419291043936_1639_m_000008 given task: attempt_1419291043936_1639_m_000002_3 2015-01-06 16:42:50,432 INFO [IPC Server handler 3 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1419291043936_1639_m_000002_3 is : 0.0 2015-01-06 16:42:50,434 FATAL [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1419291043936_1639_m_000002_3 - exited : java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:50,434 INFO [IPC Server handler 4 on 33406] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_3: Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:50,434 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1419291043936_1639_m_000002_3: Error: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:532) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:369) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:311) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:228) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:464) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1232) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:510) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:489) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2015-01-06 16:42:50,435 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP 2015-01-06 16:42:50,435 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1419291043936_1639_01_000008 taskAttempt attempt_1419291043936_1639_m_000002_3 2015-01-06 16:42:50,435 INFO ContainerLauncher #2 org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1419291043936_1639_m_000002_3 2015-01-06 16:42:50,435 INFO ContainerLauncher #2 org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sfdmgctsn005.gid.gap.com:45454 2015-01-06 16:42:50,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP 2015-01-06 16:42:50,441 INFO CommitterEvent Processor #4 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2015-01-06 16:42:50,442 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1419291043936_1639_m_000002_3 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED 2015-01-06 16:42:50,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1419291043936_1639_m_000002 Task Transitioned from RUNNING to FAILED 2015-01-06 16:42:50,443 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sfdmgctsn005.gid.gap.com 2015-01-06 16:42:50,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 4 2015-01-06 16:42:50,443 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0 2015-01-06 16:42:50,444 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from RUNNING to FAIL_ABORT 2015-01-06 16:42:50,444 INFO CommitterEvent Processor #0 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT 2015-01-06 16:42:50,455 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1419291043936_1639Job Transitioned from FAIL_ABORT to FAILED 2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry 2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true 2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 2015-01-06 16:42:50,456 INFO [Thread-79] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0 2015-01-06 16:42:50,485 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://sfdmgct:8020/user/hive/.staging/job_1419291043936_1639/job_1419291043936_1639_1.jhist to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639-1420580542797-hive-sfdmgctmn003.gid.gap.com%2D32%2Dcompactor%2Dds_infra.eve-1420580570443-3-0-FAILED-default-1420580548162.jhist_tmp 2015-01-06 16:42:50,504 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639-1420580542797-hive-sfdmgctmn003.gid.gap.com%2D32%2Dcompactor%2Dds_infra.eve-1420580570443-3-0-FAILED-default-1420580548162.jhist_tmp 2015-01-06 16:42:50,507 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://sfdmgct:8020/user/hive/.staging/job_1419291043936_1639/job_1419291043936_1639_1_conf.xml to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639_conf.xml_tmp 2015-01-06 16:42:50,524 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639_conf.xml_tmp 2015-01-06 16:42:50,530 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639.summary_tmp to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639.summary 2015-01-06 16:42:50,531 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639_conf.xml_tmp to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639_conf.xml 2015-01-06 16:42:50,533 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639-1420580542797-hive-sfdmgctmn003.gid.gap.com%2D32%2Dcompactor%2Dds_infra.eve-1420580570443-3-0-FAILED-default-1420580548162.jhist_tmp to hdfs://sfdmgct:8020/mr-history/tmp/hive/job_1419291043936_1639-1420580542797-hive-sfdmgctmn003.gid.gap.com%2D32%2Dcompactor%2Dds_infra.eve-1420580570443-3-0-FAILED-default-1420580548162.jhist 2015-01-06 16:42:50,533 INFO [Thread-79] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2015-01-06 16:42:50,535 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Task failed task_1419291043936_1639_m_000002 Job failed as tasks failed. failedMaps:1 failedReduces:0 2015-01-06 16:42:50,536 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://sfdmgctmn004.gid.gap.com:19888/jobhistory/job/job_1419291043936_1639 2015-01-06 16:42:50,539 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered. 2015-01-06 16:42:51,540 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3 CompletedReds:0 ContAlloc:7 ContRel:0 HostLocal:4 RackLocal:0 2015-01-06 16:42:51,541 INFO [Thread-79] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://sfdmgct /user/hive/.staging/job_1419291043936_1639 2015-01-06 16:42:51,543 INFO [Thread-79] org.apache.hadoop.ipc.Server: Stopping server on 33406 2015-01-06 16:42:51,544 INFO [IPC Server listener on 33406] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 33406 2015-01-06 16:42:51,545 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 2015-01-06 16:42:51,545 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
        Hide
        alangates Alan Gates added a comment -

        The issue is that since the writer died with an unclosed batch it left the orc file in a state where it cannot be read without the length file. So removing the length file means any reader will fail when reading it.

        The proper solution is for the compactor to stop at that partition until it has determined all transactions in that file have committed or aborted. Then it should compact it using the length file, but properly ignore the length file. I'll work on the fix.

        Show
        alangates Alan Gates added a comment - The issue is that since the writer died with an unclosed batch it left the orc file in a state where it cannot be read without the length file. So removing the length file means any reader will fail when reading it. The proper solution is for the compactor to stop at that partition until it has determined all transactions in that file have committed or aborted. Then it should compact it using the length file, but properly ignore the length file. I'll work on the fix.
        Hide
        jihongliu Jihong Liu added a comment -

        Make sense. It is so great if that solution can be implemented.Thanks

        Show
        jihongliu Jihong Liu added a comment - Make sense. It is so great if that solution can be implemented.Thanks
        Hide
        alangates Alan Gates added a comment -

        This patch takes a new approach. Rather than changing AcidUtils.getAcidState (as previous 2 attempts) this patch gives a new implementation of ValidTxnList that only returns isTxnRangeValid ALL or NONE, and gives NONE if there are any open transactions <= the max transaction in the range (even if it's below the range). This new implementation is used only by the compactor so that it's understanding of what files it should compact are different than what files a reader views as available for reading.

        I've also added tests to TestCompactor to test compaction during streaming and compaction after a streamer has aborted and died without cleaning up.

        Show
        alangates Alan Gates added a comment - This patch takes a new approach. Rather than changing AcidUtils.getAcidState (as previous 2 attempts) this patch gives a new implementation of ValidTxnList that only returns isTxnRangeValid ALL or NONE, and gives NONE if there are any open transactions <= the max transaction in the range (even if it's below the range). This new implementation is used only by the compactor so that it's understanding of what files it should compact are different than what files a reader views as available for reading. I've also added tests to TestCompactor to test compaction during streaming and compaction after a streamer has aborted and died without cleaning up.
        Hide
        hiveqa Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12691437/HIVE-8966.4.patch

        SUCCESS: +1 6764 tests passed

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2322/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2322/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2322/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12691437 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12691437/HIVE-8966.4.patch SUCCESS: +1 6764 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2322/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2322/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2322/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12691437 - PreCommit-HIVE-TRUNK-Build
        Hide
        alangates Alan Gates added a comment -

        Owen O'Malley pointed out that I need to change the implementation of ValidCompactorTxnList.isTxnValid to return false for aborted transactions so that aborted records aren't carried forward in compacted files.

        Show
        alangates Alan Gates added a comment - Owen O'Malley pointed out that I need to change the implementation of ValidCompactorTxnList.isTxnValid to return false for aborted transactions so that aborted records aren't carried forward in compacted files.
        Hide
        alangates Alan Gates added a comment -

        Yet another version of this patch, this one fixes the issue introduced by the last one that adds aborted records to compacted files.

        Show
        alangates Alan Gates added a comment - Yet another version of this patch, this one fixes the issue introduced by the last one that adds aborted records to compacted files.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12692048/HIVE-8966.5.patch

        ERROR: -1 due to 1 failed/errored test(s), 7330 tests executed
        Failed tests:

        org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2369/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2369/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2369/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12692048 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12692048/HIVE-8966.5.patch ERROR: -1 due to 1 failed/errored test(s), 7330 tests executed Failed tests: org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2369/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2369/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2369/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12692048 - PreCommit-HIVE-TRUNK-Build
        Hide
        owen.omalley Owen O'Malley added a comment -

        This looks good, Alan. +1

        One minor nit is that the class javadoc for ValidReadTxnList has "And" instead of the intended "An".

        Show
        owen.omalley Owen O'Malley added a comment - This looks good, Alan. +1 One minor nit is that the class javadoc for ValidReadTxnList has "And" instead of the intended "An".
        Hide
        owen.omalley Owen O'Malley added a comment -

        After a little more thought, I'm worried that someone will accidentally create a ValidCompactorTxnList and get confused by the different behavior. I think it would make sense to move it into the compactor package to minimize the chance that someone accidentally uses it by mistake.

        Show
        owen.omalley Owen O'Malley added a comment - After a little more thought, I'm worried that someone will accidentally create a ValidCompactorTxnList and get confused by the different behavior. I think it would make sense to move it into the compactor package to minimize the chance that someone accidentally uses it by mistake.
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        +1 for a branch 1.0.

        Show
        vikram.dixit Vikram Dixit K added a comment - +1 for a branch 1.0.
        Hide
        alangates Alan Gates added a comment -

        Final version of the patch. Moved ValidCompactorTxnList per Owen's request. Also made small changes to StreamingIntegrationTester to make it work properly in cases where you want it to go slowly.

        Show
        alangates Alan Gates added a comment - Final version of the patch. Moved ValidCompactorTxnList per Owen's request. Also made small changes to StreamingIntegrationTester to make it work properly in cases where you want it to go slowly.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12694321/HIVE-8966.6.patch

        ERROR: -1 due to 1 failed/errored test(s), 7370 tests executed
        Failed tests:

        TestSparkCliDriver-parallel_join1.q-avro_joins.q-groupby_ppr.q-and-12-more - did not produce a TEST-*.xml file
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2506/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2506/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2506/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12694321 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12694321/HIVE-8966.6.patch ERROR: -1 due to 1 failed/errored test(s), 7370 tests executed Failed tests: TestSparkCliDriver-parallel_join1.q-avro_joins.q-groupby_ppr.q-and-12-more - did not produce a TEST-*.xml file Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2506/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2506/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2506/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12694321 - PreCommit-HIVE-TRUNK-Build
        Hide
        brocknoland Brock Noland added a comment -

        Looks like this was committed but I am seeing:

        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure:
        [ERROR] /Users/noland/workspaces/hive-apache/hive/common/src/java/org/apache/hadoop/hive/common/ValidTxnListImpl.java:[23,8] org.apache.hadoop.hive.common.ValidTxnListImpl is not abstract and does not override abstract method getInvalidTransactions() in org.apache.hadoop.hive.common.ValidTxnList
        [ERROR] /Users/noland/workspaces/hive-apache/hive/common/src/java/org/apache/hadoop/hive/common/ValidTxnListImpl.java:[46,3] method does not override or implement a method from a supertype
        [ERROR] /Users/noland/workspaces/hive-apache/hive/common/src/java/org/apache/hadoop/hive/common/ValidTxnListImpl.java:[54,3] method does not override or implement a method from a supertype
        [ERROR] /Users/noland/workspaces/hive-apache/hive/common/src/java/org/apache/hadoop/hive/common/ValidTxnListImpl.java:[121,3] method does not override or implement a method from a supertype
        [ERROR] -> [Help 1]
        [ERROR] 
        [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
        [ERROR] Re-run Maven using the -X switch to enable full debug logging.
        [ERROR] 
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
        [ERROR] 
        [ERROR] After correcting the problems, you can resume the build with the command
        [ERROR]   mvn <goals> -rf :hive-common
        
        Show
        brocknoland Brock Noland added a comment - Looks like this was committed but I am seeing: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure: [ERROR] /Users/noland/workspaces/hive-apache/hive/common/src/java/org/apache/hadoop/hive/common/ValidTxnListImpl.java:[23,8] org.apache.hadoop.hive.common.ValidTxnListImpl is not abstract and does not override abstract method getInvalidTransactions() in org.apache.hadoop.hive.common.ValidTxnList [ERROR] /Users/noland/workspaces/hive-apache/hive/common/src/java/org/apache/hadoop/hive/common/ValidTxnListImpl.java:[46,3] method does not override or implement a method from a supertype [ERROR] /Users/noland/workspaces/hive-apache/hive/common/src/java/org/apache/hadoop/hive/common/ValidTxnListImpl.java:[54,3] method does not override or implement a method from a supertype [ERROR] /Users/noland/workspaces/hive-apache/hive/common/src/java/org/apache/hadoop/hive/common/ValidTxnListImpl.java:[121,3] method does not override or implement a method from a supertype [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-common
        Hide
        alangates Alan Gates added a comment -

        I did svn add instead of svn rm on a couple of files that moved. I'll fix it.

        Show
        alangates Alan Gates added a comment - I did svn add instead of svn rm on a couple of files that moved. I'll fix it.
        Hide
        brocknoland Brock Noland added a comment -

        thx

        Show
        brocknoland Brock Noland added a comment - thx
        Hide
        alangates Alan Gates added a comment -

        Fixed.

        Show
        alangates Alan Gates added a comment - Fixed.
        Hide
        alangates Alan Gates added a comment -

        Patch 6 checked into trunk. Patch marked for branch 1 checked into branch 1.

        Show
        alangates Alan Gates added a comment - Patch 6 checked into trunk. Patch marked for branch 1 checked into branch 1.
        Hide
        leftylev Lefty Leverenz added a comment -

        Any documentation needed?

        Show
        leftylev Lefty Leverenz added a comment - Any documentation needed?
        Hide
        alangates Alan Gates added a comment -

        Lefty Leverenz no, we just made what should have worked before work properly.

        Show
        alangates Alan Gates added a comment - Lefty Leverenz no, we just made what should have worked before work properly.
        Hide
        jihongliu Jihong Liu added a comment -

        Thanks Alan.

        Show
        jihongliu Jihong Liu added a comment - Thanks Alan.
        Hide
        leftylev Lefty Leverenz added a comment -

        Does this also need to be checked into branch-1.1 (formerly known as 0.15)?

        Show
        leftylev Lefty Leverenz added a comment - Does this also need to be checked into branch-1.1 (formerly known as 0.15)?
        Hide
        alangates Alan Gates added a comment -

        I confirmed that it is already in 1.1, based on the git logs.

        Show
        alangates Alan Gates added a comment - I confirmed that it is already in 1.1, based on the git logs.
        Hide
        thejas Thejas M Nair added a comment -

        Updating release version for jiras resolved in 1.0.0 .

        Show
        thejas Thejas M Nair added a comment - Updating release version for jiras resolved in 1.0.0 .
        Hide
        thejas Thejas M Nair added a comment -

        This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

        Show
        thejas Thejas M Nair added a comment - This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

          People

          • Assignee:
            alangates Alan Gates
            Reporter:
            jihongliu Jihong Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development