Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-9125

RSC stdout is logged twice [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: spark-branch
    • Fix Version/s: None
    • Component/s: Spark
    • Labels:
      None

      Description

      This is quite strange and I don't see the issue at first glance.

      2014-12-16 12:44:48,826 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - 2014-12-16T12:44:48.638-0500: [Full GC [PSYoungGen: 111616K->50711K(143360K)] [ParOldGen: 349385K->349385K(349696K)] 461001K->400097K(493056K) [PSPermGen: 58684K->58684K(58880K)], 0.1879000 secs] [Times: user=1.14 sys=0.00, real=0.19 secs] 
      2014-12-16 12:44:48,826 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - 2014-12-16T12:44:48.638-0500: [Full GC [PSYoungGen: 111616K->50711K(143360K)] [ParOldGen: 349385K->349385K(349696K)] 461001K->400097K(493056K) [PSPermGen: 58684K->58684K(58880K)], 0.1879000 secs] [Times: user=1.14 sys=0.00, real=0.19 secs] 
      
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - "sparkDriver-akka.actor.default-dispatcher-3" daemon prio=10 tid=0x00007f9e3c5cc000 nid=0x3698 runnable [0x00007f9e30376000]
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - "sparkDriver-akka.actor.default-dispatcher-3" daemon prio=10 tid=0x00007f9e3c5cc000 nid=0x3698 runnable [0x00007f9e30376000]
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -    java.lang.Thread.State: RUNNABLE
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -    java.lang.Thread.State: RUNNABLE
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:767)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:767)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:131)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:131)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      2014-12-16 12:44:48,554 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1050)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1050)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:941)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:941)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:955)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:955)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:397)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:397)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:287)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:287)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.<init>(CombineHiveInputFormat.java:101)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.<init>(CombineHiveInputFormat.java:101)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getCombineSplits(CombineHiveInputFormat.java:441)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getCombineSplits(CombineHiveInputFormat.java:441)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:508)
      2014-12-16 12:44:48,555 INFO  [stdout-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) -        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:508)
      

        Attachments

          Activity

            People

            • Assignee:
              jxiang Jimmy Xiang
              Reporter:
              brocknoland Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: