Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5134

Fix TestAvroStorage unit test in Spark mode

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It seems that test fails, because Avro GenericData#Record doesn't implement Serializable interface:

      2017-02-23 09:14:41,887 ERROR [main] spark.JobGraphBuilder (JobGraphBuilder.java:sparkOperToRDD(183)) - throw exception in sparkOperToRDD: 
      org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 9.0 (TID 9) had a not serializable result: org.apache.avro.generic.GenericData$Record
      Serialization stack:
      	- object not serializable (class: org.apache.avro.generic.GenericData$Record, value: {"key": "stuff in closet", "value1": {"thing": "hat", "count": 7}, "value2": {"thing": "coat", "count": 2}})
      	- field (class: org.apache.pig.impl.util.avro.AvroTupleWrapper, name: avroObject, type: interface org.apache.avro.generic.IndexedRecord)
      	- object (class org.apache.pig.impl.util.avro.AvroTupleWrapper, org.apache.pig.impl.util.avro.AvroTupleWrapper@3d3a58c1)
      	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
      	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
      

      The failing tests is a new test introduced with merging trunk to spark branch, that's why we didn't see this error before.

        Attachments

        1. PIG-5134_2.patch
          3 kB
          Nándor Kollár
        2. PIG-5134_3.patch
          4 kB
          Nándor Kollár
        3. PIG-5134.patch
          7 kB
          Nándor Kollár

        Issue Links

          Activity

            People

            • Assignee:
              nkollar Nándor Kollár
              Reporter:
              kellyzly liyunzhang

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment