Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5387

Test failures on JRE 11

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None

      Description

      I tried to compile Pig with JDK 8 and execute the test with Java 11, and faced with several test failures. For example TestCommit#testCheckin2 failed with the following exception:

      2019-05-08 16:06:09,712 WARN  [Thread-108] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1000317333_0003
      java.lang.Exception: java.io.IOException: Deserialization error: null
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
      Caused by: java.io.IOException: Deserialization error: null
      	at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:62)
      	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:183)
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.lang.NullPointerException
      	at org.apache.pig.impl.plan.Operator.hashCode(Operator.java:106)
      	at java.base/java.util.HashMap.hash(HashMap.java:339)
      	at java.base/java.util.HashMap.readObject(HashMap.java:1461)
      	at java.base/jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1160)
      

      It deserialization of one of the map plan failed, it appears we ran into JDK-8201131. I seems that the workaround in the issue report works, adding a readObject method to org.apache.pig.impl.plan.Operator:

          private void readObject(ObjectInputStream in) throws ClassNotFoundException, IOException {
              in.defaultReadObject();
          }
      

      solves the problem, however I'm not sure that this is the optimal solution.

        Attachments

        1. PIG-5387_1.patch
          5 kB
          Nándor Kollár
        2. PIG-5387_2.patch
          5 kB
          Nándor Kollár
        3. PIG-5387_3.patch
          5 kB
          Nándor Kollár

          Activity

            People

            • Assignee:
              nkollar Nándor Kollár
              Reporter:
              nkollar Nándor Kollár
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: