Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11170

​ EOFException on History server reading in progress lz4

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.5.1
    • Fix Version/s: None
    • Component/s: Web UI
    • Labels:
      None
    • Environment:

      HDP: 2.3.2.0-2950 (Hadoop 2.7.1.2.3.2.0-2950)
      Spark: 1.5.x (c27e1904)

      Description

      The Spark​ ​History server is not able to read/save the jobs history if Spark is configured to use "spark.io.compression.codec=org.apache.spark.io.LZ4CompressionCodec", it continuously generated the following error:

      ERROR 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: Exception encountered when attempting to load application log hdfs://DATA/user/spark/his
      tory/application_1444297190346_0073_1.lz4.inprogress
      java.io.EOFException: Stream ended prematurely
              at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218)
              at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150)
              at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117)
              at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
              at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
              at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
              at java.io.InputStreamReader.read(InputStreamReader.java:184)
              at java.io.BufferedReader.fill(BufferedReader.java:161)
              at java.io.BufferedReader.readLine(BufferedReader.java:324)
              at java.io.BufferedReader.readLine(BufferedReader.java:389)
              at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67)
              at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:55)
              at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:457)
              at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:292)
              at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:289)
              at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
              at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
              at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
              at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
              at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
              at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
              at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:289)
              at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$2.run(FsHistoryProvider.scala:210)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      INFO 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: Replaying log path: hdfs://DATA/user/spark/history/application_1444297190346_0072_1.lz4.i
      nprogress
      

      As a workaround setting "spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec​"​ makes the​ ​History server work correctly

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                syepes Sebastian YEPES FERNANDEZ
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: