Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12083

java.lang.IllegalArgumentException: requirement failed: Overflowed precision (q98)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 1.5.0
    • None
    • SQL

    • CentOS release 6.6

    Description

      While running with 10 users we found that one of the executor randomly hangs during q98 execution. The behavior is random in way that it happens at different time but for the same query. Tried to get a stack trace but was not successful in generating the stack trace.
      Here is the last exception that I saw before the hang:
      java.lang.IllegalArgumentException: requirement failed: Overflowed precision
      at scala.Predef$.require(Predef.scala:233)
      at org.apache.spark.sql.types.Decimal.set(Decimal.scala:111)
      at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:335)
      at org.apache.spark.sql.types.Decimal.apply(Decimal.scala)
      at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getDecimal(UnsafeRow.java:388)
      at org.apache.spark.sql.catalyst.expressions.JoinedRow.getDecimal(JoinedRow.scala:95)
      at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection.apply(Unknown Source)
      ===================================
      One of the other executor before this had the following exception:
      FetchFailed(BlockManagerId(10, d2412.halxg.cloudera.com, 45956), shuffleId=0, mapId=212, reduceId=492, message=
      org.apache.spark.shuffle.FetchFailedException: Failed to connect to d2412.halxg.cloudera.com/10.20.122.112:45956
      at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:321)
      at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:306)
      at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
      at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
      at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:173)
      at org.apache.spark.sql.execution.TungstenSort.org$apache$spark$sql$execution$TungstenSort$$executePartition$1(sort.scala:160)
      at org.apache.spark.sql.execution.TungstenSort$$anonfun$doExecute$4.apply(sort.scala:169)
      at org.apache.spark.sql.execution.TungstenSort$$anonfun$doExecute$4.apply(sort.scala:169)
      at org.apache.spark.rdd.MapPartitionsWithPreparationRDD.compute(MapPartitionsWithPreparationRDD.scala:64)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:88)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Failed to connect to d2412.halxg.cloudera.com/10.20.122.112:45956
      at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:193)
      at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156)
      at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:88)
      at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
      at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
      at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      ... 3 more
      Caused by: java.net.ConnectException: Connection refused: d2412.halxg.cloudera.com/10.20.122.112:45956
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
      at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
      at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      ... 1 more

      )
      =============================
      During stack trace this is what I found:
      Deadlock Detection:

      No deadlocks found.

      Error accessing address 0x7f428f893d80
      sun.jvm.hotspot.debugger.UnmappedAddressException: 7f428f893d80
      at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
      at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
      at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217)
      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482)
      at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
      at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
      at sun.jvm.hotspot.types.basic.BasicField.getAddress(BasicField.java:248)
      at sun.jvm.hotspot.types.basic.BasicAddressFieldWrapper.getValue(BasicAddressFieldWrapper.java:48)
      at sun.jvm.hotspot.runtime.Threads.first(Threads.java:145)
      at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
      at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
      at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
      at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
      at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
      at sun.tools.jstack.JStack.main(JStack.java:102)

      =================
      For schema and query details look at the following JIRA:
      https://issues.apache.org/jira/browse/SPARK-11873

      Attachments

        Activity

          People

            Unassigned Unassigned
            dkumar@cloudera.com Dileep Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: