Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12083

java.lang.IllegalArgumentException: requirement failed: Overflowed precision (q98)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 1.5.0
    • Fix Version/s: None
    • Component/s: SQL
    • Environment:

      CentOS release 6.6

      Description

      While running with 10 users we found that one of the executor randomly hangs during q98 execution. The behavior is random in way that it happens at different time but for the same query. Tried to get a stack trace but was not successful in generating the stack trace.
      Here is the last exception that I saw before the hang:
      java.lang.IllegalArgumentException: requirement failed: Overflowed precision
      at scala.Predef$.require(Predef.scala:233)
      at org.apache.spark.sql.types.Decimal.set(Decimal.scala:111)
      at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:335)
      at org.apache.spark.sql.types.Decimal.apply(Decimal.scala)
      at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getDecimal(UnsafeRow.java:388)
      at org.apache.spark.sql.catalyst.expressions.JoinedRow.getDecimal(JoinedRow.scala:95)
      at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection.apply(Unknown Source)
      ===================================
      One of the other executor before this had the following exception:
      FetchFailed(BlockManagerId(10, d2412.halxg.cloudera.com, 45956), shuffleId=0, mapId=212, reduceId=492, message=
      org.apache.spark.shuffle.FetchFailedException: Failed to connect to d2412.halxg.cloudera.com/10.20.122.112:45956
      at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:321)
      at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:306)
      at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
      at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
      at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:173)
      at org.apache.spark.sql.execution.TungstenSort.org$apache$spark$sql$execution$TungstenSort$$executePartition$1(sort.scala:160)
      at org.apache.spark.sql.execution.TungstenSort$$anonfun$doExecute$4.apply(sort.scala:169)
      at org.apache.spark.sql.execution.TungstenSort$$anonfun$doExecute$4.apply(sort.scala:169)
      at org.apache.spark.rdd.MapPartitionsWithPreparationRDD.compute(MapPartitionsWithPreparationRDD.scala:64)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:88)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Failed to connect to d2412.halxg.cloudera.com/10.20.122.112:45956
      at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:193)
      at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156)
      at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:88)
      at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
      at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
      at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      ... 3 more
      Caused by: java.net.ConnectException: Connection refused: d2412.halxg.cloudera.com/10.20.122.112:45956
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
      at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
      at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      ... 1 more

      )
      =============================
      During stack trace this is what I found:
      Deadlock Detection:

      No deadlocks found.

      Error accessing address 0x7f428f893d80
      sun.jvm.hotspot.debugger.UnmappedAddressException: 7f428f893d80
      at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
      at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
      at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217)
      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482)
      at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
      at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
      at sun.jvm.hotspot.types.basic.BasicField.getAddress(BasicField.java:248)
      at sun.jvm.hotspot.types.basic.BasicAddressFieldWrapper.getValue(BasicAddressFieldWrapper.java:48)
      at sun.jvm.hotspot.runtime.Threads.first(Threads.java:145)
      at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
      at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
      at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
      at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
      at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
      at sun.tools.jstack.JStack.main(JStack.java:102)

      =================
      For schema and query details look at the following JIRA:
      https://issues.apache.org/jira/browse/SPARK-11873

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dkumar@cloudera.com Dileep Kumar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: