Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3453

Fix TaskManager logs exception when sampling backpressure while task completes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.0
    • Component/s: Runtime / Coordination
    • Labels:
      None

      Description

      Backpressure sampling is interrupted when a task completes. It may be best to create a new response class for this case.

      java.lang.IllegalStateException: Cannot sample task 08f138723e8174e70f5e7ddc672f8954. Task was removed after 65 sample(s).
              at org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleStackTraceSampleMessage(TaskManager.scala:743)
              at org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:277)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
              at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
              at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
              at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
              at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
              at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
              at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
              at org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:119)
              at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
              at akka.actor.ActorCell.invoke(ActorCell.scala:487)
              at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
              at akka.dispatch.Mailbox.run(Mailbox.scala:221)
              at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
              at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
              at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
      2016-02-19 09:36:52,773 ERROR org.apache.flink.runtime.webmonitor.BackPressureStatsTracker  - Failed to gather stack trace sample.
      java.lang.RuntimeException: Discarded
              at org.apache.flink.runtime.webmonitor.StackTraceSampleCoordinator$PendingStackTraceSample.discard(StackTraceSampleCoordinator.java:394)
              at org.apache.flink.runtime.webmonitor.StackTraceSampleCoordinator.cancelStackTraceSample(StackTraceSampleCoordinator.java:249)
              at org.apache.flink.runtime.webmonitor.StackTraceSampleCoordinator$StackTraceSampleCoordinatorActor.handleMessage(StackTraceSampleCoordinator.java:462)
              at org.apache.flink.runtime.akka.FlinkUntypedActor.handleLeaderSessionID(FlinkUntypedActor.java:97)
              at org.apache.flink.runtime.akka.FlinkUntypedActor.onReceive(FlinkUntypedActor.java:68)
              at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
              at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
              at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
              at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
              at akka.actor.ActorCell.invoke(ActorCell.scala:487)
              at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
              at akka.dispatch.Mailbox.run(Mailbox.scala:221)
              at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
              at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
              at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
      Caused by: java.lang.IllegalStateException: Cannot sample task 08f138723e8174e70f5e7ddc672f8954. Task was removed after 65 sample(s).
              at org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleStackTraceSampleMessage(TaskManager.scala:743)
              at org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:277)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
              at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
              at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
              at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
              at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
              at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
              at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
              at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
              at org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:119)
              ... 9 more
      

        Attachments

          Activity

            People

            • Assignee:
              uce Ufuk Celebi
              Reporter:
              greghogan Greg Hogan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: