[APEXCORE-535] Node.teardown() should try to gracefully shutdown exectutor service

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.5.0
• Assignee: Vlad Rozov
• Reporter: Vlad Rozov
• Created: 15/Sep/16 04:43
• Updated: 10/Dec/16 18:42
• Resolved: 15/Sep/16 18:37

Description

Forceful shutdown of the executor service leads to InterruptedException if asynchronous checkpointing is in progress:

java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1281)
        at java.lang.Thread.join(Thread.java:1355)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:456)
        at org.apache.hadoop.util.Shell.run(Shell.java:379)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:305)
        at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1011)
        at org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:85)
        at org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.<init>(ChecksumFs.java:344)
        at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:390)
        at org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:575)
        at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:676)
        at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:672)
        at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
        at org.apache.hadoop.fs.FileContext.create(FileContext.java:672)
        at com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:118)
        at com.datatorrent.stram.engine.Node$CheckpointHandler.call(Node.java:667)
        at com.datatorrent.stram.engine.Node$CheckpointHandler.call(Node.java:656)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2016-09-14 22:02:51,463 [Thread-2109] WARN  util.Shell run - Error reading the error stream
java.io.IOException: Stream closed
        at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
        at java.io.BufferedReader.readLine(BufferedReader.java:310)
        at java.io.BufferedReader.readLine(BufferedReader.java:382)
        at org.apache.hadoop.util.Shell$1.run(Shell.java:431)

Issue Links

• links to: GitHub Pull Request #392