Major Description
Currently Livy unit test will output the log into unit-tests.log, but it is not separated by each test, so it is hard to identify which log belongs to which task, so here propose to add context before and after unit test, so that user could easily identify the related logs.
Here is the example of unit test output:
==== TEST OUTPUT FOR com.cloudera.livy.repl.SparkSessionSpec: 'should start in the starting or idle state' ==== 16/09/09 10:59:15.735 pool-17-thread-1 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/09/09 10:59:15.871 pool-17-thread-1 INFO SecurityManager: Changing view acls to: sshao 16/09/09 10:59:15.872 pool-17-thread-1 INFO SecurityManager: Changing modify acls to: sshao 16/09/09 10:59:15.873 pool-17-thread-1 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sshao); users with modify permissions: Set(sshao) 16/09/09 10:59:16.040 pool-17-thread-1 INFO HttpServer: Starting HTTP Server 16/09/09 10:59:16.119 pool-17-thread-1 INFO Utils: Successfully started service 'HTTP class server' on port 56636. 16/09/09 10:59:18.840 pool-17-thread-1 INFO SparkContext: Running Spark version 1.6.2 16/09/09 10:59:18.878 pool-17-thread-1 INFO SecurityManager: Changing view acls to: sshao 16/09/09 10:59:18.878 pool-17-thread-1 INFO SecurityManager: Changing modify acls to: sshao 16/09/09 10:59:18.878 pool-17-thread-1 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sshao); users with modify permissions: Set(sshao) 16/09/09 10:59:19.093 pool-17-thread-1 INFO Utils: Successfully started service 'sparkDriver' on port 56638. 16/09/09 10:59:19.695 sparkDriverActorSystem-akka.actor.default-dispatcher-3 INFO Slf4jLogger: Slf4jLogger started 16/09/09 10:59:19.734 sparkDriverActorSystem-akka.actor.default-dispatcher-3 INFO Remoting: Starting remoting 16/09/09 10:59:19.897 sparkDriverActorSystem-akka.actor.default-dispatcher-3 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.0.103:56640] 16/09/09 10:59:19.904 pool-17-thread-1 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 56640. 16/09/09 10:59:19.916 pool-17-thread-1 INFO SparkEnv: Registering MapOutputTracker 16/09/09 10:59:19.937 pool-17-thread-1 INFO SparkEnv: Registering BlockManagerMaster 16/09/09 10:59:19.953 pool-17-thread-1 INFO DiskBlockManager: Created local directory at /Users/sshao/projects/livy/repl/target/tmp/blockmgr-ba269a9b-0b0e-4d3e-86a0-6a384e1f6d6e 16/09/09 10:59:19.958 pool-17-thread-1 INFO MemoryStore: MemoryStore started with capacity 1140.4 MB 16/09/09 10:59:20.025 pool-17-thread-1 INFO SparkEnv: Registering OutputCommitCoordinator 16/09/09 10:59:20.122 pool-17-thread-1 INFO Executor: Starting executor ID driver on host localhost 16/09/09 10:59:20.130 pool-17-thread-1 INFO Executor: Using REPL class URI: http://192.168.0.103:56636 16/09/09 10:59:20.145 pool-17-thread-1 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56641. 16/09/09 10:59:20.145 pool-17-thread-1 INFO NettyBlockTransferService: Server created on 56641 16/09/09 10:59:20.146 pool-17-thread-1 INFO BlockManagerMaster: Trying to register BlockManager 16/09/09 10:59:20.149 dispatcher-event-loop-2 INFO BlockManagerMasterEndpoint: Registering block manager localhost:56641 with 1140.4 MB RAM, BlockManagerId(driver, localhost, 56641) 16/09/09 10:59:20.151 pool-17-thread-1 INFO BlockManagerMaster: Registered BlockManager 16/09/09 10:59:20.427 pool-17-thread-1 INFO SparkInterpreter: Created sql context. 16/09/09 10:59:23.108 dispatcher-event-loop-6 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/09/09 10:59:23.113 ScalaTest-main-running-SparkSessionSpec INFO MemoryStore: MemoryStore cleared 16/09/09 10:59:23.114 ScalaTest-main-running-SparkSessionSpec INFO BlockManager: BlockManager stopped 16/09/09 10:59:23.122 ScalaTest-main-running-SparkSessionSpec INFO BlockManagerMaster: BlockManagerMaster stopped 16/09/09 10:59:23.126 dispatcher-event-loop-3 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/09/09 10:59:23.128 ScalaTest-main-running-SparkSessionSpec INFO SparkContext: Successfully stopped SparkContext 16/09/09 10:59:23.133 sparkDriverActorSystem-akka.actor.default-dispatcher-2 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 16/09/09 10:59:23.137 sparkDriverActorSystem-akka.actor.default-dispatcher-2 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 16/09/09 10:59:23.173 sparkDriverActorSystem-akka.actor.default-dispatcher-2 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 16/09/09 10:59:23.182 ScalaTest-main-running-SparkSessionSpec INFO SparkSessionSpec: ==== FINISHED com.cloudera.livy.repl.SparkSessionSpec: 'should start in the starting or idle state' ====