Details
-
Sub-task
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
spark-branch
-
None
-
None
Description
The test hook_context_cs fails while checking for expected value in VerifyContentSummaryCacheHook:
TestSparkCliDriver.testCliDriver_hook_context_cs:120->runTest:146 Unexpected exception junit.framework.AssertionFailedError: expected:<1> but was:<0>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at org.apache.hadoop.hive.ql.hooks.VerifyContentSummaryCacheHook.run(VerifyContentSummaryCacheHook.java:34)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1512)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1216)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1033)
The problem is that the post-hook check in org.apache.hadoop.hive.ql.hooks.VerifyContentSummaryCacheHook fails because inputToCS.size is 0 instead of 1:
Assert.assertEquals(1, inputToCS.size());
I ran the same test with TestCliDriver where Hive will create MapRedTask objects. For a MapRedTask, the hookContext.getInputPathToContentSummary() is set in the setNumberOfReducers method, line 400:
inputSummary = Utilities.getInputSummary(driverContext.getCtx(), work.getMapWork(), null);
(the path is added in Utilities.getInputSummary(Context, MapWork, PathFilter) line: 2584).
Since we have a SparkTask, the array will be empty, hence the check for length==1 fails.
It looks somewhat related to HIVE-8504 where the test case also configures a hive.exec.post.hooks, but since it's a different post-hook class altogether, I wanted to open a separate JIRA for it.
Maybe something needs to be added to the SparkTask.