Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-8790

UT: fix hook_context_cs test case

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: spark-branch
    • Fix Version/s: None
    • Component/s: Tests
    • Labels:
      None

      Description

      The test hook_context_cs fails while checking for expected value in VerifyContentSummaryCacheHook:
      TestSparkCliDriver.testCliDriver_hook_context_cs:120->runTest:146 Unexpected exception junit.framework.AssertionFailedError: expected:<1> but was:<0>
      at junit.framework.Assert.fail(Assert.java:57)
      at junit.framework.Assert.failNotEquals(Assert.java:329)
      at junit.framework.Assert.assertEquals(Assert.java:78)
      at junit.framework.Assert.assertEquals(Assert.java:234)
      at junit.framework.Assert.assertEquals(Assert.java:241)
      at org.apache.hadoop.hive.ql.hooks.VerifyContentSummaryCacheHook.run(VerifyContentSummaryCacheHook.java:34)
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1512)
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1216)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1033)

      The problem is that the post-hook check in org.apache.hadoop.hive.ql.hooks.VerifyContentSummaryCacheHook fails because inputToCS.size is 0 instead of 1:
      Assert.assertEquals(1, inputToCS.size());

      I ran the same test with TestCliDriver where Hive will create MapRedTask objects. For a MapRedTask, the hookContext.getInputPathToContentSummary() is set in the setNumberOfReducers method, line 400:
      inputSummary = Utilities.getInputSummary(driverContext.getCtx(), work.getMapWork(), null);
      (the path is added in Utilities.getInputSummary(Context, MapWork, PathFilter) line: 2584).

      Since we have a SparkTask, the array will be empty, hence the check for length==1 fails.

      It looks somewhat related to HIVE-8504 where the test case also configures a hive.exec.post.hooks, but since it's a different post-hook class altogether, I wanted to open a separate JIRA for it.
      Maybe something needs to be added to the SparkTask.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tfriedr Thomas Friedrich
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: