Hive
  1. Hive
  2. HIVE-1129

Fix Assertion in ExecDriver.execute when assertions are enabled in HADOOP_OPTS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I noticed that when running hive CLI, assertions are not enabled, which was causing me some confusion when debugging an issue.

      So, I added the following to my environment:

      export HADOOP_OPTS="-ea -esa"

      This worked, and allowed me to see assertion failures when executing via CLI.

      But then I tried to run a test, and got an assertion failure from the following code in ExecDriver.execute:

      // Turn on speculative execution for reducers
      HiveConf.setVar(job, HiveConf.ConfVars.HADOOPSPECULATIVEEXECREDUCERS,
      HiveConf.getVar(job, HiveConf.ConfVars.HIVESPECULATIVEEXECREDUCERS));

      The assertion says it should be using getBoolVar/setBoolVar instead.

        Activity

        Hide
        Zheng Shao added a comment -

        Committed. Thanks John!

        Show
        Zheng Shao added a comment - Committed. Thanks John!
        Hide
        Zheng Shao added a comment -

        +1. That makes sense to me.
        Please open a follow-up as an "improvement".

        Show
        Zheng Shao added a comment - +1. That makes sense to me. Please open a follow-up as an "improvement".
        Hide
        John Sichi added a comment -

        Resubmitting patch as is. If you agree with me, I'll create a followup for adding the setters; otherwise, cancel it and I'll combine the two changes.

        Show
        John Sichi added a comment - Resubmitting patch as is. If you agree with me, I'll create a followup for adding the setters; otherwise, cancel it and I'll combine the two changes.
        Hide
        John Sichi added a comment -

        In the spirit of keeping patch transactions small and focused, maybe it would be better to commit this patch as is and assign me a separate JIRA for adding and using the setter methods? The reason I say this is that my patch followed the existing pattern here in the same file:

        if(work.getMinSplitSize() != null)

        { job.setInt(HiveConf.ConfVars.MAPREDMINSPLITSIZE.varname, work.getMinSplitSize().intValue()); }

        So that code will need to be changed too as part of consistently using the new setters, and that change is unrelated to my fix for the existing bug with assertions.

        Show
        John Sichi added a comment - In the spirit of keeping patch transactions small and focused, maybe it would be better to commit this patch as is and assign me a separate JIRA for adding and using the setter methods? The reason I say this is that my patch followed the existing pattern here in the same file: if(work.getMinSplitSize() != null) { job.setInt(HiveConf.ConfVars.MAPREDMINSPLITSIZE.varname, work.getMinSplitSize().intValue()); } So that code will need to be changed too as part of consistently using the new setters, and that change is unrelated to my fix for the existing bug with assertions.
        Hide
        Zheng Shao added a comment -

        Hi John, can you add the setBoolVar() function to HiveConf class just like the getBoolVar()? That makes the code easier to read and also more consistent.

        Show
        Zheng Shao added a comment - Hi John, can you add the setBoolVar() function to HiveConf class just like the getBoolVar()? That makes the code easier to read and also more consistent.
        Hide
        John Sichi added a comment -

        Here's a patch for getting/setting the variables in such a way that the assertions are not triggered.

        Show
        John Sichi added a comment - Here's a patch for getting/setting the variables in such a way that the assertions are not triggered.
        Hide
        Namit Jain added a comment -

        A parameter exists for enabling speculative execution for reducers in hive - the reason we added it about a month back was that we
        wanted to test it easily – there were some bugs in speculative execution for reducers in hadoop 17, which is why we had disabled it.
        Now that those bugs have been solved, we wanted to test it gradually.

        Show
        Namit Jain added a comment - A parameter exists for enabling speculative execution for reducers in hive - the reason we added it about a month back was that we wanted to test it easily – there were some bugs in speculative execution for reducers in hadoop 17, which is why we had disabled it. Now that those bugs have been solved, we wanted to test it gradually.
        Hide
        John Sichi added a comment -

        Haha, you're right, I should get my eyes checked.

        Show
        John Sichi added a comment - Haha, you're right, I should get my eyes checked.
        Hide
        Paul Yang added a comment -

        Aren't those two different variables - getVar is accessing HIVESPECULATIVEEXECREDUCERS, while the setVar is setting HADOOPSPECULATIVEEXECREDUCERS.

        Show
        Paul Yang added a comment - Aren't those two different variables - getVar is accessing HIVESPECULATIVEEXECREDUCERS, while the setVar is setting HADOOPSPECULATIVEEXECREDUCERS.
        Hide
        John Sichi added a comment -

        Also, I don't really understand what this code is doing. Why is doing setVar using the current result of getVar? Doesn't that mean it has no effect?

        Show
        John Sichi added a comment - Also, I don't really understand what this code is doing. Why is doing setVar using the current result of getVar? Doesn't that mean it has no effect?
        Hide
        John Sichi added a comment -

        Here's the assertion stack:

        [junit] Exception in thread "main" java.lang.AssertionError
        [junit] at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:292)
        [junit] at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:569)
        [junit] at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:946)

        Show
        John Sichi added a comment - Here's the assertion stack: [junit] Exception in thread "main" java.lang.AssertionError [junit] at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:292) [junit] at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:569) [junit] at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:946)

          People

          • Assignee:
            John Sichi
            Reporter:
            John Sichi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development