Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4175

IOBE may occur in Calcite RexProgramBuilder when queries are submitted concurrently

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      distribution

    • Flags:
      Important

      Description

      I queryed a sql just like `selelct v from table limit 1`,I get a error:

      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IndexOutOfBoundsException: Index: 68, Size: 67

      After debug, I found there is a bug in calcite parse:
      first we look line 72 in org.apache.calcite.rex.RexProgramBuilder

         registerInternal(RexInputRef.of(i, fields), false);
      

      there we get RexInputRef from RexInputRef.of, and it has a method named createName(int idex), here NAMES is SelfPopulatingList.class. SelfPopulatingList.class describe as Thread-safe list, but in fact it is Thread-unsafe. when NAMES.get(index) is called distributed, it gets a error. We hope SelfPopulatingList.class to be {$0 $1 $2 ....$n}, but when it called distributed, it may be {$0,$1...$29,$30...$59,$30,$31...$59...}.
      We see method registerInternal

      private RexLocalRef registerInternal(RexNode expr, boolean force) {
      expr = simplify(expr);
      
          RexLocalRef ref;
          final Pair<String, String> key;
          if (expr instanceof RexLocalRef) {
            key = null;
            ref = (RexLocalRef) expr;
          } else {
            key = RexUtil.makeKey(expr);
            ref = exprMap.get(key);
          }
          if (ref == null) {
            if (validating) {
              validate(
                  expr,
                  exprList.size());
            }
      

      Here makeKey(expr) hope to get different key, however it get same key, so addExpr(expr) called less, in this method

      RexLocalRef ref;
          final int index = exprList.size();
          exprList.add(expr);
          ref =
              new RexLocalRef(
                  index,
                  expr.getType());
          localRefList.add(ref);
          return ref;
      

      localRefList get error size, so in line 939,

      final RexLocalRef ref = localRefList.get(index);
      

      throw IndexOutOfBoundsException

      bugfix:
      We can't change origin code of calcite before they fix this bug, so we can init NAMEs in RexLocalRef on start. Just add

      RexInputRef.createName(2048);
      

      on Bootstrap.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                huntersjm huntersjm
                Reviewer:
                Krystal
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: