Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4470

Predicate hits NumberFormatException due to constant folding.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Frontend
    • Labels:

      Description

      Impala folds constants in a few specific places. For example, for predicates assigned to Kudu scan nodes, or for predicates used as partition filters. This constant folding may result in queries not being executable if the evaluation result is NaN or infinity (and possibly other special values).

      The underlying problem is that our NumeritLiteral has no good way of representing NaN or infinity.

      Repro:

      explain select * from functional.alltypes
      where year < (cast(1.1000000238418579 as double) / cast(0 as double));
      ERROR: NumberFormatException: Infinite or NaN
      

      Stack:

      I1110 13:04:23.117324 16414 jni-util.cc:169] java.lang.NumberFormatException: Infinite or NaN
      	at java.math.BigDecimal.<init>(BigDecimal.java:808)
      	at org.apache.impala.analysis.LiteralExpr.create(LiteralExpr.java:202)
      	at org.apache.impala.analysis.Expr.foldConstantChildren(Expr.java:1219)
      	at org.apache.impala.planner.HdfsPartitionPruner.canEvalUsingPartitionMd(HdfsPartitionPruner.java:182)
      	at org.apache.impala.planner.HdfsPartitionPruner.prunePartitions(HdfsPartitionPruner.java:123)
      	at org.apache.impala.planner.SingleNodePlanner.createHdfsScanPlan(SingleNodePlanner.java:1199)
      	at org.apache.impala.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:1259)
      	at org.apache.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:1482)
      	at org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:757)
      	at org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:597)
      	at org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:248)
      	at org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:145)
      	at org.apache.impala.planner.Planner.createPlan(Planner.java:91)
      	at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:987)
      	at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1067)
      	at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:155)
      

      The easiest solution is to consider the constant folding a failure for special evaluation results like NaN/infinity and fall back to executing the original expression.
      As a result, the index-based partition pruning may not kick in or predicates may not be pushed to Kudu. These cases seem exotic enough for this behavior to be acceptable in most cases.

      Alternatively, we could work on proper representation of NaN and infinity in our NumericLiteral.

        Activity

        Hide
        alex.behm Alexander Behm added a comment -

        commit f5e660dd6eabc667094d8bc40810602cd7fa56d4
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Thu Nov 10 13:50:20 2016 -0800

        IMPALA-4470: Avoid creating a NumericLiteral from NaN/infinity/-0.

        Our NumericLiteral is backed by a BigDecimal which cannot
        represent the special float values NaN, infinity or negative zero.
        As a result, when evaluating constant expressions from the FE we
        hit an exception when trying to create a NumericLiteral from
        a NaN or infinity value. Before, negative zero would silently
        get converted to zero which is dangerous.

        The fix is to treat the expr evaluation as a failure and not
        replace the constant Expr with a LiteralExpr.

        Change-Id: I8243b2ee9fa9c470d078b385583f2f48b606a230
        Reviewed-on: http://gerrit.cloudera.org:8080/5050
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        alex.behm Alexander Behm added a comment - commit f5e660dd6eabc667094d8bc40810602cd7fa56d4 Author: Alex Behm <alex.behm@cloudera.com> Date: Thu Nov 10 13:50:20 2016 -0800 IMPALA-4470 : Avoid creating a NumericLiteral from NaN/infinity/-0. Our NumericLiteral is backed by a BigDecimal which cannot represent the special float values NaN, infinity or negative zero. As a result, when evaluating constant expressions from the FE we hit an exception when trying to create a NumericLiteral from a NaN or infinity value. Before, negative zero would silently get converted to zero which is dangerous. The fix is to treat the expr evaluation as a failure and not replace the constant Expr with a LiteralExpr. Change-Id: I8243b2ee9fa9c470d078b385583f2f48b606a230 Reviewed-on: http://gerrit.cloudera.org:8080/5050 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development