Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28528 Fix Idempotence for Once batches in Catalyst optimizer
  3. SPARK-28306

Once optimizer rule NormalizeFloatingNumbers is not idempotent

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      When the rule NormalizeFloatingNumbers is called multiple times, it will add additional transform operator to an expression, which is not appropriate. To fix it, we have to make it idempotent, i.e. yield the same logical plan regardless of multiple runs.

      Attachments

        Activity

          People

            manifoldQAQ Yesheng Ma
            manifoldQAQ Yesheng Ma
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: