Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22984

Fix incorrect bitmap copying and offset shifting in GenerateUnsafeRowJoiner

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.5.0, 1.6.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0
    • Fix Version/s: 2.2.2, 2.3.0
    • Component/s: SQL
    • Labels:
    • Target Version/s:

      Description

      The following query returns an incorrect answer:

      set spark.sql.autoBroadcastJoinThreshold=-1;
      
      create table a as select * from values 1;
      create table b as select * from values 2;
      
      SELECT
      t3.col1,
      t1.col1
      FROM a t1
      CROSS JOIN b t2
      CROSS JOIN b t3
      

      This should return the row 2, 1 but instead it returns null, 1. If you permute the order of the columns in the select statement or the order of the joins then it returns a valid answer (i.e. one without incorrect NULLs).

      This turns out to be due to two longstanding bugs in GenerateUnsafeRowJoiner, which I'll describe in more detail in my PR.

        Attachments

          Activity

            People

            • Assignee:
              joshrosen Josh Rosen
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: