Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13736 Big-Endian plataform issues
  3. SPARK-12319

ExchangeCoordinatorSuite fails on big-endian platforms

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: 1.6.0
    • Fix Version/s: None
    • Component/s: SQL
    • Environment:

      Problems apparent on BE, LE could be impacted too

    • Flags:
      Important

      Description

      JIRA to cover endian specific problems - since testing 1.6 I've noticed problems with DataFrames on BE platforms, e.g. https://issues.apache.org/jira/browse/SPARK-9858

      Josh Rosen Yin Huai

      Current progress: using com.google.common.io.LittleEndianDataInputStream and com.google.common.io.LittleEndianDataOutputStream within UnsafeRowSerializer fixes three test failures in ExchangeCoordinatorSuite but I'm concerned around performance/wider functional implications

      "org.apache.spark.sql.DatasetAggregatorSuite.typed aggregation: class input with reordering" fails as we expect "one, 1" but instead get "one, 9" - we believe the issue lies within BitSetMethods.java, specifically around: return (wi << 6) + subIndex + java.lang.Long.numberOfTrailingZeros(word);

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              aroberts Adam Roberts
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: