Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23899 Built-in SQL Function Improvement
  3. SPARK-28962

High-order function: filter(array<T>, function<T, Int, boolean>) → array<T>

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      It's helpful to have access to the index when using the filter function. For instance, we're using SparkSQL to manipulate genomic data. We store some fields in a long array that has the same length for every row in the DataFrame. We compute aggregates that are per array position (so we look at the kth element for each row's array) and then want to filter each row's array by looking values in the aggregate array.

        Attachments

          Activity

            People

            • Assignee:
              hhd Henry Davidge
              Reporter:
              hhd Henry Davidge
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: