Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9930 Feature transformers in 1.6
  3. SPARK-8345

Add an SQL node as a feature transformer

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.6.0
    • ML
    • None

    Description

      Some simple feature transformations can take leverage on SQL operators. Users do not need to create an ML transformer for each of them. We can have an SQL transformer that executes an SQL command which operates on the input dataframe.

      val sql = new SQL()
        .setStatement("SELECT *, length(text) AS text_length FROM __THIS__")
      

      where "_THIS_" will be replaced by a temp table that represents the DataFrame.

      Attachments

        Issue Links

          Activity

            People

              yanboliang Yanbo Liang
              mengxr Xiangrui Meng
              Xiangrui Meng Xiangrui Meng
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: