Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8159

Improve expression function coverage (Spark 1.5)

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0
    • SQL
    • None

    Description

      This is an umbrella ticket to track new expressions we are adding to SQL/DataFrame.

      For each new expression, we should:
      1. Add a new Expression implementation in org.apache.spark.sql.catalyst.expressions
      2. If applicable, implement the code generated version (by implementing genCode).
      3. Add comprehensive unit tests (for all the data types the expressions support).
      4. If applicable, add a new function for DataFrame in org.apache.spark.sql.functions, and python/pyspark/sql/functions.py for Python.

      For date/time functions, put them in expressions/datetime.scala, and create a DateTimeFunctionSuite.scala for testing.

      Attachments

        Issue Links

        duplicates

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-9845 Add built-in UDF

        • Major - Major loss of function.
        • Resolved
        is blocked by

        Sub-task - The sub-task of the issue SPARK-7285 Audit missing Hive functions

        • Major - Major loss of function.
        • Resolved
        is related to

        Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-7075 Project Tungsten (Spark 1.5 Phase 1)

        • Major - Major loss of function.
        • Resolved

        Umbrella - An overarching type made of sub-tasks SPARK-8829 Improve expression performance

        • Major - Major loss of function.
        • Resolved
        relates to

        Umbrella - An overarching type made of sub-tasks SPARK-9571 Improve expression function coverage

        • Major - Major loss of function.
        • Resolved
        1.
        		 date/time function: unix_timestamp Sub-task Resolved Adrian Wang Actions
        2.
        		 date/time function: from_unixtime Sub-task Resolved Adrian Wang Actions
        3.
        		 date/time function: to_date Sub-task Resolved Adrian Wang Actions
        4.
        		 date/time function: year Sub-task Resolved Tarek Auel Actions
        5.
        		 date/time function: quarter Sub-task Resolved Tarek Auel Actions
        6.
        		 date/time function: month Sub-task Resolved Tarek Auel Actions
        7.
        		 date/time function: day / dayofmonth Sub-task Resolved Tarek Auel Actions
        8.
        		 date/time function: hour Sub-task Resolved Tarek Auel Actions
        9.
        		 date/time function: minute Sub-task Resolved Tarek Auel Actions
        10.
        		 date/time function: second Sub-task Resolved Tarek Auel Actions
        11.
        		 date/time function: weekofyear Sub-task Resolved Tarek Auel Actions
        12.
        		 date/time function: datediff Sub-task Resolved Adrian Wang Actions
        13.
        		 date/time function: date_add Sub-task Resolved Adrian Wang Actions
        14.
        		 date/time function: date_sub Sub-task Resolved Adrian Wang Actions
        15.
        		 date/time function: from_utc_timestamp Sub-task Resolved Adrian Wang Actions
        16.
        		 date/time function: to_utc_timestamp Sub-task Resolved Adrian Wang Actions
        17.
        		 date/time function: current_date Sub-task Resolved Adrian Wang Actions
        18.
        		 date/time function: current_timestamp Sub-task Resolved Adrian Wang Actions
        19.
        		 math function: round Sub-task Resolved Yijie Shen Actions
        20.
        		 date/time function: add_months Sub-task Resolved Adrian Wang Actions
        21.
        		 date/time function: last_day Sub-task Resolved Adrian Wang Actions
        22.
        		 date/time function: next_day Sub-task Resolved Adrian Wang Actions
        23.
        		 date/time function: trunc Sub-task Resolved Adrian Wang Actions
        24.
        		 date/time function: months_between Sub-task Resolved Adrian Wang Actions
        25.
        		 date/time function: date_format Sub-task Resolved Tarek Auel Actions
        26.
        		 conditional function: if Sub-task Resolved Reynold Xin Actions
        27.
        		 conditional functions: greatest Sub-task Resolved Adrian Wang Actions
        28.
        		 conditional function: least Sub-task Resolved Adrian Wang Actions
        29.
        		 conditional function: nvl Sub-task Resolved Reynold Xin Actions
        30.
        		 math function: bin Sub-task Resolved L. C. Hsieh Actions
        31.
        		 math function: ceiling Sub-task Resolved Reynold Xin Actions
        32.
        		 math function: conv Sub-task Resolved zhichao-li Actions
        33.
        		 math function: degrees Sub-task Resolved Reynold Xin Actions
        34.
        		 math function: radians Sub-task Resolved Reynold Xin Actions
        35.
        		 math function: e Sub-task Resolved Adrian Wang Actions
        36.
        		 math function: factorial Sub-task Resolved zhichao-li Actions
        37.
        		 math function: hex Sub-task Resolved zhichao-li Actions
        38.
        		 math function: pi Sub-task Resolved Adrian Wang Actions
        39.
        		 math function: rename log -> ln Sub-task Resolved Reynold Xin Actions
        40.
        		 math function: log2 Sub-task Resolved Adrian Wang Actions
        41.
        		 math function: log Sub-task Resolved L. C. Hsieh Actions
        42.
        		 math function: negative Sub-task Resolved Reynold Xin Actions
        43.
        		 math function: positive Sub-task Resolved zhichao-li Actions
        44.
        		 math function: pmod Sub-task Resolved zhichao-li Actions
        45.
        		 math function: alias power / pow Sub-task Resolved Reynold Xin Actions
        46.
        		 math function: shiftleft Sub-task Resolved Tarek Auel Actions
        47.
        		 math function: shiftright Sub-task Resolved Tarek Auel Actions
        48.
        		 math function: alias sign / signum Sub-task Resolved Reynold Xin Actions
        49.
        		 math function: shiftrightunsigned Sub-task Resolved zhichao-li Actions
        50.
        		 math function: unhex Sub-task Resolved zhichao-li Actions
        51.
        		 conditional function: isnull Sub-task Resolved Reynold Xin Actions
        52.
        		 conditional function: isnotnull Sub-task Resolved Reynold Xin Actions
        53.
        		 complex function: size Sub-task Resolved Pedro Rodriguez Actions
        54.
        		 complex function: array_contains Sub-task Resolved Pedro Rodriguez Actions
        55.
        		 complex function: sort_array Sub-task Resolved Cheng Hao Actions
        56.
        		 misc function: md5 Sub-task Resolved Qian, Shilei Actions
        57.
        		 misc function: sha1 / sha Sub-task Resolved Tarek Auel Actions
        58.
        		 misc function: crc32 Sub-task Resolved Tarek Auel Actions
        59.
        		 misc function: sha2 Sub-task Resolved L. C. Hsieh Actions
        60.
        		 string function: ascii Sub-task Resolved Cheng Hao Actions
        61.
        		 string function: base64 Sub-task Resolved Cheng Hao Actions
        62.
        		 string function: concat Sub-task Resolved Reynold Xin Actions
        63.
        		 string function: concat_ws Sub-task Resolved Reynold Xin Actions
        64.
        		 string function: decode Sub-task Resolved Cheng Hao Actions
        65.
        		 string function: encode Sub-task Resolved Cheng Hao Actions
        66.
        		 string function: format_number Sub-task Resolved Cheng Hao Actions
        67.
        		 string function: get_json_object Sub-task Resolved Nathan Howell Actions
        68.
        		 string function: instr Sub-task Resolved Cheng Hao Actions
        69.
        		 string function: length Sub-task Resolved Cheng Hao Actions
        70.
        		 string function: locate Sub-task Resolved Cheng Hao Actions
        71.
        		 string function: alias lower/lcase Sub-task Resolved Reynold Xin Actions
        72.
        		 string function: alias upper / ucase Sub-task Resolved Reynold Xin Actions
        73.
        		 string function: lpad Sub-task Resolved Cheng Hao Actions
        74.
        		 string function: ltrim Sub-task Resolved Cheng Hao Actions
        75.
        		 string function: printf Sub-task Resolved Cheng Hao Actions
        76.
        		 string function: regexp_extract Sub-task Resolved Cheng Hao Actions
        77.
        		 string function: regexp_replace Sub-task Resolved Cheng Hao Actions
        78.
        		 string function: repeat Sub-task Resolved Cheng Hao Actions
        79.
        		 string function: reverse Sub-task Resolved Cheng Hao Actions
        80.
        		 string function: rpad Sub-task Resolved Cheng Hao Actions
        81.
        		 string function: rtrim Sub-task Resolved Cheng Hao Actions
        82.
        		 string function: space Sub-task Resolved Cheng Hao Actions
        83.
        		 string function: split Sub-task Resolved Cheng Hao Actions
        84.
        		 string function: substr/substring should also support binary type Sub-task Resolved Cheng Hao Actions
        85.
        		 string function: substring_index Sub-task Resolved Cheng Hao Actions
        86.
        		 string function: trim Sub-task Resolved Cheng Hao Actions
        87.
        		 string function: unbase64 Sub-task Resolved Cheng Hao Actions
        88.
        		 string function: initcap Sub-task Resolved Cheng Hao Actions
        89.
        		 string function: levenshtein Sub-task Resolved Tarek Auel Actions
        90.
        		 string function: soundex Sub-task Resolved Cheng Hao Actions
        91.
        		 udf_round_3 test fails Sub-task Resolved Yijie Shen Actions
        92.
        		 udf_struct test failure Sub-task Resolved Yijie Shen Actions
        93.
        		 Add unit tests for abs Sub-task Resolved Reynold Xin Actions
        94.
        		 Add unit tests for +, -, *, /, % Sub-task Resolved Reynold Xin Actions
        95.
        		 Move sqrt into math Sub-task Resolved L. C. Hsieh Actions
        96.
        		 improve unit test for MaxOf and MinOf Sub-task Resolved Wenchen Fan Actions
        97.
        		 complex type constructors: struct and named_struct Sub-task Resolved Yijie Shen Actions
        98.
        		 Remove e and pi from DataFrame functions Sub-task Resolved Reynold Xin Actions
        99.
        		 Add python API for hex/unhex Sub-task Resolved Davies Liu Actions
        100.
        		 Date/time function and data type design Sub-task Resolved Reynold Xin Actions
        101.
        		 Improve unit test coverage for bitwise expressions Sub-task Resolved Reynold Xin Actions
        102.
        		 MonotonicallyIncreasingID and SparkPartitionID should be marked as nondeterministic Sub-task Resolved Reynold Xin Actions
        103.
        		 Improve unit test coverage for null expressions Sub-task Resolved Reynold Xin Actions
        104.
        		 use UTC Calendar in `stringToDate` Sub-task Resolved Wenchen Fan Actions
        105.
        		 add and improve tests for nondeterministic expressions Sub-task Resolved Wenchen Fan Actions
        106.
        		 Add unit test for null inputs for date functions Sub-task Resolved Yijie Shen Actions
        107.
        		 Add test cases for null inputs for expression unit tests Sub-task Resolved Yijie Shen Actions
        108.
        		 Audit expression unit tests to make sure we pass the proper numeric ranges Sub-task Resolved Yijie Shen Actions
        109.
        		 Audit expression unit tests to test for non-foldable codegen path Sub-task Resolved Yijie Shen Actions
        110.
        		 Add TernaryExpression to simplify implementations Sub-task Resolved Davies Liu Actions
        111.
        		 Create Python API for all SQL functions Sub-task Resolved Davies Liu Actions
        112.
        		 DateTimeUtils cleanup Sub-task Resolved Yijie Shen Actions
        113.
        		 minor bug fix in expressions Sub-task Resolved Yijie Shen Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rxin Reynold Xin
            rxin Reynold Xin
            Votes:
            0 Vote for this issue
            Watchers:
            19 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Agile

                Completed Sprint:
                Spark 1.5 release ended 14/Aug/15
                View on Board

                Slack

                  Issue deployment