Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21914

Running examples as tests in SQL builtin function documentation

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      It looks we have added many examples in ExpressionDescription for builtin functions.
      Actually, if I have seen correctly, we have fixed many examples so far in some minor PRs and sometimes require to add the examples as tests sql and golden files.

      As we have formatted examples in ExpressionDescription.examples - https://github.com/apache/spark/blob/ba327ee54c32b11107793604895bd38559804858/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java#L44-L50, and we have `SQLQueryTestSuite`, I think we could run the examples as tests like Python's doctests.

      Rough way I am thinking:

      1. Loads the example in ExpressionDescription.
      2. identify queries by >.
      3. identify the rest of them as the results.
      4. run the examples by reusing SQLQueryTestSuite if possible.
      5. compare the output by reusing SQLQueryTestSuite if possible.

      Advantages of doing this I could think for now:

      • Reduce the number of PRs to fix the examples
      • De-duplicate the test cases that should be added into sql and golden files.
      • Correct documentation with correct examples.
      • Reduce reviewing costs for documentation fix PRs.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maxgekk Max Gekk
                Reporter:
                hyukjin.kwon Hyukjin Kwon
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: