Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11337

Make example code in user guide testable

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: Documentation
    • Labels:
      None

      Description

      The example code in the user guide is embedded in the markdown and hence it is not easy to test. It would be nice to automatically test them. This JIRA is to discuss options to automate example code testing and see what we can do in Spark 1.6.

      One option I propose is to move actual example code to spark/examples and test compilation in Jenkins builds. Then in the markdown, we can reference part of the code to show in the user guide. This requires adding a Jekyll tag that is similar to https://github.com/jekyll/jekyll/blob/master/lib/jekyll/tags/include.rb, e.g., called include_example.

      {% include_example scala/org/apache/spark/examples/ml/KMeansExample.scala %}
      

      Jekyll will find `examples/src/main/scala/org/apache/spark/examples/ml/KMeansExample.scala` and pick code blocks marked "example" and put them under `

      {% highlight %}

      ` in the markdown. We can discuss the syntax for marker comments.

      Sub-tasks are created to move example code from user guide to `examples/`.

      self-check list for contributors in this JIRA

      • Be sure to match Scala/Java/Python code style guide. If unsure of a code style, please refer to other merged example code under examples/.
      • Remove useless imports
      • It's better to have a side-effect operation at the end of each example code, usually it's a
        print(...)
      • Make sure the code example is runnable without error.
      • After finishing code migration, use
        cd docs; SKIP_API=1 jekyll serve

        the check the webpage at http://127.0.0.1:4000 to see whether the generated html looks good.

        Attachments

          Issue Links

          1.
          Implement a Jekyll tag to include example code in user guide Sub-task Resolved Xusen Yin
          2.
          Substitute code examples of tf-idf and word2vec with include_example Sub-task Resolved Xusen Yin
          3.
          Add link to the source file at the end of included example code Sub-task Closed Xusen Yin
          4.
          Replace example code in mllib-frequent-pattern-mining.md using include_example Sub-task Resolved Pravin Gadakh
          5.
          Replace example code in mllib-linear-methods.md using include_example Sub-task Resolved Dongjoon Hyun
          6.
          Replace example code in mllib-decision-tree.md using include_example Sub-task Resolved Xusen Yin
          7.
          Include_example should support labels to cut out different parts in one example code Sub-task Closed Unassigned
          8.
          Replace example code in mllib-naive-bayes.md/mllib-isotonic-regression.md using include_example Sub-task Resolved Rishabh Bhardwaj
          9.
          Blank lines should be reserved in include_example Sub-task Resolved Xusen Yin
          10.
          Replace example code in mllib-ensembles.md using include_example Sub-task Resolved Rishabh Bhardwaj
          11.
          Include path to the source file in generated example code Sub-task Resolved Xusen Yin
          12.
          Replace example code in mllib-collaborative-filtering.md using include_example Sub-task Resolved Rishabh Bhardwaj
          13.
          Replace example code in mllib-evaluation-metrics.md using include_example Sub-task Resolved Vikas Nelamangala
          14.
          Replace example code in mllib-optimization.md using include_example Sub-task Resolved Pravin Gadakh
          15.
          Replace example code in ml-features.md using include_example Sub-task Resolved somil deshmukh
          16.
          Replace example code in ml-decision-tree.md using include_example Sub-task Resolved Sachin Aggarwal
          17.
          Replace example code in ml-ensembles.md using include_example Sub-task Resolved Xusen Yin
          18.
          Replace example code in ml-linear-methods.md using include_example Sub-task Resolved Xusen Yin
          19.
          Follow-up: Refine example code in ml-features.md Sub-task Resolved Xusen Yin
          20.
          Replace example code in ml-guide.md using include_example Sub-task Resolved Devaraj K
          21.
          Replace example code in mllib-clustering.md using include_example Sub-task Resolved Xin Ren
          22.
          Replace example code in mllib-collaborative-filtering.md using include_example Sub-task Closed Xin Ren
          23.
          Replace example code in mllib-data-types.md using include_example Sub-task Resolved Xin Ren
          24.
          Replace example code in mllib-dimensionality-reduction.md using include_example Sub-task Resolved Devaraj K
          25.
          Replace example code in mllib-feature-extraction.md using include_example Sub-task Resolved Xin Ren
          26.
          Replace example code in mllib-pmml-model-export.md using include_example Sub-task Resolved Xin Ren
          27.
          Replace example code in mllib-statistics.md using include_example Sub-task Resolved Xin Ren
          28.
          Missing Python code example in model selection user guide Sub-task Resolved Jeremy
          29.
          Vector serialization error in example code of ModelSelectionViaTrainValidationSplitExample and JavaModelSelectionViaTrainValidationSplitExample Sub-task Closed Unassigned

            Activity

              People

              • Assignee:
                yinxusen Xusen Yin
                Reporter:
                mengxr Xiangrui Meng
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: