[SPARK-10382] Implement a Jekyll tag to include example code in user guide

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Documentation, ML, MLlib
• Target Version/s: 1.6.0
• Assignee: Xusen Yin
• Reporter: Xiangrui Meng
• Shepherd: Xiangrui Meng
• Created: 01/Sep/15 00:22
• Updated: 28/Nov/15 15:26
• Resolved: 23/Oct/15 15:31

Description

The example code in the user guide is embedded in the markdown and hence it is not easy to test. It would be nice to automatically test them. This JIRA is to discuss options to automate example code testing and see what we can do in Spark 1.6.

One option I propose is to move actual example code to spark/examples and test compilation in Jenkins builds. Then in the markdown, we can reference part of the code to show in the user guide. This requires adding a Jekyll tag that is similar to https://github.com/jekyll/jekyll/blob/master/lib/jekyll/tags/include.rb, e.g., called include_example.

{% include_example scala ml.KMeansExample guide %}

Jekyll will find `examples/src/main/scala/org/apache/spark/examples/ml/KMeansExample.scala` and pick code blocks marked "guide" and put them under `

{% highlight %}

` in the markdown. We can discuss the syntax for marker comments.

Just one way to implement this. It would be nice to hear more ideas.

Issue Links

• is related to: SPARK-11336 Include path to the source file in generated example code
• is related to: SPARK-11289 Substitute code examples of tf-idf and word2vec with include_example
• is related to: SPARK-11297 code example generated by include_example is not exactly the same with {% highlight %}
• is related to: SPARK-10383 Sync example code between API doc and user guide
• relates to: SPARK-12035 Add more debug information in include_example tag of Jekyll
• links to: [Github] Pull Request #9109 (yinxusen)