[SPARK-25232] Support Full-Text Search in Spark SQL

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.1
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Lijie Xu
• Created: 24/Aug/18 15:59
• Updated: 12/Dec/22 18:11
• Resolved: 08/Oct/19 05:44

Description

Full-text search (i.e., keyword search) is widely used in search engines and relational databases such as MATCH() ... AGAINST operator in MySQL (https://dev.mysql.com/doc/en/fulltext-search.html), Text query in Oracle (https://docs.oracle.com/cd/B28359_01/text.111/b28303/query.htm#g1016054), and text search in PostgreSQL (https://www.postgresql.org/docs/9.5/static/textsearch.html). However, it is not natively supported in Spark SQL. We propose an approach to implement this full-text search in Spark SQL.

Our proposed approach is detailed  at https://github.com/JerryLead/Misc/blob/master/FullTextSearch/Full-text-issue-2018.pdf

and the prototype is available at https://github.com/bigdata-iscas/SparkFullTextQuery/tree/like_explorer