[CALCITE-6498] Elasticsearch multi-field mappings do not work

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.38.0
• Component/s: elasticsearch-adapter
• Labels: pull-request-available
• Assignee: Tim Grein
• Reporter: Tim Grein
• Created: 24/Jul/24 14:28
• Updated: 5 days ago 22:41
• Resolved: 5 days ago 22:41

Description

EmbeddedElasticsearchPolicy#
applyMapping tries to apply a multi-field mapping by adding another properties field inside a parent field mapping. In Elasticsearch you need to use fields rather than properties for a multi-field mapping to work.
 
I've tested this with ES 7 and ES 6 (only including ES 7 example here in the issue description; for ES 6 you need to wrap everything under "mappings" in "_doc"):

ES 7:

PUT /some_index
{
  "mappings": {
    "properties": {
      "some_field": {
        "type": "text",
        "properties": {
          "keyword": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

will lead to the following error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "Mapping definition for [some_field] has unsupported parameters:  [properties : {keyword={type=keyword}}]"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "Failed to parse mapping [_doc]: Mapping definition for [some_field] has unsupported parameters:  [properties : {keyword={type=keyword}}]",
    "caused_by" : {
      "type" : "mapper_parsing_exception",
      "reason" : "Mapping definition for [some_field] has unsupported parameters:  [properties : {keyword={type=keyword}}]"
    }
  },
  "status" : 400
} 

 
Successful request:

PUT /some_index
{
  "mappings": {
    "properties": {
      "some_field": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      }
    }
  }
} 

You'll encounter this error also, if you adapt the test data in ElasticsearchAdapterTest to include nested fields:

@BeforeAll
public static void setupInstance() throws Exception {
  // "city.keyword" is a nested field with type "keyword"
  final Map<String, String> mapping =
      ImmutableMap.of("city", "text", "city.keyword", "keyword", "state", "keyword", "pop", "long"); 

Error:

{...

{"type":"mapper_parsing_exception","reason":"unknown parameter [properties] on mapper [city] of type [text]"}

...} 

Looking at ElasticsearchJson#visitMappingProperties I assume this will lead to a similar issue, which I'll double-check (we probably need to distinguish between nested fields and multi-field mappings explicitly).

This is related to https://issues.apache.org/jira/browse/CALCITE-3027 as you want to detect, if you perform a "LIKE" operator on a purely "text" mapped field, which will lead to weird semantics ("text" mapped fields are analyzed/broken up into several tokens), if you do not prevent it. Usually you have a multi-field mapping for a field containing text ("text" mapping for full text search and a "keyword" mapping you can use for aggregations, wildcard queries etc.). So this is rather important to work correctly overall and in the tests.

Issue Links

• blocks: CALCITE-6582 Release Calcite 1.38.0
• links to: GitHub Pull Request #3885