[FLINK-10801] Fix sql client integrate elasticsearch connector test failure

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Component/s: Test Infrastructure
• Labels: pull-request-available
• Assignee: Piotr Nowojski
• Reporter: vinoyang
• Created: 06/Nov/18 09:36
• Updated: 09/Nov/18 08:54
• Resolved: 09/Nov/18 08:54

Description

It usually reports : 

FAIL SQL Client Elasticsearch Upsert: Output hash mismatch. Got 6187222e109ee9222e6b2f117742070c, expected 982cb32908def9801e781381c1b8a8db.
head hexdump of actual:
0000000 { \n " h i t s " : { \n 
0000010 " t o t a l " : 3 , \n
0000020 " m a x _ s c o r e " 
0000030 : 1 . 0 , \n " h i t s
0000040 " : [ \n { \n 
0000050 " _ i n d e x " :
0000060 " m y _ u s e r s " , \n 
0000070 " _ t y p e " : "
0000080 u s e r " , \n "
0000090 _ i d " : " 1 _ B o b "
00000a0 , \n " _ s c o r
00000b0 e " : 1 . 0 , \n 
00000ba

the actual hash means : 

{
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my_users",
        "_type" : "user",
        "_id" : "1_Bob  ",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 1,
          "user_name" : "Bob  ",
          "user_count" : 1
        }
      },
      {
        "_index" : "my_users",
        "_type" : "user",
        "_id" : "22_Alice",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 22,
          "user_name" : "Alice",
          "user_count" : 1
        }
      },
      {
        "_index" : "my_users",
        "_type" : "user",
        "_id" : "42_Greg ",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 42,
          "user_name" : "Greg ",
          "user_count" : 3
        }
      }
    ]
  }
}

the expected hash code means : 

{
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my_users",
        "_type" : "user",
        "_id" : "1_Bob  ",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 1,
          "user_name" : "Bob  ",
          "user_count" : 2
        }
      },
      {
        "_index" : "my_users",
        "_type" : "user",
        "_id" : "22_Alice",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 22,
          "user_name" : "Alice",
          "user_count" : 1
        }
      },
      {
        "_index" : "my_users",
        "_type" : "user",
        "_id" : "42_Greg ",
        "_score" : 1.0,
        "_source" : {
          "user_id" : 42,
          "user_name" : "Greg ",
          "user_count" : 3
        }
      }
    ]
  }
}

It seems that the user count for "Bob" is off by 1.

The speculation is due to the premature acquisition of aggregated statistics from Elasticsearch.

 

Issue Links

• links to: GitHub Pull Request #7060