Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10801

Fix sql client integrate elasticsearch connector test failure

    XMLWordPrintableJSON

    Details

      Description

      It usually reports : 

      FAIL SQL Client Elasticsearch Upsert: Output hash mismatch. Got 6187222e109ee9222e6b2f117742070c, expected 982cb32908def9801e781381c1b8a8db.
      head hexdump of actual:
      0000000 { \n " h i t s " : { \n 
      0000010 " t o t a l " : 3 , \n
      0000020 " m a x _ s c o r e " 
      0000030 : 1 . 0 , \n " h i t s
      0000040 " : [ \n { \n 
      0000050 " _ i n d e x " :
      0000060 " m y _ u s e r s " , \n 
      0000070 " _ t y p e " : "
      0000080 u s e r " , \n "
      0000090 _ i d " : " 1 _ B o b "
      00000a0 , \n " _ s c o r
      00000b0 e " : 1 . 0 , \n 
      00000ba
      

      the actual hash means : 

      {
        "hits" : {
          "total" : 3,
          "max_score" : 1.0,
          "hits" : [
            {
              "_index" : "my_users",
              "_type" : "user",
              "_id" : "1_Bob  ",
              "_score" : 1.0,
              "_source" : {
                "user_id" : 1,
                "user_name" : "Bob  ",
                "user_count" : 1
              }
            },
            {
              "_index" : "my_users",
              "_type" : "user",
              "_id" : "22_Alice",
              "_score" : 1.0,
              "_source" : {
                "user_id" : 22,
                "user_name" : "Alice",
                "user_count" : 1
              }
            },
            {
              "_index" : "my_users",
              "_type" : "user",
              "_id" : "42_Greg ",
              "_score" : 1.0,
              "_source" : {
                "user_id" : 42,
                "user_name" : "Greg ",
                "user_count" : 3
              }
            }
          ]
        }
      }
      
      

      the expected hash code means : 

      {
        "hits" : {
          "total" : 3,
          "max_score" : 1.0,
          "hits" : [
            {
              "_index" : "my_users",
              "_type" : "user",
              "_id" : "1_Bob  ",
              "_score" : 1.0,
              "_source" : {
                "user_id" : 1,
                "user_name" : "Bob  ",
                "user_count" : 2
              }
            },
            {
              "_index" : "my_users",
              "_type" : "user",
              "_id" : "22_Alice",
              "_score" : 1.0,
              "_source" : {
                "user_id" : 22,
                "user_name" : "Alice",
                "user_count" : 1
              }
            },
            {
              "_index" : "my_users",
              "_type" : "user",
              "_id" : "42_Greg ",
              "_score" : 1.0,
              "_source" : {
                "user_id" : 42,
                "user_name" : "Greg ",
                "user_count" : 3
              }
            }
          ]
        }
      }
      

      It seems that the user count for "Bob" is off by 1.

      The speculation is due to the premature acquisition of aggregated statistics from Elasticsearch.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pnowojski Piotr Nowojski
                Reporter:
                yanghua vinoyang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: