Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16257

Intermittent issue with incorrect resultset with Spark



    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.1.0
    • None
    • Hive
    • None


      This issue is highly intermittent that only seems to occurs with spark engine when the query has a GROUPBY clause. The following is the testcase.

      drop table if exists test_hos_sample;
      create table test_hos_sample (name string, val1 decimal(18,2), val2 decimal(20,3));
      insert into test_hos_sample values ('test1',101.12,102.123),('test1',101.12,102.123),('test2',102.12,103.234),('test1',101.12,102.123),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test3',103.52,102.345),('test4',104.52,104.456),('test4',104.52,104.456),('test5',105.52,105.567),('test3',103.52,102.345),('test5',105.52,105.567);
      set hive.execution.engine=spark;
      select  name, val1,val2 from test_hos_sample group by name, val1, val2;

      Expected Results:

      name    val1    val2
      test5   105.52  105.567
      test3   103.52  102.345
      test1   101.12  102.123
      test4   104.52  104.456
      test2   102.12  103.234

      Incorrect results once in a while:

      name    val1    val2
      test5   105.52  105.567
      test3   103.52  102.345
      test1   104.52  102.123
      test4   104.52  104.456
      test2   102.12  103.234

      1) Not reproducible with HoMR.
      2) Not an issue when running from spark-shell.
      3) Not reproducible when the column data type is String or double. Only reproducible with decimal data types. Also works fine for decimal datatype if you cast decimal as string on read and cast it back to decimal on select.
      4) Occurs with parquet and text file format as well. (havent tried with other formats).
      5) Occurs in both scenarios when table data is within encryption zone and outside.
      6) Even in clusters where this is reproducible, this occurs once in like 20 times or more.
      7) Occurs with both Beeline and Hive CLI.
      8) Reproducible only when there is a a groupby clause.


        1. failed_yarnlogs.log
          249 kB
          Naveen Gangam
        2. success_yarnlogs.log
          178 kB
          Naveen Gangam

        Issue Links



              ngangam Naveen Gangam
              ngangam Naveen Gangam
              0 Vote for this issue
              6 Start watching this issue