Hive
  1. Hive
  2. HIVE-6984

Analyzing partitioned table with NULL values for the partition column failed with NPE

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.14.0
    • Component/s: Statistics
    • Labels:
      None

      Description

      The following describes how to produce the bug:

      hive> desc test2;
      name                	string              	                    
      age                 	int                 	                    
      
      hive> select * from test2;
      6666666666666666666	NULL
      5555555555555555555	NULL
      tom	15
      john	NULL
      mayr	40
      	30
      	NULL
      
      hive> create table test3(name string) partitioned by (age int);
      
      hive> from test2 insert overwrite table test3 partition(age) select test2.name, test2.age;
      Loading data to table default.test3 partition (age=null)
      	Loading partition {age=40}
      	Loading partition {age=__HIVE_DEFAULT_PARTITION__}
      	Loading partition {age=30}
      	Loading partition {age=15}
      Partition default.test3{age=15} stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
      Partition default.test3{age=30} stats: [numFiles=1, numRows=1, totalSize=1, rawDataSize=0]
      Partition default.test3{age=40} stats: [numFiles=1, numRows=1, totalSize=5, rawDataSize=4]
      Partition default.test3{age=__HIVE_DEFAULT_PARTITION__} stats: [numFiles=1, numRows=4, totalSize=46, rawDataSize=42]
      
      hive> analyze table test3 partition(age) compute statistics;
      ...
      Task with the most failures(4): 
      -----
      Diagnostic Messages for this Task:
      java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
      	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
      	at org.apache.hadoop.mapred.Child.main(Child.java:262)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
      	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
      	at org.apache.hado
      
      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
      

      The following is the stack trace in mapper log:

      2014-04-28 15:39:25,073 FATAL org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
      	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
      	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
      	at org.apache.hadoop.mapred.Child.main(Child.java:262)
      Caused by: java.lang.NullPointerException
      	at org.apache.hadoop.hive.ql.exec.TableScanOperator.gatherStats(TableScanOperator.java:149)
      	at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
      	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
      	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
      	... 9 more
      
      1. HIVE-6984.1.patch
        18 kB
        Xuefu Zhang
      2. HIVE-6984.2.patch
        49 kB
        Xuefu Zhang
      3. HIVE-6984.patch
        8 kB
        Xuefu Zhang

        Activity

        Xuefu Zhang created issue -
        Xuefu Zhang made changes -
        Field Original Value New Value
        Description The following describes how to produce the bug:
        {code}
        hive> desc test2;
        name string
        age int

        hive> select * from test2;
        Query ID = xzhang_20140428154343_bb921bce-7282-47b6-bfe8-669c7edaabe7
        OK
        6666666666666666666 NULL
        5555555555555555555 NULL
        tom 15
        john NULL
        mayr 40
        30
        NULL

        hive> create table test3(name string) partitioned by (age int);

        hive> from test2 insert overwrite table test3 partition(age) select test2.name, test2.age;
        Loading data to table default.test3 partition (age=null)
        Loading partition {age=40}
        Loading partition {age=__HIVE_DEFAULT_PARTITION__}
        Loading partition {age=30}
        Loading partition {age=15}
        Partition default.test3{age=15} stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
        Partition default.test3{age=30} stats: [numFiles=1, numRows=1, totalSize=1, rawDataSize=0]
        Partition default.test3{age=40} stats: [numFiles=1, numRows=1, totalSize=5, rawDataSize=4]
        Partition default.test3{age=__HIVE_DEFAULT_PARTITION__} stats: [numFiles=1, numRows=4, totalSize=46, rawDataSize=42]

        hive> analyze table test3 partition(age) compute statistics;

        Task with the most failures(4):
        -----
        Task ID:
          task_201403041834_0063_m_000000

        -----
        Diagnostic Messages for this Task:
        java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
        Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
        at org.apache.hado

        FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
        {code}

        The following is the stack trace in mapper log:
        {code}
        2014-04-28 15:39:25,073 FATAL org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
        Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.gatherStats(TableScanOperator.java:149)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
        ... 9 more
        {code}
        The following describes how to produce the bug:
        {code}
        hive> desc test2;
        name string
        age int

        hive> select * from test2;
        6666666666666666666 NULL
        5555555555555555555 NULL
        tom 15
        john NULL
        mayr 40
        30
        NULL

        hive> create table test3(name string) partitioned by (age int);

        hive> from test2 insert overwrite table test3 partition(age) select test2.name, test2.age;
        Loading data to table default.test3 partition (age=null)
        Loading partition {age=40}
        Loading partition {age=__HIVE_DEFAULT_PARTITION__}
        Loading partition {age=30}
        Loading partition {age=15}
        Partition default.test3{age=15} stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
        Partition default.test3{age=30} stats: [numFiles=1, numRows=1, totalSize=1, rawDataSize=0]
        Partition default.test3{age=40} stats: [numFiles=1, numRows=1, totalSize=5, rawDataSize=4]
        Partition default.test3{age=__HIVE_DEFAULT_PARTITION__} stats: [numFiles=1, numRows=4, totalSize=46, rawDataSize=42]

        hive> analyze table test3 partition(age) compute statistics;
        ...
        Task with the most failures(4):
        -----
        Diagnostic Messages for this Task:
        java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
        Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
        at org.apache.hado

        FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
        {code}

        The following is the stack trace in mapper log:
        {code}
        2014-04-28 15:39:25,073 FATAL org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
        Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.gatherStats(TableScanOperator.java:149)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
        ... 9 more
        {code}
        Xuefu Zhang made changes -
        Assignee Xuefu Zhang [ xuefuz ]
        Xuefu Zhang made changes -
        Attachment HIVE-6984.patch [ 12642543 ]
        Xuefu Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Xuefu Zhang made changes -
        Attachment HIVE-6984.1.patch [ 12642553 ]
        Xuefu Zhang made changes -
        Attachment HIVE-6984.2.patch [ 12642690 ]
        Xuefu Zhang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.14.0 [ 12326450 ]
        Resolution Fixed [ 1 ]
        Thejas M Nair made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Xuefu Zhang
            Reporter:
            Xuefu Zhang
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development