Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4051

build cube met the InvalidProtocolBufferException

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: v2.1.0
    • Fix Version/s: None
    • Component/s: Job Engine
    • Labels:
    • Environment:
      Kylin 2.1, hbase 1.2.0-cdh5.10.1, hadoop-2.5.0-cdh5.3.2, hive 1.2.1

      Description

      Dears,

      When I tried to build a cube, met the following errors, does anyone meet the same error before, hope to get your feedback soon, thanks in advance~

      Error: java.io.IOException: java.lang.reflect.InvocationTargetException
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:295)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:242)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:356)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:591)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:438)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at java.security.AccessController.doPrivileged(Native Method)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at javax.security.auth.Subject.doAs(Subject.java:422)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
      2019-06-19 15:57:05,569 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : Caused by: java.lang.reflect.InvocationTargetException
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:281)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : ... 11 more
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : Caused by: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: streams[3].kind, streams[8].kind, streams[11].kind, streams[14
      ].kind, streams[21].kind, streams[24].kind, streams[28].kind

      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:71)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.parseFrom(OrcProto.java:8878)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripeFooter(RecordReaderImpl.java:2174)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2505)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2949)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2991)
      2019-06-19 15:57:05,570 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:284)
      2019-06-19 15:57:05,571 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:480)
      2019-06-19 15:57:05,571 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:214)
      2019-06-19 15:57:05,571 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:146)
      2019-06-19 15:57:05,571 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:997)
      2019-06-19 15:57:05,571 INFO [Job 5e7639c4-660a-46dd-852e-62bf3fc4a713-1559] hive.CreateFlatHiveTableStep:38 : at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)

       

       

       

       

      followings is cube json

      { "uuid": "1300457a-2ef9-415e-bf90-9fa310b71b26", "last_modified": 1560930896836, "version": "2.1.0", "name": "sales_value_details_info_day", "is_draft": false, "model_name": "sales_value_details_info_day", "description": "sales_value_details_info_day", "null_string": null, "dimensions": [ \{ "name": "PO_NO", "table": "SALES_VALUE_DETAILS_INFO_DAY", "column": "PO_NO", "derived": null }

      ,
      {
      "name": "ITEM_CODE",
      "table": "SALES_VALUE_DETAILS_INFO_DAY",
      "column": "ITEM_CODE",
      "derived": null
      },
      {
      "name": "ITEM_DESC",
      "table": "SALES_VALUE_DETAILS_INFO_DAY",
      "column": "ITEM_DESC",
      "derived": null
      },
      {
      "name": "BRAND_CODE",
      "table": "SALES_VALUE_DETAILS_INFO_DAY",
      "column": "BRAND_CODE",
      "derived": null
      },
      {
      "name": "BRAND_NAME",
      "table": "SALES_VALUE_DETAILS_INFO_DAY",
      "column": "BRAND_NAME",
      "derived": null
      },
      {
      "name": "DT",
      "table": "SALES_VALUE_DETAILS_INFO_DAY",
      "column": "DT",
      "derived": null
      }
      ],
      "measures": [
      {
      "name": "COUNT",
      "function": {
      "expression": "COUNT",
      "parameter":

      { "type": "constant", "value": "1" }

      ,
      "returntype": "bigint"
      }
      },
      {
      "name": "NET_SALES_COST",
      "function": {
      "expression": "SUM",
      "parameter":

      { "type": "column", "value": "SALES_VALUE_DETAILS_INFO_DAY.NET_SALES_COST" }

      ,
      "returntype": "decimal(19,4)"
      }
      },
      {
      "name": "NET_SALES_MONEY",
      "function": {
      "expression": "SUM",
      "parameter":

      { "type": "column", "value": "SALES_VALUE_DETAILS_INFO_DAY.NET_SALES_MONEY" }

      ,
      "returntype": "decimal(19,4)"
      }
      },
      {
      "name": "SALES_COST",
      "function": {
      "expression": "SUM",
      "parameter":

      { "type": "column", "value": "SALES_VALUE_DETAILS_INFO_DAY.SALES_COST" }

      ,
      "returntype": "decimal(19,4)"
      }
      }
      ],
      "dictionaries": [],
      "rowkey": {
      "rowkey_columns": [

      { "column": "SALES_VALUE_DETAILS_INFO_DAY.PO_NO", "encoding": "dict", "isShardBy": false }

      ,
      {
      "column": "SALES_VALUE_DETAILS_INFO_DAY.ITEM_CODE",
      "encoding": "dict",
      "isShardBy": false
      },
      {
      "column": "SALES_VALUE_DETAILS_INFO_DAY.ITEM_DESC",
      "encoding": "dict",
      "isShardBy": false
      },
      {
      "column": "SALES_VALUE_DETAILS_INFO_DAY.BRAND_CODE",
      "encoding": "dict",
      "isShardBy": false
      },
      {
      "column": "SALES_VALUE_DETAILS_INFO_DAY.BRAND_NAME",
      "encoding": "dict",
      "isShardBy": false
      },
      {
      "column": "SALES_VALUE_DETAILS_INFO_DAY.DT",
      "encoding": "dict",
      "isShardBy": false
      }
      ]
      },
      "hbase_mapping": {
      "column_family": [
      {
      "name": "F1",
      "columns": [

      { "qualifier": "M", "measure_refs": [ "_COUNT_", "NET_SALES_COST", "NET_SALES_MONEY", "SALES_COST" ] }

      ]
      }
      ]
      },
      "aggregation_groups": [
      {
      "includes": [
      "SALES_VALUE_DETAILS_INFO_DAY.PO_NO",
      "SALES_VALUE_DETAILS_INFO_DAY.ITEM_CODE",
      "SALES_VALUE_DETAILS_INFO_DAY.ITEM_DESC",
      "SALES_VALUE_DETAILS_INFO_DAY.BRAND_CODE",
      "SALES_VALUE_DETAILS_INFO_DAY.BRAND_NAME",
      "SALES_VALUE_DETAILS_INFO_DAY.DT"
      ],
      "select_rule":

      { "hierarchy_dims": [], "mandatory_dims": [ "SALES_VALUE_DETAILS_INFO_DAY.DT" ], "joint_dims": [] }

      }
      ],
      "signature": "xKEMvizWVPEl6EMWHB+SGg==",
      "notify_list": [],
      "status_need_notify": [
      "ERROR",
      "DISCARDED",
      "SUCCEED"
      ],
      "partition_date_start": 1560643200000,
      "partition_date_end": 3153600000000,
      "auto_merge_time_ranges": [],
      "retention_range": 0,
      "cuboid_cut_size_mb": null,
      "exclude_cuboid_layer": null,
      "engine_type": 2,
      "storage_type": 2,
      "override_kylin_properties": {
      "kylin.engine.mr.config-override.fs.defaultFS": "hdfs://bipcluster",
      "kylin.engine.mr.config-override.mapreduce.jobhistory.address": "sd-hadoop-journalnode-71-21.idc.vip.com:10020",
      "kylin.engine.mr.config-override.mapreduce.jobhistory.webapp.address": "sd-hadoop-journalnode-71-21.idc.vip.com:19888"
      },
      "cuboid_black_list": [],
      "parent_forward": 3
      }

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              xiangakun xiangakun
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: