Uploaded image for project: 'Apache HAWQ'
  1. Apache HAWQ
  2. HAWQ-1674

Pxf error "expecting union description" when accessing partitioned hive table with avro schema

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: PXF
    • Labels:
      None

      Description

      Hi,

       

      i created a partitioned table in hive with an avro schema

      {
          "type"     : "record",
          "name"     : "DavidSimpleData",
          "namespace": "namespace",
          "doc"      : "TestSchema",
          "fields"   : [
              {
                  "name": "meta_extracted_at_time",
                  "type": "long"
              },
              {
                  "name": "status",
                  "type": {
                      "type"   : "enum",
                      "name"   : "ToDoStatus",
                      "symbols": [
                          "HIDDEN",
                          "ACTIONABLE",
                          "DONE",
                          "ARCHIVED",
                          "DELETED"
                      ]
                  }
              }
          ]
      }

       

      create external table staging.david_simple
      PARTITIONED BY(year int, month int)
      row format serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
      stored as inputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
      outputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
      LOCATION '/user/talend/staging/david_simple'
      tblproperties ('avro.schema.url'='http://schemaservice-stage.local:2876/schema-repo/david_simple/id/4')

       

      The hawq table is created with

      create external table history.ext_test_david_simple (
          meta_extracted_at_time bigint,
          status varchar (100),
          year int,
          month int
          ) LOCATION ('pxf://pim01.local:51200/staging.david_simple?PROFILE=Hive')
          FORMAT 'custom' (formatter = 'pxfwritable_import');

       

      When selecting from the table The hawq error i get is

       

      [XX000] ERROR: remote component error (500) from 'ip:51200': type Exception report message Found namespace.ToDoStatus, expecting union description The server encountered an internal error that prevented it from fulfilling this request. exception java.io.IOException: Found namespace.ToDoStatus, expecting union (libchurl.c:897) (seg11 pid03.cyberservices.local:40000 pid=371019) (dispatcher.c:1805) Detail: External table ext_test_david_simple

       

      Selecting on hive give correct results so it seems the pxf is producing the error somehow.

       

        Attachments

          Activity

            People

            • Assignee:
              espino Ed Espino
              Reporter:
              Xcellence David
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: