Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13203

STRUCT type not allowing more than 266 fields. Table get created with more fields but throws error while querying against the table. SHOW CREATE TABLE skips having some fields in STRUCT.

    Details

    • Type: Bug
    • Status: Open
    • Priority: Trivial
    • Resolution: Unresolved
    • Affects Version/s: 1.2.0
    • Fix Version/s: None
    • Component/s: Database/Schema
    • Labels:
      None

      Description

      I am trying to create a table with multiple STRUCT column types, one of which having more than 300 fields.

      I get the table created with no problem, but when i load i face few errors in loading up the data.when i check SHOW CREATE TABLE TABLENAME, i get fields only till 266th column and rest of them do not appear in the table DDL.(check the issue at StateOrProvinc, search for StateOrProvinc in the second DDL).

      Would like to know, if there is any restriction on the number of fields to be using in STRUCT datatype.

      EXAMPLE:

      BELOW IS THE DDL i run in hive and the second below DLL is the TABLE DDL i get from SHOW CREATE TABLE.

      CREATE TABLE property_feature.property_feature_new(
      `_id` STRUCT<`mls_sysid`:String,`type`:String>,
      all_attrs STRUCT<`1`: String ,`10`: String ,`100`: String ,`105`: String ,`113`: String ,`118`: String ,`126`: String ,`1299`: String ,`1301`: String ,`1305`: String ,`1307`: String ,`1319`: String ,`1325`: String ,`1327`: String ,`1329`: String ,`1339`: String ,`1341`: String ,`1345`: String ,`1351`: String ,`1353`: String ,`1359`: String ,`1363`: String ,`1365`: String ,`1367`: String ,`1373`: String ,`1377`: String ,`1397`: String ,`1401`: String ,`1403`: String ,`1405`: String ,`1409`: String ,`1411`: String ,`1413`: String ,`1417`: String ,`1419`: String ,`1421`: String ,`1423`: String ,`1431`: String ,`1437`: String ,`1439`: String ,`144`: String ,`1441`: String ,`145`: String ,`1451`: String ,`1453`: String ,`146`: String ,`1467`: String ,`1469`: String ,`1483`: String ,`1485`: String ,`1491`: String ,`1493`: String ,`15`: String ,`1505`: String ,`1519`: String ,`1521`: String ,`1523`: String ,`155`: String ,`156`: String ,`158`: String ,`1586`: String ,`1678`: String ,`169`: String ,`17`: String ,`177`: String ,`1783`: String ,`1785`: String ,`1787`: String ,`1789`: String ,`1793`: String ,`1795`: String ,`1797`: String ,`1799`: String ,`1801`: String ,`1803`: String ,`1805`: String ,`1807`: String ,`1809`: String ,`1811`: String ,`1815`: String ,`1818`: String ,`1826`: String ,`1829`: String ,`1874`: String ,`1876`: String ,`1878`: String ,`1880`: String ,`1882`: String ,`207`: String ,`215`: String ,`226`: String ,`2293`: String ,`232`: String ,`239`: String ,`241`: String ,`245`: String ,`2456`: String ,`248`: String ,`2482`: String ,`2490`: String ,`2491`: String ,`2492`: String ,`2493`: String ,`2495`: String ,`25`: String ,`2502`: String ,`2503`: String ,`2517`: String ,`2522`: String ,`255`: String ,`256`: String ,`2573`: String ,`2574`: String ,`260`: String ,`263`: String ,`265`: String ,`276`: String ,`278`: String ,`286`: String ,`293`: String ,`296`: String ,`299`: String ,`3`: String ,`30`: String ,`307`: String ,`312`: String ,`320`: String ,`328`: String ,`340`: String ,`348`: String ,`354`: String ,`363`: String ,`364`: String ,`369`: String ,`375`: String ,`376`: String ,`383`: String ,`401`: String ,`405`: String ,`409`: String ,`412`: String ,`42`: String ,`423`: String ,`444`: String ,`445`: String ,`446`: String ,`447`: String ,`450`: String ,`451`: String ,`452`: String ,`453`: String ,`455`: String ,`458`: String ,`461`: String ,`463`: String ,`464`: String ,`51`: String ,`543472980`: String ,`57`: String ,`75`: String ,`90`: String ,`92`: String ,`AVMYN`: String ,`AccessibilityAmenities`: String ,`AddressPublic`: String ,`Amenities`: String ,`AmountofWaterFront`: String ,`AppliancesIncluded`: String ,`AppraiserInitials`: String ,`AppraiserName`: String ,`Basement`: String ,`BathsFull`: String ,`BathsHalf`: String ,`BathsTotal`: String ,`BedsTotal`: String ,`Block`: String ,`Book`: String ,`BuildingNumber`: String ,`City`: String ,`CloseDate`: String ,`ClosePrice`: String ,
      `CoListAgentDirectWorkPhone`: String ,`CoListAgentEmail`: String ,`CoListAgentFullName`: String ,`CoListAgentMLSID`: String ,`CoListAgent_MUI`: String ,`CoListOfficeMLSID`: String ,`CoListOfficeName`: String ,`CoListOfficePhone`: String ,`CoListOffice_MUI`: String ,`Color`: String ,`CommentsYN`: String ,`Construction`: String ,`Cooling`: String ,`CountyOrParish`: String ,`DOM`: String ,`Deed`: String ,`Directions`: String ,`Driveway`: String ,`Electric`: String ,`Equipment`: String ,`Exterior`: String ,`Fireplace`: String ,`FloorNumber`: String ,`Floors`: String ,`FoundationMaterials`: String ,`FullTaxAmount`: String ,`Gas`: String ,`HeatFuel`: String ,`HeatSystem`: String ,`Internet`: String ,`ListAgentDirectWorkPhone`: String ,`ListAgentEmail`: String ,`ListAgentFullName`: String ,`ListAgentMLSID`: String ,`ListAgent_MUI`: String ,`ListOfficeMLSID`: String ,`ListOfficeName`: String ,`ListOfficePhone`: String ,`ListOffice_MUI`: String ,`ListPrice`: String ,`ListingContractDate`: String ,`Location`: String ,`Lot`: String ,`LotSizeArea`: String ,`MLSNumber`: String ,`Make`: String ,`MapNumber`: String ,`MatrixModifiedDT`: String ,`MobileHomeIncludes`: String ,`Model`: String ,`MonthlyAssociationFee`: String ,`NeighborhoodAssoc`: String ,`NumberOfRooms`: String ,`OneBedroomUnits`: String ,`OriginalListPrice`: String ,`Page`: String ,`Parking`: String ,`PhotoCount`: String ,`PhotoModificationTimestamp`: String ,`PostalCode`: String ,`PropertySubType`: String ,`PropertyType`: String ,`PublicAddress`: String ,`RecreationalWater`: String ,`Remarks`: String ,`Restrictions`: String ,`RoadFrontage`: String ,`Roads`: String ,`Roof`: String ,`SchoolDistrict`: String ,`Seasonal`: String ,`SellingAgentDirectWorkPhone`: String ,`SellingAgentEmail`: String ,`SellingAgentFullName`: String ,`SellingAgentMLSID`: String ,`SellingAgent_MUI`: String ,`SellingOfficeMLSID`: String ,`SellingOfficeName`: String ,`SellingOfficePhone`: String ,`SellingOffice_MUI`: String ,`SerialNumber`: String ,`Site`: String ,`SoldTerms`: String ,`SourceofSquareFootage`: String ,`SqFtFinishedTotal`: String ,`StateOrProvince`: String ,`Status`: String ,`Style`: String ,`SupplementPublicCount`: String ,`Surveyed`: String ,`TaxYear`: String ,`ThreeBedroomUnits`: String ,`Transportation`: String ,`TwoBedroomUnits`: String ,`UnitType`: String ,`UnitsInBuilding`: String ,`VehicleStorage`: String ,`View`: String ,`VirtualTour`: String ,`WasteWaterDisposal`: String ,`Water`: String ,`WaterBodyName`: String ,`WaterBodyType`: String ,`WaterFrontage`: String ,`WaterFrontageOwned`: String ,`WaterFrontageShared`: String ,`WaterHeater`: String ,`YearBuilt`: String ,`Zoning`: String ,`matrix_unique_id`: String ,`removed_created_at`: String ,`sysid`: String>
      basement` string
      ) STORED AS parquet;

      CREATE TABLE `property_feature`(
      `_id` struct<mls_sysid:string,type:string>,
      `all_attrs` struct<1:string,10:string,100:string,105:string,113:string,118:string,126:string,1299:string,
      1301:string,1305:string,1307:string,1319:string,1325:string,1327:string,1329:string,
      1339:string,1341:string,1345:string,1351:string,1353:string,1359:string,1363:string,
      1365:string,1367:string,1373:string,1377:string,1397:string,1401:string,1403:string,
      1405:string,1409:string,1411:string,1413:string,1417:string,1419:string,1421:string,
      1423:string,1431:string,1437:string,1439:string,144:string,1441:string,145:string,1451:string,
      1453:string,146:string,1467:string,1469:string,1483:string,1485:string,1491:string,1493:string,
      15:string,1505:string,1519:string,1521:string,1523:string,155:string,156:string,158:string,
      1586:string,1678:string,169:string,17:string,177:string,1783:string,1785:string,1787:string,
      1789:string,1793:string,1795:string,1797:string,1799:string,1801:string,1803:string,1805:string,
      1807:string,1809:string,1811:string,1815:string,1818:string,1826:string,1829:string,1874:string,
      1876:string,1878:string,1880:string,1882:string,207:string,215:string,226:string,2293:string,
      232:string,239:string,241:string,245:string,2456:string,248:string,2482:string,2490:string,
      2491:string,2492:string,2493:string,2495:string,25:string,2502:string,2503:string,2517:string,
      2522:string,255:string,256:string,2573:string,2574:string,260:string,263:string,265:string,
      276:string,278:string,286:string,293:string,296:string,299:string,3:string,30:string,307:string,
      312:string,320:string,328:string,340:string,348:string,354:string,363:string,364:string,369:string,
      375:string,376:string,383:string,401:string,405:string,409:string,412:string,42:string,423:string,444:string,445:string,446:string,447:string,450:string,451:string,452:string,453:string,455:string,458:string,461:string,463:string,464:string,51:string,543472980:string,57:string,75:string,90:string,92:string,AVMYN:string,AccessibilityAmenities:string,
      AddressPublic:string,Amenities:string,AmountofWaterFront:string,AppliancesIncluded:string,AppraiserInitials:string,AppraiserName:string,Basement:string,BathsFull:string,BathsHalf:string,BathsTotal:string,BedsTotal:string,Block:string,Book:string,BuildingNumber:string,City:string,CloseDate:string,ClosePrice:string,CoListAgentDirectWorkPhone:string,CoListAgentEmail:string,CoListAgentFullName:string,CoListAgentMLSID:string,CoListAgent_MUI:string,CoListOfficeMLSID:string,CoListOfficeName:string,CoListOfficePhone:string,
      CoListOffice_MUI:string,Color:string,CommentsYN:string,Construction:string,Cooling:string,CountyOrParish:string,DOM:string,Deed:string,Directions:string,Driveway:string,Electric:string,Equipment:string,Exterior:string,Fireplace:string,FloorNumber:string,Floors:string,FoundationMaterials:string,FullTaxAmount:string,Gas:string,HeatFuel:string,HeatSystem:string,Internet:string,ListAgentDirectWorkPhone:string,ListAgentEmail:string,ListAgentFullName:string,ListAgentMLSID:string,ListAgent_MUI:string,ListOfficeMLSID:string,ListOfficeName:string,ListOfficePhone:string,ListOffice_MUI:string,ListPrice:string,ListingContractDate:string,Location:string,Lot:string,LotSizeArea:string,MLSNumber:string,Make:string,MapNumber:string,MatrixModifiedDT:string,MobileHomeIncludes:string,Model:string,MonthlyAssociationFee:string,NeighborhoodAssoc:string,NumberOfRooms:string,OneBedroomUnits:string,OriginalListPrice:string,Page:string,Parking:string,PhotoCount:string,PhotoModificationTimestamp:string,PostalCode:string,PropertySubType:string,PropertyType:string,PublicAddress:string,RecreationalWater:string,Remarks:string,Restrictions:string,RoadFrontage:string,Roads:string,Roof:string,SchoolDistrict:string,Seasonal:string,SellingAgentDirectWorkPhone:string,SellingAgentEmail:string,SellingAgentFullName:string,SellingAgentMLSID:string,SellingAgent_MUI:string,SellingOfficeMLSID:string,SellingOfficeName:string,SellingOfficePhone:string,SellingOffice_MUI:string,SerialNumber:string,Site:string,SoldTerms:string,
      SourceofSquareFootage:string,SqFtFinishedTotal:string,StateOrProvinc,
      `basement` string
      ) STORED AS parquet;

      Please help out in case if there is any workaround for it or any other restriction details.

        Attachments

          Activity

            People

            • Assignee:
              Satyajit Satyajit varma
              Reporter:
              Satyajit Satyajit varma
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated: