Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6114

Pig with widerows=true and batch size = 1 works incorrectly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Low
    • Resolution: Fixed
    • 1.2.12, 2.0.2
    • None
    • None
    • Low

    Description

      If I run the demo pig scripts, I end up with a column family with 6 fairly wide rows. If I load and dump those rows with widerows=true or set the cassandra.range.batch.size=1, the dump returns the correct values. However, if I set both of those, it does not. So in the case of a batch size of 1, wide rows support is broken.

      So it's relatively simple to reproduce from the demo data:

      grunt> SET cassandra.range.batch.size 1                                                
      grunt> rows = LOAD 'cassandra://PigDemo/Scores' using CassandraStorage();                
      grunt> dump rows;
      ...
      (sylvain,{(4,),(7,),(10,),(21,),(24,),(46,),(47,),(49,),(51,),(52,),(67,),(68,),(72,),(73,),(82,),(83,),(86,),(98,),(101,),(105,),(108,),(112,),(114,),(124,),(125,),(136,),(139,),(145,),(150,),(151,),(153,),(165,),(167,),(171,),(178,),(182,),(202,),(211,),(212,),(215,),(226,),(237,),(242,),(243,),(255,),(261,),(273,),(282,),(300,),(307,),(308,),(311,),(312,),(313,),(316,),(317,),(332,),(337,),(338,),(348,),(355,),(360,),(361,),(373,),(375,),(377,),(384,),(401,),(404,),(412,),(418,),(429,),(436,),(441,),(451,),(453,),(461,),(473,),(478,),(483,),(485,),(486,),(489,),(509,),(511,),(516,),(517,),(521,),(536,),(541,),(543,),(545,),(550,),(583,),(587,),(592,),(611,),(613,),(622,),(625,),(627,),(633,),(648,),(649,),(651,),(659,),(665,),(668,),(670,),(672,),(679,),(688,),(692,),(700,),(703,),(707,),(709,),(730,),(731,),(738,),(740,),(744,),(750,),(759,),(764,),(766,),(768,),(774,),(776,),(778,),(779,),(788,),(795,),(796,),(813,),(821,),(825,),(830,),(831,),(835,),(843,),(846,),(847,),(848,),(851,),(862,),(863,),(872,),(878,),(881,),(883,),(884,),(888,),(905,),(906,),(916,),(921,),(926,),(928,),(944,),(946,),(947,),(952,),(954,),(972,),(973,),(974,),(976,),(978,),(982,),(991,)})
      (brandon,{(6,),(7,),(14,),(15,),(25,),(36,),(37,),(38,),(46,),(53,),(57,),(65,),(74,),(75,),(84,),(91,),(104,),(120,),(128,),(137,),(148,),(159,),(171,),(174,),(176,),(179,),(183,),(192,),(195,),(201,),(205,),(210,),(216,),(222,),(223,),(243,),(255,),(264,),(271,),(287,),(290,),(308,),(309,),(326,),(343,),(347,),(356,),(359,),(360,),(363,),(367,),(368,),(378,),(398,),(400,),(402,),(410,),(412,),(419,),(427,),(429,),(447,),(449,),(462,),(464,),(468,),(470,),(472,),(480,),(482,),(506,),(511,),(520,),(521,),(522,),(524,),(535,),(548,),(553,),(565,),(569,),(571,),(573,),(575,),(583,),(584,),(595,),(597,),(606,),(608,),(634,),(646,),(650,),(654,),(667,),(673,),(677,),(686,),(690,),(692,),(713,),(715,),(721,),(723,),(736,),(737,),(752,),(753,),(758,),(759,),(764,),(766,),(767,),(776,),(778,),(786,),(812,),(816,),(818,),(823,),(826,),(832,),(838,),(842,),(860,),(873,),(879,),(918,),(919,),(935,),(941,),(942,),(948,),(956,),(961,),(966,),(973,),(974,),(977,),(979,),(983,),(984,),(986,),(995,),(997,)})
      (jake,{(1,),(7,),(10,),(14,),(29,),(52,),(54,),(65,),(67,),(78,),(82,),(83,),(89,),(97,),(100,),(115,),(126,),(140,),(141,),(145,),(214,),(221,),(230,),(231,),(232,),(241,),(245,),(247,),(265,),(266,),(269,),(271,),(282,),(286,),(288,),(299,),(316,),(323,),(331,),(332,),(335,),(338,),(348,),(353,),(355,),(364,),(367,),(371,),(379,),(398,),(409,),(420,),(428,),(429,),(439,),(443,),(450,),(454,),(467,),(477,),(482,),(488,),(490,),(502,),(503,),(512,),(520,),(521,),(535,),(536,),(541,),(548,),(552,),(557,),(560,),(596,),(600,),(604,),(606,),(611,),(613,),(621,),(624,),(630,),(635,),(641,),(647,),(655,),(660,),(665,),(674,),(676,),(690,),(693,),(694,),(704,),(719,),(720,),(724,),(731,),(749,),(751,),(763,),(765,),(767,),(771,),(779,),(782,),(784,),(789,),(793,),(797,),(798,),(801,),(802,),(806,),(820,),(825,),(839,),(845,),(848,),(856,),(865,),(866,),(867,),(870,),(876,),(887,),(891,),(901,),(905,),(908,),(922,),(929,),(944,),(960,),(964,),(980,),(988,),(996,)})
      (eric,{(14,),(17,),(23,),(25,),(26,),(34,),(42,),(43,),(57,),(64,),(68,),(80,),(88,),(93,),(100,),(114,),(131,),(132,),(134,),(143,),(146,),(147,),(156,),(157,),(170,),(171,),(172,),(177,),(186,),(197,),(198,),(206,),(209,),(223,),(224,),(233,),(236,),(241,),(251,),(252,),(255,),(263,),(266,),(267,),(268,),(272,),(277,),(280,),(289,),(293,),(294,),(297,),(301,),(306,),(310,),(312,),(321,),(326,),(333,),(334,),(335,),(345,),(357,),(362,),(363,),(370,),(380,),(389,),(392,),(393,),(401,),(420,),(431,),(462,),(464,),(465,),(471,),(484,),(486,),(490,),(493,),(504,),(505,),(509,),(515,),(521,),(534,),(538,),(547,),(554,),(557,),(561,),(564,),(572,),(573,),(578,),(582,),(584,),(590,),(598,),(599,),(603,),(605,),(609,),(618,),(634,),(636,),(639,),(648,),(656,),(661,),(667,),(671,),(674,),(675,),(687,),(713,),(721,),(733,),(736,),(763,),(767,),(776,),(785,),(787,),(809,),(813,),(826,),(829,),(830,),(832,),(840,),(841,),(844,),(846,),(854,),(855,),(876,),(890,),(892,),(902,),(910,),(930,),(934,),(938,),(940,),(943,),(955,),(959,),(965,),(966,),(968,),(972,),(980,),(985,),(989,)})
      (jonathan,{(17,),(18,),(31,),(34,),(37,),(40,),(67,),(69,),(75,),(93,),(111,),(124,),(127,),(128,),(137,),(142,),(168,),(178,),(190,),(193,),(194,),(207,),(211,),(216,),(221,),(229,),(237,),(242,),(252,),(253,),(264,),(265,),(267,),(270,),(272,),(274,),(276,),(278,),(280,),(283,),(297,),(299,),(300,),(302,),(303,),(309,),(311,),(318,),(323,),(329,),(330,),(332,),(344,),(346,),(351,),(354,),(358,),(361,),(363,),(366,),(367,),(374,),(378,),(379,),(386,),(389,),(392,),(395,),(398,),(404,),(424,),(426,),(429,),(434,),(439,),(443,),(445,),(448,),(472,),(477,),(494,),(500,),(504,),(522,),(525,),(538,),(539,),(541,),(548,),(553,),(557,),(560,),(563,),(566,),(567,),(578,),(591,),(593,),(595,),(599,),(605,),(610,),(626,),(635,),(636,),(640,),(642,),(644,),(649,),(660,),(662,),(663,),(667,),(674,),(690,),(706,),(708,),(712,),(716,),(723,),(733,),(741,),(747,),(758,),(765,),(797,),(798,),(801,),(822,),(827,),(828,),(837,),(850,),(863,),(867,),(894,),(895,),(896,),(904,),(911,),(917,),(932,),(949,),(951,),(952,),(958,),(969,),(974,),(983,),(985,),(988,),(989,),(996,),(1000,)})
      (gary,{(3,),(13,),(21,),(23,),(33,),(36,),(44,),(45,),(48,),(62,),(65,),(68,),(75,),(80,),(81,),(90,),(111,),(113,),(119,),(123,),(137,),(149,),(152,),(153,),(157,),(161,),(166,),(178,),(179,),(180,),(183,),(184,),(188,),(189,),(191,),(197,),(199,),(200,),(204,),(212,),(221,),(229,),(239,),(265,),(270,),(272,),(276,),(279,),(282,),(295,),(296,),(304,),(305,),(314,),(326,),(329,),(335,),(342,),(345,),(346,),(362,),(370,),(371,),(375,),(380,),(382,),(387,),(389,),(390,),(393,),(399,),(403,),(406,),(414,),(417,),(424,),(428,),(445,),(458,),(462,),(486,),(490,),(492,),(495,),(499,),(500,),(507,),(514,),(520,),(542,),(550,),(551,),(570,),(571,),(572,),(574,),(577,),(588,),(604,),(614,),(619,),(626,),(634,),(640,),(648,),(659,),(663,),(684,),(687,),(690,),(694,),(715,),(741,),(750,),(765,),(772,),(776,),(781,),(782,),(783,),(785,),(789,),(802,),(806,),(812,),(816,),(820,),(829,),(836,),(843,),(850,),(855,),(868,),(873,),(875,),(889,),(900,),(904,),(922,),(928,),(929,),(935,),(946,),(949,),(954,),(956,),(959,),(960,),(962,),(992,)})
      
      grunt> SET cassandra.range.batch.size 1                                                
      grunt> rows = LOAD 'cassandra://PigDemo/Scores?widerows=true' using CassandraStorage();
      grunt> dump rows;
      ...
      (jonathan,{(17,)})
      (sylvain,{(4,)})
      

      When I try with set batch size to something higher than 1 and it works fine.

      Attachments

        1. 6114.txt
          0.9 kB
          Alex Liu

        Activity

          People

            alexliu68 Alex Liu
            alexliu68 Alex Liu
            Alex Liu
            Jonathan Ellis
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: