Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2614

AvroStorage crashes on LOADING a single bad error

    XMLWordPrintableJSON

Details

    Description

      AvroStorage dies when a single bad record exists, such as one with missing fields. This is very bad on 'big data,' where bad records are inevitable. See discussion at http://www.quora.com/Big-Data/In-Big-Data-ETL-how-many-records-are-an-acceptable-loss for more theory.

      Attachments

        1. PIG-2614_0.patch
          29 kB
          Jonathan Coveney
        2. PIG-2614_1.patch
          29 kB
          Jonathan Coveney
        3. PIG-2614_2.patch
          19 kB
          Cheolsoo Park
        4. test_avro_files.tar.gz
          0.6 kB
          Cheolsoo Park

        Issue Links

          is superceded by

          New Feature - A new feature of the product, which has yet to be developed. PIG-3059 Global configurable minimum 'bad record' thresholds

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. PIG-2909 Add a new option for ignoring corrupted files to AvroStorage load func

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jcoveney Jonathan Coveney
              russell.jurney Russell Jurney
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: