Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3059

Global configurable minimum 'bad record' thresholds

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.11
    • None
    • impl
    • None

    Description

      See PIG-2614.

      Pig dies when one record in a LOAD of a billion records fails to parse. This is almost certainly not the desired behavior. elephant-bird and some other storage UDFs have minimum thresholds in terms of percent and count that must be exceeded before a job will fail outright.

      We need these limits to be configurable for Pig, globally. I've come to realize what a major problem Pig's crashing on bad records is for new Pig users. I believe this feature can greatly improve Pig.

      An example of a config would look like:

      pig.storage.bad.record.threshold=0.01
      pig.storage.bad.record.min=100

      A thorough discussion of this issue is available here: http://www.quora.com/Big-Data/In-Big-Data-ETL-how-many-records-are-an-acceptable-loss

      Attachments

        1. PIG-3059-2.patch
          33 kB
          Cheolsoo Park
        2. PIG-3059.patch
          24 kB
          Cheolsoo Park
        3. avro_test_files-2.tar.gz
          20 kB
          Cheolsoo Park

        Issue Links

          is duplicated by

          New Feature - A new feature of the product, which has yet to be developed. PIG-3404 Improve Pig to ignore bad files or inaccessible files or folders

          • Major - Major loss of function.
          • Open
          is related to

          New Feature - A new feature of the product, which has yet to be developed. AVRO-806 add a column-major file format

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. PIG-3015 Rewrite of AvroStorage

          • Major - Major loss of function.
          • Closed
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. PIG-2614 AvroStorage crashes on LOADING a single bad error

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Review Board

          Activity

            People

              Unassigned Unassigned
              russell.jurney Russell Jurney
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: