Pig
  1. Pig
  2. PIG-2925

Extremely long JobConf values should not be added to Streaming environment

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      PIG-2900 made Pig match Hadoop's practice of making all JobConf properties available as environment variables for pig streaming.

      As it happens, ProcessBuilder creates a long string with all environment variables and their values on it, then runs it; having some of the weightier properties like serialized plans in the environment can make the total command length exceed OS limits.

      We propose adding a blacklist of properties which should not be added to streaming environment.

      1. PIG-2925.patch
        4 kB
        Dmitriy V. Ryaboy

        Issue Links

          Activity

          Hide
          Dmitriy V. Ryaboy added a comment -

          Decided it's easier to set the max length, and allow users to bump it up or down as needed, than to mess with explicit blacklists. Too hard to control what everyone might decide to shove into their config.

          Show
          Dmitriy V. Ryaboy added a comment - Decided it's easier to set the max length, and allow users to bump it up or down as needed, than to mess with explicit blacklists. Too hard to control what everyone might decide to shove into their config.
          Hide
          Jonathan Coveney added a comment -

          We are instead using the method in PIG-2971

          Show
          Jonathan Coveney added a comment - We are instead using the method in PIG-2971

            People

            • Assignee:
              Dmitriy V. Ryaboy
              Reporter:
              Dmitriy V. Ryaboy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development