Pig
  1. Pig
  2. PIG-3552

UriUtil used by reducer estimator should support viewfs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.1
    • Component/s: impl
    • Labels:
      None
    • Environment:

      Pig 0.11+

    • Patch Info:
      Patch Available

      Description

      On Hadoop2 running viewfs throws error

      2013-10-30 02:12:46,916 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
      2013-10-30 02:12:46,917 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1610612736 maxReducers=999 totalInputFileSize=-1
      2013-10-30 02:12:46,917 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Could not estimate number of reducers and no requested or default parallelism set. Defaulting to 1 reducer.
      

      This is caused by the fact that UriUtil.isHDFSFileOrLocalOrS3N only supports files that start with "/", "hdfs:", "file:", "s3n:" but not "viewfs:".

      Trivial fix, patch attached.

      Testing: ant test-commit passes.

      1. UriUtil_tests.patch
        2 kB
        Akihiro Matsukawa
      2. UriUtil_enable_viewfs.patch
        1 kB
        Akihiro Matsukawa

        Activity

        Hide
        Aniket Mokashi added a comment -

        committed to trunk and branch-0.12. Thanks Akihiro Matsukawa!

        Show
        Aniket Mokashi added a comment - committed to trunk and branch-0.12. Thanks Akihiro Matsukawa !
        Hide
        Aniket Mokashi added a comment -

        +1. New change is trivial (no refactor), so will not commit a new test.

        Show
        Aniket Mokashi added a comment - +1. New change is trivial (no refactor), so will not commit a new test.
        Hide
        Akihiro Matsukawa added a comment -

        Sorry was on branch-0.11 rather than trunk. Patches updated.

        Show
        Akihiro Matsukawa added a comment - Sorry was on branch-0.11 rather than trunk. Patches updated.
        Hide
        Aniket Mokashi added a comment -

        Akihiro Matsukawa, patch doesn't apply cleanly to trunk. Can you take a look?

        Show
        Aniket Mokashi added a comment - Akihiro Matsukawa , patch doesn't apply cleanly to trunk. Can you take a look?
        Hide
        Koji Noguchi added a comment -

        Rohini Palaniswamy, Koji Noguchi, I'm curious, why didn't you hit this one when running with hadoop 2?

        We haven't started using viewfs on production yet.

        Show
        Koji Noguchi added a comment - Rohini Palaniswamy, Koji Noguchi, I'm curious, why didn't you hit this one when running with hadoop 2? We haven't started using viewfs on production yet.
        Hide
        Aniket Mokashi added a comment -

        +1. Can you add a test before we commit this?

        Rohini Palaniswamy, Koji Noguchi, I'm curious, why didn't you hit this one when running with hadoop 2?

        Show
        Aniket Mokashi added a comment - +1. Can you add a test before we commit this? Rohini Palaniswamy , Koji Noguchi , I'm curious, why didn't you hit this one when running with hadoop 2?

          People

          • Assignee:
            Akihiro Matsukawa
            Reporter:
            Akihiro Matsukawa
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development