Bigtop
  1. Bigtop
  2. BIGTOP-894

Pig compilation fails on RPM systems on Bigtop trunk

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 1.0.0
    • Component/s: debian, rpm
    • Labels:
      None

      Description

      BIGTOP-870 bumped up version of Pig from 0.10 to 0.11 for Bigtop 0.6 release. However, while that seems to build Pig fine on Debian based systems, Pig compilation fails on RPM based systems with an error like:

      04:27:41  compile:
      04:27:41       [echo]  *** Compiling Pig UDFs ***
      04:27:41      [javac] /mnt/jenkins/workspace/Bigtop-trunk-Pig/label/centos6/build/pig/rpm/BUILD/pig-0.11.0/contrib/piggybank/java/build.xml:93: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
      04:27:41      [javac] Compiling 158 source files to /mnt/jenkins/workspace/Bigtop-trunk-Pig/label/centos6/build/pig/rpm/BUILD/pig-0.11.0/contrib/piggybank/java/build/classes
      04:27:41      [javac] /mnt/jenkins/workspace/Bigtop-trunk-Pig/label/centos6/build/pig/rpm/BUILD/pig-0.11.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:31: unmappable character for encoding ASCII
      04:27:41      [javac]  * Note this function checks for Integer range ???2,147,483,648 to 2,147,483,647.
      04:27:41      [javac]                                                ^
      04:27:41      [javac] /mnt/jenkins/workspace/Bigtop-trunk-Pig/label/centos6/build/pig/rpm/BUILD/pig-0.11.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:31: unmappable character for encoding ASCII
      04:27:41      [javac]  * Note this function checks for Integer range ???2,147,483,648 to 2,147,483,647.
      04:27:41      [javac]                                                 ^
      04:27:41      [javac] /mnt/jenkins/workspace/Bigtop-trunk-Pig/label/centos6/build/pig/rpm/BUILD/pig-0.11.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:31: unmappable character for encoding ASCII
      04:27:41      [javac]  * Note this function checks for Integer range ???2,147,483,648 to 2,147,483,647.
      04:27:41      [javac]                                                  ^
      04:27:42      [javac] 3 errors
      
      1. 0001-BIGTOP-894-Use-C.UTF-8-locale.patch
        1 kB
        Olaf Flebbe
      2. BIGTOP-894.2.patch
        1 kB
        Mark Grover
      3. BIGTOP-894.1.patch
        10 kB
        Mark Grover

        Issue Links

          Activity

          Hide
          Roman Shaposhnik added a comment -

          +1 and committed!

          Show
          Roman Shaposhnik added a comment - +1 and committed!
          Hide
          Olaf Flebbe added a comment -

          Roman Shaposhnik Patch appended

          Show
          Olaf Flebbe added a comment - Roman Shaposhnik Patch appended
          Hide
          Roman Shaposhnik added a comment -

          Olaf Flebbe any chance you can provide a patch so I can review and commit it?

          Show
          Roman Shaposhnik added a comment - Olaf Flebbe any chance you can provide a patch so I can review and commit it?
          Hide
          Olaf Flebbe added a comment -

          — a/bigtop-packages/src/common/pig/do-component-build
          +++ b/bigtop-packages/src/common/pig/do-component-build
          @@ -39,7 +39,7 @@ sed -i -e '/.<dependency./

          {N; s/.*<dependency.*\n.*hadoop20.*//g}

          ' ivy.xml
          BUILD_OPTS="-Djavac.version=1.6 -Dforrest.home=$

          {FORREST_HOME}

          -Dversion=$

          {PIG_VERSION}

          -Dhadoopversion=23"

          1. FIXME: BIGTOP-894. Please get rid of the line below when Bigtop integrates a version of Pig that includes the fix for PIG-3262
            -export LC_ALL=en_US.UTF-8
            +export LC_ALL=C.UTF-8
          1. FIXME: there's a weird synchronization issue between ivy and Maven caches wrt. jetty
          2. If we don't do the following the build fails with (note zip instead of jar):
          Show
          Olaf Flebbe added a comment - — a/bigtop-packages/src/common/pig/do-component-build +++ b/bigtop-packages/src/common/pig/do-component-build @@ -39,7 +39,7 @@ sed -i -e '/. <dependency. / {N; s/.*<dependency.*\n.*hadoop20.*//g} ' ivy.xml BUILD_OPTS="-Djavac.version=1.6 -Dforrest.home=$ {FORREST_HOME} -Dversion=$ {PIG_VERSION} -Dhadoopversion=23" FIXME: BIGTOP-894 . Please get rid of the line below when Bigtop integrates a version of Pig that includes the fix for PIG-3262 -export LC_ALL=en_US.UTF-8 +export LC_ALL=C.UTF-8 FIXME: there's a weird synchronization issue between ivy and Maven caches wrt. jetty If we don't do the following the build fails with (note zip instead of jar):
          Hide
          Olaf Flebbe added a comment -

          Sorry, this change breaks non US compile jobs, since en_US.utf8 locale is not available for instance in old europe.

          export LC_ALL=C.UTF-8

          is needed.

          Show
          Olaf Flebbe added a comment - Sorry, this change breaks non US compile jobs, since en_US.utf8 locale is not available for instance in old europe. export LC_ALL=C.UTF-8 is needed.
          Hide
          Roman Shaposhnik added a comment -

          +1 and committed!

          Show
          Roman Shaposhnik added a comment - +1 and committed!
          Hide
          Mark Grover added a comment -

          Patch with a simpler workaround as suggested by Bruno Mahé

          export LC_ALL=en_US.UTF-8
          
          Show
          Mark Grover added a comment - Patch with a simpler workaround as suggested by Bruno Mahé export LC_ALL=en_US.UTF-8
          Hide
          Mark Grover added a comment -

          Built on RHEL5 and Ubuntu 12.04.

          Feedback/nitpicks welcome!

          Show
          Mark Grover added a comment - Built on RHEL5 and Ubuntu 12.04. Feedback/nitpicks welcome!
          Hide
          Mark Grover added a comment -

          From the looks of it, it looks like the pig contib code is being compiled without the javac encoding being set. The default encoding on Debian systems seems to work the compilation but that on RPM based systems doesn't. There should be a deterministic way of setting the encoding of the code and a good default value (UTF8?) so this doesn't happen.

          Show
          Mark Grover added a comment - From the looks of it, it looks like the pig contib code is being compiled without the javac encoding being set. The default encoding on Debian systems seems to work the compilation but that on RPM based systems doesn't. There should be a deterministic way of setting the encoding of the code and a good default value (UTF8?) so this doesn't happen.

            People

            • Assignee:
              Mark Grover
              Reporter:
              Mark Grover
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development