Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4526

Make setting up the build environment easier

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      A docker based build environment was added to allow for easier and more consistent builds.

      Description

      In AVRO-1537 and HADOOP-11843 a docker based solution was created to setup all the tools for doing a full build. This enables much easier reproduction of any issues and getting up and running for new developers.

      This issue is to 'copy/port' that setup into the pig project.

      1. PIG-4526-2015-04-30-1632.patch
        10 kB
        Niels Basjes
      2. PIG-4526-2015-05-01-1545.patch
        10 kB
        Niels Basjes
      3. PIG-4526-2015-05-03-0910.patch
        10 kB
        Niels Basjes
      4. PIG-4526-2016-02-24-1310.patch
        11 kB
        Niels Basjes
      5. PIG-4526-2016-04-08.patch
        13 kB
        Niels Basjes
      6. PIG-4526-2016-04-26.patch
        15 kB
        Niels Basjes

        Issue Links

          Activity

          Hide
          rohini Rohini Palaniswamy added a comment -

          Thanks Niels Basjes. Making it a markdown file was a good idea. This is good enough to start with.

          +1. Committed to trunk.

          Show
          rohini Rohini Palaniswamy added a comment - Thanks Niels Basjes . Making it a markdown file was a good idea. This is good enough to start with. +1. Committed to trunk.
          Hide
          nielsbasjes Niels Basjes added a comment -

          Rohini Palaniswamy I renamed the file to .md and added what I think is a good enough first version of this description.
          Please let me know what you think.

          P.S. When you decide to commit please make sure the shell script is committed with execute rights.

          Show
          nielsbasjes Niels Basjes added a comment - Rohini Palaniswamy I renamed the file to .md and added what I think is a good enough first version of this description. Please let me know what you think. P.S. When you decide to commit please make sure the shell script is committed with execute rights.
          Hide
          nielsbasjes Niels Basjes added a comment -

          Sure, will do.

          Show
          nielsbasjes Niels Basjes added a comment - Sure, will do.
          Hide
          rohini Rohini Palaniswamy added a comment -

          Niels Basjes,
          Can you add information on building the regular way with just ant in BUILDING.txt similar to Maven build goals section in https://svn.apache.org/repos/asf/hadoop/common/trunk/BUILDING.txt. Key ones are good enough and any additions we can do in a separate jira. Also move current contents to a Building with Docker section.

          Show
          rohini Rohini Palaniswamy added a comment - Niels Basjes , Can you add information on building the regular way with just ant in BUILDING.txt similar to Maven build goals section in https://svn.apache.org/repos/asf/hadoop/common/trunk/BUILDING.txt . Key ones are good enough and any additions we can do in a separate jira. Also move current contents to a Building with Docker section.
          Hide
          nielsbasjes Niels Basjes added a comment -

          This version is capable of building the entire build running the tests and including building all of the documentation.

          Show
          nielsbasjes Niels Basjes added a comment - This version is capable of building the entire build running the tests and including building all of the documentation.
          Hide
          nielsbasjes Niels Basjes added a comment -

          Fixing the patch to solve PIG-3906

          Show
          nielsbasjes Niels Basjes added a comment - Fixing the patch to solve PIG-3906
          Hide
          nielsbasjes Niels Basjes added a comment -

          This will really help in making builds more reproducible.

          Show
          nielsbasjes Niels Basjes added a comment - This will really help in making builds more reproducible.
          Hide
          nielsbasjes Niels Basjes added a comment -

          I ran these commands successfully (Hadoop 23):

          ANT='ant -Dhadoopversion=23 -Djavac.args="-Xlint -Xmaxwarns 100"'
          ${ANT} clean piggybank jar compile-test
          cd contrib/piggybank/java && ${ANT} test
          ${ANT} test-commit
          
          Show
          nielsbasjes Niels Basjes added a comment - I ran these commands successfully (Hadoop 23): ANT='ant -Dhadoopversion=23 -Djavac.args= "-Xlint -Xmaxwarns 100" ' ${ANT} clean piggybank jar compile-test cd contrib/piggybank/java && ${ANT} test ${ANT} test-commit
          Hide
          nielsbasjes Niels Basjes added a comment -

          Ran into the same problem as HADOOP-11936

          Show
          nielsbasjes Niels Basjes added a comment - Ran into the same problem as HADOOP-11936
          Hide
          nielsbasjes Niels Basjes added a comment -

          Added the USER environment variable.
          The failing test now passed.

          Show
          nielsbasjes Niels Basjes added a comment - Added the USER environment variable. The failing test now passed.
          Hide
          nielsbasjes Niels Basjes added a comment -

          Cause of unit test failure: Inside the docker setup the environment variable USER was not set.

          Show
          nielsbasjes Niels Basjes added a comment - Cause of unit test failure: Inside the docker setup the environment variable USER was not set.
          Hide
          nielsbasjes Niels Basjes added a comment -

          Over night I ran ant -Dhadoopversion=23 -Djavac.args="-Xlint -Xmaxwarns 1000" clean jar test-core.
          This took 386 minutes 53 seconds and resulted in only 1 test failing:
          org.apache.pig.test.TestScriptUDF testPythonBuiltinModuleImport1

          Show
          nielsbasjes Niels Basjes added a comment - Over night I ran ant -Dhadoopversion=23 -Djavac.args="-Xlint -Xmaxwarns 1000" clean jar test-core . This took 386 minutes 53 seconds and resulted in only 1 test failing: org.apache.pig.test.TestScriptUDF testPythonBuiltinModuleImport1
          Hide
          nielsbasjes Niels Basjes added a comment -

          This can successfully run ant -Dhadoopversion=23 -Djavac.args="-Xlint -Xmaxwarns 1000" clean jar test-commit

          Show
          nielsbasjes Niels Basjes added a comment - This can successfully run ant -Dhadoopversion=23 -Djavac.args="-Xlint -Xmaxwarns 1000" clean jar test-commit
          Hide
          nielsbasjes Niels Basjes added a comment -

          Looks like I created a reproduction path for BIGTOP-1303 and PIG-3906

          Show
          nielsbasjes Niels Basjes added a comment - Looks like I created a reproduction path for BIGTOP-1303 and PIG-3906
          Hide
          nielsbasjes Niels Basjes added a comment -

          Updated to forrest 0.9 because 0.8 can only run with Java 1.5 ...

          Current state:

          ant docs

          still fails with a bunch of broken links all relating to PDF files.
          /home/nbasjes/pig/src/docs/build/site/broken-links.xml

          <broken-links>                                                                                       
            <link message="No pipeline matched request: admin.pdf">admin.pdf</link>                            
            <link message="No pipeline matched request: perf.pdf">perf.pdf</link>                              
            <link message="No pipeline matched request: basic.pdf">basic.pdf</link>                            
            <link message="No pipeline matched request: cont.pdf">cont.pdf</link>                              
            <link message="No pipeline matched request: linkmap.pdf">linkmap.pdf</link>                        
            <link message="No pipeline matched request: index.pdf">index.pdf</link>                            
            <link message="No pipeline matched request: func.pdf">func.pdf</link>                              
            <link message="No pipeline matched request: pig-index.pdf">pig-index.pdf</link>                    
            <link message="No pipeline matched request: start.pdf">start.pdf</link>                            
            <link message="No pipeline matched request: udf.pdf">udf.pdf</link>                                
            <link message="No pipeline matched request: test.pdf">test.pdf</link>                              
            <link message="No pipeline matched request: cmds.pdf">cmds.pdf</link>                              
          </broken-links>                                                                                      
          
          Show
          nielsbasjes Niels Basjes added a comment - Updated to forrest 0.9 because 0.8 can only run with Java 1.5 ... Current state: ant docs still fails with a bunch of broken links all relating to PDF files. /home/nbasjes/pig/src/docs/build/site/broken-links.xml <broken-links> <link message= "No pipeline matched request: admin.pdf" >admin.pdf</link> <link message= "No pipeline matched request: perf.pdf" >perf.pdf</link> <link message= "No pipeline matched request: basic.pdf" >basic.pdf</link> <link message= "No pipeline matched request: cont.pdf" >cont.pdf</link> <link message= "No pipeline matched request: linkmap.pdf" >linkmap.pdf</link> <link message= "No pipeline matched request: index.pdf" >index.pdf</link> <link message= "No pipeline matched request: func.pdf" >func.pdf</link> <link message= "No pipeline matched request: pig-index.pdf" >pig-index.pdf</link> <link message= "No pipeline matched request: start.pdf" >start.pdf</link> <link message= "No pipeline matched request: udf.pdf" >udf.pdf</link> <link message= "No pipeline matched request: test.pdf" >test.pdf</link> <link message= "No pipeline matched request: cmds.pdf" >cmds.pdf</link> </broken-links>
          Hide
          nielsbasjes Niels Basjes added a comment -

          Submitted bug report and patch for this issue: PIG-4527

          Show
          nielsbasjes Niels Basjes added a comment - Submitted bug report and patch for this issue: PIG-4527
          Hide
          nielsbasjes Niels Basjes added a comment -

          Please test and post your findings.
          I just found that running

          ant docs

          shows non-ACSII characters in the Javadoc in

          contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java
          
          * Note this function checks for Integer range <E2><88><92>2,147,483,648 to 2,147,483,647.
          
          Show
          nielsbasjes Niels Basjes added a comment - Please test and post your findings. I just found that running ant docs shows non-ACSII characters in the Javadoc in contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java * Note this function checks for Integer range <E2><88><92>2,147,483,648 to 2,147,483,647.
          Hide
          michaelthoward Michael Howard added a comment -

          Niels,
          I am a pig newbie who has recently struggled with getting a dev environment set up.
          Thank you for your work to make it easier for developers to get started.

          Show
          michaelthoward Michael Howard added a comment - Niels, I am a pig newbie who has recently struggled with getting a dev environment set up. Thank you for your work to make it easier for developers to get started.
          Hide
          nielsbasjes Niels Basjes added a comment -

          First draft patch that can build the entire project.
          Needs further testing.

          Show
          nielsbasjes Niels Basjes added a comment - First draft patch that can build the entire project. Needs further testing.

            People

            • Assignee:
              nielsbasjes Niels Basjes
              Reporter:
              nielsbasjes Niels Basjes
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development