Sqoop
  1. Sqoop
  2. SQOOP-514

Layout changes for Sqoop 2 distribution

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 1.99.1
    • Component/s: sqoop2-build, sqoop2-dist
    • Labels:
      None

      Description

      The current layout of the generated tarball is very different from the source layout. This is a request to change the layout so that it is closer to the source layout to help facilitate easy build and testing.

      1. bugSQOOP-514.patch
        8 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Arvind Prabhakar created issue -
          Hide
          Arvind Prabhakar added a comment -

          The current layout generates the following structure:

          .
          ├── bin
          │   └── sqoop.sh
          ├── client
          │   └── lib
          │       ├── ...
          │       └── sqoop-common-2.0.0-SNAPSHOT.jar
          └── server
              ├── LICENSE
              ├── NOTICE
              ├── RELEASE-NOTES
              ├── RUNNING.txt
              ├── bin
              │   ├── catalina.sh
              │   ├── ...
              │   ├── setenv.bat
              │   ├── setenv.sh
              │   └── ...
              ├── conf
              │   ├── catalina.properties
              │   ├── ...
              │   ├── sqoop.properties
              │   ├── sqoop_bootstrap.properties
              │   └── ...
              ├── lib
              │   ├── ...
              │   └── tomcat-util.jar
              ├── logs
              ├── temp
              │   └── safeToDelete.tmp
              ├── webapps
              │   ├── ROOT
              │   └── sqoop.war
              └── work
          
          
          

          This is very different from the source layout which is of the form:

          .
          ├── DISCLAIMER.txt
          ├── LICENSE.txt
          ├── NOTICE.txt
          ├── README.txt
          ├── client
          │   └── ...
          ├── common
          │   └── ...
          ├── connector
          │   └── ...
          ├── core
          │   └── ...
          ├── dist
          │   └── ...
          ├── docs
          │   └── ...
          ├── pom.xml
          ├── repository
          │   └── ...
          ├── server
          │   └── ...
          ├── spi
          │   └── ...
          └── target
              └── ...
          
          

          It will be good to follow the same source layout with added directories at the top to help make sure that top level files exist in the distribution at the necessary location among other benefits.

          One suggested layout would be as follows:

          .
          ├── DISCLAIMER.txt
          ├── LICENSE.txt
          ├── NOTICE.txt
          ├── README.txt
          ├── bin
          │   └── sqoop
          ├── client
          │   └── ...
          ├── common
          │   └── ...
          ├── config
          |   ├── sqoop_bootstrap.properties
          │   └── sqoop.properties
          ├── connector
          │   └── ...
          ├── core
          │   └── ...
          ├── dist
          │   └── ...
          ├── docs
          │   └── ...
          ├── pom.xml
          ├── repository
          │   └── ...
          ├── server
          │   └── ...
          ├── spi
          │   └── ...
          ├── target
          |   └── ...
          ├── lib
          |   └── <all compiled libs>
          └── webserver
              └── <tomcat specific files>
          
          

          With this layout the config files, bin scripts will remain in the top level directory and will be able to run directly form there. Also this distribution will be identical to the source distribution without the compiled binaries.

          Show
          Arvind Prabhakar added a comment - The current layout generates the following structure: . ├── bin │   └── sqoop.sh ├── client │   └── lib │   ├── ... │   └── sqoop-common-2.0.0-SNAPSHOT.jar └── server ├── LICENSE ├── NOTICE ├── RELEASE-NOTES ├── RUNNING.txt ├── bin │   ├── catalina.sh │   ├── ... │   ├── setenv.bat │   ├── setenv.sh │   └── ... ├── conf │   ├── catalina.properties │   ├── ... │   ├── sqoop.properties │   ├── sqoop_bootstrap.properties │   └── ... ├── lib │   ├── ... │   └── tomcat-util.jar ├── logs ├── temp │   └── safeToDelete.tmp ├── webapps │   ├── ROOT │   └── sqoop.war └── work This is very different from the source layout which is of the form: . ├── DISCLAIMER.txt ├── LICENSE.txt ├── NOTICE.txt ├── README.txt ├── client │   └── ... ├── common │   └── ... ├── connector │   └── ... ├── core │   └── ... ├── dist │   └── ... ├── docs │   └── ... ├── pom.xml ├── repository │   └── ... ├── server │   └── ... ├── spi │   └── ... └── target └── ... It will be good to follow the same source layout with added directories at the top to help make sure that top level files exist in the distribution at the necessary location among other benefits. One suggested layout would be as follows: . ├── DISCLAIMER.txt ├── LICENSE.txt ├── NOTICE.txt ├── README.txt ├── bin │   └── sqoop ├── client │   └── ... ├── common │   └── ... ├── config | ├── sqoop_bootstrap.properties │   └── sqoop.properties ├── connector │   └── ... ├── core │   └── ... ├── dist │   └── ... ├── docs │   └── ... ├── pom.xml ├── repository │   └── ... ├── server │   └── ... ├── spi │   └── ... ├── target |   └── ... ├── lib |   └── <all compiled libs> └── webserver    └── <tomcat specific files> With this layout the config files, bin scripts will remain in the top level directory and will be able to run directly form there. Also this distribution will be identical to the source distribution without the compiled binaries.
          Hide
          Jarek Jarcec Cecho added a comment -

          Do we need to follow the same directory structure for binary artifact as is in source artifact?

          I kind of like this division present in binary artifact into client and server (and common bin) directory. I believe that it will be very easy for end user to simply take it and deploy it.

          Show
          Jarek Jarcec Cecho added a comment - Do we need to follow the same directory structure for binary artifact as is in source artifact? I kind of like this division present in binary artifact into client and server (and common bin) directory. I believe that it will be very easy for end user to simply take it and deploy it.
          Hide
          Arvind Prabhakar added a comment -

          @Jarcec - typically most Hadoop related projects tend to create the binary distribution as a superset of source distribution. You could for example, take a release's binary distribution which contains sources and build it with any changes that you need. Sqoop 1 has had a similar layout for binary distribution in all its releases.

          The other aspect of this is that there are top-level files such as LICENSE, NOTICE, README etc which need to reside at the root of the distribution - much like they reside at the root of the source distribution as well. Having a bin distribution that includes the source distribution as well in it will keep things updated as and when these top-level files change.

          Show
          Arvind Prabhakar added a comment - @Jarcec - typically most Hadoop related projects tend to create the binary distribution as a superset of source distribution. You could for example, take a release's binary distribution which contains sources and build it with any changes that you need. Sqoop 1 has had a similar layout for binary distribution in all its releases. The other aspect of this is that there are top-level files such as LICENSE, NOTICE, README etc which need to reside at the root of the distribution - much like they reside at the root of the source distribution as well. Having a bin distribution that includes the source distribution as well in it will keep things updated as and when these top-level files change.
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you Arvind for your comment. Your arguments make sense to me.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Thank you Arvind for your comment. Your arguments make sense to me. Jarcec
          Jarek Jarcec Cecho made changes -
          Field Original Value New Value
          Assignee Jarek Jarcec Cecho [ jarcec ]
          Jarek Jarcec Cecho made changes -
          Remote Link This issue links to "Review board (Web Link)" [ 11434 ]
          Jarek Jarcec Cecho made changes -
          Attachment bugSQOOP-514.patch [ 12553238 ]
          Jarek Jarcec Cecho made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Bilung Lee added a comment -

          Patch is in. Thanks.

          Show
          Bilung Lee added a comment - Patch is in. Thanks.
          Bilung Lee made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Jarek Jarcec Cecho made changes -
          Fix Version/s 1.99.1 [ 12323641 ]
          Fix Version/s 2.0.0 [ 12319272 ]

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Arvind Prabhakar
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development