Sqoop
  1. Sqoop
  2. SQOOP-1192

Add option "--skip-dist-cache" to allow Sqoop not copying jars in %SQOOP_HOME%\lib folder when launched by Oozie and use Oozie share lib

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None

      Description

      Now Sqoop will copy jar files in %SQOOP_HOME%\lib folder to the job cache every time a Sqoop job is launched. When Oozie launch a Sqoop job, this behavior can be optimized by add these jars in Oozie Sqoop sharelib. In this case, the jar files in share lib only needed be localized to each worker node once and reuse by all Sqoop job launched by Oozie. This can reduce massive disk I/O on worker node when using Sqoop by Oozie. To enable this, Sqoop need to have an option which enable the job to skip adding lib jars to the job cache. For now, this option should only be used by Oozie started Sqoop job. The patch attached introduce "--skip-dist-cache" option to enable this feature.

      1. SQOOP-1192.3.patch
        5 kB
        Shuaishuai Nie
      2. SQOOP-1192.2.patch
        4 kB
        Shuaishuai Nie
      3. SQOOP-1192.1.patch
        4 kB
        Shuaishuai Nie

        Issue Links

          Activity

          Jarek Jarcec Cecho made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 1.4.5 [ 12324765 ]
          Resolution Fixed [ 1 ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.3.patch [ 12608365 ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.3.patch [ 12608352 ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.3.patch [ 12608352 ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.3.patch [ 12608348 ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.3.patch [ 12608348 ]
          Shuaishuai Nie made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Shuaishuai Nie made changes -
          Assignee Shuaishuai Nie [ shuainie ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.2.patch [ 12603887 ]
          Shuaishuai Nie made changes -
          Remote Link This issue links to "Review board link (Web Link)" [ 12632 ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.1.patch [ 12602614 ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.1.patch [ 12601561 ]
          Shuaishuai Nie made changes -
          Link This issue depends upon OOZIE-1521 [ OOZIE-1521 ]
          Shuaishuai Nie made changes -
          Link This issue is depended upon by OOZIE-1521 [ OOZIE-1521 ]
          Shuaishuai Nie made changes -
          Link This issue depends upon OOZIE-1521 [ OOZIE-1521 ]
          Shuaishuai Nie made changes -
          Link This issue is depended upon by OOZIE-1521 [ OOZIE-1521 ]
          Shuaishuai Nie made changes -
          Attachment SQOOP-1192.1.patch [ 12601561 ]
          Shuaishuai Nie made changes -
          Field Original Value New Value
          Description Now Sqoop will copy jar files in %SQOOP_HOME%\lib folder to the job cache every time a Sqoop job is launched. When Oozie launch a Sqoop job, this behavior can be optimized by add these jars in Oozie Sqoop sharelib. In this case, the jar files in share lib only needed be localized to each worker node once and reuse by all Sqoop job launched by Oozie. This can reduce massive disk I/O on worker node when using Sqoop by Oozie. To enable this, Sqoop need to have an option which enable the job to skip adding lib jars to the job cache. For now, this option should only be used by Oozie started Sqoop job. The patch attached introduce "--skip-dist-cache" option to enable this feature.
          Shuaishuai Nie created issue -

            People

            • Assignee:
              Shuaishuai Nie
              Reporter:
              Shuaishuai Nie
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development