Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1619 Enhancing sharelib support
  3. OOZIE-1584

Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Supported features.

      1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at server starts.

      2. Admin copies sharelib to new timestamp directory(using sharelibCLI) and issues invalidate command(using oozieCLI) to Oozie server. Oozie picks latest shared lib.

      3.Oozie server doesn't auto deploy sharelib at startup. It picks the latest sharelib jar pushed by admin(based on timestamp).

      DFS after generating launcher jar by ooozie server and sharelib by cli
      -------------------
      makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
      /user/purushah/share/lib/
      /user/purushah/share/lib/launcher_20131017092254
      /user/purushah/share/lib/launcher_20131017093814
      /user/purushah/share/lib/launcher_20131017094652
      /user/purushah/share/lib/launcher_20131017094836
      /user/purushah/share/lib/launcher_20131017095549
      /user/purushah/share/lib/lib_20131017092806
      makebag-lm:example purushah$

      Purging.
      -------------
      There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
      directory and purging happens for both at startup.
      Anything older than ( defined in property) is purged.
      We always keep 2 set of directory, purging happens if number of sharelib
      directory > 2.

      Multiple version.
      -------------------
      To support multiple version, sharelib will be in below format.

      Sharelib will cache all versions and return list of dfs files for a particular
      version.

      makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
      /user/purushah/share/lib/
      Found 15 items
      /user/purushah/share/lib/launcher_20131017092254
      /user/purushah/share/lib/launcher_20131017093814
      /user/purushah/share/lib/launcher_20131017094652
      /user/purushah/share/lib/launcher_20131017094836
      /user/purushah/share/lib/launcher_20131017095549
      /user/purushah/share/lib/lib_20131017092806/pig/...
      /user/purushah/share/lib/lib_20131017092806/pig_9/...
      /user/purushah/share/lib/lib_20131017092806/pig_10/...
      makebag-lm:example purushah$

        Attachments

        1. OOZIE-1584_v2.patch
          57 kB
          Purshotam Shah
        2. OOZIE-1584_v3.patch
          59 kB
          Purshotam Shah
        3. OOZIE-1584_v4.patch
          63 kB
          Purshotam Shah
        4. OOZIE-1584_v5.patch
          62 kB
          Purshotam Shah
        5. OOZIE-1584_v6.patch
          67 kB
          Purshotam Shah

          Issue Links

            Activity

              People

              • Assignee:
                puru Purshotam Shah
                Reporter:
                puru Purshotam Shah
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: