Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2512

ShareLibservice returns incorrect path for jar

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.3.0
    • None
    • None

    Description

      If we have oozie.service.ShareLibService.mapping.file setting pointing to a metafile, then ShareLibServe loads paths from that file.

      We can mention path to a directory or path to a direct file.

      When path to a directory is mentioned, the paths are populated correctly, but not when we mentioned direct path for a file.

      Consider following paths:

      • /sharelib/pig/
        • pig.jar
        • some.jar
      • /sharelib/spark
        • spark-assembly.jar

      In metafile, we have

      oozie.pig=/sharelib/pig/
      oozie.spark=/sharelib/spark/spark-assembly.jar
      

      Now, the SharelibService calculates the paths as
      pig - hdfs://clustername.com:8020/sharelib/pig/pig.jar,hdfs://clustername.com:8020/sharelib/pig/some.jar
      spark - /sharelib/spark/spark-assembly.jar

      The spark path does not have hdfs prefixed.

      Later on, when we run a spark action, it fails with

      Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File file:/sharelib/spark/spark-assembly.jar does not exist
      java.io.FileNotFoundException: File file:/sharelib/spark/spark-assembly.jar does not exist
      

      Remember, if we already mentioned hdfs prefixed paths in metafile, then it works fine.

      Attachments

        1. OOZIE-2512-1.patch
          3 kB
          Satish Saley
        2. OOZIE-2512-2.patch
          4 kB
          Satish Saley
        3. OOZIE-2512-3.patch
          4 kB
          Satish Saley
        4. OOZIE-2512-amend-1.patch
          1.0 kB
          Satish Saley
        5. OOZIE-2512-amend-2.patch
          5 kB
          Satish Saley
        6. OOZIE-2512-amend-3.patch
          6 kB
          Satish Saley
        7. OOZIE-2512-amend-4.patch
          6 kB
          Satish Saley
        8. OOZIE-2512-amend-5.patch
          7 kB
          Satish Saley

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            satishsaley Satish Saley
            satishsaley Satish Saley
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment