Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3363

Hadoop's cleanup of local directory in uber mode causing failures for sqoop actions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 4.3.0, 5.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      See related task OOZIE-2536 and discussion https://issues.apache.org/jira/browse/OOZIE-2536?focusedCommentId=16617206&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16617206

      the same issue cleaning up local directories also affects sqoop jobs that create the sqoop.xml file inbetween.

      2018-09-15 00:31:09,416 [AsyncDispatcher event handler] INFO  org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl  - Num completed Tasks: 1
      2018-09-15 00:31:09,417 [AsyncDispatcher event handler] INFO  org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl  - job_1535972033593_259806Job Transitioned from RUNNING to COMMITTING
      2018-09-15 00:31:09,419 [CommitterEvent Processor #1] INFO  org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler  - Processing the event EventType: JOB_COMMIT
      2018-09-15 00:31:09,455 [uber-SubtaskRunner] WARN  org.apache.hadoop.mapred.LocalContainerLauncher  - Unable to delete unexpected local file/dir .action.xml.crc: insufficient permissions?
      2018-09-15 00:31:09,455 [uber-SubtaskRunner] WARN  org.apache.hadoop.mapred.LocalContainerLauncher  - Unable to delete unexpected local file/dir .action.xml.crc: insufficient permissions?
      2018-09-15 00:31:09,459 [CommitterEvent Processor #1] FATAL org.apache.hadoop.conf.Configuration  - error parsing conf sqoop-site.xml
      java.io.FileNotFoundException: /appdata/hdfs/v7/yarn/nm/usercache/SEM/appcache/application_1535972033593_259806/container_e100_1535972033593_259806_01_000001/sqoop-site.xml (No such file or directory)
      	at java.io.FileInputStream.open(Native Method)
      	at java.io.FileInputStream.<init>(FileInputStream.java:146)
      	at java.io.FileInputStream.<init>(FileInputStream.java:101)
      	at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
      	at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
      	at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2483)
      	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2554)
      	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2507)
      	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2413)
      	at org.apache.hadoop.conf.Configuration.get(Configuration.java:984)
      	at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1034)
      	at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1254)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
      	at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.touchz(CommitterEventHandler.java:268)
      	at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:282)
      	at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                z17 Liz Szilagyi
                Reporter:
                cvaliente Clemens Valiente
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: