Oozie
  1. Oozie
  2. OOZIE-1211

oozie does not support duplicated dataset

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: trunk, 3.3.1
    • Fix Version/s: 3.3.2
    • Component/s: coordinator
    • Labels:
      None

      Description

      in the example below, dataset B is defined in both coordinator.xml and include dataset file. oozie fails to submit job with error:

      Error: E0803 : E0803: IO error, null

      coordinator.xml:
      <datasets>
      <include>$

      {include_ds_files}</include>
      <dataset name="B" frequency="${coord:minutes(5)}" initial-instance="${ds_start}" timezone="${timezone}">
      <uri-template>hcat://${hcat_server}/${hcat_out_db}/${hcat_out_table}/datestamp=${YEAR}${MONTH}${DAY}${HOUR}${MINUTE};region=${hcat_region}</uri-template>
      </dataset>
      </datasets>

      include_ds_files:
      <datasets>
      <include>${include_ds_files}

      </include>
      <dataset name="B" frequency="$

      {coord:minutes(5)}

      " initial-instance="$

      {ds_start}

      " timezone="$

      {timezone}

      ">
      <uri-template>hcat://$

      {hcat_server}

      /$

      {hcat_out_db}

      /$

      {hcat_out_table}

      /datestamp=$

      {YEAR}

      $

      {MONTH}

      $

      {DAY}

      $

      {HOUR}

      $

      {MINUTE}

      ;region=$

      {hcat_region}

      </uri-template>
      </dataset>
      </datasets>

      oozie log:
      2013-02-07 01:02:01,308 WARN CoordSubmitXCommand:544 - USER[mchiang] GROUP[users] TOKEN[-] APP[-] JOB[-] ACTION[-] ERROR:
      java.util.ConcurrentModificationException
      at org.jdom.ContentList$FilterListIterator.checkConcurrentModification(ContentList.java:940)
      at org.jdom.ContentList$FilterListIterator.nextIndex(ContentList.java:829)
      at org.jdom.ContentList$FilterListIterator.hasNext(ContentList.java:785)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.removeDataSet(CoordSubmitXCommand.java:1017)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.includeDataSets(CoordSubmitXCommand.java:952)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.basicResolveAndIncludeDS(CoordSubmitXCommand.java:558)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:225)
      at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:81)
      at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:29)
      at org.apache.oozie.command.XCommand.call(XCommand.java:277)
      at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:375)
      at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:219)
      at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:83)
      at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:100)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
      at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:286)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at com.yahoo.oozie.security.authentication.filter.YAuthFilter$2.doFilter(YAuthFilter.java:135)
      at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372)
      at com.yahoo.oozie.security.authentication.filter.YAuthFilter.doFilter(YAuthFilter.java:139)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:68)
      at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:52)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:447)
      at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:219)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:104)
      at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:50)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:104)
      at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:65)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:219)
      at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:143)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at yjava.tomcat.valves.YahooConnectionValve.invoke(YahooConnectionValve.java:191)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      at java.lang.Thread.run(Thread.java:662)
      2013-02-07 01:02:01,309 ERROR CoordSubmitXCommand:538 - USER[mchiang] GROUP[users] TOKEN[-] APP[-] JOB[-] ACTION[-] XException,
      org.apache.oozie.command.CommandException: E0803: IO error, null
      at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:297)
      at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:81)
      at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:29)
      at org.apache.oozie.command.XCommand.call(XCommand.java:277)
      at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:375)
      at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:219)
      at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:83)
      at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:100)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
      at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:286)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at com.yahoo.oozie.security.authentication.filter.YAuthFilter$2.doFilter(YAuthFilter.java:135)
      at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372)
      at com.yahoo.oozie.security.authentication.filter.YAuthFilter.doFilter(YAuthFilter.java:139)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:68)
      at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:52)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:447)
      at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:219)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:104)
      at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:50)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:104)
      at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:65)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:219)
      at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:143)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at yjava.tomcat.valves.YahooConnectionValve.invoke(YahooConnectionValve.java:191)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.util.ConcurrentModificationException
      at org.jdom.ContentList$FilterListIterator.checkConcurrentModification(ContentList.java:940)
      at org.jdom.ContentList$FilterListIterator.nextIndex(ContentList.java:829)
      at org.jdom.ContentList$FilterListIterator.hasNext(ContentList.java:785)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.removeDataSet(CoordSubmitXCommand.java:1017)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.includeDataSets(CoordSubmitXCommand.java:952)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.basicResolveAndIncludeDS(CoordSubmitXCommand.java:558)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:225)
      ... 53 more
      2013-02-07 01:02:01,310 WARN V1JobsServlet:544 - USER[mchiang] GROUP[users] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://gsbl90547.blue.ygrid.yahoo.com:4080/oozie/v1/jobs?action=start] error[E0803], E0803: IO error, null
      org.apache.oozie.servlet.XServletException: E0803: IO error, null
      at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:224)
      at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:83)
      at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:100)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
      at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:286)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at com.yahoo.oozie.security.authentication.filter.YAuthFilter$2.doFilter(YAuthFilter.java:135)
      at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372)
      at com.yahoo.oozie.security.authentication.filter.YAuthFilter.doFilter(YAuthFilter.java:139)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:68)
      at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:52)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:447)
      at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:219)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:104)
      at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:50)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:104)
      at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:65)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
      at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:219)
      at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:143)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at yjava.tomcat.valves.YahooConnectionValve.invoke(YahooConnectionValve.java:191)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: org.apache.oozie.CoordinatorEngineException: E0803: IO error, null
      at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:378)
      at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:219)
      ... 48 more
      Caused by: org.apache.oozie.command.CommandException: E0803: IO error, null
      at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:297)
      at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:81)
      at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:29)
      at org.apache.oozie.command.XCommand.call(XCommand.java:277)
      at org.apache.oozie.CoordinatorEngine.submitJob(CoordinatorEngine.java:375)
      ... 49 more
      Caused by: java.util.ConcurrentModificationException
      at org.jdom.ContentList$FilterListIterator.checkConcurrentModification(ContentList.java:940)
      at org.jdom.ContentList$FilterListIterator.nextIndex(ContentList.java:829)
      at org.jdom.ContentList$FilterListIterator.hasNext(ContentList.java:785)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.removeDataSet(CoordSubmitXCommand.java:1017)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.includeDataSets(CoordSubmitXCommand.java:952)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.basicResolveAndIncludeDS(CoordSubmitXCommand.java:558)
      at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:225)
      ... 53 more

      1. OOZIE-1211-3.patch
        10 kB
        jun aoki
      2. OOZIE-1211-2.patch
        10 kB
        jun aoki
      3. OOZIE-1211.patch
        10 kB
        jun aoki

        Activity

        Hide
        jun aoki added a comment -

        So if I read it right, the expected result is the B in the coordinator.xml file show be honored. Am I right?
        http://oozie.apache.org/docs/3.3.1/CoordinatorFunctionalSpec.html#a10.1.1._Dataset_Names_Collision_Resolution

        Show
        jun aoki added a comment - So if I read it right, the expected result is the B in the coordinator.xml file show be honored. Am I right? http://oozie.apache.org/docs/3.3.1/CoordinatorFunctionalSpec.html#a10.1.1._Dataset_Names_Collision_Resolution
        Hide
        Virag Kothari added a comment -

        Correct. The expected result is B in coordinator.xml and the duplicate B in dataset file should be ignored.
        It seems like the collection is modified while it is being iterated over causing ConcurrentModificationException

        Show
        Virag Kothari added a comment - Correct. The expected result is B in coordinator.xml and the duplicate B in dataset file should be ignored. It seems like the collection is modified while it is being iterated over causing ConcurrentModificationException
        Hide
        jun aoki added a comment -

        I removed <include> tag from the include file, and am able to reproduce it, whereas the original Description contains <include> tag in the both files. I assume I get it right.

        include.xml
        <datasets> 
        <!--	<include>some_path</include> Removed from the original description -->
        	<dataset name="B" frequency="${coord:days(7)}" initial-instance="2009-02-01T01:00Z" timezone="UTC">
        		<uri-template>file:///tmp/coord/workflows/${YEAR}/${DAY}</uri-template>
        	</dataset> 
        </datasets>
        
        coordinator.xml
        <datasets> 
        	<include>include.xml</include>
        	<dataset name="B" frequency="${coord:days(7)}" initial-instance="2009-02-01T01:00Z" timezone="UTC">
        		<uri-template>file:///tmp/coord/workflows/${YEAR}/${DAY}</uri-template>
        	</dataset> 
        </datasets>
        
        Show
        jun aoki added a comment - I removed <include> tag from the include file, and am able to reproduce it, whereas the original Description contains <include> tag in the both files. I assume I get it right. include.xml <datasets> <!-- <include>some_path</include> Removed from the original description --> <dataset name= "B" frequency= "${coord:days(7)}" initial-instance= "2009-02-01T01:00Z" timezone= "UTC" > <uri-template>file: ///tmp/coord/workflows/${YEAR}/${DAY}</uri-template> </dataset> </datasets> coordinator.xml <datasets> <include>include.xml</include> <dataset name= "B" frequency= "${coord:days(7)}" initial-instance= "2009-02-01T01:00Z" timezone= "UTC" > <uri-template>file: ///tmp/coord/workflows/${YEAR}/${DAY}</uri-template> </dataset> </datasets>
        Hide
        Virag Kothari added a comment -

        Correct. The problem is due to the duplicate dataset and <include> shouldn't matter.
        Thanks for working on this.

        Show
        Virag Kothari added a comment - Correct. The problem is due to the duplicate dataset and <include> shouldn't matter. Thanks for working on this.
        Hide
        jun aoki added a comment -

        Virag, thank you for sharing the information and clarifications.
        Please review once my patch passes the precommit.

        Show
        jun aoki added a comment - Virag, thank you for sharing the information and clarifications. Please review once my patch passes the precommit.
        Hide
        Virag Kothari added a comment -

        looks good..
        one question, any reason for decrementing 'i' in 'if'?

        for (int i = datasetElementsLength - 1; i >= 0; i--) {
                    Element eDataset = datasetElements.get(i);
                     if (eDataset.getAttributeValue("name").equals(name)) {
                         eDataset.detach();
                       i--;
                     }
                 }
        

        Also, Oozie doesn't have the convention of including JIRA number in java-docs

        /**
            * https://issues.apache.org/jira/browse/OOZIE-1211
        
        Show
        Virag Kothari added a comment - looks good.. one question, any reason for decrementing 'i' in 'if'? for ( int i = datasetElementsLength - 1; i >= 0; i--) { Element eDataset = datasetElements.get(i); if (eDataset.getAttributeValue( "name" ).equals(name)) { eDataset.detach(); i--; } } Also, Oozie doesn't have the convention of including JIRA number in java-docs /** * https: //issues.apache.org/jira/browse/OOZIE-1211
        Hide
        Hadoop QA added a comment -

        Testing JIRA OOZIE-1211

        Cleaning local svn workspace

        ----------------------------

        +1 PATCH_APPLIES
        +1 CLEAN
        +1 RAW_PATCH_ANALYSIS
        . +1 the patch does not introduce any @author tags
        . +1 the patch does not introduce any tabs
        . +1 the patch does not introduce any trailing spaces
        . +1 the patch does not introduce any line longer than 132
        . +1 the patch does adds/modifies 1 testcase(s)
        +1 RAT
        . +1 the patch does not seem to introduce new RAT warnings
        +1 JAVADOC
        . +1 the patch does not seem to introduce new Javadoc warnings
        +1 COMPILE
        . +1 HEAD compiles
        . +1 patch compiles
        . +1 the patch does not seem to introduce new javac warnings
        +1 BACKWARDS_COMPATIBILITY
        . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
        . +1 the patch does not modify JPA files
        +1 TESTS
        . Tests run: 951
        +1 DISTRO
        . +1 distro tarball builds with the patch

        ----------------------------
        +1 Overall result, good!, no -1s

        The full output of the test-patch run is available at

        . https://builds.apache.org/job/oozie-trunk-precommit-build/322/

        Show
        Hadoop QA added a comment - Testing JIRA OOZIE-1211 Cleaning local svn workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . +1 the patch does adds/modifies 1 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 951 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- +1 Overall result, good!, no -1s The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/322/
        Hide
        jun aoki added a comment -

        Virag, thank you for your feedback.

        If incrementing, you are looking for C and remove it,

        i == 0

        i 1 2 3 4 5
        Data A B C D E

        i == 3, and C is removed

        i 1 2 3 4 5
        Data A B D E

        the next i, i == 4 is examining E and D is not skipped.

        Show
        jun aoki added a comment - Virag, thank you for your feedback. If incrementing, you are looking for C and remove it, i == 0 i 1 2 3 4 5 Data A B C D E i == 3, and C is removed i 1 2 3 4 5 Data A B D E the next i, i == 4 is examining E and D is not skipped.
        Hide
        jun aoki added a comment -

        but the backward search avoid this issue. (and it was the root cause of this bug with Iterator)

        For the ticket number, I made the test that tests for a very specific case fixed in this ticket.
        So if anybody has to read through the test, it is better to have a link to this ticket and get the context behind.
        But I understand Oozie does not mandate a link to tickets so if you really don't think it is appropriate I can take it out. Let me know.

        Otherwise, the precommit passed so please commit the patch.

        Show
        jun aoki added a comment - but the backward search avoid this issue. (and it was the root cause of this bug with Iterator) For the ticket number, I made the test that tests for a very specific case fixed in this ticket. So if anybody has to read through the test, it is better to have a link to this ticket and get the context behind. But I understand Oozie does not mandate a link to tickets so if you really don't think it is appropriate I can take it out. Let me know. Otherwise, the precommit passed so please commit the patch.
        Hide
        jun aoki added a comment -

        In my two previous comment, I meant the next i, when i == 4, E is examined but D is skipped. sorry my typo.

        Show
        jun aoki added a comment - In my two previous comment, I meant the next i, when i == 4, E is examined but D is skipped. sorry my typo.
        Hide
        Virag Kothari added a comment -

        I didn't ask about incrementing and was probably not clear.
        My question was why is the counter 'i' decremented in 'if' as it is already decremented as part of the for loop.
        Also, it seems that there can be max one duplicate, so can we get away by returning as soon as the duplicate is found. This will not give the chance for the iterator to throw concurrentModificationException as it would not that the list is changed.
        I believe that the case for multiple duplicates is already handled before it reaches removeDateSet(). But can you make sure about that? Thanks

         for (Element eDataset : (List<Element>) eDatasets.getChildren("dataset", eDatasets.getNamespace())) {
                    if (eDataset.getAttributeValue("name").equals(name)) {
                        eDataset.detach();
                        return; //
        
                    }
                }
        
        Show
        Virag Kothari added a comment - I didn't ask about incrementing and was probably not clear. My question was why is the counter 'i' decremented in 'if' as it is already decremented as part of the for loop. Also, it seems that there can be max one duplicate, so can we get away by returning as soon as the duplicate is found. This will not give the chance for the iterator to throw concurrentModificationException as it would not that the list is changed. I believe that the case for multiple duplicates is already handled before it reaches removeDateSet(). But can you make sure about that? Thanks for (Element eDataset : (List<Element>) eDatasets.getChildren( "dataset" , eDatasets.getNamespace())) { if (eDataset.getAttributeValue( "name" ).equals(name)) { eDataset.detach(); return ; // } }
        Hide
        jun aoki added a comment -

        Hi Virag, thank you for catching the i-- thing and it is not necessary, my bad.
        The reason I intended through out all elements is I had an impression that dataset can be duplicate since it is in List. (If it were HashSet or something I would've seen the uniqueness guaranteed).
        I found includeOneDSFile should set it unique, as you pointed out.

        Agreed and fixed the patch.

        Show
        jun aoki added a comment - Hi Virag, thank you for catching the i-- thing and it is not necessary, my bad. The reason I intended through out all elements is I had an impression that dataset can be duplicate since it is in List. (If it were HashSet or something I would've seen the uniqueness guaranteed). I found includeOneDSFile should set it unique, as you pointed out. Agreed and fixed the patch.
        Hide
        Virag Kothari added a comment -

        +1 pending jenkins
        Agree that it was hard to follow the code the way it is written

        Show
        Virag Kothari added a comment - +1 pending jenkins Agree that it was hard to follow the code the way it is written
        Hide
        Hadoop QA added a comment -

        Testing JIRA OOZIE-1211

        Cleaning local svn workspace

        ----------------------------

        +1 PATCH_APPLIES
        +1 CLEAN
        +1 RAW_PATCH_ANALYSIS
        . +1 the patch does not introduce any @author tags
        . +1 the patch does not introduce any tabs
        . +1 the patch does not introduce any trailing spaces
        . +1 the patch does not introduce any line longer than 132
        . +1 the patch does adds/modifies 1 testcase(s)
        +1 RAT
        . +1 the patch does not seem to introduce new RAT warnings
        +1 JAVADOC
        . +1 the patch does not seem to introduce new Javadoc warnings
        +1 COMPILE
        . +1 HEAD compiles
        . +1 patch compiles
        . +1 the patch does not seem to introduce new javac warnings
        +1 BACKWARDS_COMPATIBILITY
        . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
        . +1 the patch does not modify JPA files
        -1 TESTS
        . Tests run: 951
        . Tests failed: 0
        . Tests errors: 1

        . The patch failed the following testcases:

        .

        +1 DISTRO
        . +1 distro tarball builds with the patch

        ----------------------------
        -1 Overall result, please check the reported -1(s)

        The full output of the test-patch run is available at

        . https://builds.apache.org/job/oozie-trunk-precommit-build/323/

        Show
        Hadoop QA added a comment - Testing JIRA OOZIE-1211 Cleaning local svn workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . +1 the patch does adds/modifies 1 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS . Tests run: 951 . Tests failed: 0 . Tests errors: 1 . The patch failed the following testcases: . +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/323/
        Hide
        jun aoki added a comment -

        The test failure looks irrelevant. Running again.

        Show
        jun aoki added a comment - The test failure looks irrelevant. Running again.
        Hide
        Virag Kothari added a comment -

        It is ok to just specify that the failure is unrelated.
        For this we can have another run as you have re-attached the patch.

        Show
        Virag Kothari added a comment - It is ok to just specify that the failure is unrelated. For this we can have another run as you have re-attached the patch.
        Hide
        Hadoop QA added a comment -

        Testing JIRA OOZIE-1211

        Cleaning local svn workspace

        ----------------------------

        +1 PATCH_APPLIES
        +1 CLEAN
        +1 RAW_PATCH_ANALYSIS
        . +1 the patch does not introduce any @author tags
        . +1 the patch does not introduce any tabs
        . +1 the patch does not introduce any trailing spaces
        . +1 the patch does not introduce any line longer than 132
        . +1 the patch does adds/modifies 1 testcase(s)
        +1 RAT
        . +1 the patch does not seem to introduce new RAT warnings
        +1 JAVADOC
        . +1 the patch does not seem to introduce new Javadoc warnings
        +1 COMPILE
        . +1 HEAD compiles
        . +1 patch compiles
        . +1 the patch does not seem to introduce new javac warnings
        +1 BACKWARDS_COMPATIBILITY
        . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
        . +1 the patch does not modify JPA files
        +1 TESTS
        . Tests run: 951
        +1 DISTRO
        . +1 distro tarball builds with the patch

        ----------------------------
        +1 Overall result, good!, no -1s

        The full output of the test-patch run is available at

        . https://builds.apache.org/job/oozie-trunk-precommit-build/324/

        Show
        Hadoop QA added a comment - Testing JIRA OOZIE-1211 Cleaning local svn workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . +1 the patch does adds/modifies 1 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 951 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- +1 Overall result, good!, no -1s The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/324/
        Hide
        Virag Kothari added a comment -

        Committed to trunk and hcat-intre. Thanks Jun!

        Show
        Virag Kothari added a comment - Committed to trunk and hcat-intre. Thanks Jun!
        Hide
        Robert Kanter added a comment -

        Closing issue; Oozie 3.3.2 is released

        Show
        Robert Kanter added a comment - Closing issue; Oozie 3.3.2 is released

          People

          • Assignee:
            jun aoki
            Reporter:
            michelle chiang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development