Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-572

If #link is missing from uri format of -cacheArchive then streaming does not throw error.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: contrib/streaming
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Improved streaming job failure when #link is missing from uri format of -cacheArchive. Earlier it used to fail when launching individual tasks, now it fails during job submission itself.

      Description

      Ran hadoop streaming command as -:
      bin/hadoop jar contrib/streaming/hadoop-*-streaming.jar -input in -output out -mapper "xargs cat" -reducer "bin/cat" -cahceArchive hdfs://h/pathofJarFile
      Streaming submits job to jobtracker and map fails.
      For similar with -cacheFile -:
      bin/hadoop jar contrib/streaming/hadoop-*-streaming.jar -input in -output out -mapper "xargs cat" -reducer "bin/cat" -cahceFile hdfs://h/pathofFile
      followinng error is repoerted back -:
      [
      You need to specify the uris as hdfs://host:port/#linkname,Please specify a different link name for all of your caching URIs
      ]

      Streaming should check about present #link after uri of cacheArchive and should throw proper error .

      1. patch-572-ydist.txt
        9 kB
        Amareshwari Sriramadasu
      2. patch-572-2.txt
        9 kB
        Amareshwari Sriramadasu
      3. patch-572-1.txt
        10 kB
        Amareshwari Sriramadasu
      4. patch-572.txt
        9 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Ravi Gummadi added a comment -

        patch-572-ydist.txt for earlier version of hadoop(and not for commit here) looks good.
        +1.

        Show
        Ravi Gummadi added a comment - patch-572-ydist.txt for earlier version of hadoop(and not for commit here) looks good. +1.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch for Yahoo! distribution

        Show
        Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I just committed this to trunk. Thanks Amareshwari!

        Show
        Vinod Kumar Vavilapalli added a comment - I just committed this to trunk. Thanks Amareshwari!
        Hide
        Amareshwari Sriramadasu added a comment -

        Hudson could not succeed running tests. I ran the tests on my local machine. All the tests passed, expect TestSimulatorDeterministicReplay timeout (MAPREDUCE-1834).

        Show
        Amareshwari Sriramadasu added a comment - Hudson could not succeed running tests. I ran the tests on my local machine. All the tests passed, expect TestSimulatorDeterministicReplay timeout ( MAPREDUCE-1834 ).
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12446002/patch-572-2.txt
        against trunk revision 949931.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/549/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/549/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/549/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/549/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446002/patch-572-2.txt against trunk revision 949931. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/549/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/549/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/549/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/549/console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        +1 for the latest patch. Running it through Hudson. It seems to have lost its sanity of late, showing wierdness towards patch runs for e.g. here, but trying anyways..

        Show
        Vinod Kumar Vavilapalli added a comment - +1 for the latest patch. Running it through Hudson. It seems to have lost its sanity of late, showing wierdness towards patch runs for e.g. here , but trying anyways..
        Hide
        Amareshwari Sriramadasu added a comment -

        Why is conversion of fragment name to lowercase needed ?

        This is done to ignore-case among fragment names.

        Don't we want to support fragment names "Archive" and "archive" as 2 different names ?

        Currently, No. These names are not allowed to be two different symlink names. I think this is done to support operating systems like Windows.

        Show
        Amareshwari Sriramadasu added a comment - Why is conversion of fragment name to lowercase needed ? This is done to ignore-case among fragment names. Don't we want to support fragment names "Archive" and "archive" as 2 different names ? Currently, No. These names are not allowed to be two different symlink names. I think this is done to support operating systems like Windows.
        Hide
        Ravi Gummadi added a comment -

        Why is conversion of fragment name to lowercase needed ?
        Don't we want to support fragment names "Archive" and "archive" as 2 different names ?

        Show
        Ravi Gummadi added a comment - Why is conversion of fragment name to lowercase needed ? Don't we want to support fragment names "Archive" and "archive" as 2 different names ?
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch replaces ArrayList with HashSet and does a contains check before adding to the Set.

        Show
        Amareshwari Sriramadasu added a comment - Patch replaces ArrayList with HashSet and does a contains check before adding to the Set.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Looked at the patch. When we 'check if there is any conflict in fragment names', we can do better than the O (n^2) comparisons to verify if there is any duplicate, for e.g. while even iterating the files/archives to see if any fragment is null, we can put them in a map keyed by fragment name and fail immediately when we encounter duplicates on further iterations?

        Granted this is not in any critical section, I am checking if we can incorporate a minor performance improvement now that the code in question is touched..

        Show
        Vinod Kumar Vavilapalli added a comment - Looked at the patch. When we 'check if there is any conflict in fragment names', we can do better than the O (n^2) comparisons to verify if there is any duplicate, for e.g. while even iterating the files/archives to see if any fragment is null, we can put them in a map keyed by fragment name and fail immediately when we encounter duplicates on further iterations? Granted this is not in any critical section, I am checking if we can incorporate a minor performance improvement now that the code in question is touched..
        Hide
        Giridharan Kesavan added a comment -

        I ve resubmitted the patch to the hudson server as I dont see any trace on the 202 build at all.
        The resubmit helped.

        Show
        Giridharan Kesavan added a comment - I ve resubmitted the patch to the hudson server as I dont see any trace on the 202 build at all. The resubmit helped.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12445297/patch-572-1.txt
        against trunk revision 947758.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/203/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/203/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/203/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/203/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445297/patch-572-1.txt against trunk revision 947758. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/203/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/203/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/203/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/203/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        The patch should not introduce any failures. Changes are same as in earlier patch, expect for indentation.
        I could not view the testReport from the above link. Giri, Can you please help?

        Show
        Amareshwari Sriramadasu added a comment - The patch should not introduce any failures. Changes are same as in earlier patch, expect for indentation. I could not view the testReport from the above link. Giri, Can you please help?
        Hide
        Ravi Gummadi added a comment -

        Latest patch looks good.
        +1

        Show
        Ravi Gummadi added a comment - Latest patch looks good. +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12445297/patch-572-1.txt
        against trunk revision 947112.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/202/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/202/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/202/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/202/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445297/patch-572-1.txt against trunk revision 947112. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/202/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/202/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/202/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/202/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch adds unit test for TrackerDistributedCacheManager.checkURIs.
        I ran streaming command with -cacheArchive, without passing symlink name. It throws the error properly.

        Show
        Amareshwari Sriramadasu added a comment - Patch adds unit test for TrackerDistributedCacheManager.checkURIs. I ran streaming command with -cacheArchive, without passing symlink name. It throws the error properly.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch corrects the indentation

        Show
        Amareshwari Sriramadasu added a comment - Patch corrects the indentation
        Hide
        Ravi Gummadi added a comment -

        Patch looks good to me.
        Very minor comment:

        if (frag1.equalsIgnoreCase(frag2))
          return false;
        

        Please add curly braces for the if statement.

        Show
        Ravi Gummadi added a comment - Patch looks good to me. Very minor comment: if (frag1.equalsIgnoreCase(frag2)) return false ; Please add curly braces for the if statement.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12445034/patch-572.txt
        against trunk revision 946833.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/198/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/198/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/198/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/198/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445034/patch-572.txt against trunk revision 946833. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/198/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/198/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/198/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/198/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Test failures are not related to the patch. They failed because of NoClassDefFoundError.
        Re-running through hudson.

        Show
        Amareshwari Sriramadasu added a comment - Test failures are not related to the patch. They failed because of NoClassDefFoundError. Re-running through hudson.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12445034/patch-572.txt
        against trunk revision 946526.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/195/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/195/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/195/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/195/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445034/patch-572.txt against trunk revision 946526. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/195/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/195/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/195/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/195/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch fixing the bug.

        Show
        Amareshwari Sriramadasu added a comment - Patch fixing the bug.
        Hide
        Karam Singh added a comment -

        Looking at code -:
        StreamJob.java (line 845)-:
        [
        boolean b = DistributedCache.checkURIs(fileURIs, archiveURIs);
        if (!b)
        fail(LINK_URI);
        }

        ]

        It is observed the StreamJob.java is calling checkURIs of Distributed.
        Looking at ChekURIs code from org.apahe.hadoop.DistributedCache.java (line 716 onwrds) -:
        [
        if (uriFiles != null){
        for (int i = 0; i < uriFiles.length; i++){
        String frag1 = uriFiles[i].getFragment();
        if (frag1 == null)
        return false;
        for (int j=i+1; j < uriFiles.length; j++)

        { String frag2 = uriFiles[j].getFragment(); if (frag2 == null) return false; if (frag1.equalsIgnoreCase(frag2)) return false; }

        if (uriArchives != null){
        for (int j = 0; j < uriArchives.length; j++){
        String frag2 = uriArchives[j].getFragment();
        if (frag2 == null)

        { return false; }

        if (frag1.equalsIgnoreCase(frag2))
        return false;
        for (int k=j+1; k < uriArchives.length; k++)

        { String frag3 = uriArchives[k].getFragment(); if (frag3 == null) return false; if (frag2.equalsIgnoreCase(frag3)) return false; }

        }
        }
        }
        }
        return true;
        ]

        It seems that if uriFiles is null it does no checks for uriArchives. So if -cacheFile option is not present then it will validate cacheArchive uris

        Show
        Karam Singh added a comment - Looking at code -: StreamJob.java (line 845)-: [ boolean b = DistributedCache.checkURIs(fileURIs, archiveURIs); if (!b) fail(LINK_URI); } ] It is observed the StreamJob.java is calling checkURIs of Distributed. Looking at ChekURIs code from org.apahe.hadoop.DistributedCache.java (line 716 onwrds) -: [ if (uriFiles != null){ for (int i = 0; i < uriFiles.length; i++){ String frag1 = uriFiles [i] .getFragment(); if (frag1 == null) return false; for (int j=i+1; j < uriFiles.length; j++) { String frag2 = uriFiles[j].getFragment(); if (frag2 == null) return false; if (frag1.equalsIgnoreCase(frag2)) return false; } if (uriArchives != null){ for (int j = 0; j < uriArchives.length; j++){ String frag2 = uriArchives [j] .getFragment(); if (frag2 == null) { return false; } if (frag1.equalsIgnoreCase(frag2)) return false; for (int k=j+1; k < uriArchives.length; k++) { String frag3 = uriArchives[k].getFragment(); if (frag3 == null) return false; if (frag2.equalsIgnoreCase(frag3)) return false; } } } } } return true; ] It seems that if uriFiles is null it does no checks for uriArchives. So if -cacheFile option is not present then it will validate cacheArchive uris

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Karam Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development