Hadoop Common
  1. Hadoop Common
  2. HADOOP-8409

Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for windows

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1-win
    • Component/s: fs, test, util
    • Labels:
      None
    • Target Version/s:

      Description

      There are multiple places in prod and test code where Windows paths are not handled properly. From a high level this could be summarized with:
      1. Windows paths are not necessarily valid DFS paths (while Unix paths are)
      2. Windows paths are not necessarily valid URIs (while Unix paths are)

      #1 causes a number of tests to fail because they implicitly assume that local paths are valid DFS paths (by extracting the DFS test path from for example "test.build.data" property)

      #2 causes issues when URIs are directly created on path strings passed in by the user

      1. HADOOP-8409-branch-1-win.2.patch
        4 kB
        Ivan Mitic
      2. HADOOP-8409-branch-1-win.3.patch
        4 kB
        Ivan Mitic
      3. HADOOP-8409-branch-1-win.4.patch
        4 kB
        Ivan Mitic
      4. HADOOP-8409-branch-1-win.patch
        24 kB
        Ivan Mitic

        Issue Links

          Activity

          Hide
          Sanjay Radia added a comment -

          Committed to hadoop-1 windows branch.
          Thanks Ivan.

          Show
          Sanjay Radia added a comment - Committed to hadoop-1 windows branch. Thanks Ivan.
          Hide
          Sanjay Radia added a comment -

          Rename title to reflect the narrower scope.

          Show
          Sanjay Radia added a comment - Rename title to reflect the narrower scope.
          Hide
          Ivan Mitic added a comment -

          Running through dos2unix.

          Show
          Ivan Mitic added a comment - Running through dos2unix.
          Hide
          Daryn Sharp added a comment -

          +1 Patch looks good as-is.

          In a prior code, you were showing that new File("some/path").toString() would return backslashes. I didn't know if it would help to have a Path ctor take a File and invoke the ctor for a uri/string. Just an idea to cut down on redundant code in the callers and reduce the chance someone will break windows when they don't know they have to specially convert files to paths.

          Show
          Daryn Sharp added a comment - +1 Patch looks good as-is. In a prior code, you were showing that new File("some/path").toString() would return backslashes. I didn't know if it would help to have a Path ctor take a File and invoke the ctor for a uri/string. Just an idea to cut down on redundant code in the callers and reduce the chance someone will break windows when they don't know they have to specially convert files to paths.
          Hide
          Ivan Mitic added a comment -

          bp. Not a big deal, but Path(parent,child) can be used in a couple places instead of a string concat.
          Fixed

          bp. Would a Path#toFile() that I mentioned in a related jira maybe help in a few cases?
          Hm... not sure I see it. Did you maybe think of using new Path(file).toUri() instead?

          Show
          Ivan Mitic added a comment - bp. Not a big deal, but Path(parent,child) can be used in a couple places instead of a string concat. Fixed bp. Would a Path#toFile() that I mentioned in a related jira maybe help in a few cases? Hm... not sure I see it. Did you maybe think of using new Path(file).toUri() instead?
          Hide
          Ivan Mitic added a comment -

          Attaching new patch.

          Show
          Ivan Mitic added a comment - Attaching new patch.
          Hide
          Daryn Sharp added a comment -

          Not a big deal, but Path(parent,child) can be used in a couple places instead of a string concat. Would a Path#toFile() that I mentioned in a related jira maybe help in a few cases?

          Show
          Daryn Sharp added a comment - Not a big deal, but Path(parent,child) can be used in a couple places instead of a string concat. Would a Path#toFile() that I mentioned in a related jira maybe help in a few cases?
          Hide
          Ivan Mitic added a comment -

          Attaching patch that fixes test failures in TestGenericOptionsParser and TestCommandLineJobSubmission.

          As GenericOptionsParser only accepts valid URIs, and File#toString() always normalizes the path to the native OS path, we must first covert a File to an URI before we can pass it to GenericOptionsParser.

          For illustration purposes, on Windows:

          new File("some/path")#toString() == some\path
          new File("c:/some/path")#toString() == c:\some\path
          new File("/c:/some/path")#toString() == c:\some\path
          
          Show
          Ivan Mitic added a comment - Attaching patch that fixes test failures in TestGenericOptionsParser and TestCommandLineJobSubmission. As GenericOptionsParser only accepts valid URIs, and File#toString() always normalizes the path to the native OS path, we must first covert a File to an URI before we can pass it to GenericOptionsParser. For illustration purposes, on Windows: new File( "some/path" )#toString() == some\path new File( "c:/some/path" )#toString() == c:\some\path new File( "/c:/some/path" )#toString() == c:\some\path
          Hide
          Ivan Mitic added a comment -

          Thanks Daryn, that makes sense. We finally decided not to further extend the support for backslash. This means that input paths to GenericOptionsParser (files/archives/libjars) must be valid URIs on Windows!

          I will repurpose this Jira to fix test failures in TestGenericOptionsParser and TestCommandLineJobSubmission in line with this decision.

          Also, I have created two new Jiras based on the original patch attached here, as these issue can generally stand on their own:

          Separately, we are still discussing possible solutions for HADOOP-8139. My goal was to break down the original problem into smaller isolated pieces so that we can make incremental progress in parallel.

          Show
          Ivan Mitic added a comment - Thanks Daryn, that makes sense. We finally decided not to further extend the support for backslash. This means that input paths to GenericOptionsParser (files/archives/libjars) must be valid URIs on Windows! I will repurpose this Jira to fix test failures in TestGenericOptionsParser and TestCommandLineJobSubmission in line with this decision. Also, I have created two new Jiras based on the original patch attached here, as these issue can generally stand on their own: HADOOP-8487 MAPREDUCE-4321 Separately, we are still discussing possible solutions for HADOOP-8139 . My goal was to break down the original problem into smaller isolated pieces so that we can make incremental progress in parallel.
          Hide
          Daryn Sharp added a comment -

          To further clarify, nothing else that uses a Path is prepared to handle fragments, so the fragment will be silently munched off the uri. In the prior example, the path effectively becomes "." which is the cwd. Trying to remove a qualified path such as "some/path/#1" would remove the parent directory. I think it's prudent to not spread the DC's semantics of fragments throughout hadoop when nothing else is prepared to deal with fragments.

          Show
          Daryn Sharp added a comment - To further clarify, nothing else that uses a Path is prepared to handle fragments, so the fragment will be silently munched off the uri. In the prior example, the path effectively becomes "." which is the cwd. Trying to remove a qualified path such as "some/path/#1" would remove the parent directory. I think it's prudent to not spread the DC's semantics of fragments throughout hadoop when nothing else is prepared to deal with fragments.
          Hide
          Daryn Sharp added a comment -

          Ah, I see. While it saves a little bit of code in the GenericOptionsParser, I don't think it warrants a very dangerous and incompatible change to Path. The use of the fragment in a URI is very specific behavior to the DC, and not outside of that context.

          Let's take an example: I create a directory called "#1". I assume that I can run "hadoop fs -rm -r '#1'". By supporting fragments in paths, that now becomes a recursive removal of your entire home directory!

          Show
          Daryn Sharp added a comment - Ah, I see. While it saves a little bit of code in the GenericOptionsParser , I don't think it warrants a very dangerous and incompatible change to Path . The use of the fragment in a URI is very specific behavior to the DC, and not outside of that context. Let's take an example: I create a directory called "#1". I assume that I can run "hadoop fs -rm -r '#1'". By supporting fragments in paths, that now becomes a recursive removal of your entire home directory!
          Hide
          Ivan Mitic added a comment -

          In the meantime, would you please elaborate on how uri fragments are related to supporting symlinks?

          Sure. Symlinks/fragments are used for Distributed Cache. For example, to add files to Distributed Cache, you would do something like this in code:

          DistributedCache.addCacheArchive(new URI("s3://bucket/path/to/archive.zip#directory"), job);
          

          and this works fine. However, if you try to do something like this:

          DistributedCache.addCacheArchive(new Path("s3://bucket/path/to/archive.zip#directory").toUri(), job);
          

          this will fail because Path object does not support fragments. In this case, ‘#’ sign will be encoded into the URI path: s3://bucket/path/to/archive.zip%23directory

          I run into this while fixing TestGenericOptionsParser that was failing on Windows. However, I found some forum posts where people actually used the incorrect pattern. This might be a good change orthogonally to what we do for paths. If you agree, maybe split fragment support into a new Jira?

          Please also look at the issues raised in HADOOP-8139 and the reasons why we did not support windows paths on HDFS.

          Thanks Suresh, I am aware of the issues raised here.

          Show
          Ivan Mitic added a comment - In the meantime, would you please elaborate on how uri fragments are related to supporting symlinks? Sure. Symlinks/fragments are used for Distributed Cache . For example, to add files to Distributed Cache, you would do something like this in code: DistributedCache.addCacheArchive( new URI( "s3: //bucket/path/to/archive.zip#directory" ), job); and this works fine. However, if you try to do something like this: DistributedCache.addCacheArchive( new Path( "s3: //bucket/path/to/archive.zip#directory" ).toUri(), job); this will fail because Path object does not support fragments. In this case, ‘#’ sign will be encoded into the URI path: s3://bucket/path/to/archive.zip%23directory I run into this while fixing TestGenericOptionsParser that was failing on Windows. However, I found some forum posts where people actually used the incorrect pattern. This might be a good change orthogonally to what we do for paths. If you agree, maybe split fragment support into a new Jira? Please also look at the issues raised in HADOOP-8139 and the reasons why we did not support windows paths on HDFS. Thanks Suresh, I am aware of the issues raised here.
          Hide
          Suresh Srinivas added a comment -

          In line with this, I spent quite a bit of time thinking about pros and cons to having Path object support backslash VS not. Both approaches have legitimate pros and cons. Once I sum them up on my end, I'll reply back.

          @Please also look at the issues raised in HADOOP-8139 and the reasons why we did not support windows paths on HDFS.

          Show
          Suresh Srinivas added a comment - In line with this, I spent quite a bit of time thinking about pros and cons to having Path object support backslash VS not. Both approaches have legitimate pros and cons. Once I sum them up on my end, I'll reply back. @Please also look at the issues raised in HADOOP-8139 and the reasons why we did not support windows paths on HDFS.
          Hide
          Daryn Sharp added a comment -

          You'll need to persuade people like Suresh about expanding support for \ . In the meantime, would you please elaborate on how uri fragments are related to supporting symlinks?

          Show
          Daryn Sharp added a comment - You'll need to persuade people like Suresh about expanding support for \ . In the meantime, would you please elaborate on how uri fragments are related to supporting symlinks?
          Hide
          Ivan Mitic added a comment -

          Thanks guys for your comments and valid concerns! Daryn, before you brought up the issue with the backslash, I was building based on the assumption that it is supported by Hadoop (hence the disconnect).

          In line with this, I spent quite a bit of time thinking about pros and cons to having Path object support backslash VS not. Both approaches have legitimate pros and cons. Once I sum them up on my end, I'll reply back.

          Show
          Ivan Mitic added a comment - Thanks guys for your comments and valid concerns! Daryn, before you brought up the issue with the backslash, I was building based on the assumption that it is supported by Hadoop (hence the disconnect). In line with this, I spent quite a bit of time thinking about pros and cons to having Path object support backslash VS not. Both approaches have legitimate pros and cons. Once I sum them up on my end, I'll reply back.
          Hide
          Daryn Sharp added a comment -

          My concern is this patch appears to be adding further support for "
          " paths by not constructing URIs. While it's debatable which Path ctor should be used, I'm leary of expanding compatibility with "
          " paths when it's supposed to be removed...

          Show
          Daryn Sharp added a comment - My concern is this patch appears to be adding further support for " " paths by not constructing URIs. While it's debatable which Path ctor should be used, I'm leary of expanding compatibility with " " paths when it's supposed to be removed...
          Hide
          Bikas Saha added a comment -

          For a moment lets retrace back to where the comments started.
          When I looked at the code it did not seem to me that the changes were allowing support for a\b\c type Windows paths. It seemed like some places in Hadoop dev/test code were concatenating/manipulating paths which did not deal with the c: properly when running on windows.
          Is that not the case?
          Which particular change are we discussing here?

          Show
          Bikas Saha added a comment - For a moment lets retrace back to where the comments started. When I looked at the code it did not seem to me that the changes were allowing support for a\b\c type Windows paths. It seemed like some places in Hadoop dev/test code were concatenating/manipulating paths which did not deal with the c: properly when running on windows. Is that not the case? Which particular change are we discussing here?
          Hide
          Daryn Sharp added a comment -

          Aksing users to enter input paths in form "c:/some/path" does not seem like the right thing to do. Please let me know if you agree with me here.

          Unfortunately I disagree...

          HADOOP-8139 has a long discussion, but the short summary is: Hadoop uses URIs, so the hadoop veterans decided the hacky support for \ needs to be removed. Although "dir/file" might look like a native path, it's still a URI being prepending with the default fs URI. I spun off HADOOP-8164 to fix the meta-char quoting issue caused by the windows hack – but bug persists on windows!

          HADOOP-8139 was linked to the window umbrella jira (HADOOP-8079) over a year ago, with the reporter Alexander quoted as "We are only using backslashes () for escaping meta-characters and not as alternative path-separators. So, things should work in a uniform manner."

          Of note, it looks like for at least the past 6 years, MS has said URIs using
          is deprecated? http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx

          Actually, this does not work for paths that are symlinks. For example, new Path("/some/path#symlink") will encode the "#" character internally, so we lose the symlink behavior.

          I'm sorry, but you've completely lost me on how URI fragments are related to symlinks? Is this a windows convention?

          Show
          Daryn Sharp added a comment - Aksing users to enter input paths in form "c:/some/path" does not seem like the right thing to do. Please let me know if you agree with me here. Unfortunately I disagree... HADOOP-8139 has a long discussion, but the short summary is: Hadoop uses URIs, so the hadoop veterans decided the hacky support for \ needs to be removed. Although "dir/file" might look like a native path, it's still a URI being prepending with the default fs URI. I spun off HADOOP-8164 to fix the meta-char quoting issue caused by the windows hack – but bug persists on windows! HADOOP-8139 was linked to the window umbrella jira ( HADOOP-8079 ) over a year ago, with the reporter Alexander quoted as "We are only using backslashes () for escaping meta-characters and not as alternative path-separators. So, things should work in a uniform manner." Of note, it looks like for at least the past 6 years, MS has said URIs using is deprecated? http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx Actually, this does not work for paths that are symlinks. For example, new Path("/some/path#symlink") will encode the "#" character internally, so we lose the symlink behavior. I'm sorry, but you've completely lost me on how URI fragments are related to symlinks? Is this a windows convention?
          Hide
          Ivan Mitic added a comment -

          Saved the best for last

          Daryn, thanks for bringing up HADOOP-8139, as it is indeed something we want to address on Windows.

          Let's first make sure that I understood the problem correctly. The Jira is about '\' character being used as an escape character for metachars, and replace("
          ", "/") in Path breaks this. Your current fix in 0.23 addresses the problem in Unix by not doing this "problematic" replace, but leaves Windows with the problem. Please correct me if I'm mistaken, as it's a long discussion.

          > After a long discussion in HADOOP-8139, it was decided that only RFC standard URIs will be supported by hadoop. Paths using "\" are not going to be supported.
          @Daryn: I would prefer to move the discussion in a direction of how to support "\" by Hadoop on Windows, and work with the community on the acceptable solution. Aksing users to enter input paths in form "c:/some/path" does not seem like the right thing to do. Please let me know if you agree with me here. I would prefer if we address HADOOP-8139 in a separate change, as this change moves us forward with Windows support, and does not break Unix behavior.

          file:/// should not allow authority or port - it is for local file systems.
          @Sanjay: I was just trying to illustrate the problem, sorry for the confusion.

          There's no reason you have to, you can always use new Path(String).
          @Daryn: Actually, this does not work for paths that are symlinks. For example, new Path("/some/path#symlink") will encode the "#" character internally, so we lose the symlink behavior. This is why I believe this is a good change. If you take a look at changes I've done to GenericOptionsParser.java you can see how this simplifies things on the call site.

          Show
          Ivan Mitic added a comment - Saved the best for last Daryn, thanks for bringing up HADOOP-8139 , as it is indeed something we want to address on Windows. Let's first make sure that I understood the problem correctly. The Jira is about '\' character being used as an escape character for metachars, and replace(" ", "/") in Path breaks this. Your current fix in 0.23 addresses the problem in Unix by not doing this "problematic" replace, but leaves Windows with the problem. Please correct me if I'm mistaken, as it's a long discussion. > After a long discussion in HADOOP-8139 , it was decided that only RFC standard URIs will be supported by hadoop. Paths using "\" are not going to be supported. @Daryn: I would prefer to move the discussion in a direction of how to support "\" by Hadoop on Windows, and work with the community on the acceptable solution. Aksing users to enter input paths in form "c:/some/path" does not seem like the right thing to do. Please let me know if you agree with me here. I would prefer if we address HADOOP-8139 in a separate change, as this change moves us forward with Windows support, and does not break Unix behavior. file:/// should not allow authority or port - it is for local file systems. @Sanjay: I was just trying to illustrate the problem, sorry for the confusion. There's no reason you have to, you can always use new Path(String). @Daryn: Actually, this does not work for paths that are symlinks. For example, new Path("/some/path#symlink") will encode the "#" character internally, so we lose the symlink behavior. This is why I believe this is a good change. If you take a look at changes I've done to GenericOptionsParser.java you can see how this simplifies things on the call site.
          Hide
          Daryn Sharp added a comment -

          file:/// should not allow authority or port - it is for local file systems.

          @Ivan: If null authority for file isn't already being blocked, this may be as easy as overriding LocalFileSystem#checkPath(Path) to ensure the authority is null prior to calling super.checkPath(Path)

          However I am fine with letting this into the branch temporarily so that you can get most unit tests pass. But it will have to be fixed before we merge with and of the mainlines.

          I have the impression that most reviewers glaze over while looking at tests, so I think we should have a new jira linked to the umbrella jira (for the windows changes) to ensure using test.build.dir doesn't get forgotten.

          Show
          Daryn Sharp added a comment - file:/// should not allow authority or port - it is for local file systems. @Ivan: If null authority for file isn't already being blocked, this may be as easy as overriding LocalFileSystem#checkPath(Path) to ensure the authority is null prior to calling super.checkPath(Path) However I am fine with letting this into the branch temporarily so that you can get most unit tests pass. But it will have to be fixed before we merge with and of the mainlines. I have the impression that most reviewers glaze over while looking at tests, so I think we should have a new jira linked to the umbrella jira (for the windows changes) to ensure using test.build.dir doesn't get forgotten.
          Hide
          Sanjay Radia added a comment -

          Path(URI) should continue to be supported.

          > getUriPath: The problematic URI would be the following: "file://test:8000/c:/some/path"
          file:/// should not allow authority or port - it is for local file systems.

          > System.getProperty("test.build.dir", "build/test")
          Agree with Daryn. However I am fine with letting this into the branch temporarily so that you can get most unit tests pass. But it will have to be fixed before we merge with and of the mainlines. (This is commit then review branch.)

          Show
          Sanjay Radia added a comment - Path(URI) should continue to be supported. > getUriPath: The problematic URI would be the following: "file://test:8000/c:/some/path" file:/// should not allow authority or port - it is for local file systems. > System.getProperty("test.build.dir", "build/test") Agree with Daryn. However I am fine with letting this into the branch temporarily so that you can get most unit tests pass. But it will have to be fixed before we merge with and of the mainlines. (This is commit then review branch.)
          Hide
          Daryn Sharp added a comment -

          The problem is that Windows paths have the following "natural" form: "c:\some\path" in which case the URI creation fails.

          After a long discussion in HADOOP-8139, it was decided that only RFC standard URIs will be supported by hadoop. Paths using "\" are not going to be supported.

          why having to create an URI first before creating a Path?

          There's no reason you have to, you can always use new Path(String). Although new Path(File#toUri()) might be useful. I don't have access to windows, but I thought File considered /c:/some/path to be a valid path?

          /tmp: would it make sense to have the test name embedded in the DFS path? That should solve the concurrent problem.

          System.getProperty("test.build.dir", "build/test") is used pretty pervasively by the tests. Is there a complication with using it for these tests?

          Show
          Daryn Sharp added a comment - The problem is that Windows paths have the following "natural" form: "c:\some\path" in which case the URI creation fails. After a long discussion in HADOOP-8139 , it was decided that only RFC standard URIs will be supported by hadoop. Paths using "\" are not going to be supported. why having to create an URI first before creating a Path? There's no reason you have to, you can always use new Path(String) . Although new Path(File#toUri()) might be useful. I don't have access to windows, but I thought File considered /c:/some/path to be a valid path? /tmp: would it make sense to have the test name embedded in the DFS path? That should solve the concurrent problem. System.getProperty("test.build.dir", "build/test") is used pretty pervasively by the tests. Is there a complication with using it for these tests?
          Hide
          Ivan Mitic added a comment -

          Thanks for the feedback, again!

          The problem is that Windows paths have the following "natural" form: "c:\some\path" in which case the URI creation fails. I actually believe that this is a good change as it simplifies the code on the call site (why having to create an URI first before creating a Path?). On the other hand, if you can see some problems that could come out of this, that would be of great help. Let me know if this makes sense.

          getUriPath: The problematic URI would be the following: "file://test:8000/c:/some/path" in which case the path value would be "/c:/some/path" what does not work well for shell. I will likely revisit this specific problem, as I believe that we can find a better solution.

          /tmp: would it make sense to have the test name embedded in the DFS path? That should solve the concurrent problem.

          Show
          Ivan Mitic added a comment - Thanks for the feedback, again! The problem is that Windows paths have the following "natural" form: "c:\some\path" in which case the URI creation fails. I actually believe that this is a good change as it simplifies the code on the call site (why having to create an URI first before creating a Path?). On the other hand, if you can see some problems that could come out of this, that would be of great help. Let me know if this makes sense. getUriPath: The problematic URI would be the following: "file://test:8000/c:/some/path" in which case the path value would be "/c:/some/path" what does not work well for shell. I will likely revisit this specific problem, as I believe that we can find a better solution. /tmp: would it make sense to have the test name embedded in the DFS path? That should solve the concurrent problem.
          Hide
          Daryn Sharp added a comment -

          Please remove the incompatible support for path fragments in URIs. I'm not sure where this would be useful?

          I don't think Path(URI) should be deprecated. Would you please illustrate windows paths that aren't valid URIs that motivate the change? Ie. "URIs are generally not well suited for paths on all platforms". I was under the impression that URIs are intended to be platform independent?

          I'm not sure I understand the need for getUriPath. Doesn't "c:/some/path" parse as a valid URI whose path is "/some/path"?

          Please don't change tests to be hardcoded to "/tmp". It causes collisions with concurrent test execution...

          Show
          Daryn Sharp added a comment - Please remove the incompatible support for path fragments in URIs. I'm not sure where this would be useful? I don't think Path(URI) should be deprecated. Would you please illustrate windows paths that aren't valid URIs that motivate the change? Ie. "URIs are generally not well suited for paths on all platforms". I was under the impression that URIs are intended to be platform independent? I'm not sure I understand the need for getUriPath . Doesn't "c:/some/path" parse as a valid URI whose path is "/some/path"? Please don't change tests to be hardcoded to "/tmp". It causes collisions with concurrent test execution...
          Hide
          Ivan Mitic added a comment -

          I believe it makes sense to deprecate Path(URI) for the reasons I mentioned in code. I am open to discussing other options or revisiting this if you see some problems down the road.

          Show
          Ivan Mitic added a comment - I believe it makes sense to deprecate Path(URI) for the reasons I mentioned in code. I am open to discussing other options or revisiting this if you see some problems down the road.
          Hide
          Bikas Saha added a comment -

          Looks good.

          Do you think the URI->Path conversion should be deprecated?

          Show
          Bikas Saha added a comment - Looks good. Do you think the URI->Path conversion should be deprecated?
          Hide
          Ivan Mitic added a comment -

          Attaching patch.

          Show
          Ivan Mitic added a comment - Attaching patch.

            People

            • Assignee:
              Ivan Mitic
              Reporter:
              Ivan Mitic
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development