Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      The CredentialShell needs end user documentation for the website.

      1. HADOOP-10922-1.patch
        4 kB
        Larry McCay
      2. HADOOP-10922-2.patch
        4 kB
        Larry McCay
      3. HADOOP-10922-3.patch
        4 kB
        Larry McCay

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1874 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/)
          HADOOP-10922. User documentation for CredentialShell. Contributed by Larry McCay. (wang: rev b6d3230e41c78750b6dfd679f24852f22947b5a5)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1874 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/ ) HADOOP-10922 . User documentation for CredentialShell. Contributed by Larry McCay. (wang: rev b6d3230e41c78750b6dfd679f24852f22947b5a5) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1899 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1899/)
          HADOOP-10922. User documentation for CredentialShell. Contributed by Larry McCay. (wang: rev b6d3230e41c78750b6dfd679f24852f22947b5a5)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1899 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1899/ ) HADOOP-10922 . User documentation for CredentialShell. Contributed by Larry McCay. (wang: rev b6d3230e41c78750b6dfd679f24852f22947b5a5) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #683 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/683/)
          HADOOP-10922. User documentation for CredentialShell. Contributed by Larry McCay. (wang: rev b6d3230e41c78750b6dfd679f24852f22947b5a5)

          • hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #683 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/683/ ) HADOOP-10922 . User documentation for CredentialShell. Contributed by Larry McCay. (wang: rev b6d3230e41c78750b6dfd679f24852f22947b5a5) hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Larry McCay added a comment -

          Thanks, Andrew Wang!
          I will carve out some time to get the key provider one done as well.

          Show
          Larry McCay added a comment - Thanks, Andrew Wang ! I will carve out some time to get the key provider one done as well.
          Hide
          Andrew Wang added a comment -

          Committed to trunk and branch-2, thanks again Larry.

          Show
          Andrew Wang added a comment - Committed to trunk and branch-2, thanks again Larry.
          Hide
          Andrew Wang added a comment -

          Sorry for the massive delay in reviewing this. LGTM +1, I'll commit this shortly. Thanks for sticking with this Larry!

          Show
          Andrew Wang added a comment - Sorry for the massive delay in reviewing this. LGTM +1, I'll commit this shortly. Thanks for sticking with this Larry!
          Hide
          Larry McCay added a comment -
          Show
          Larry McCay added a comment - Andrew Wang ?
          Hide
          Larry McCay added a comment -

          Andrew Wang - can I get a review on this, please?
          Thanks!

          Show
          Larry McCay added a comment - Andrew Wang - can I get a review on this, please? Thanks!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667285/HADOOP-10922-3.patch
          against trunk revision 7498dd7.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4675//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4675//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667285/HADOOP-10922-3.patch against trunk revision 7498dd7. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4675//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4675//console This message is automatically generated.
          Hide
          Larry McCay added a comment -

          Andrew Wang - can you please give this latest patch a review?
          Thanks!

          Show
          Larry McCay added a comment - Andrew Wang - can you please give this latest patch a review? Thanks!
          Hide
          Larry McCay added a comment -

          Realized that previous patch had dashes in front of the subcommands which is wrong. This patch corrects that issue.

          Show
          Larry McCay added a comment - Realized that previous patch had dashes in front of the subcommands which is wrong. This patch corrects that issue.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667092/HADOOP-10922-2.patch
          against trunk revision a23144f.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4667//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4667//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667092/HADOOP-10922-2.patch against trunk revision a23144f. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4667//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4667//console This message is automatically generated.
          Hide
          Larry McCay added a comment -

          Is there any point in "submitting patch" for doc only changes?

          Show
          Larry McCay added a comment - Is there any point in "submitting patch" for doc only changes?
          Hide
          Larry McCay added a comment -

          New patch addresses Andrew Wang's review comments.

          Show
          Larry McCay added a comment - New patch addresses Andrew Wang 's review comments.
          Hide
          Larry McCay added a comment -

          Thank you for the detailed review.
          I also anticipated moving a good bit of this to the design doc but it seemed some of the background was needed here in order to understand the provider.path config.

          All of your points are clear and reasonable and I will make the changes. I need to figure out how to do the monospaced thing in apt.

          b. What does "comma separated list of url syntax" mean? Should this say "comma-separated list of URIs"?
          Is only the first provider consulted? That's what the example's descriptive text seems to indicate.

          The comma-separated list determines the providers that are returned by the getProviders call. Once you get a list of providers consulting the is up to the consumer of the API. The Configuration.getPassword method actually consults them all until it finds the alias in a provider and returns that one.

          Hope that makes sense and I will try and convey that in the documentation and clarify the example description so that when using the CLI you really should indicate specifically which provider you want with the -provider option and only provide one. For background info, the CLI implementation will affect the first non-transient provider in the path that it encounters. Transient providers are providers like UserProvider since they are not long term storage for passwords but are instead available for job access and cleaned up when done. That is definitely description for the design document.

          Show
          Larry McCay added a comment - Thank you for the detailed review. I also anticipated moving a good bit of this to the design doc but it seemed some of the background was needed here in order to understand the provider.path config. All of your points are clear and reasonable and I will make the changes. I need to figure out how to do the monospaced thing in apt. b. What does "comma separated list of url syntax" mean? Should this say "comma-separated list of URIs"? Is only the first provider consulted? That's what the example's descriptive text seems to indicate. The comma-separated list determines the providers that are returned by the getProviders call. Once you get a list of providers consulting the is up to the consumer of the API. The Configuration.getPassword method actually consults them all until it finds the alias in a provider and returns that one. Hope that makes sense and I will try and convey that in the documentation and clarify the example description so that when using the CLI you really should indicate specifically which provider you want with the -provider option and only provide one. For background info, the CLI implementation will affect the first non-transient provider in the path that it encounters. Transient providers are providers like UserProvider since they are not long term storage for passwords but are instead available for job access and cleaned up when done. That is definitely description for the design document.
          Hide
          Andrew Wang added a comment -

          Thanks Larry, this is a nice start. I foresee this moving into the design doc when it's ready and simply linking to that, sorta like the Hadoop archive section in the command manual. However, we can get this committed first if you like, and move it over later in HADOOP-11031.

          A few review comments:

          • Rather than ending with "by them", better to flip it to "separation of applications and how they store their required passwords and secrets".
          • "CredentialProvider API" rather than "credential provider API"
          • "hadoop" should be capitalized
          • I like to monospace everything that's part of a CLI command, a file path, hostname, config option, etc. We can skip doing it for the first column of the table, but I think it should apply everywhere else.
          • The "Usage" needs to be adjusted, since we have a number of suboptions that are exclusive, e.g. you can't specify both -create and -delete at the same time. Maybe better to just say <subcommand>, and rely on the table.
          • "comma separated" -> "comma-separated"
          • "url" should be all caps
          • What does "comma separated list of url syntax" mean? Should this say "comma-separated list of URIs"?
          • Is only the first provider consulted? That's what the example's descriptive text seems to indicate.
          Show
          Andrew Wang added a comment - Thanks Larry, this is a nice start. I foresee this moving into the design doc when it's ready and simply linking to that, sorta like the Hadoop archive section in the command manual. However, we can get this committed first if you like, and move it over later in HADOOP-11031 . A few review comments: Rather than ending with "by them", better to flip it to "separation of applications and how they store their required passwords and secrets". "CredentialProvider API" rather than "credential provider API" "hadoop" should be capitalized I like to monospace everything that's part of a CLI command, a file path, hostname, config option, etc. We can skip doing it for the first column of the table, but I think it should apply everywhere else. The "Usage" needs to be adjusted, since we have a number of suboptions that are exclusive, e.g. you can't specify both -create and -delete at the same time. Maybe better to just say <subcommand> , and rely on the table. "comma separated" -> "comma-separated" "url" should be all caps What does "comma separated list of url syntax" mean? Should this say "comma-separated list of URIs"? Is only the first provider consulted? That's what the example's descriptive text seems to indicate.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12665418/HADOOP-10922-1.patch
          against trunk revision b03653f.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4599//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4599//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665418/HADOOP-10922-1.patch against trunk revision b03653f. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4599//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4599//console This message is automatically generated.
          Hide
          Larry McCay added a comment -

          Filed HADOOP-11031 for design document.

          Show
          Larry McCay added a comment - Filed HADOOP-11031 for design document.
          Hide
          Larry McCay added a comment -

          Andrew Wang Can you take a look at this patch and see if it is a good start for credential providers? I would like to get the key provider command done as well before tackling the design docs for each. Let me know if that makes sense to you.

          Show
          Larry McCay added a comment - Andrew Wang Can you take a look at this patch and see if it is a good start for credential providers? I would like to get the key provider command done as well before tackling the design docs for each. Let me know if that makes sense to you.
          Hide
          Larry McCay added a comment -

          This patch provides a somewhat detailed description of the credential command.

          Show
          Larry McCay added a comment - This patch provides a somewhat detailed description of the credential command.
          Hide
          Larry McCay added a comment -

          I see, Andrew Wang. I'll try and get them both done but may file a separate jira for the command manual if we have to break it up into separate passes.

          Show
          Larry McCay added a comment - I see, Andrew Wang . I'll try and get them both done but may file a separate jira for the command manual if we have to break it up into separate passes.
          Hide
          Andrew Wang added a comment -

          I was hoping for separate pages, since I'm not aware of a nice document for downstreams and 3rd parties that might be interested in using or implementing these APIs. It'd be sort of like a design doc, but with more practical information.

          As an example, there was a discussion on HADOOP-10970 about KP URIs. HADOOP-10904 is another similar example for CP URIs. These are the sorts of usecases and design decisions that I'd like to see covered. Without the backstory, it's hard for others to put them into context.

          It'd be good of course to add blurbs to the CommandManual too.

          Show
          Andrew Wang added a comment - I was hoping for separate pages, since I'm not aware of a nice document for downstreams and 3rd parties that might be interested in using or implementing these APIs. It'd be sort of like a design doc, but with more practical information. As an example, there was a discussion on HADOOP-10970 about KP URIs. HADOOP-10904 is another similar example for CP URIs. These are the sorts of usecases and design decisions that I'd like to see covered. Without the backstory, it's hard for others to put them into context. It'd be good of course to add blurbs to the CommandManual too.
          Hide
          Larry McCay added a comment -

          Hi Andrew Wang - I have been hoping to get to this in the next few days. Sorry for the delay. One thing that I was wondering about was whether we want separate pages for these or to simply add the key and credential commands to the CommandManual?

          Show
          Larry McCay added a comment - Hi Andrew Wang - I have been hoping to get to this in the next few days. Sorry for the delay. One thing that I was wondering about was whether we want separate pages for these or to simply add the key and credential commands to the CommandManual?
          Hide
          Andrew Wang added a comment -

          Hey Larry, any update on this or HADOOP-10923? Thanks.

          Show
          Andrew Wang added a comment - Hey Larry, any update on this or HADOOP-10923 ? Thanks.
          Hide
          Andrew Wang added a comment -

          I'm not sure there is a guide out there, but you could take a look at HDFS-6394 where I wrote up encryption documentation. Basically, look for a folder with apt.vm files in hadoop-common and add your new doc file there, then link it from the main site.

          Show
          Andrew Wang added a comment - I'm not sure there is a guide out there, but you could take a look at HDFS-6394 where I wrote up encryption documentation. Basically, look for a folder with apt.vm files in hadoop-common and add your new doc file there, then link it from the main site.
          Hide
          Larry McCay added a comment -

          Hi Andrew Wang - noob question. I can't seem to find a page that describes the process for contributing user documentation patches. Can you link me?

          Show
          Larry McCay added a comment - Hi Andrew Wang - noob question. I can't seem to find a page that describes the process for contributing user documentation patches. Can you link me?
          Hide
          Andrew Wang added a comment -

          Thanks Larry

          Show
          Andrew Wang added a comment - Thanks Larry
          Hide
          Larry McCay added a comment -

          Okay - now that the dashes are messed up - I mean aligned - I can do it.

          Show
          Larry McCay added a comment - Okay - now that the dashes are messed up - I mean aligned - I can do it.
          Hide
          Andrew Wang added a comment -

          Hey Larry, that'd be great. I actually just merged the single dash JIRA for CredShell (HADOOP-10900), so we should be good to go.

          I'm planning to backport KeyShell to branch-2 when HDFS-6134 is backported, so docs for the two are about equivalent urgency.

          Show
          Andrew Wang added a comment - Hey Larry, that'd be great. I actually just merged the single dash JIRA for CredShell ( HADOOP-10900 ), so we should be good to go. I'm planning to backport KeyShell to branch-2 when HDFS-6134 is backported, so docs for the two are about equivalent urgency.
          Hide
          Larry McCay added a comment -

          Hi Andrew Wang - I can put this on my current tasks for credential provider api. Of course, it doesn't make much sense in documenting until the dashes are aligned with KeyShell now - do we have a jira for addressing that?

          I assume CredentialShell has a higher priority since it is in branch-2.

          Show
          Larry McCay added a comment - Hi Andrew Wang - I can put this on my current tasks for credential provider api. Of course, it doesn't make much sense in documenting until the dashes are aligned with KeyShell now - do we have a jira for addressing that? I assume CredentialShell has a higher priority since it is in branch-2.
          Hide
          Andrew Wang added a comment -

          Larry McCay or Owen O'Malley, same as for HADOOP-10923, any interest in picking this up?

          Show
          Andrew Wang added a comment - Larry McCay or Owen O'Malley , same as for HADOOP-10923 , any interest in picking this up?

            People

            • Assignee:
              Larry McCay
              Reporter:
              Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development