Hadoop Common
  1. Hadoop Common
  2. HADOOP-10607

Create an API to Separate Credentials/Password Storage from Applications

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: security
    • Labels:
      None

      Description

      As with the filesystem API, we need to provide a generic mechanism to support multiple credential storage mechanisms that are potentially from third parties.

      We need the ability to eliminate the storage of passwords and secrets in clear text within configuration files or within code.

      Toward that end, I propose an API that is configured using a list of URLs of CredentialProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries.

      Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system.

      A CredShell CLI will also be included in this patch which provides the ability to manage the credentials within the stores.

      1. 10607.patch
        56 kB
        Larry McCay
      2. 10607-10.patch
        67 kB
        Larry McCay
      3. 10607-11.patch
        68 kB
        Larry McCay
      4. 10607-12.patch
        69 kB
        Larry McCay
      5. 10607-2.patch
        56 kB
        Larry McCay
      6. 10607-3.patch
        56 kB
        Larry McCay
      7. 10607-4.patch
        56 kB
        Larry McCay
      8. 10607-5.patch
        56 kB
        Larry McCay
      9. 10607-6.patch
        60 kB
        Larry McCay
      10. 10607-7.patch
        60 kB
        Larry McCay
      11. 10607-8.patch
        60 kB
        Larry McCay
      12. 10607-9.patch
        60 kB
        Larry McCay
      13. 10607-branch-2.patch
        64 kB
        Larry McCay

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Alejandro Abdelnur made changes -
          Fix Version/s 3.0.0 [ 12320357 ]
          Thejas M Nair made changes -
          Link This issue relates to HADOOP-10904 [ HADOOP-10904 ]
          Thejas M Nair made changes -
          Link This issue relates to HIVE-7634 [ HIVE-7634 ]
          Josh Elser made changes -
          Link This issue relates to ACCUMULO-2464 [ ACCUMULO-2464 ]
          Steve Loughran made changes -
          Link This issue is depended upon by SLIDER-263 [ SLIDER-263 ]
          Steve Loughran made changes -
          Link This issue is depended upon by SLIDER-254 [ SLIDER-254 ]
          Owen O'Malley made changes -
          Fix Version/s 2.6.0 [ 12327179 ]
          Fix Version/s 2.5.0 [ 12326263 ]
          Owen O'Malley made changes -
          Fix Version/s 2.5.0 [ 12326263 ]
          Benoy Antony made changes -
          Link This issue relates to HADOOP-10829 [ HADOOP-10829 ]
          Benoy Antony made changes -
          Link This issue relates to HADOOP-10830 [ HADOOP-10830 ]
          Benoy Antony made changes -
          Link This issue relates to HADOOP-10831 [ HADOOP-10831 ]
          Benoy Antony made changes -
          Link This issue relates to HADOOP-10833 [ HADOOP-10833 ]
          Benoy Antony made changes -
          Link This issue relates to HADOOP-10834 [ HADOOP-10834 ]
          Larry McCay made changes -
          Attachment 10607-branch-2.patch [ 12651600 ]
          Owen O'Malley made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-12.patch [ 12650097 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-11.patch [ 12649866 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-10.patch [ 12649841 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Link This issue is related to HIVE-7175 [ HIVE-7175 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-9.patch [ 12648265 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-8.patch [ 12648180 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-7.patch [ 12647891 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-6.patch [ 12647855 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Link This issue is related to HADOOP-9534 [ HADOOP-9534 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-5.patch [ 12645438 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607-4.patch [ 12645376 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Attachment 10607-3.patch [ 12645221 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Larry McCay made changes -
          Attachment 10607-2.patch [ 12645101 ]
          Larry McCay made changes -
          Summary Create an API to separate Credentials/Password Storage from Applications Create an API to Separate Credentials/Password Storage from Applications
          Larry McCay made changes -
          Issue Type Bug [ 1 ] New Feature [ 2 ]
          Larry McCay made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Larry McCay made changes -
          Attachment 10607.patch [ 12644760 ]
          Larry McCay made changes -
          Link This issue relates to HADOOP-10141 [ HADOOP-10141 ]
          Larry McCay made changes -
          Description As with the filesystem API, we need to provide a generic mechanism to support multiple credential storage mechanisms that are potentially from third parties.

          We need the ability to eliminate the storage of passwords and secrets in clear text within configuration files or within code.

          Toward that end, I propose an API that is configured using a list of URLs of CredentialProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries.

          Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system.

          As with the filesystem API, we need to provide a generic mechanism to support multiple credential storage mechanisms that are potentially from third parties.

          We need the ability to eliminate the storage of passwords and secrets in clear text within configuration files or within code.

          Toward that end, I propose an API that is configured using a list of URLs of CredentialProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries.

          Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system.

          A CredShell CLI will also be included in this patch which provides the ability to manage the credentials within the stores.

          Larry McCay made changes -
          Assignee Owen O'Malley [ owen.omalley ] Larry McCay [ lmccay ]
          Larry McCay made changes -
          Link This issue is cloned as HADOOP-10141 [ HADOOP-10141 ]
          Larry McCay made changes -
          Description As with the filesystem API, we need to provide a generic mechanism to support multiple key storage mechanisms that are potentially from third parties.

          An additional requirement for long term data lakes is to keep multiple versions of each key so that keys can be rolled periodically without requiring the entire data set to be re-written. Rolling keys provides containment in the event of keys being leaked.

          Toward that end, I propose an API that is configured using a list of URLs of KeyProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries.

          Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system.

          As with the filesystem API, we need to provide a generic mechanism to support multiple credential storage mechanisms that are potentially from third parties.

          We need the ability to eliminate the storage of passwords and secrets in clear text within configuration files or within code.

          Toward that end, I propose an API that is configured using a list of URLs of CredentialProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries.

          Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system.

          Larry McCay made changes -
          Field Original Value New Value
          Link This issue is cloned as HADOOP-10141 [ HADOOP-10141 ]
          Larry McCay created issue -

            People

            • Assignee:
              Larry McCay
              Reporter:
              Larry McCay
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development