Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4680

Audit logging of delegation tokens for MR tracing

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.1.1-beta
    • Component/s: namenode, security
    • Labels:
      None

      Description

      HDFS audit logging tracks HDFS operations made by different users, e.g. creation and deletion of files. This is useful for after-the-fact root cause analysis and security. However, logging merely the username is insufficient for many usecases. For instance, it is common for a single user to run multiple MapReduce jobs (I believe this is the case with Hive). In this scenario, given a particular audit log entry, it is difficult to trace it back to the MR job or task that generated that entry.

      I see a number of potential options for implementing this.

      1. Make an optional "client name" field part of the NN RPC format. We already pass a clientName as a parameter in many RPC calls, so this would essentially make it standardized. MR tasks could then set this field to the job and task ID.
      2. This could be generalized to a set of optional key-value tags in the NN RPC format, which would then be audit logged. This has standalone benefits outside of just verifying MR task ids.
      3. Neither of the above two options actually securely verify that MR clients are who they claim they are. Doing this securely requires the JobTracker to sign MR task attempts, and then having the NN verify this signature. However, this is substantially more work, and could be built on after idea #2.

      Thoughts welcomed.

      1. hdfs-4680-5.patch
        17 kB
        Andrew Wang
      2. hdfs-4680-4.patch
        14 kB
        Andrew Wang
      3. hdfs-4680-3.patch
        11 kB
        Andrew Wang
      4. hdfs-4680-2.patch
        12 kB
        Andrew Wang
      5. hdfs-4680-1.patch
        5 kB
        Andrew Wang

        Issue Links

          Activity

          Andrew Wang created issue -
          Hide
          Steve Loughran added a comment -

          Rather than client name, I'd recommend a workflow ID that can be propagated across multiple Oozie, pig, hive queries, helping you track it down not to a user, but to a specific job.

          This would aid not just auditing of client use, but cost auditing of entire workflows

          Show
          Steve Loughran added a comment - Rather than client name, I'd recommend a workflow ID that can be propagated across multiple Oozie, pig, hive queries, helping you track it down not to a user, but to a specific job. This would aid not just auditing of client use, but cost auditing of entire workflows
          Hide
          Daryn Sharp added a comment -

          I believe modifying the RPC layer and expecting clients to somehow reliably propagate a client id will be extremely challenging. The easier way to implement is via inclusion in the UGI. How to propagate it in the UGI? Use delegation tokens since the remote UGI is constructed via the token. Embedding the client id in the token has the benefit that the client id can't be changed or impersonated. Job submission could request tokens with a client id of the job id. Lest you point out: but insecure clusters don't use tokens! I'm trying to get back to my almost complete SASL changes that will use tokens regardless of security settings.

          Show
          Daryn Sharp added a comment - I believe modifying the RPC layer and expecting clients to somehow reliably propagate a client id will be extremely challenging. The easier way to implement is via inclusion in the UGI. How to propagate it in the UGI? Use delegation tokens since the remote UGI is constructed via the token. Embedding the client id in the token has the benefit that the client id can't be changed or impersonated. Job submission could request tokens with a client id of the job id. Lest you point out: but insecure clusters don't use tokens! I'm trying to get back to my almost complete SASL changes that will use tokens regardless of security settings.
          Hide
          Kihwal Lee added a comment -

          Rather than client name, I'd recommend a workflow ID that can be propagated across multiple Oozie, pig, hive queries, helping you track it down not to a user, but to a specific job.

          Currently DFSClient constructs an ID string from conf.get("mapreduce.task.attempt.id", "NONMAPREDUCE"). This may be generalized to include any type of tracing ID from upper layer.

          Show
          Kihwal Lee added a comment - Rather than client name, I'd recommend a workflow ID that can be propagated across multiple Oozie, pig, hive queries, helping you track it down not to a user, but to a specific job. Currently DFSClient constructs an ID string from conf.get("mapreduce.task.attempt.id", "NONMAPREDUCE") . This may be generalized to include any type of tracing ID from upper layer.
          Hide
          Todd Lipcon added a comment -

          I believe modifying the RPC layer and expecting clients to somehow reliably propagate a client id will be extremely challenging.

          I don't think it's too bad using threadlocals, etc. I have a prototype of Dapper-style tracing which hooks into Hadoop RPC and it only took me a day to get working. If folks would find this integration useful, I'm happy to work on cleaning up the patch later this spring and work on getting it merged.

          But, I didn't solve the security issue there.

          The major question I have is whether there is value in having data in the audit log which is not in fact authenticated. We'd want to present it as such, eg by phrasing it as: "clientID: DFSClient_12345 (client-provided trace tags: job=job_3219234, oozie=workflow_234234, hive query=SELECT *...)". By phrasing it as "client-provided tags" we know that it should be used for informational data only, and depending on whether the client code is trusted, shouldn't be used for billback without first verifying that indeed job_3219234 was run by the user who claimed it.

          Show
          Todd Lipcon added a comment - I believe modifying the RPC layer and expecting clients to somehow reliably propagate a client id will be extremely challenging. I don't think it's too bad using threadlocals, etc. I have a prototype of Dapper-style tracing which hooks into Hadoop RPC and it only took me a day to get working. If folks would find this integration useful, I'm happy to work on cleaning up the patch later this spring and work on getting it merged. But, I didn't solve the security issue there. The major question I have is whether there is value in having data in the audit log which is not in fact authenticated. We'd want to present it as such, eg by phrasing it as: "clientID: DFSClient_12345 (client-provided trace tags: job=job_3219234, oozie=workflow_234234, hive query=SELECT *...)". By phrasing it as "client-provided tags" we know that it should be used for informational data only, and depending on whether the client code is trusted, shouldn't be used for billback without first verifying that indeed job_3219234 was run by the user who claimed it.
          Hide
          Daryn Sharp added a comment -

          Would a combination of embedding the clientID in the token (so it's authenticated), and perhaps passing the client tags in the RPC be a reasonable approach? I thought dapper was based on web servlets, so could you elaborate on how your prototype uses dapper?

          Show
          Daryn Sharp added a comment - Would a combination of embedding the clientID in the token (so it's authenticated), and perhaps passing the client tags in the RPC be a reasonable approach? I thought dapper was based on web servlets, so could you elaborate on how your prototype uses dapper?
          Hide
          Todd Lipcon added a comment -

          Would a combination of embedding the clientID in the token (so it's authenticated), and perhaps passing the client tags in the RPC be a reasonable approach?

          I'm not quite following the idea here. You're suggesting that the HDFS delegation token included in the job would have the job ID? I don't think that really works, since the HDFS delegation token is fetched by the client code at job submission time. Thus, the client could pick whatever job ID it wants to "fake", and there's no way for the NN to authenticate the job ID.

          At the end of the day, the JT/RM is the one who can validate that a given job belongs to a given user, so it needs to somehow sign this before the client passes it to the NN, whether in the form of a signed MR token or some other infrastructure. Right?

          I thought dapper was based on web servlets, so could you elaborate on how your prototype uses dapper?

          Maybe we're talking about different dappers. http://research.google.com/pubs/pub36356.html is the one I'm referring to.

          Show
          Todd Lipcon added a comment - Would a combination of embedding the clientID in the token (so it's authenticated), and perhaps passing the client tags in the RPC be a reasonable approach? I'm not quite following the idea here. You're suggesting that the HDFS delegation token included in the job would have the job ID? I don't think that really works, since the HDFS delegation token is fetched by the client code at job submission time. Thus, the client could pick whatever job ID it wants to "fake", and there's no way for the NN to authenticate the job ID. At the end of the day, the JT/RM is the one who can validate that a given job belongs to a given user, so it needs to somehow sign this before the client passes it to the NN, whether in the form of a signed MR token or some other infrastructure. Right? I thought dapper was based on web servlets, so could you elaborate on how your prototype uses dapper? Maybe we're talking about different dappers. http://research.google.com/pubs/pub36356.html is the one I'm referring to.
          Hide
          Andrew Wang added a comment -

          I'm leaning toward the unauthenticated solution in route #2: stuffing some optional k/v pairs in the RPC format and displaying them as "client provided tags" in the audit log. It's still useful for debugging/tracing, pretty simple, and these RPC fields could possibly be co-opted by Todd's Dapper patch later on.

          If this sounds good, let's defer the token solution to another JIRA. It seems like we need some more related work first anyway, and also some more thinking about how to implement it securely.

          Show
          Andrew Wang added a comment - I'm leaning toward the unauthenticated solution in route #2: stuffing some optional k/v pairs in the RPC format and displaying them as "client provided tags" in the audit log. It's still useful for debugging/tracing, pretty simple, and these RPC fields could possibly be co-opted by Todd's Dapper patch later on. If this sounds good, let's defer the token solution to another JIRA. It seems like we need some more related work first anyway, and also some more thinking about how to implement it securely.
          Hide
          Daryn Sharp added a comment -

          Todd Lipcon Yes, the client would be able to name itself when acquiring tokens. We really don't want something MR specific, so we doing want the RM signing things and establishing a trust relationship with the NN, etc. If things "play nice", ie. MR always uses the job id, then it at least provides traceability between tokens and the operations performed. The basic idea is that a task can't forge what the job submitter described itself as.

          Andrew Wang Adding arbitrary KVPs to the RPC protocol seems prone to abuse. Ex. consuming large amounts of memory in the NN, spamming the audit logs, allow spoofing of log entries unless things like newlines are filtered out, etc.

          How are you proposing to propagate these KVPs between job client, tasks, pig jobs, oozie jobs, tasks running hadoop fs commands, etc? I skimmed the Dapper article but I'm not sure exactly how it solves the problem.

          Show
          Daryn Sharp added a comment - Todd Lipcon Yes, the client would be able to name itself when acquiring tokens. We really don't want something MR specific, so we doing want the RM signing things and establishing a trust relationship with the NN, etc. If things "play nice", ie. MR always uses the job id, then it at least provides traceability between tokens and the operations performed. The basic idea is that a task can't forge what the job submitter described itself as. Andrew Wang Adding arbitrary KVPs to the RPC protocol seems prone to abuse. Ex. consuming large amounts of memory in the NN, spamming the audit logs, allow spoofing of log entries unless things like newlines are filtered out, etc. How are you proposing to propagate these KVPs between job client, tasks, pig jobs, oozie jobs, tasks running hadoop fs commands, etc? I skimmed the Dapper article but I'm not sure exactly how it solves the problem.
          Hide
          Andrew Wang added a comment -

          Daryn Sharp I skimmed the Dapper paper a bit too, it only traces within apps that all use the same Dapper-instrumented RPC framework. REST calls and raw sockets for instance are not traced. They also only pass the trace and span ids in the RPC, while user-provided data is logged locally and re-associated with a given trace+span later. This avoids the problem you brought up of arbitrary user data bloating RPCs.

          I agree that arbitrary KVPs is probably not a good idea. Is adding just the client name field to the RPC format the next-lowest-cost solution? It'd still need to be length-limited and sanitized for special characters, but those are solvable issues. The client name seems like a useful enough bit of audit information that it deserves special attention like this.

          Show
          Andrew Wang added a comment - Daryn Sharp I skimmed the Dapper paper a bit too, it only traces within apps that all use the same Dapper-instrumented RPC framework. REST calls and raw sockets for instance are not traced. They also only pass the trace and span ids in the RPC, while user-provided data is logged locally and re-associated with a given trace+span later. This avoids the problem you brought up of arbitrary user data bloating RPCs. I agree that arbitrary KVPs is probably not a good idea. Is adding just the client name field to the RPC format the next-lowest-cost solution? It'd still need to be length-limited and sanitized for special characters, but those are solvable issues. The client name seems like a useful enough bit of audit information that it deserves special attention like this.
          Hide
          Daryn Sharp added a comment -

          I'm not sure the DFSClient name will do what you want. It's unreliable because it's composed of:
          "DFSClient_" + conf.get("mapreduce.task.attempt.id", "NONMAPREDUCE") + "" + DFSUtil.getRandom().nextInt() + "" + Thread.currentThread().getId();

          Ensuring propagation of the client name via subsequent RPC calls seems difficult if not impossible to do reliably. Relying on a conf value is fragile, ex. commands that don't use a job conf (such as, I think, hadoop fs), won't include the attempt id. Hence why I suggested embedding a "client name" in the token (actually token identifier) so it cannot be subsequently altered or spoofed - you just have to trust the requestor of the token to submit a sane value. This requires an API change though.

          The simplest solution not requiring API changes may be to log tokenIdentifier#getSequenceNumber() to allow collation of all access for a given token (ie. job). Traceability to a job will be a challenge.

          Show
          Daryn Sharp added a comment - I'm not sure the DFSClient name will do what you want. It's unreliable because it's composed of: "DFSClient_" + conf.get("mapreduce.task.attempt.id", "NONMAPREDUCE") + " " + DFSUtil.getRandom().nextInt() + " " + Thread.currentThread().getId(); Ensuring propagation of the client name via subsequent RPC calls seems difficult if not impossible to do reliably. Relying on a conf value is fragile, ex. commands that don't use a job conf (such as, I think, hadoop fs), won't include the attempt id. Hence why I suggested embedding a "client name" in the token (actually token identifier) so it cannot be subsequently altered or spoofed - you just have to trust the requestor of the token to submit a sane value. This requires an API change though. The simplest solution not requiring API changes may be to log tokenIdentifier#getSequenceNumber() to allow collation of all access for a given token (ie. job). Traceability to a job will be a challenge.
          Sandy Ryza made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-9775 [ HADOOP-9775 ]
          Hide
          Andrew Wang added a comment -

          Based on discussion with Daryn, Tucu, and Sandy, we've come up with another approach beyond delegation token sequence numbers. The attached patch instead logs the MD5 hash of the token's #getBytes method.

          No tests as this only kicks in when access is done with a delegation token, and I couldn't find any existing unit tests that did this. I have tested this manually with a small MR cluster, and saw hashes being printed in the audit log.

          Show
          Andrew Wang added a comment - Based on discussion with Daryn, Tucu, and Sandy, we've come up with another approach beyond delegation token sequence numbers. The attached patch instead logs the MD5 hash of the token's #getBytes method. No tests as this only kicks in when access is done with a delegation token, and I couldn't find any existing unit tests that did this. I have tested this manually with a small MR cluster, and saw hashes being printed in the audit log.
          Andrew Wang made changes -
          Attachment hdfs-4680-1.patch [ 12595129 ]
          Andrew Wang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 3.0.0 [ 12320356 ] 3.0.0, 2.3.0 [ 12320356, 12324588 ]
          Andrew Wang made changes -
          Summary Audit logging of client names Audit logging of delegation tokens for MR tracing
          Andrew Wang made changes -
          Link This issue relates to MAPREDUCE-5379 [ MAPREDUCE-5379 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12595129/hdfs-4680-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestFsck

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4751//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4751//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595129/hdfs-4680-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestFsck +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4751//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4751//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          For the record: This is an offline compromise to decoding the identifier. I don't like this approach and find it marginally useful. I like the concept of this feature for chargeback but don't want a feeble dependency on the job conf and history files - or even tailored to MR.

          Other dislikes are it's not a forward mapping. It's not a descriptive string, ex. jobId for MR or a query id for hive. If the NN is being spammed/abused, I want to immediately know the responsible job so it can be killed. Traceability with this patch requires joining the audit logs with the job history files post-job completion.

          A "lying" client will be able to abuse any solution. This approach allows any task or sub-jobs to strip out or alter the tracking info from the job conf and break the traceability. Embedding a tracking id in the token identifier provides only one point to lie: initial token acquisition during submission. If you are using oozie to control job submission then you can "trust" that oozie isn't going to lie and jobs are incapable of lying.

          That said, comments on the patch:

          1. Make it a configurable option defaulting to off. I don't want the penalty for something of little to no value to us.
          2. UGI.getCurrentUser isn't cheap. The caller of logAuditEvent has the UGI, but it's passing just the username. Pass the actual ugi down instead to avoid an unnecessary lookup.
          3. Don't look for a token if the user isn't authed with a token.
          4. There should be one and only one token ident in the UGI but I suppose paranoia is good. However it should be looking specifically for DelegationTokenIdentifier, not the abstract.
          5. It's costly to compute the md5sum for every single client connection. Store it in the DelegationTokenInformation when the token is created and query the dtsm during logging.
          6. I'd prefer it's called something generic like "trackingId" so it can be reused when we actually make it a useful forward mapping.

          Completely untested code that illustrates some of the above points:

          if (someConfValue && ugi.getAuthenticationMethod() == AuthenticationMethod.TOKEN) {
            for (TokenIdentifier tokenId : ugi.getTokenIdentifiers()) {
              if (tokenId instanceof DelegationTokenIdentifier) {
                sb.append("\ttrackingId=").append(dtsm.getTrackingId(tokenId));
                break;
              }
            }
          }
          
          Show
          Daryn Sharp added a comment - For the record: This is an offline compromise to decoding the identifier. I don't like this approach and find it marginally useful. I like the concept of this feature for chargeback but don't want a feeble dependency on the job conf and history files - or even tailored to MR. Other dislikes are it's not a forward mapping. It's not a descriptive string, ex. jobId for MR or a query id for hive. If the NN is being spammed/abused, I want to immediately know the responsible job so it can be killed. Traceability with this patch requires joining the audit logs with the job history files post-job completion. A "lying" client will be able to abuse any solution. This approach allows any task or sub-jobs to strip out or alter the tracking info from the job conf and break the traceability. Embedding a tracking id in the token identifier provides only one point to lie: initial token acquisition during submission. If you are using oozie to control job submission then you can "trust" that oozie isn't going to lie and jobs are incapable of lying. That said, comments on the patch: Make it a configurable option defaulting to off . I don't want the penalty for something of little to no value to us. UGI.getCurrentUser isn't cheap. The caller of logAuditEvent has the UGI, but it's passing just the username. Pass the actual ugi down instead to avoid an unnecessary lookup. Don't look for a token if the user isn't authed with a token. There should be one and only one token ident in the UGI but I suppose paranoia is good. However it should be looking specifically for DelegationTokenIdentifier, not the abstract. It's costly to compute the md5sum for every single client connection. Store it in the DelegationTokenInformation when the token is created and query the dtsm during logging. I'd prefer it's called something generic like "trackingId" so it can be reused when we actually make it a useful forward mapping. Completely untested code that illustrates some of the above points: if (someConfValue && ugi.getAuthenticationMethod() == AuthenticationMethod.TOKEN) { for (TokenIdentifier tokenId : ugi.getTokenIdentifiers()) { if (tokenId instanceof DelegationTokenIdentifier) { sb.append( "\ttrackingId=" ).append(dtsm.getTrackingId(tokenId)); break ; } } }
          Andrew Wang made changes -
          Attachment hdfs-4680-2.patch [ 12596376 ]
          Hide
          Andrew Wang added a comment -

          Thanks for the review Daryn, and providing the backstory/analysis. I agree it's kind of a stopgap measure.

          Your pointers were super useful for this newest patch, I think they're all addressed with these caveats:

          • forgoes the paranoid check for multiple tokens, unnecessary like you said
          • I didn't want to modify the public AuditLogger interface, so there's some instanceof ugliness.
          • Even with the conf option off, the MD5 is still calculated on the token being added to the dtsm. I can propagate the conf option down to the dtsm if this matters.
          • I intentionally left the conf option out of hdfs-default.xml, since it's only useful for expert users.
          Show
          Andrew Wang added a comment - Thanks for the review Daryn, and providing the backstory/analysis. I agree it's kind of a stopgap measure. Your pointers were super useful for this newest patch, I think they're all addressed with these caveats: forgoes the paranoid check for multiple tokens, unnecessary like you said I didn't want to modify the public AuditLogger interface, so there's some instanceof ugliness. Even with the conf option off, the MD5 is still calculated on the token being added to the dtsm. I can propagate the conf option down to the dtsm if this matters. I intentionally left the conf option out of hdfs-default.xml, since it's only useful for expert users.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12596376/hdfs-4680-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4772//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4772//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596376/hdfs-4680-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4772//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4772//console This message is automatically generated.
          Hide
          Andrew Wang added a comment -

          WebHdfs test failures look like some kind of environmental problem, unrelated to the patch.

          Show
          Andrew Wang added a comment - WebHdfs test failures look like some kind of environmental problem, unrelated to the patch.
          Hide
          Aaron T. Myers added a comment -

          The core changes look good to me, though it looks like you left some debugging output in the test accidentally. +1 from me once this is addressed.

          Daryn, does the content of the patch look good to you? If I don't hear in the next day or two, I think we should go ahead and commit this patch.

          Show
          Aaron T. Myers added a comment - The core changes look good to me, though it looks like you left some debugging output in the test accidentally. +1 from me once this is addressed. Daryn, does the content of the patch look good to you? If I don't hear in the next day or two, I think we should go ahead and commit this patch.
          Hide
          Andrew Wang added a comment -

          Thanks for the review ATM, here's a v3 that removes the extraneous printlns.

          Show
          Andrew Wang added a comment - Thanks for the review ATM, here's a v3 that removes the extraneous printlns.
          Andrew Wang made changes -
          Attachment hdfs-4680-3.patch [ 12596951 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12596951/hdfs-4680-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4786//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4786//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596951/hdfs-4680-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4786//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4786//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          I'll try to review this afternoon.

          Show
          Daryn Sharp added a comment - I'll try to review this afternoon.
          Hide
          Daryn Sharp added a comment -

          I just glanced over it. Question: if the trackingId is in the token, why is it extracted and stored in the DelegationTokenInformation? Currently that object tracks info about token idents that isn't in the token ident itself. I need to look at the linked jiras to get the big picture.

          Show
          Daryn Sharp added a comment - I just glanced over it. Question: if the trackingId is in the token, why is it extracted and stored in the DelegationTokenInformation? Currently that object tracks info about token idents that isn't in the token ident itself. I need to look at the linked jiras to get the big picture.
          Hide
          Andrew Wang added a comment -

          I did that based on your previous review feedback:

          It's costly to compute the md5sum for every single client connection. Store it in the DelegationTokenInformation when the token is created and query the dtsm during logging.

          Do you want to change this up?

          Show
          Andrew Wang added a comment - I did that based on your previous review feedback: It's costly to compute the md5sum for every single client connection. Store it in the DelegationTokenInformation when the token is created and query the dtsm during logging. Do you want to change this up?
          Hide
          Daryn Sharp added a comment -

          I made the mistake of looking at the raw patch instead of applying it.

          With the way you've done it, I think we may be able to simplify it. The instanceof for the default audit logger seems like it can/should be avoided. It appears you did this in part to avoid the performance hit of looking up the token identifier and its tracking id every time you log a message. We should probably think of a way to avoid that.

          Off the top of my head, conceptually it would be ideal if the connection knew the trackingId, and the audit logger would simply log it if not null. I'll think about it more today since I'm trying to contemplate how a forward lookup would be an easy drop-in in the future and if there would be any rolling upgrade issues.

          Show
          Daryn Sharp added a comment - I made the mistake of looking at the raw patch instead of applying it. With the way you've done it, I think we may be able to simplify it. The instanceof for the default audit logger seems like it can/should be avoided. It appears you did this in part to avoid the performance hit of looking up the token identifier and its tracking id every time you log a message. We should probably think of a way to avoid that. Off the top of my head, conceptually it would be ideal if the connection knew the trackingId, and the audit logger would simply log it if not null. I'll think about it more today since I'm trying to contemplate how a forward lookup would be an easy drop-in in the future and if there would be any rolling upgrade issues.
          Hide
          Daryn Sharp added a comment -

          I also noticed that ADTSM is taking the md5sum penalty for every token generation regardless of the conf setting.

          Show
          Daryn Sharp added a comment - I also noticed that ADTSM is taking the md5sum penalty for every token generation regardless of the conf setting.
          Hide
          Aaron T. Myers added a comment -

          I'll think about it more today since I'm trying to contemplate how a forward lookup would be an easy drop-in in the future and if there would be any rolling upgrade issues.

          Hey Daryn, have you given this issue any more thought? I'd really like to be able to make some progress on this issue, but currently it's not clear what a design would look like that would be acceptable to you.

          Show
          Aaron T. Myers added a comment - I'll think about it more today since I'm trying to contemplate how a forward lookup would be an easy drop-in in the future and if there would be any rolling upgrade issues. Hey Daryn, have you given this issue any more thought? I'd really like to be able to make some progress on this issue, but currently it's not clear what a design would look like that would be acceptable to you.
          Hide
          Andrew Wang added a comment -

          Hey folks, here's a revised patch which avoids MD5 overhead when the conf option is disabled. Pushing down the conf option was somewhat ugly; ideally this change would only touch HDFS' DTSM, but some of the DelegationTokenInformation add hooks are in ADTSM.

          The instanceof for the default audit logger seems like it can/should be avoided...

          I wanted to avoid modifying AuditLogger, since it's a public interface and there can be external implementations. I think this means compatibility issues, but suggestions welcome.

          it would be ideal if the connection knew the trackingId...

          I'm not sure the best way of doing this, but again, suggestions welcome.

          Show
          Andrew Wang added a comment - Hey folks, here's a revised patch which avoids MD5 overhead when the conf option is disabled. Pushing down the conf option was somewhat ugly; ideally this change would only touch HDFS' DTSM, but some of the DelegationTokenInformation add hooks are in ADTSM. The instanceof for the default audit logger seems like it can/should be avoided... I wanted to avoid modifying AuditLogger, since it's a public interface and there can be external implementations. I think this means compatibility issues, but suggestions welcome. it would be ideal if the connection knew the trackingId... I'm not sure the best way of doing this, but again, suggestions welcome.
          Andrew Wang made changes -
          Attachment hdfs-4680-4.patch [ 12601099 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12601099/hdfs-4680-4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4922//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4922//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12601099/hdfs-4680-4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4922//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4922//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          The latest patch looks pretty good to me. My only suggestion is that I think we should make it possible for the pluggable audit logger implementations to be passed the tracing ID as well.

          Thanks, Andrew.

          Show
          Aaron T. Myers added a comment - The latest patch looks pretty good to me. My only suggestion is that I think we should make it possible for the pluggable audit logger implementations to be passed the tracing ID as well. Thanks, Andrew.
          Hide
          Andrew Wang added a comment -

          Thanks Aaron, new patch introduces a new HdfsAuditLogger abstract class for external implementions.

          Show
          Andrew Wang added a comment - Thanks Aaron, new patch introduces a new HdfsAuditLogger abstract class for external implementions.
          Andrew Wang made changes -
          Attachment hdfs-4680-5.patch [ 12602089 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12602089/hdfs-4680-5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4940//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4940//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12602089/hdfs-4680-5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4940//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4940//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          +1, the latest patch looks good to me.

          Thanks, Andrew.

          Show
          Aaron T. Myers added a comment - +1, the latest patch looks good to me. Thanks, Andrew.
          Hide
          Andrew Wang added a comment -

          Thanks ATM (and everyone who's looked at this), committed to trunk and branch-2.

          Show
          Andrew Wang added a comment - Thanks ATM (and everyone who's looked at this), committed to trunk and branch-2.
          Andrew Wang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.3.0 [ 12324588 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4400 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4400/)
          HDFS-4680. Audit logging of delegation tokens for MR tracing. (Andrew Wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522012)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenIdentifier.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HdfsAuditLogger.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4400 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4400/ ) HDFS-4680 . Audit logging of delegation tokens for MR tracing. (Andrew Wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522012 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenIdentifier.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HdfsAuditLogger.java
          Hide
          Andrew Wang added a comment -

          And additionally committed to branch-2.1-beta, CHANGES.txt updated.

          Show
          Andrew Wang added a comment - And additionally committed to branch-2.1-beta, CHANGES.txt updated.
          Andrew Wang made changes -
          Fix Version/s 2.1.1-beta [ 12324809 ]
          Fix Version/s 2.3.0 [ 12324588 ]
          Target Version/s 3.0.0, 2.3.0 [ 12320356, 12324588 ] 3.0.0, 2.3.0, 2.1.1-beta [ 12320356, 12324588, 12324809 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4401 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4401/)
          Move HDFS-4680 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522049)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4401 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4401/ ) Move HDFS-4680 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522049 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #330 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/330/)
          Move HDFS-4680 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522049)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HDFS-4680. Audit logging of delegation tokens for MR tracing. (Andrew Wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522012)
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenIdentifier.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HdfsAuditLogger.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #330 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/330/ ) Move HDFS-4680 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522049 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-4680 . Audit logging of delegation tokens for MR tracing. (Andrew Wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522012 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenIdentifier.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HdfsAuditLogger.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1520 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1520/)
          Move HDFS-4680 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522049)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HDFS-4680. Audit logging of delegation tokens for MR tracing. (Andrew Wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522012)
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenIdentifier.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HdfsAuditLogger.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1520 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1520/ ) Move HDFS-4680 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522049 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-4680 . Audit logging of delegation tokens for MR tracing. (Andrew Wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522012 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenIdentifier.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HdfsAuditLogger.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1546 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1546/)
          Move HDFS-4680 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522049)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            HDFS-4680. Audit logging of delegation tokens for MR tracing. (Andrew Wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522012)
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenIdentifier.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HdfsAuditLogger.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1546 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1546/ ) Move HDFS-4680 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522049 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt HDFS-4680 . Audit logging of delegation tokens for MR tracing. (Andrew Wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522012 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenIdentifier.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HdfsAuditLogger.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Andrew Wang
              Reporter:
              Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development