Kafka
  1. Kafka
  2. KAFKA-566

Add last modified time to the TopicMetadataRequest

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      To support KAFKA-560 it would be nice to have a last modified time in the TopicMetadataRequest. This would be the timestamp of the last append to the log as taken from stat on the final log segment.

      Implementation would involve
      1. Adding a new field to TopicMetadataRequest
      2. Adding a method Log.lastModified: Long to get the last modified time from a log

      This timestamp would, of course, be subject to error in the event that the file was touched without modification, but I think that is actually okay since it provides a manual way to avoid gc'ing a topic that you know you will want.

      It is debatable whether this should go in 0.8. It would be nice to add the field to the metadata request, at least, as that change should be easy and would avoid needing to bump the version in the future.

        Issue Links

          Activity

          Hide
          Neha Narkhede added a comment -

          Sriharsha Chintalapani Thanks for taking this on.

          Show
          Neha Narkhede added a comment - Sriharsha Chintalapani Thanks for taking this on.
          Hide
          Sriharsha Chintalapani added a comment -

          Jun Rao Neha Narkhede If no one actively working on this JIRA i am interested in taking it. Assigning it to myself please change it if necessary.

          Show
          Sriharsha Chintalapani added a comment - Jun Rao Neha Narkhede If no one actively working on this JIRA i am interested in taking it. Assigning it to myself please change it if necessary.
          Hide
          Neha Narkhede added a comment -

          0.8.1 will include delete topic. Probably ok to move the TTL feature to 0.8.2

          Show
          Neha Narkhede added a comment - 0.8.1 will include delete topic. Probably ok to move the TTL feature to 0.8.2
          Hide
          Neha Narkhede added a comment -

          TopicMetadataRequest is the right class to look at.

          Show
          Neha Narkhede added a comment - TopicMetadataRequest is the right class to look at.
          Hide
          Madhusudan C.S. added a comment - - edited

          Makes sense, I will see if I can implement them. Btw, do we already have getMetadataRequest call implemented? I looked it up and I could not find it. Or does TopicMetadataRequest translate to getMetadataRequest for clients?

          Show
          Madhusudan C.S. added a comment - - edited Makes sense, I will see if I can implement them. Btw, do we already have getMetadataRequest call implemented? I looked it up and I could not find it. Or does TopicMetadataRequest translate to getMetadataRequest for clients?
          Hide
          Jun Rao added a comment -

          Madhusudan,

          Your understanding is correct. However, there is a bigger issue. The problem is that TopicMetadataRequest is designed to be served on any broker since it just gets all topic and partition info from ZK. To get log specific information such as lastModified time, the request has to be served on the broker to which a partition is assigned. So, it's likely that we will need a separate request type, something like getReplicaInforRequest. The client will first issue the getMetedataRequest to figure out which broker is hosting a partition and then issue the getReplicaInforRequest directly to the broker to get the physical information related to the replica.

          Show
          Jun Rao added a comment - Madhusudan, Your understanding is correct. However, there is a bigger issue. The problem is that TopicMetadataRequest is designed to be served on any broker since it just gets all topic and partition info from ZK. To get log specific information such as lastModified time, the request has to be served on the broker to which a partition is assigned. So, it's likely that we will need a separate request type, something like getReplicaInforRequest. The client will first issue the getMetedataRequest to figure out which broker is hosting a partition and then issue the getReplicaInforRequest directly to the broker to get the physical information related to the replica.
          Hide
          Madhusudan C.S. added a comment -

          I was tying to implement the steps described here and I had a few questions about it. I am new to Kafka's code base, so please help me.

          TopicMetadataRequest returns multiple topics as strings and in addition each topic can have multiple partitions and the last modified timestamp is per log I guess? And there exists logs for every partition for each topic? So what do we have to return here? Isn't PartitionMetadata which is embedded in TopicMetadata a better place to return this?

          Show
          Madhusudan C.S. added a comment - I was tying to implement the steps described here and I had a few questions about it. I am new to Kafka's code base, so please help me. TopicMetadataRequest returns multiple topics as strings and in addition each topic can have multiple partitions and the last modified timestamp is per log I guess? And there exists logs for every partition for each topic? So what do we have to return here? Isn't PartitionMetadata which is embedded in TopicMetadata a better place to return this?

            People

            • Assignee:
              Sriharsha Chintalapani
              Reporter:
              Jay Kreps
              Reviewer:
              Neha Narkhede
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development