Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2506

Make Kafka Spout log its assigned partition

    Details

      Description

      On using storm-kafka KafkaSpout, the generated worker logs have information about the task index rather than task ID while printing the assigned kafka partitions. It is difficult to fetch the executor ID reading a particular kafka partition if only task index is present in the logs.

      Example from generated worker.log while using storm-kafka module's KafkaSpout. There were a total of 16 tasks and 2 executors assigned for this topology.
      The executor IDs as indicated by storm-UI were (5 12) and (13 20).

      This problem has been eliminated in the KafkaSpout from storm-kafka-client module. Logs generated with the new KafkaSpout print the mapping between executor ID and the kafka parititions. The task ID is still not printed in the log line which prints this mapping.

      Example from worker.log while using storm-kafka module's KafkaSpout. There were a total of 16 tasks and 2 executors for this topology as well.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                srishtyagrawal9@gmail.com Srishty Agrawal
                Reporter:
                srishtyagrawal9@gmail.com Srishty Agrawal
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 7h 40m
                  7h 40m