Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1519

Console consumer: expose configuration option to enable/disable writing the line separator

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.8.1.1
    • Fix Version/s: 0.8.2.0
    • Component/s: consumer
    • Labels:

      Description

      The current console consumer includes a DefaultMessageFormatter, which exposes a few user-configurable options which can be set on the command line via -property, e.g. "-property line.separator=XYZ".

      Unfortunately, the current implementation does not allow the user to completely disable writing any such line separator. However, this functionality would be helpful to enable users to capture data "as is" from a Kafka topic to snapshot file. Capturing data "as is" – without an artificial line separator – is particularly nice for data in a binary format (including Avro).

      No workaround

      A potential workaround would be to pass an empty string as the property value of "line.separator", but this doesn't work in the current implementation.

      The following variants throw an "Invalid parser arguments" exception:

      --property line.separator=       # "nothing"
      --property line.separator=""     # double quotes
      --property line.separator=''     # single quotes
      

      Escape tricks via a backslash don't work either.

      If there actually is a workaround please let me know.

      How to fix

      We can introduce a "print.line" option to enable/disable writing "line.separator" similar to how the code already uses "print.key" to enable/disable writing "key.separator".

      This change is trivial. To preserve backwards compatibility, the "print.line" option would be set to true by default (unlike the "print.key" option, which defaults to false).

      Alternatives

      Apart from modifying the built-in DefaultMessageFormatter, users could of course implement their own custom MessageFormatter. But given that it's a) a trivial change to the DefaultMessageFormatter and b) a nice user feature I'd say changing the built-in DefaultMessageFormatter would be the better approach. This way, Kafka would support writing data as-is to a file out of the box.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gwenshap Gwen Shapira
                Reporter:
                miguno Michael Noll
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: