Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3369

Reduce the data size sink from Kafka topic to HDFS

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v2.4.0
    • Component/s: NRT Streaming
    • Labels:
      None

      Description

      When building a cube from Kafka topic, the first step is to sink the Kafka data to HDFS. In today's implementation, it will persist all the fields of a message to disk. While in many cases, only a couple of fields will be needed for cubing; Today's behavior wastes network bandwidth and disk space.

        Attachments

          Activity

            People

            • Assignee:
              shaofengshi Shao Feng Shi
              Reporter:
              shaofengshi Shao Feng Shi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: