Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4743

NPE when some kafka partition's offset is not exists in checkpoint

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • v3.0.0, v3.1.0
    • Future
    • Real-time Streaming
    • None
    • Centos 7.4.1
      hbase 1.2.4
      hive 2.0.1
      hadoop 2.7.2
    • Patch

    Description

      We have used streaming cube to statistics data from Kafka.
      If the data partitions of kafka topic is imbalanced (one or more partition is empty),it will cause the cube's checkpoint contains one or more partition's offset is empty.
      For some reason the receiver node be restarted,it will fail to initialization kafka connector because offset is null.

      Attachments

        Activity

          People

            ggkE GuKe
            ggkE GuKe
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 72h
                72h
                Remaining:
                Remaining Estimate - 72h
                72h
                Logged:
                Time Spent - Not Specified
                Not Specified