Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-2256

Exception in Druid Adapter: JsonSegmentMetadata, numRows field out of range



    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.16.0
    • None
    • druid-adapter
    • None


      When trying to connect to a Druid datasource, Calcite pulls metadata from Druid as a first action. As Druid is a database for big-data, it is not uncommon that a datasource would contain billions of records. The metadata about a datasource returns a records count, namely the total number of records residing in that datasource. This is returned in a property named "numRows".

      The Calcite Druid Adapter tries to read the numRows field (that Druid returns within the response Json document of a metadata request) into an integer field which overflows for record counts bigger than 4 billion and triggers the below exception

      Caused by: com.fasterxml.jackson.databind.JsonMappingException: Numeric value (13452491153) out of range of int
       at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 565, column: 26]
       at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 565, column: 15] (through reference chain: java.util.ArrayList[0]->org.apache.calcite.adapter.druid.DruidConnectionImpl$JsonSegmentMetadata["numRows"])

      In class org.apache.calcite.adapter.druid,DruidConnectionImpl, inner class JsonSegmentMetadata has integer fields for reading in Druid metadata like size or numRows which aught to be of type long so that they can hold larger numbers:

      private static class JsonSegmentMetadata {
        public String id;
        public List<String> intervals;
        public Map<String, JsonColumn> columns;
        *public int size;*
        *public int numRows;*
        public Map<String, JsonAggregator> aggregators;




            Unassigned Unassigned
            wtff SaschaC
            0 Vote for this issue
            1 Start watching this issue