Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3499

Backend cannot write catalog-update topic more than 2GB to jByteArray

    Details

      Description

      I've encountered a issue after impalad restarts. Logging shows:

      E0509 11:44:08.168478 72547 impala-server.cc:1325] There was an error processing the impalad catalog update. Requesting a full topic update to recover: NegativeArraySizeException: null

      This may be related to size of catalog-update topic in statestored exceeding 2GB.
      In updateCatalogCache, serialized data passed through JNI are represented as byte[], which cannot exceed that limit.

      When I restart catalogd (that is, reset catalog-update topic), everything goes fine.

      I'm assuming that metadata larger than 2GB can be a common case in massive scale usage.
      Is there any possible approach to fix this?
      Thanks.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                HuaisiXu Huaisi Xu
                Reporter:
                He Tianyi He Tianyi
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: