Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39988

LevelDBIterator not close after used in `RemoteBlockPushResolver`, `YarnShuffleService` and `ExternalShuffleBlockResolver`

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Spark Core, YARN
    • None

    Description

      For example:

      @VisibleForTesting
      static ConcurrentMap<AppExecId, ExecutorShuffleInfo> reloadRegisteredExecutors(DB db)
          throws IOException {
        ConcurrentMap<AppExecId, ExecutorShuffleInfo> registeredExecutors = Maps.newConcurrentMap();
        if (db != null) {
          DBIterator itr = db.iterator();
          itr.seek(APP_KEY_PREFIX.getBytes(StandardCharsets.UTF_8));
          while (itr.hasNext()) {
            Map.Entry<byte[], byte[]> e = itr.next();
            String key = new String(e.getKey(), StandardCharsets.UTF_8);
            if (!key.startsWith(APP_KEY_PREFIX)) {
              break;
            }
            AppExecId id = parseDbAppExecKey(key);
            logger.info("Reloading registered executors: " +  id.toString());
            ExecutorShuffleInfo shuffleInfo = mapper.readValue(e.getValue(), ExecutorShuffleInfo.class);
            registeredExecutors.put(id, shuffleInfo);
          }
        }
        return registeredExecutors;
      } 

      DBIterator in above code should call close after used, otherwise, there is no other place to close it

       

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: