[HDDS-8914] Datanode may fail to start due to duplicate VolumeInfoMetrics

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Ozone Datanode
• Labels: pull-request-available
• Target Version/s: 1.4.0, 1.3.1
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 23/Jun/23 08:24
• Updated: 12/Jul/23 07:11
• Resolved: 26/Jun/23 18:38

Description

https://github.com/apache/ozone/blob/d66d15fbea56b1e1d3c9dd5c2c55707bb34dd24d/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/MutableVolumeSet.java#L169-L190

      try {
        StorageLocation location = StorageLocation.parse(locationString);


        StorageVolume volume = volumeFactory.createVolume(
            location.getUri().getPath(), location.getStorageType());


        LOG.info("Added Volume : {} to VolumeSet",
            volume.getStorageDir().getPath());


        if (!volume.getStorageDir().mkdirs() &&
            !volume.getStorageDir().exists()) {
          throw new IOException("Failed to create storage dir " +
              volume.getStorageDir());
        }
        volumeMap.put(volume.getStorageDir().getPath(), volume);
        volumeStateMap.get(volume.getStorageType()).add(volume);
      } catch (IOException e) {
        StorageVolume volume =
            volumeFactory.createFailedVolume(locationString);
        failedVolumeMap.put(locationString, volume);
        LOG.error("Failed to parse the storage location: " + locationString, e);
      }

Any IOException thrown from/after volumeFactory.createVolume results in the creation of another, failed volume for the same dir (volumeFactory.createFailedVolume).

Metrics is registered for each HddsVolume in its constructor. The metrics for the original volume should be unregistered, otherwise we get:

MetricsException: Metrics source VolumeInfoMetrics-/data/hdds/... already exists

and datanode fails to start.

Issue Links

• duplicates: HDDS-8507 Ozone Datanode fails with Metrics source path already exists
• is related to: HDDS-7086 DataNode UI to show failed volumes too
• links to: GitHub Pull Request #4966