[HDDS-1680] Create missing parent directories during the creation of HddsVolume dirs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.1
• Labels: pull-request-available
• Target Version/s: 0.4.1
• Sprint: Done
• Assignee: Marton Elek
• Reporter: Marton Elek
• Created: 13/Jun/19 12:27
• Updated: 05/Aug/19 14:45
• Resolved: 13/Jun/19 23:18

Description

I started to execute all the unit tests continuously (in kubernetes with argo workflow).

Until now I got the following failures (number of failures / unit test name):

```
1 org.apache.hadoop.fs.ozone.contract.ITestOzoneContractMkdir
1 org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRename
3 org.apache.hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware
31 org.apache.hadoop.ozone.container.common.TestDatanodeStateMachine
31 org.apache.hadoop.ozone.container.common.volume.TestVolumeSet
1 org.apache.hadoop.ozone.freon.TestDataValidateWithSafeByteOperations
```

TestVolumeSet is also failed locally:

2019-06-13 14:23:18,637 ERROR volume.VolumeSet (VolumeSet.java:initializeVolumeSet(184)) - Failed to parse the storage location: /home/elek/projects/hadoop/hadoop-hdds/container-service/target/test-dir/dfs
java.io.IOException: Cannot create directory /home/elek/projects/hadoop/hadoop-hdds/container-service/target/test-dir/dfs/hdds
	at org.apache.hadoop.ozone.container.common.volume.HddsVolume.initialize(HddsVolume.java:208)
	at org.apache.hadoop.ozone.container.common.volume.HddsVolume.<init>(HddsVolume.java:179)
	at org.apache.hadoop.ozone.container.common.volume.HddsVolume.<init>(HddsVolume.java:72)
	at org.apache.hadoop.ozone.container.common.volume.HddsVolume$Builder.build(HddsVolume.java:156)
	at org.apache.hadoop.ozone.container.common.volume.VolumeSet.createVolume(VolumeSet.java:311)
	at org.apache.hadoop.ozone.container.common.volume.VolumeSet.initializeVolumeSet(VolumeSet.java:165)
	at org.apache.hadoop.ozone.container.common.volume.VolumeSet.<init>(VolumeSet.java:130)
	at org.apache.hadoop.ozone.container.common.volume.VolumeSet.<init>(VolumeSet.java:109)
	at org.apache.hadoop.ozone.container.common.volume.TestVolumeSet.testFailVolumes(TestVolumeSet.java:232)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

The problem here is that the parent directory of the volume dir is missing. I propose to use hddsRootDir.mkdirs() instead of hddsRootDir.mkdir() which creates the missing parent directories.

Issue Links

• links to: GitHub Pull Request #961