[CARBONDATA-3904] insert into data got Failed to create directory path /d

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Component/s: core
• Environment: spark-2.4.5 ; hadoop 2.7.3 ; carbondata2.0.1
• Assignee: Unassigned
• Reporter: XiaoWen
• Created: 16/Jul/20 03:35
• Updated: 16/Jul/20 07:19

Description

insert data

spark.sql("INSERT OVERWRITE TABLE ods.test_table SELECT * FROM ods.socol_cmdinfo")

 check logs from spark application on yarn

$ yarn logs -applicationId application_1592787941917_4116

found a lot this error messages

20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
20/07/15 17:12:45 ERROR FileFactory:  Failed to create directory path /d

core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java

public static void createDirectoryAndSetPermission(String directoryPath, FsPermission permission)
      throws IOException {
    FileFactory.FileType fileType = FileFactory.getFileType(directoryPath);
    switch (fileType) {
      case S3:
      case HDFS:
      case ALLUXIO:
      case VIEWFS:
      case CUSTOM:
      case HDFS_LOCAL:
        try {
          Path path = new Path(directoryPath);
          FileSystem fs = path.getFileSystem(getConfiguration());
          if (!fs.exists(path)) {
            fs.mkdirs(path);
            fs.setPermission(path, permission);
          }
        } catch (IOException e) {
          LOGGER.error("Exception occurred : " + e.getMessage(), e);
          throw e;
        }
        return;
      case LOCAL:
      default:
        directoryPath = FileFactory.getUpdatedFilePath(directoryPath);
        File file = new File(directoryPath);
        if (!file.mkdirs()) {
          LOGGER.error(" Failed to create directory path " + directoryPath);
        }    }
  }

 

I output the variable directoryPath and fileType

if (!file.mkdirs()) {
          //  check variables
          LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
          LOGGER.error(" Failed to create directory path " + directoryPath);
        }

add line 

LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");

got echo on yarn logs

2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]

....

 

Why fileType is LOCAL?

I have never set this value: directoryPath = [/d]?

But the data is inserted normally and can be queried normally.