Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3904

insert into data got Failed to create directory path /d

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None
    • Environment:
      spark-2.4.5
      hadoop 2.7.3
      carbondata2.0.1

      Description

      insert data

      spark.sql("INSERT OVERWRITE TABLE ods.test_table SELECT * FROM ods.socol_cmdinfo")
      

       check logs from spark application on yarn

      $ yarn logs -applicationId application_1592787941917_4116

      found a lot this error messages

      20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 16:59:45 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 16:59:51 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:00:00 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:00:35 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:02:47 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:03:36 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:09:55 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:10:05 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:11:08 ERROR FileFactory:  Failed to create directory path /d
      20/07/15 17:12:45 ERROR FileFactory:  Failed to create directory path /d
      
      core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
      
      public static void createDirectoryAndSetPermission(String directoryPath, FsPermission permission)
            throws IOException {
          FileFactory.FileType fileType = FileFactory.getFileType(directoryPath);
          switch (fileType) {
            case S3:
            case HDFS:
            case ALLUXIO:
            case VIEWFS:
            case CUSTOM:
            case HDFS_LOCAL:
              try {
                Path path = new Path(directoryPath);
                FileSystem fs = path.getFileSystem(getConfiguration());
                if (!fs.exists(path)) {
                  fs.mkdirs(path);
                  fs.setPermission(path, permission);
                }
              } catch (IOException e) {
                LOGGER.error("Exception occurred : " + e.getMessage(), e);
                throw e;
              }
              return;
            case LOCAL:
            default:
              directoryPath = FileFactory.getUpdatedFilePath(directoryPath);
              File file = new File(directoryPath);
              if (!file.mkdirs()) {
                LOGGER.error(" Failed to create directory path " + directoryPath);
              }    }
        }
      

       

      I output the variable directoryPath and fileType

      if (!file.mkdirs()) {
                //  check variables
                LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");
                LOGGER.error(" Failed to create directory path " + directoryPath);
              }
      

      add line 

      LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" + fileType.toString() + "]");

      got echo on yarn logs

      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
      2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]

      ....

       

      Why fileType is LOCAL?

      I have never set this value: directoryPath = [/d]?

      But the data is inserted normally and can be queried normally.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hugess1f XiaoWen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: