Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-42901

Should move message StorageLevel from `base.proto` to a separate file

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.5.0
    • Connect
    • None

    Description

      https://github.com/apache/spark/pull/40510 introduce `message StorageLevel` to `base.proto`, but if we try to import `base.proto` in `catalog.proto` to reuse `StorageLevel` in `message CacheTable` and run `build/sbt "connect-common/compile" to compile, there will be following message in compile log:

       

      spark/connect/base.proto:23:1: File recursively imports itself: spark/connect/base.proto -> spark/connect/commands.proto -> spark/connect/relations.proto -> spark/connect/catalog.proto -> spark/connect/base.proto
      spark/connect/catalog.proto:22:1: Import "spark/connect/base.proto" was not found or had errors.
      spark/connect/catalog.proto:144:12: "spark.connect.DataType" seems to be defined in "spark/connect/types.proto", which is not imported by "spark/connect/catalog.proto".  To use it here, please add the necessary import.
      spark/connect/catalog.proto:161:12: "spark.connect.DataType" seems to be defined in "spark/connect/types.proto", which is not imported by "spark/connect/catalog.proto".  To use it here, please add the necessary import.
      spark/connect/relations.proto:25:1: Import "spark/connect/catalog.proto" was not found or had errors.
      spark/connect/relations.proto:84:5: "Catalog" is not defined.
      spark/connect/commands.proto:22:1: Import "spark/connect/relations.proto" was not found or had errors.
      spark/connect/commands.proto:63:3: "Relation" is not defined.
      spark/connect/commands.proto:81:3: "Relation" is not defined.
      spark/connect/commands.proto:142:3: "Relation" is not defined.
      spark/connect/base.proto:23:1: Import "spark/connect/commands.proto" was not found or had errors.
      spark/connect/base.proto:25:1: Import "spark/connect/relations.proto" was not found or had errors.
      .... 

      So we should move `message StorageLevel` from `base.proto` to a separate file to avoid this issue

       

       

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: