Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31084

spark on k8s Exception "Database xxx not found" when hive MetaStoreClient lost connection

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 2.4.4
    • Fix Version/s: None
    • Component/s: Kubernetes, Spark Core
    • Labels:
      None
    • Environment:

      spark 2.4.4

       

       

       

      Description

      org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke:184 | MetaStoreClient lost connection. Attempting to reconnect.org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
      06-03-2020 12:56:09 CST stage1 INFO - Caused by: java.net.SocketException: Connection reset
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - at java.net.SocketInputStream.read(SocketInputStream.java:210)
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - at java.net.SocketInputStream.read(SocketInputStream.java:141)
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - ... 70 more

      06-03-2020 12:56:09 CST stage1 INFO - 2020-03-06 04:56:06 [INFO] – org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open:376 | Trying to connect to metastore with URI thrift://hive-metastore-server:9083
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - 2020-03-06 04:56:06 [INFO] – org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open:472 | Connected to metastore.
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - 2020-03-06 04:56:14 [WARN] – org.apache.spark.internal.Logging$class.logWarning:87 | Kubernetes client has been closed (this is expected if the application is shutting down.)
      06-03-2020 12:56:09 CST stage1 INFO - 2020-03-06 04:56:14 [WARN] – org.apache.spark.internal.Logging$class.logWarning:87 | Kubernetes client has been closed (this is expected if the application is shutting down.)
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - Exception in thread "main" org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'xxx' not found;
      06-03-2020 12:56:09 CST stage1 INFO -
      06-03-2020 12:56:09 CST stage1 INFO - at org.apache.spark.sql.catalyst.catalog.SessionCatalog.org$apache$spark$sql$catalyst$catalog$SessionCatalog$$requireDbExists(SessionCatalog.scala:178)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              liuxiuyuan92 liuxiuyuan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: