[IMPALA-7668] close() URLClassLoaders after usage.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.12.0, Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Frontend
• Epic Color: ghx-label-2
• Assignee: Bharath Vissapragada
• Reporter: Bharath Vissapragada
• Created: 05/Oct/18 06:46
• Updated: 27/Dec/18 22:24
• Resolved: 23/Oct/18 23:41

Description

There are a few places in the code that uses URLClassLoaders to load some java classes at runtime. One example is when loading Java UDFs at startup.

public static List<Function> extractFunctions(String db,
      ...........
      URL[] classLoaderUrls = new URL[] {new URL(localJarPath.toString())};
      URLClassLoader urlClassLoader = new URLClassLoader(classLoaderUrls);

Starting JDK7, URLClassloader lets the caller close all the closeables opened by it, avoiding bugs like FD leaks etc.

https://docs.oracle.com/javase/7/docs/api/java/net/URLClassLoader.html#close()

We have seen issues like lingering FDs from this code using certain versions of JDKs where the FDs of temporary jars (copied to /tmp) by this code are not closed and hence their space from disk is not claimed causing disk space issues.

Usually, error message looks like the following (shows up as "out of disk space in /tmp").

I1101 12:29:09.738245 3067 jni-util.cc:230] org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:256)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:417)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:213)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:164)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:145)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:405)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:246)
at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:285)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:70)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:121)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2123)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2092)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2068)
at org.apache.impala.common.FileSystemUtil.copyToLocal(FileSystemUtil.java:418)
at org.apache.impala.catalog.CatalogServiceCatalog.extractFunctions(CatalogServiceCatalog.java:851)
at org.apache.impala.catalog.CatalogServiceCatalog.loadJavaFunctions(CatalogServiceCatalog.java:944)
at org.apache.impala.catalog.CatalogServiceCatalog.refreshFunctions(CatalogServiceCatalog.java:978)
at org.apache.impala.service.CatalogOpExecutor.execResetMetadata(CatalogOpExecutor.java:3091)
at org.apache.impala.service.JniCatalog.resetMetadata(JniCatalog.java:160)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:254)
... 27 more
 I1101 12:29:09.755692 3067 status.cc:125] FSError: java.io.IOException: No space left on device
CAUSED BY: IOException: No space left on device
    @ 0x96628a impala::Status::Status()
    @ 0xced58d impala::JniUtil::GetJniExceptionMsg()
    @ 0x9578e0 impala::Catalog::ResetMetadata()
    @ 0x94a68f CatalogServiceThriftIf::ResetMetadata()
    @ 0x9fc997 impala::CatalogServiceProcessor::process_ResetMetadata()
    @ 0x9faa6f impala::CatalogServiceProcessor::dispatchCall()
    @ 0x933f2a apache::thrift::TDispatchProcessor::process()
    @ 0xb17c19 apache::thrift::server::TAcceptQueueServer::Task::run()
    @ 0xb10399 impala::ThriftThread::RunRunnable()
    @ 0xb11192 boost::detail::function::void_function_obj_invoker0<>::invoke()
    @ 0xd602cf impala::Thread::SuperviseThread()
    @ 0xd60aca boost::detail::thread_data<>::run()
    @ 0x12d758a (unknown)
    @ 0x3430807aa1 (unknown)
    @ 0x34304e8c4d (unknown)
 E1101 12:29:09.756640 3067 catalog-server.cc:84] FSError: java.io.IOException: No space left on device
CAUSED BY: IOException: No space left on device

/tmp looks filled up

catalogd 3634 impala 275r REG 253,5 11961993 44 /tmp/7c1acc9e-2b82-4668-a099-4fb3b96b328f.jar (deleted)
catalogd 3634 impala 277r REG 253,5 11961993 46 /tmp/f4a49e7d-f5a9-4ae5-9291-62ce39233d0b.jar (deleted)
catalogd 3634 impala 279r REG 253,5 11961993 67 /tmp/ceaa27a6-37a9-4139-afb0-694949f8d5ad.jar (deleted)
catalogd 3634 impala 281r REG 253,5 11961993 47 /tmp/fbc12d6d-aeab-47c7-944c-f4580927f24f.jar (deleted)
catalogd 3634 impala 282r REG 253,5 11961993 48 /tmp/793815e9-c0a2-46c6-a4f8-64e3ff3e3942.jar (deleted)
catalogd 3634 impala 284r REG 253,5 49661 49 /tmp/256be23b-5853-42cc-8922-44e84d31f84c.jar (deleted)
catalogd 3634 impala 285r REG 253,5 11961993 69 /tmp/1136b8dd-aeff-4f5d-bf6e-99c26b0e7980.jar (deleted)
catalogd 3634 impala 286r REG 253,5 49661 50 /tmp/7e1555a7-e9d3-4136-ad22-5cb41e51fdf4.jar (deleted)
catalogd 3634 impala 287r REG 253,5 11961988 88 /tmp/62659aa9-b313-4670-ad04-06336e35ae3e.jar (deleted)
catalogd 3634 impala 288r REG 253,5 11961993 51 /tmp/64c267d8-7c66-4c7a-b66d-0bd32743d290.jar (deleted)
catalogd 3634 impala 289r REG 253,5 11961993 70 /tmp/119ddf11-6617-4691-8856-56387bdd218d.jar (deleted)
catalogd 3634 impala 290r REG 253,5 49661 52 /tmp/114f90b9-c40e-429d-a1ee-d62ad75247f4.jar (deleted)
catalogd 3634 impala 291r REG 253,5 49661 53 /tmp/1e74761b-09ee-4916-9ebe-05c78c93fa20.jar (deleted)
catalogd 3634 impala 292r REG 253,5 49661 54 /tmp/54b722b3-d995-48f4-a5f9-6e29084a3a18.jar (deleted)
catalogd 3634 impala 293r REG 253,5 49661 55 /tmp/3458c0ea-cf95-4f2e-b47d-ce5573a30843.jar (deleted)
catalogd 3634 impala 303r REG 253,5 11961993 56 /tmp/d08e5ec7-407a-4036-8a1e-f59c9c5a56ef.jar (deleted)
catalogd 3634 impala 304r REG 253,5 11961993 60 /tmp/0b06d239-9025-4ac4-9d42-9a172a6fd9a7.jar (deleted)
catalogd 3634 impala 305r REG 253,5 11961988 89 /tmp/9f510691-1cd9-4e6d-80e7-2d56a939e283.jar (deleted)
catalogd 3634 impala 306r REG 253,5 11961993 63 /tmp/e664a87e-0734-458b-9e54-f61dbdf30e1b.jar (deleted)
catalogd 3634 impala 307r REG 253,5 11961993 66 /tmp/29720d48-4b1a-476b-a8ce-f9e3c42006e6.jar (deleted)
catalogd 3634 impala 308r REG 253,5 11961993 71 /tmp/27f98197-80fb-4704-a887-d7d3458dfd11.jar (deleted)
catalogd 3634 impala 309r REG 253,5 49660 72 /tmp/2fb57d8b-5962-4927-8482-58a2d16fa6ca.jar (deleted)
catalogd 3634 impala 310r REG 253,5 11961988 102 /tmp/ea7aa558-dd70-4e4f-b438-8a145d187044.jar (deleted)
catalogd 3634 impala 311r REG 253,5 49660 73 /tmp/fd499d2b-c042-4f5e-ab04-0508444061dc.jar (deleted)
catalogd 3634 impala 312r REG 253,5 11961988 90 /tmp/b92b2beb-9fd3-49ba-b999-b1053877e4cf.jar (deleted)
catalogd 3634 impala 313r REG 253,5 11961993 74 /tmp/f76647c5-c8d6-494c-bb1f-a6b346d6636e.jar (deleted)
catalogd 3634 impala 314r REG 253,5 49660 80 /tmp/d8796b89-1bf7-44e1-80b1-4c91acb695f7.jar (deleted)
catalogd 3634 impala 315r REG 253,5 49660 81 /tmp/97685f6a-40ca-449c-9f45-ca5c2ff65f8b.jar (deleted)
catalogd 3634 impala 316r REG 253,5 49660 82 /tmp/174c33ad-ab2f-4150-b147-273c3f9e9c3d.jar (deleted)
catalogd 3634 impala 317r REG 253,5 49660 83 /tmp/97592f9c-7f9b-4857-908a-639bc588009d.jar (deleted)
catalogd 3634 impala 318r REG 253,5 11961988 91 /tmp/a3a6dffc-8aee-424a-877c-09701f751905.jar (deleted)
catalogd 3634 impala 319r REG 253,5 11961993 84 /tmp/313d1b3d-a8ba-4211-8d8b-ef63c0a6085d.jar (deleted)
catalogd 3634 impala 320r REG 253,5 11961993 85 /tmp/f579a717-c10d-44c0-ac03-375a1cb94395.jar (deleted)
catalogd 3634 impala 321r REG 253,5 11961993 86 /tmp/8b8465a3-62e0-45f6-989e-59c08ebc8535.jar (deleted)