[IGNITE-12254] IO errors during write header of WAL files in FSYNC mode should be handled by failure handler

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.8
• Assignee: Aleksey Plekhanov
• Reporter: Aleksey Plekhanov
• Created: 01/Oct/19 15:55
• Updated: 15/Oct/19 06:42
• Resolved: 14/Oct/19 13:05

Description

Currently, such errors can hang the cluster.

Reproducer:

    @Test
    public void testWalFsyncIOError() throws Exception {
        cleanPersistenceDir();

        IgniteConfiguration cfg = new IgniteConfiguration();

        cfg.setCacheConfiguration(new CacheConfiguration(DEFAULT_CACHE_NAME).setAtomicityMode(ATOMIC));

        cfg.setDataStorageConfiguration(
            new DataStorageConfiguration()
                .setDefaultDataRegionConfiguration(
                    new DataRegionConfiguration()
                        .setMaxSize(100L * 1024 * 1024)
                        .setPersistenceEnabled(true))
                .setWalMode(WALMode.FSYNC)
                .setWalSegmentSize(512 * 1024)
                .setWalBufferSize(512 * 1024));

        IgniteEx ignite0 = startGrid(new IgniteConfiguration(cfg).setIgniteInstanceName("ignite0"));
        IgniteEx ignite1 = startGrid(new IgniteConfiguration(cfg).setIgniteInstanceName("ignite1"));

        ignite0.cluster().active(true);

        IgniteCache cache = ignite0.cache(DEFAULT_CACHE_NAME);

        for (int i = 0; i < 1_000; i++)
            cache.put(i, "Test value " + i);

        ((FileWriteAheadLogManager)ignite1.context().cache().context().wal()).setFileIOFactory(new FileIOFactory() {
            FileIOFactory delegateFactory = new RandomAccessFileIOFactory();
            @Override public FileIO create(File file, OpenOption... modes) throws IOException {
                final FileIO delegate = delegateFactory.create(file, modes);

                return new FileIODecorator(delegate) {
                    @Override public int write(ByteBuffer srcBuf) throws IOException {
                        throw new IOException("No space left on device");
                    }
                };
            }
        });

        for (int i = 0; i < 2_000; i++)
            try {
                cache.put(i, "Test value " + i);
            }
            catch (Exception ignore) {
            }
    }

Issue Links

• links to: GitHub Pull Request #6934