[FLINK-27928] External Resource Framework: 'external-resources' name delimitation not working as expected

• Type: Bug
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.15.0
• Component/s: Runtime / Configuration
• Labels: auto-deprioritized-major auto-deprioritized-minor external-resources
• Assignee: Unassigned
• Reporter: James Cho
• Created: 07/Jun/22 05:54
• Updated: 19/Aug/23 10:35

Description

For flink version 1.15.0, the configuration `external-resources` delimiter separation(";") functionality between external resource names is not working as expected. 

 

Reproducing the bug setting up a flink cluster running using docker with two external resources named `abc` and `efg`:

docker-compose.yml:

 

services:
  jobmanager:
    image: flink:1.15.0-scala_2.12
    ports:
      - "8081:8081"
      - "6123:6123"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager

  taskmanager:
    image: flink:1.15.0-scala_2.12
    depends_on:
      - jobmanager
    command: taskmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 8
        external-resources: abc;efg
        external-resource.abc.amount: 1
        external-resource.efg.amount: 1

 

Related task manager logs:

 

root@2873c5ae72c2:/opt/flink# cat log/flink* | grep ExternalResourceUtils
2022-06-07 05:42:34,232 WARN  org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - The amount of the abc;efg should be configured. Will ignore that resource.
2022-06-07 05:42:34,232 INFO  org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: []
2022-06-07 05:42:34,236 WARN  org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not find driver class name for abc;efg. Please make sure external-resource.abc;efg.driver-factory.class is configured.
2022-06-07 05:42:34,240 WARN  org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - The amount of the abc;efg should be configured. Will ignore that resource.

Notice flink recognizes `abc;efg` as a single external resource rather than as `abc` and `efg`

 

 

The same issue does not exist in flink version 1.14.4

docker-compose.yml:

services:
  jobmanager:
    image: flink:1.14.4-scala_2.12
    ports:
      - "8081:8081"
      - "6123:6123"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager

  taskmanager:
    image: flink:1.14.4-scala_2.12
    depends_on:
      - jobmanager
    command: taskmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 8
        external-resources: abc;efg
        external-resource.abc.amount: 1
        external-resource.efg.amount: 1 

related tm logs

docker logs ... | grep ExternalResourceUtils
2022-06-07 05:48:13,531 INFO  org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: [abc, efg]
2022-06-07 05:48:13,534 WARN  org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not find driver class name for abc. Please make sure external-resource.abc.driver-factory.class is configured.
2022-06-07 05:48:13,534 WARN  org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Could not find driver class name for efg. Please make sure external-resource.efg.driver-factory.class is configured.