[FLINK-29311] Unable to use http for Kinesis consumer when testing locally

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.15.3
• Component/s: Connectors / Kinesis, Stateful Functions

Environment

Here is my Kinesis module code

 

kind: io.statefun.kinesis.v1/ingress
spec:
  id: com.sentiment
  awsRegion:
    id: us-east-1
    type: custom-endpoint
    customEndpoint: https://localstack:4566
  awsCredentials:
    type: basic
    id: us-east-1
    accessKeyId: key
    secretAccessKey: secret
  startupPosition:
    type: latest
  streams:
    - stream: customer-details
      valueType: SentimentEvent
      targets:
        - sentiment
  clientConfigProperties:
    - SocketTimeout: 9999
    - MaxConnections: 15

And I am using  the latest apache/statefun Docker image within docker compose

• Assignee: Unassigned
• Reporter: Nicolas Pocock
• Created: 15/Sep/22 09:06
• Updated: 11/Nov/22 08:36

Description

I am attempting to run the StateFun docker image locally in a docker-compose stack. I have Kinesis running using Localstack and want to use it as a consumer. 

The documentation says to use a custom endpoint you can use the following config with the io.statefun.kinesis.v1/ingress module:

awsCredentials:
  type: custom-endpoint
  endpoint: https://localhost:4567
  id: us-west-1

The above errors as the custom endpoint property is on the AWS region instead so it needs to be:

awsRegion:
  id: us-east-1
  type: custom-endpoint
  endpoint: http://localstack:4566

However, I am unable to use HTTP as it complains that:

Caused by: java.lang.IllegalArgumentException: Invalid service endpoint url: http://localstack:4566; Only custom service endpoints using HTTPS are supported

I tried using HTTPS but it errors with a certificate issue, I think this is an issue with Localstack.

Some of the docs for the Kinesis connectors show HTTP being used with a custom endpoint

https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kinesis/#using-custom-kinesis-endpoints

It would be nice if we can use HTTP when testing locally

I see from this issue https://github.com/localstack/localstack/issues/893  there is a configuration option TRUST_ALL_CERTIFICATES that can be provided to the NettyNioAsyncHttpClient that allows the above to work for local dev.