P3 Description
If a DoFn's `ProcessElement`uses an emitter `func(MyType)` where `MyType` is not a concrete type, (such as having function type fields, which are not valid) the pipeline can fail with incomprehensible errors that don't explain the problem correctly. eg.
{{panic: inserting ParDo in scope *mypackage.studyGenerationElement
constructing DoFn
invalid DoFn
caused by:
method ProcessElement invalid
caused by:
bad parameter type for reflect.methodValueCall: func(*mypackage.studyGenerationElement)
goroutine 740 [running]:
panic(0x55e4c34f3e80, 0xc005952680)
go/gc/src/runtime/panic.go:1066+0x47e fp=0xc0061a18c8 sp=0xc0061a17d8 pc=0x55e49ff4b67e
testing.tRunner.func1.1(0x55e4c34f3e80, 0xc005952680)
go/gc/src/testing/testing.go:1076 +0x30d fp=0xc0061a1980 sp=0xc0061a18c8 pc=0x55e4a097254d
testing.tRunner.func1(0xc005046900)
go/gc/src/testing/testing.go:1079 +0x41a fp=0xc0061a1a28 sp=0xc0061a1980 pc=0x55e4a097297a
runtime.call32(0x0, 0x55e4c3f0e388, 0xc0050dbf38, 0x800000008)
go/gc/src/runtime/asm_amd64.s:540 +0x40 fp=0xc0061a1a58 sp=0xc0061a1a28 pc=0x55e49ff86fc0
runtime.reflectcallSave(0xc0061a1b88, 0x55e4c3f0e388, 0xc0050dbf38, 0x55e400000008)
go/gc/src/runtime/panic.go:881 +0x5a fp=0xc0061a1a88 sp=0xc0061a1a58 pc=0x55e49ff4b13a
runtime.runOpenDeferFrame(0xc0035bc180, 0xc0050dbef0, 0xc0061a1c08)
go/gc/src/runtime/panic.go:855 +0x2cd fp=0xc0061a1b18 sp=0xc0061a1a88 pc=0x55e49ff4afed
panic(0x55e4c34f3e80, 0xc005952680)
go/gc/src/runtime/panic.go:969 +0x175 fp=0xc0061a1c08 sp=0xc0061a1b18 pc=0x55e49ff4b375
...pkg/beam/beam.MustN(...)...pkg/beam/util.go:104
...pkg/beam/beam.ParDo(0xc00507d9e0, 0xc0052a7b30, 0x55e4c30faae0, 0xc0053244e0, 0xc0053245a0, 0x0, 0x0, 0x0, 0x0)
...pkg/beam/pardo.go:358 +0x135 fp=0xc0061a1c80 sp=0xc0061a1c08 pc=0x55e4a0f3cc95
}}
In particular this message is generated by funcx.New and occurs because we don't maintain an error state when trying to analyze the DoFn.
We would catch most errors like this if we provide a justification for it better.
This will likely mean reimplementing the IsConcrete function and similar in terms of errors with the explanations, and having a justification handler to return that error from funcx.New if the type is invalid. This is subtle as while most errors are due to a type not being concrete, but the type might be nested in a emitter or iterator function.
