utility to enable flexible ETL scenarios, supports golang plug-in for built-in consumer|transformer|producer options
Go has supported plugins for a while now since v1.8.
These are like compiled Go objects which can be loaded in runtime from a file, path of which can inferred dynamically during runtime as well.
So they are like dynamically linked libraries but better, these are dynamically loaded libraries. No linking required during compile time.
They fundamentally work like shared objects in linux, but Go main program don’t need to know of all plug-ins it can use during compile time.
above mentioned aspects are iteration of same feature, it helps your logic flow become truly generic and highly extensible
- people can manage plug-in with sensitive domain logic internally and use it safely in higher context
- people can contribute to capabilities of main workflow without worrying of it being merged first
To understand Go plugins separately, Francis Campoy’s simple demo can be looked at. KrakenD, api gateway uses Go plugins as here, if need to look at a real world project.
If you want to write a flow which needs all custom components where it can read from a file, check if each line is an existing path and produce that path mapped to its existence to stdout.