Support for multiple Spark versions

[gerashegalov]

Problem
We need to be able to support multiple Spark versions to avoid painful upgrades

Solution

• Create build profiles capturing dependencies of a particular Spark version. Add a Shim library with the stable facade translating API's that change let us say from 2.3.2 to 2.4. x to 3.x

Alternatives
continue being in lock step with Spark and suffer

Additional context
Support for wider variety of users with differing pace of Spark version adoption

[koertkuipers]

not just multiple spark versions... also multiple scala versions

[SemanticBeeng]

Maybe we can reuse the approach from frameless https://github.com/typelevel/frameless

Frameless intentionally does not have a compile dependency on Spark. This essentially allows you to use any version of Frameless with any version of Spark.

Since the 0.9.x release, Frameless is compiled only against Scala 2.12.x.