Spark API reference#
This package contains things that are useful when extending Flytekit.
|
Optionally creates a new spark session and returns it. |
Extend this abstract class, implement the decode function, and register your concrete class with the StructuredDatasetTransformerEngine class in order for the core flytekit type engine to handle dataframe libraries. |
|
|
Use this to configure a SparkContext for a your task. |
|
Implements how SparkDataFrame should be read using the |
|
Implements how SparkDataFrame should be written to using |
Transforms Spark DataFrame's to and from a Schema (typed/untyped) |