flytekitplugins.spark.SparkDataFrameSchemaReader

class flytekitplugins.spark.SparkDataFrameSchemaReader(from_path, cols, fmt)

Implements how SparkDataFrame should be read using the open method of FlyteSchema

Methods

Parameters

• from_path (str) –

• cols (Optional[Dict[str, type]]) –

• fmt (flytekit.types.schema.types.SchemaFormat) –

all(**kwargs)

Return type

pyspark.sql.dataframe.DataFrame

iter(**kwargs)

Return type

Generator[flytekit.core.type_engine.T, None, None]

Attributes

column_names

from_path