flytekitplugins.spark.SparkDataFrameSchemaWriter

class flytekitplugins.spark.SparkDataFrameSchemaWriter(to_path, cols, fmt)

Implements how SparkDataFrame should be written to using open method of FlyteSchema

Methods

Parameters

• to_path (str) –

• cols (Optional[Dict[str, type]]) –

• fmt (flytekit.types.schema.types.SchemaFormat) –

write(*dfs, **kwargs)

Parameters

dfs (pyspark.sql.dataframe.DataFrame) –

Attributes

column_names

to_path