flytekitplugins.spark.SparkDataFrameTransformer#

class flytekitplugins.spark.SparkDataFrameTransformer[source]#

Transforms Spark DataFrame’s to and from a Schema (typed/untyped)

Methods

assert_type(t, v)#
Parameters

  • t (Type[flytekit.core.type_engine.T]) –

  • v (flytekit.core.type_engine.T) –

get_literal_type(t)[source]#

Converts the python type to a Flyte LiteralType

Parameters

t (Type[pyspark.sql.dataframe.DataFrame]) –

Return type

flytekit.models.types.LiteralType

guess_python_type(literal_type)#

Converts the Flyte LiteralType to a python object type.

Parameters

literal_type (flytekit.models.types.LiteralType) –

Return type

Type[flytekit.core.type_engine.T]

to_html(ctx, python_val, expected_python_type)#

Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div

Parameters
Return type

str

to_literal(ctx, python_val, python_type, expected)[source]#

Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch :param ctx: A FlyteContext, useful in accessing the filesystem and other attributes :param python_val: The actual value to be transformed :param python_type: The assumed type of the value (this matches the declared type on the function) :param expected: Expected Literal Type

Parameters
Return type

flytekit.models.literals.Literal

to_python_value(ctx, lv, expected_python_type)[source]#

Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised :param ctx: FlyteContext :param lv: The received literal Value :param expected_python_type: Expected native python type that should be returned

Parameters
Return type

flytekit.core.type_engine.T

Attributes

name
python_type

This returns the python type

type_assertions_enabled

Indicates if the transformer wants type assertions to be enabled at the core type engine layer