flytekitplugins.spark.SparkDataFrameTransformer#
- class flytekitplugins.spark.SparkDataFrameTransformer[source]#
Transforms Spark DataFrame’s to and from a Schema (typed/untyped)
Methods
- assert_type(t, v)#
- Parameters
t (Type[flytekit.core.type_engine.T]) –
v (flytekit.core.type_engine.T) –
- get_literal_type(t)[source]#
Converts the python type to a Flyte LiteralType
- Parameters
t (Type[pyspark.sql.dataframe.DataFrame]) –
- Return type
- guess_python_type(literal_type)#
Converts the Flyte LiteralType to a python object type.
- Parameters
literal_type (flytekit.models.types.LiteralType) –
- Return type
Type[flytekit.core.type_engine.T]
- to_html(ctx, python_val, expected_python_type)#
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div
- Parameters
python_val (flytekit.core.type_engine.T) –
expected_python_type (Type[flytekit.core.type_engine.T]) –
- Return type
- to_literal(ctx, python_val, python_type, expected)[source]#
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch :param ctx: A FlyteContext, useful in accessing the filesystem and other attributes :param python_val: The actual value to be transformed :param python_type: The assumed type of the value (this matches the declared type on the function) :param expected: Expected Literal Type
- Parameters
python_val (pyspark.sql.dataframe.DataFrame) –
python_type (Type[pyspark.sql.dataframe.DataFrame]) –
expected (flytekit.models.types.LiteralType) –
- Return type
- to_python_value(ctx, lv, expected_python_type)[source]#
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised :param ctx: FlyteContext :param lv: The received literal Value :param expected_python_type: Expected native python type that should be returned
- Parameters
expected_python_type (Type[pyspark.sql.dataframe.DataFrame]) –
- Return type
flytekit.core.type_engine.T
Attributes
- name#
- python_type#
This returns the python type
- type_assertions_enabled#
Indicates if the transformer wants type assertions to be enabled at the core type engine layer