flytekitplugins.vaex.ParquetToVaexDataFrameDecodingHandler¶

class flytekitplugins.vaex.ParquetToVaexDataFrameDecodingHandler[source]¶

Methods

decode(ctx, flyte_value, current_task_metadata)[source]¶

This is code that will be called by the dataset transformer engine to ultimately translate from a Flyte Literal value into a Python instance.

Parameters:

ctx (FlyteContext) – A FlyteContext, useful in accessing the filesystem and other attributes
flyte_value (StructuredDataset) – This will be a Flyte IDL StructuredDataset Literal - do not confuse this with the StructuredDataset class defined also in this module.
current_task_metadata (StructuredDatasetMetadata) – Metadata object containing the type (and columns if any) for the currently executing task. This type may have more or less information than the type information bundled inside the incoming flyte_value.

Returns:

This function can either return an instance of the dataframe that this decoder handles, or an iterator of those dataframes.

Return type:

DataFrameLocal

Attributes