1.16.10

flytekit.types.structured.basic_dfs

Directory

Classes

Class Description
ArrowToParquetEncodingHandler Helper class that provides a standard way to create an ABC using.
CSVToPandasDecodingHandler Helper class that provides a standard way to create an ABC using.
PandasToCSVEncodingHandler Helper class that provides a standard way to create an ABC using.
PandasToParquetEncodingHandler Helper class that provides a standard way to create an ABC using.
ParquetToArrowDecodingHandler Helper class that provides a standard way to create an ABC using.
ParquetToPandasDecodingHandler Helper class that provides a standard way to create an ABC using.

Methods

Method Description
get_pandas_storage_options()

Variables

Property Type Description
CSV str
PARQUET str
T TypeVar

Methods

get_pandas_storage_options() 

def get_pandas_storage_options(
    uri: str,
    data_config: flytekit.configuration.DataConfig,
    anonymous: bool,
) -> typing.Optional[typing.Dict]
Parameter Type Description
uri str
data_config flytekit.configuration.DataConfig
anonymous bool

flytekit.types.structured.basic_dfs.ArrowToParquetEncodingHandler

Helper class that provides a standard way to create an ABC using inheritance.

def ArrowToParquetEncodingHandler()

Extend this abstract class, implement the encode function, and register your concrete class with the StructuredDatasetTransformerEngine class in order for the core flytekit type engine to handle dataframe libraries. This is the encoding interface, meaning it is used when there is a Python value that the flytekit type engine is trying to convert into a Flyte Literal. For the other way, see the StructuredDatasetEncoder

Methods

Method Description
encode() Even if the user code returns a plain dataframe instance, the dataset transformer engine will wrap the.

encode() 

def encode(
    ctx: flytekit.core.context_manager.FlyteContext,
    structured_dataset: flytekit.types.structured.structured_dataset.StructuredDataset,
    structured_dataset_type: flytekit.models.types.StructuredDatasetType,
) -> flytekit.models.literals.StructuredDataset

Even if the user code returns a plain dataframe instance, the dataset transformer engine will wrap the incoming dataframe with defaults set for that dataframe type. This simplifies this function’s interface as a lot of data that could be specified by the user using the

TODO: Do we need to add a flag to indicate if it was wrapped by the transformer or by the user?

Parameter Type Description
ctx flytekit.core.context_manager.FlyteContext
structured_dataset flytekit.types.structured.structured_dataset.StructuredDataset This is a StructuredDataset wrapper object. See more info above.
structured_dataset_type flytekit.models.types.StructuredDatasetType This the StructuredDatasetType, as found in the LiteralType of the interface of the task that invoked this encoding call. It is passed along to encoders so that authors of encoders can include it in the returned literals.StructuredDataset. See the IDL for more information on why this literal in particular carries the type information along with it. If the encoder doesn’t supply it, it will also be filled in after the encoder runs by the transformer engine. :return: This function should return a StructuredDataset literal object. Do not confuse this with the StructuredDataset wrapper class used as input to this function - that is the user facing Python class. This function needs to return the IDL StructuredDataset.

Properties

Property Type Description
protocol
python_type
supported_format

flytekit.types.structured.basic_dfs.CSVToPandasDecodingHandler

Helper class that provides a standard way to create an ABC using inheritance.

def CSVToPandasDecodingHandler()

Extend this abstract class, implement the decode function, and register your concrete class with the StructuredDatasetTransformerEngine class in order for the core flytekit type engine to handle dataframe libraries. This is the decoder interface, meaning it is used when there is a Flyte Literal value, and we have to get a Python value out of it. For the other way, see the StructuredDatasetEncoder

Methods

Method Description
decode() This is code that will be called by the dataset transformer engine to ultimately translate from a Flyte Literal.

decode() 

def decode(
    ctx: flytekit.core.context_manager.FlyteContext,
    flyte_value: flytekit.models.literals.StructuredDataset,
    current_task_metadata: flytekit.models.literals.StructuredDatasetMetadata,
) -> pd.DataFrame

This is code that will be called by the dataset transformer engine to ultimately translate from a Flyte Literal value into a Python instance.

Parameter Type Description
ctx flytekit.core.context_manager.FlyteContext A FlyteContext, useful in accessing the filesystem and other attributes
flyte_value flytekit.models.literals.StructuredDataset This will be a Flyte IDL StructuredDataset Literal - do not confuse this with the StructuredDataset class defined also in this module.
current_task_metadata flytekit.models.literals.StructuredDatasetMetadata Metadata object containing the type (and columns if any) for the currently executing task. This type may have more or less information than the type information bundled inside the incoming flyte_value. :return: This function can either return an instance of the dataframe that this decoder handles, or an iterator of those dataframes.

Properties

Property Type Description
protocol
python_type
supported_format

flytekit.types.structured.basic_dfs.PandasToCSVEncodingHandler

Helper class that provides a standard way to create an ABC using inheritance.

def PandasToCSVEncodingHandler()

Extend this abstract class, implement the encode function, and register your concrete class with the StructuredDatasetTransformerEngine class in order for the core flytekit type engine to handle dataframe libraries. This is the encoding interface, meaning it is used when there is a Python value that the flytekit type engine is trying to convert into a Flyte Literal. For the other way, see the StructuredDatasetEncoder

Methods

Method Description
encode() Even if the user code returns a plain dataframe instance, the dataset transformer engine will wrap the.

encode() 

def encode(
    ctx: flytekit.core.context_manager.FlyteContext,
    structured_dataset: flytekit.types.structured.structured_dataset.StructuredDataset,
    structured_dataset_type: flytekit.models.types.StructuredDatasetType,
) -> flytekit.models.literals.StructuredDataset

Even if the user code returns a plain dataframe instance, the dataset transformer engine will wrap the incoming dataframe with defaults set for that dataframe type. This simplifies this function’s interface as a lot of data that could be specified by the user using the

TODO: Do we need to add a flag to indicate if it was wrapped by the transformer or by the user?

Parameter Type Description
ctx flytekit.core.context_manager.FlyteContext
structured_dataset flytekit.types.structured.structured_dataset.StructuredDataset This is a StructuredDataset wrapper object. See more info above.
structured_dataset_type flytekit.models.types.StructuredDatasetType This the StructuredDatasetType, as found in the LiteralType of the interface of the task that invoked this encoding call. It is passed along to encoders so that authors of encoders can include it in the returned literals.StructuredDataset. See the IDL for more information on why this literal in particular carries the type information along with it. If the encoder doesn’t supply it, it will also be filled in after the encoder runs by the transformer engine. :return: This function should return a StructuredDataset literal object. Do not confuse this with the StructuredDataset wrapper class used as input to this function - that is the user facing Python class. This function needs to return the IDL StructuredDataset.

Properties

Property Type Description
protocol
python_type
supported_format

flytekit.types.structured.basic_dfs.PandasToParquetEncodingHandler

Helper class that provides a standard way to create an ABC using inheritance.

def PandasToParquetEncodingHandler()

Extend this abstract class, implement the encode function, and register your concrete class with the StructuredDatasetTransformerEngine class in order for the core flytekit type engine to handle dataframe libraries. This is the encoding interface, meaning it is used when there is a Python value that the flytekit type engine is trying to convert into a Flyte Literal. For the other way, see the StructuredDatasetEncoder

Methods

Method Description
encode() Even if the user code returns a plain dataframe instance, the dataset transformer engine will wrap the.

encode() 

def encode(
    ctx: flytekit.core.context_manager.FlyteContext,
    structured_dataset: flytekit.types.structured.structured_dataset.StructuredDataset,
    structured_dataset_type: flytekit.models.types.StructuredDatasetType,
) -> flytekit.models.literals.StructuredDataset

Even if the user code returns a plain dataframe instance, the dataset transformer engine will wrap the incoming dataframe with defaults set for that dataframe type. This simplifies this function’s interface as a lot of data that could be specified by the user using the

TODO: Do we need to add a flag to indicate if it was wrapped by the transformer or by the user?

Parameter Type Description
ctx flytekit.core.context_manager.FlyteContext
structured_dataset flytekit.types.structured.structured_dataset.StructuredDataset This is a StructuredDataset wrapper object. See more info above.
structured_dataset_type flytekit.models.types.StructuredDatasetType This the StructuredDatasetType, as found in the LiteralType of the interface of the task that invoked this encoding call. It is passed along to encoders so that authors of encoders can include it in the returned literals.StructuredDataset. See the IDL for more information on why this literal in particular carries the type information along with it. If the encoder doesn’t supply it, it will also be filled in after the encoder runs by the transformer engine. :return: This function should return a StructuredDataset literal object. Do not confuse this with the StructuredDataset wrapper class used as input to this function - that is the user facing Python class. This function needs to return the IDL StructuredDataset.

Properties

Property Type Description
protocol
python_type
supported_format

flytekit.types.structured.basic_dfs.ParquetToArrowDecodingHandler

Helper class that provides a standard way to create an ABC using inheritance.

def ParquetToArrowDecodingHandler()

Extend this abstract class, implement the decode function, and register your concrete class with the StructuredDatasetTransformerEngine class in order for the core flytekit type engine to handle dataframe libraries. This is the decoder interface, meaning it is used when there is a Flyte Literal value, and we have to get a Python value out of it. For the other way, see the StructuredDatasetEncoder

Methods

Method Description
decode() This is code that will be called by the dataset transformer engine to ultimately translate from a Flyte Literal.

decode() 

def decode(
    ctx: flytekit.core.context_manager.FlyteContext,
    flyte_value: flytekit.models.literals.StructuredDataset,
    current_task_metadata: flytekit.models.literals.StructuredDatasetMetadata,
) -> pa.Table

This is code that will be called by the dataset transformer engine to ultimately translate from a Flyte Literal value into a Python instance.

Parameter Type Description
ctx flytekit.core.context_manager.FlyteContext A FlyteContext, useful in accessing the filesystem and other attributes
flyte_value flytekit.models.literals.StructuredDataset This will be a Flyte IDL StructuredDataset Literal - do not confuse this with the StructuredDataset class defined also in this module.
current_task_metadata flytekit.models.literals.StructuredDatasetMetadata Metadata object containing the type (and columns if any) for the currently executing task. This type may have more or less information than the type information bundled inside the incoming flyte_value. :return: This function can either return an instance of the dataframe that this decoder handles, or an iterator of those dataframes.

Properties

Property Type Description
protocol
python_type
supported_format

flytekit.types.structured.basic_dfs.ParquetToPandasDecodingHandler

Helper class that provides a standard way to create an ABC using inheritance.

def ParquetToPandasDecodingHandler()

Extend this abstract class, implement the decode function, and register your concrete class with the StructuredDatasetTransformerEngine class in order for the core flytekit type engine to handle dataframe libraries. This is the decoder interface, meaning it is used when there is a Flyte Literal value, and we have to get a Python value out of it. For the other way, see the StructuredDatasetEncoder

Methods

Method Description
decode() This is code that will be called by the dataset transformer engine to ultimately translate from a Flyte Literal.

decode() 

def decode(
    ctx: flytekit.core.context_manager.FlyteContext,
    flyte_value: flytekit.models.literals.StructuredDataset,
    current_task_metadata: flytekit.models.literals.StructuredDatasetMetadata,
) -> pd.DataFrame

This is code that will be called by the dataset transformer engine to ultimately translate from a Flyte Literal value into a Python instance.

Parameter Type Description
ctx flytekit.core.context_manager.FlyteContext A FlyteContext, useful in accessing the filesystem and other attributes
flyte_value flytekit.models.literals.StructuredDataset This will be a Flyte IDL StructuredDataset Literal - do not confuse this with the StructuredDataset class defined also in this module.
current_task_metadata flytekit.models.literals.StructuredDatasetMetadata Metadata object containing the type (and columns if any) for the currently executing task. This type may have more or less information than the type information bundled inside the incoming flyte_value. :return: This function can either return an instance of the dataframe that this decoder handles, or an iterator of those dataframes.

Properties

Property Type Description
protocol
python_type
supported_format