Data Types and IO#
Flyte being a data-aware orchestration platform, types play a vital role within it. This section provides an introduction to the wide range of data types that Flyte supports. These types serve a dual purpose by not only validating the data but also enabling seamless transfer of data between local and cloud storage. They enable:
Data lineage
Memoization
Auto parallelization
Simplifying access to data
Auto generated CLI and launch UI
For a more comprehensive understanding of how Flyte manages data, refer to the Understand How Flyte Handles Data guide.
Mapping Python to Flyte types#
Flytekit automatically translates most Python types into Flyte types. Here’s a breakdown of these mappings:
Python Type |
Flyte Type |
Conversion |
Comment |
---|---|---|---|
|
|
Automatic |
Use Python 3 type hints. |
|
|
Automatic |
Use Python 3 type hints. |
|
|
Automatic |
Use Python 3 type hints. |
|
|
Automatic |
Use Python 3 type hints. |
|
|
Not Supported |
You have the option to employ your own custom type transformer. |
|
NA |
Not Supported |
You have the option to employ your own custom type transformer. |
|
|
Automatic |
Use Python 3 type hints. |
|
|
Automatic |
Use Python 3 type hints. |
|
|
Automatic |
Use Python 3 type hints. |
|
|
Automatic |
Use |
|
|
Automatic |
Use |
File / file-like / |
|
Automatic |
If you’re using |
Directory |
|
Automatic |
When using |
|
|
Automatic |
Use |
|
JSON ( |
Automatic |
Use |
|
|
Automatic |
The class should be a pure value class that inherits from Mashumaro’s DataClassJSONMixin,
and be annotated with the |
|
File |
Automatic |
Use |
|
Structured Dataset |
Automatic |
Use |
|
Structured Dataset |
To utilize the type, install the |
Use |
|
|
To utilize the type, install the |
Use |
|
File |
To utilize the type, install the |
Use |
|
File |
To utilize the type, install the |
Use |
|
File |
To utilize the type, install the |
Use |
User defined types |
Any |
Custom transformers |
The |