flytekit.core.data_persistence.FileAccessProvider#
- class flytekit.core.data_persistence.FileAccessProvider(local_sandbox_dir, raw_output_prefix, data_config=None)[source]#
This is the class that is available through the FlyteContext and can be used for persisting data to the remote durable store.
- Parameters
local_sandbox_dir (Union[str, os.PathLike]) – A local temporary working directory, that should be used to store data
raw_output_prefix (str) –
data_config (Optional[flytekit.configuration.DataConfig]) –
Methods
- construct_random_path(persist, file_path_or_file_name=None)[source]#
Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name
- Parameters
persist (flytekit.core.data_persistence.DataPersistence) –
- Return type
- download_directory(remote_path, local_path)[source]#
Downloads directory from given remote to local path
- get_data(remote_path, local_path, is_multipart=False)[source]#
- Parameters
remote_path (Text) –
local_path (Text) –
is_multipart (bool) –
- get_random_local_path(file_path_or_file_name=None)[source]#
Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name
- get_random_remote_path(file_path_or_file_name=None)[source]#
Constructs a randomized path on the configured raw_output_prefix (persistence layer). the random bit is a UUID and allows for disambiguating paths within the same directory.
Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name
- static is_remote(path)[source]#
Deprecated. Lets find a replacement
- Parameters
path (Union[str, os.PathLike]) –
- Return type
- put_data(local_path, remote_path, is_multipart=False)[source]#
The implication here is that we’re always going to put data to the remote location, so we .remote to ensure we don’t use the true local proxy if the remote path is a file://
- Parameters
local_path (Text) –
remote_path (Text) –
is_multipart (bool) –
- upload_directory(local_path, remote_path)[source]#
- Parameters
local_path (Text) –
remote_path (Text) –
Attributes
- data_config#
- local_access#
- local_sandbox_dir#
- raw_output_prefix#