flytekit.core.data_persistence.FileAccessProvider#
- class flytekit.core.data_persistence.FileAccessProvider(local_sandbox_dir, raw_output_prefix, data_config=None)[source]#
This is the class that is available through the FlyteContext and can be used for persisting data to the remote durable store.
- Parameters
local_sandbox_dir (Union[str, os.PathLike]) – A local temporary working directory, that should be used to store data
raw_output_prefix (str) –
data_config (Optional[flytekit.configuration.DataConfig]) –
Methods
- download_directory(remote_path, local_path)[source]#
Downloads directory from given remote to local path
- get_random_local_path(file_path_or_file_name=None)[source]#
Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name
- get_random_remote_path(file_path_or_file_name=None)[source]#
Constructs a randomized path on the configured raw_output_prefix (persistence layer). the random bit is a UUID and allows for disambiguating paths within the same directory.
Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name
- static is_remote(path)[source]#
Deprecated. Let’s find a replacement
- Parameters
path (Union[str, os.PathLike]) –
- Return type
- put_data(local_path, remote_path, is_multipart=False, **kwargs)[source]#
The implication here is that we’re always going to put data to the remote location, so we .remote to ensure we don’t use the true local proxy if the remote path is a file://
- Parameters
local_path (Union[str, os.PathLike]) –
remote_path (str) –
is_multipart (bool) –
- static strip_file_header(path, trim_trailing_sep=False)[source]#
Drops file:// if it exists from the file
- upload_directory(local_path, remote_path)[source]#
- Parameters
local_path (Text) –
remote_path (Text) –
Attributes
- data_config#
- local_access#
- local_sandbox_dir#
This is a context based temp dir.
- raw_output_prefix#