Jupyter Notebook Tasks#
In this example, we will show how to create a flyte task that runs a simple notebook, accepts one input variable, transforms it, and produces one output. This can be generalized to multiple inputs and outputs.
import math import os import pathlib from flytekit import kwtypes, task, workflow from flytekitplugins.papermill import NotebookTask
How to specify inputs and outputs#
After you are satisfied with the notebook, ensure that the first cell only has the input variables for the notebook. Now add the tag
parametersfor the first cell.
Typically at the last cell of the notebook (which does not need to be the last cell), add a tag
outputsfor the intended cell.
In a python file, create a new task at the
modulelevel. An example task is shown below:
nb = NotebookTask( name="simple-nb", notebook_path=os.path.join(pathlib.Path(__file__).parent.absolute(), "nb_simple.ipynb"), render_deck=True, inputs=kwtypes(v=float), outputs=kwtypes(square=float), )
Note the notebook_path. This is the absolute path to the actual notebook.
Note the inputs and outputs. The variable names match the variable names in the jupyter notebook.
You can see the notebook on Flyte deck if
render_deckis set to true.
You can definitely declare other tasks and seamlessly work with notebook tasks. The example below shows how to declare a task that accepts the squared value from the notebook and provides a sqrt:
@task def square_root_task(f: float) -> float: return math.sqrt(f)
Now treat the notebook task as a regular task:
@workflow def nb_to_python_wf(f: float) -> float: out = nb(v=f) return square_root_task(f=out.square)
And execute the task locally as well:
if __name__ == "__main__": print(nb_to_python_wf(f=3.14))
Why Are There 3 Outputs?#
On executing, you should see 3 outputs instead of the expected one, because this task generates 2 implicit outputs.
One of them is the executed notebook (captured) and a rendered (HTML) of the executed notebook. In this case they are called