Implementing the agent metadata service

About the agent metadata service

Before FlytePropeller sends a request to the agent server, it needs to know four things:

  • The name of the agent

  • Which task category the agent supports

  • The version of the task category

  • Whether the agent executes tasks synchronously or asynchronously

After FlytePropeller obtains this metadata, it can send a request to the agent deployment using the correct gRPC method.

Note

  • An agent can support multiple task categories.

  • We will use the combination of [task category][version] to identify the specific agent’s deployment and know whether the task is synchronous or asynchronous in FlytePropeller.

  • The task category is task_type in flytekit.

Using the BigQuery Agent as an example:

  • The agent’s name is BigQuery Agent.

  • The agent supports bigquery_query_job_task.

  • The agent’s version is 0.

  • By default, the agent executes tasks asynchronously.

Implement the agent metadata service

To implement the agent metadata service, you must do two things:

  1. Implement the agent metadata service.

  2. Add the agent metadata service to the agent server.

You can refer to base_agent.py, agent_service.py, and serve.py to see how the agent metadata service is implemented in flytekit’s agent server.

Those gRPC methods are generated by flyteidl and you can import them from here.

Note

You can search the keyword metadata to find implementations in those files.