Configuring Logging Links in UI

Oftentimes to debug your workflows in production, you want to access logs from your tasks as they run. These logs are different from the core Flyte platform logs, are specific to execution, and may vary from plugin to plugin; for example, Spark may have driver and executor logs.

Every organization potentially uses different log aggregators, making it hard to create a one-size-fits-all solution. Some examples of the log aggregators include cloud-hosted solutions like AWS CloudWatch, GCP Stackdriver, Splunk, Datadog, etc.

Flyte does not have an opinion here and provides a simplified interface to configure your log provider. Flyte-sandbox ships with the Kubernetes dashboard to visualize the logs. This may not be safe for production; hence we recommend users explore other log aggregators.

How Do I configure?

To configure your log provider, the provider needs to support URL links that are shareable and can be templatized. The templating engine has access to these parameters.

The parameters can be used to generate a unique URL to the logs using a templated URI that pertain to a specific task. The templated URI has access to the following parameters:

Parameters to generate a templated URI

Parameter	Description
{{ .podName }}	Gets the pod name as it shows in k8s dashboard
{{ .namespace }}	K8s namespace where the pod runs
{{ .containerName }}	The container name that generated the log
{{ .containerId }}	The container id docker/crio generated at run time
{{ .logName }}	A deployment specific name where to expect the logs to be
{{ .hostname }}	The hostname where the pod is running and logs reside
{{ .podUnixStartTime }}	The pod creation time (in unix seconds, not millis)
{{ .podUnixFinishTime }}	Don’t have a good mechanism for this yet, but approximating with time.Now for now

The parameterization engine uses Golangs native templating format and hence uses {{ }}. An example configuration can be seen as follows:

task_logs:
  plugins:
    logs:
      displayName: <name-to-show>
      templateUris:
        - "https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/flyte-production/kubernetes;stream=var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log"
        - "https://some-other-source/home?region=us-east-1#logEventViewer:group=/flyte-production/kubernetes;stream=var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log"
      messageFormat: "json" # "unknown" | "csv" | "json"

This code snippet will output two logs per task that use the log plugin. However, not all task types use the log plugin; for example, the Sagemaker plugin uses the log output provided by Sagemaker, and the Snowflake plugin will use a link to the snowflake console.