Neptune plugin#

Tags: Integration, Data, Metrics, Intermediate

Neptune is an experiment tracker for large-scale model training. It allows AI researchers to monitor their model training in real time, visualize and compare experiments, and collaborate on them with a team. This plugin enables seamless use of Neptune within Flyte by configuring links between the two platforms. You can find more information about how to use Neptune in their documentation.

Installation#

To install the Flyte Neptune plugin, run the following command:

pip install flytekitplugins-neptune

Example usage#

For a usage example, see the Neptune example.

Local testing#

To run Neptune example locally:

  1. Create an account on Neptune.

  2. Create a project on Neptune.

  3. In the example, set NEPTUNE_PROJECT to your project name.

  4. Add a secret using Flyte’s Secrets manager with key="neptune-api-token" and group="neptune-api-group"

  5. If you want to see the dynamic log links in the UI, then add the configuration in the next section.

Flyte deployment configuration#

To enable dynamic log links, add the plugin to Flyte’s configuration file:

plugins:
  logs:
    dynamic-log-links:
      - neptune-run-id:
          displayName: Neptune
          templateUris: "{{ .taskConfig.host }}/{{ .taskConfig.project }}?query=(%60flyte%2Fexecution_id%60%3Astring%20%3D%20%22{{ .executionName }}-{{ .nodeId }}-{{ .taskRetryAttempt }}%22)&lbViewUnpacked=true"