flytekitplugins.spark.Spark#

class flytekitplugins.spark.Spark(spark_conf=None, hadoop_conf=None)[source]#

Use this to configure a SparkContext for a your task. Task’s marked with this will automatically execute natively onto K8s as a distributed execution of spark

Parameters
  • spark_conf (Optional[Dict[str, str]]) – Dictionary of spark config. The variables should match what spark expects

  • hadoop_conf (Optional[Dict[str, str]]) – Dictionary of hadoop conf. The variables should match a typical hadoop configuration for spark

Return type

None

Methods

Attributes

hadoop_conf: Optional[Dict[str, str]] = None#
spark_conf: Optional[Dict[str, str]] = None#