flytekitplugins.spark.new_spark_session

flytekitplugins.spark.new_spark_session(name, conf=None)

Optionally creates a new spark session and returns it. In cluster mode (running in hosted flyte, this will disregard the spark conf passed in)

This method is safe to be used from any other method. That is one reason why, we have duplicated this code fragment with the pre-execute. For example in the notebook scenario we might want to call it from a separate kernel

Parameters:

• name (str)

• conf (Dict[str, str] | None)