org.apache.spark.sql

Class SparkSessionExtensions

Object

org.apache.spark.sql.SparkSessionExtensions

public class SparkSessionExtensions
extends Object

:: Experimental :: Holder for injection points to the SparkSession. We make NO guarantee about the stability regarding binary compatibility and source compatibility of methods here.

This current provides the following extension points:

• Analyzer Rules.
• Check Analysis Rules.
• Optimizer Rules.
• Planning Strategies.
• Customized Parser.
• (External) Catalog listeners.
• Columnar Rules.

The extensions can be used by calling withExtensions on the SparkSession.Builder, for example:

   SparkSession.builder()
     .master("...")
     .config("...", true)
     .withExtensions { extensions =>
       extensions.injectResolutionRule { session =>
         ...
       }
       extensions.injectParser { (session, parser) =>
         ...
       }
     }
     .getOrCreate()
 

The extensions can also be used by setting the Spark SQL configuration property spark.sql.extensions. Multiple extensions can be set using a comma-separated list. For example:

   SparkSession.builder()
     .master("...")
     .config("spark.sql.extensions", "org.example.MyExtensions")
     .getOrCreate()

   class MyExtensions extends Function1[SparkSessionExtensions, Unit] {
     override def apply(extensions: SparkSessionExtensions): Unit = {
       extensions.injectResolutionRule { session =>
         ...
       }
       extensions.injectParser { (session, parser) =>
         ...
       }
     }
   }
 

Note that none of the injected builders should assume that the SparkSession is fully initialized and should not touch the session's internals (e.g. the SessionState).

Constructor Summary

Constructors

Constructor and Description
SparkSessionExtensions()

Method Summary

Modifier and Type	Method and Description
void	injectCheckRule(scala.Function1<SparkSession,scala.Function1<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,scala.runtime.BoxedUnit>> builder) Inject an check analysis Rule builder into the SparkSession.
void	injectColumnar(scala.Function1<SparkSession,org.apache.spark.sql.execution.ColumnarRule> builder) Inject a rule that can override the columnar execution of an executor.
void	injectFunction(scala.Tuple3<org.apache.spark.sql.catalyst.FunctionIdentifier,org.apache.spark.sql.catalyst.expressions.ExpressionInfo,scala.Function1<scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression>,org.apache.spark.sql.catalyst.expressions.Expression>> functionDescription) Injects a custom function into the FunctionRegistry at runtime for all sessions.
void	injectOptimizerRule(scala.Function1<SparkSession,org.apache.spark.sql.catalyst.rules.Rule<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>> builder) Inject an optimizer Rule builder into the SparkSession.
void	injectParser(scala.Function2<SparkSession,org.apache.spark.sql.catalyst.parser.ParserInterface,org.apache.spark.sql.catalyst.parser.ParserInterface> builder) Inject a custom parser into the SparkSession.
void	injectPlannerStrategy(scala.Function1<SparkSession,org.apache.spark.sql.execution.SparkStrategy> builder) Inject a planner Strategy builder into the SparkSession.
void	injectPostHocResolutionRule(scala.Function1<SparkSession,org.apache.spark.sql.catalyst.rules.Rule<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>> builder) Inject an analyzer Rule builder into the SparkSession.
void	injectResolutionRule(scala.Function1<SparkSession,org.apache.spark.sql.catalyst.rules.Rule<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>> builder) Inject an analyzer resolution Rule builder into the SparkSession.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkSessionExtensions

public SparkSessionExtensions()

Method Detail

injectCheckRule

public void injectCheckRule(scala.Function1<SparkSession,scala.Function1<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,scala.runtime.BoxedUnit>> builder)

Inject an check analysis Rule builder into the SparkSession. The injected rules will be executed after the analysis phase. A check analysis rule is used to detect problems with a LogicalPlan and should throw an exception when a problem is found.

Parameters:

builder - (undocumented)

injectColumnar

public void injectColumnar(scala.Function1<SparkSession,org.apache.spark.sql.execution.ColumnarRule> builder)

Inject a rule that can override the columnar execution of an executor.

Parameters:

builder - (undocumented)

injectFunction

public void injectFunction(scala.Tuple3<org.apache.spark.sql.catalyst.FunctionIdentifier,org.apache.spark.sql.catalyst.expressions.ExpressionInfo,scala.Function1<scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression>,org.apache.spark.sql.catalyst.expressions.Expression>> functionDescription)

Injects a custom function into the FunctionRegistry at runtime for all sessions.

Parameters:

functionDescription - (undocumented)

injectOptimizerRule

public void injectOptimizerRule(scala.Function1<SparkSession,org.apache.spark.sql.catalyst.rules.Rule<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>> builder)

Inject an optimizer Rule builder into the SparkSession. The injected rules will be executed during the operator optimization batch. An optimizer rule is used to improve the quality of an analyzed logical plan; these rules should never modify the result of the LogicalPlan.

Parameters:

builder - (undocumented)

injectParser

public void injectParser(scala.Function2<SparkSession,org.apache.spark.sql.catalyst.parser.ParserInterface,org.apache.spark.sql.catalyst.parser.ParserInterface> builder)

Inject a custom parser into the SparkSession. Note that the builder is passed a session and an initial parser. The latter allows for a user to create a partial parser and to delegate to the underlying parser for completeness. If a user injects more parsers, then the parsers are stacked on top of each other.

Parameters:

builder - (undocumented)

injectPlannerStrategy

public void injectPlannerStrategy(scala.Function1<SparkSession,org.apache.spark.sql.execution.SparkStrategy> builder)

Inject a planner Strategy builder into the SparkSession. The injected strategy will be used to convert a LogicalPlan into a executable SparkPlan.

Parameters:

builder - (undocumented)

injectPostHocResolutionRule

public void injectPostHocResolutionRule(scala.Function1<SparkSession,org.apache.spark.sql.catalyst.rules.Rule<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>> builder)

Inject an analyzer Rule builder into the SparkSession. These analyzer rules will be executed after resolution.

Parameters:

builder - (undocumented)

injectResolutionRule

public void injectResolutionRule(scala.Function1<SparkSession,org.apache.spark.sql.catalyst.rules.Rule<org.apache.spark.sql.catalyst.plans.logical.LogicalPlan>> builder)

Inject an analyzer resolution Rule builder into the SparkSession. These analyzer rules will be executed as part of the resolution phase of analysis.

Parameters:

builder - (undocumented)