IEP-24: SQL Partition Pruning

[Vladimir Ozerov]

Igniters,

I prepared and IEP-24 [1] for so-called "partition pruning" optimization
for our SQL engine, which will allow us to determine target nodes
containing query data prior to query execution. We already use this
optimization for very simple scenarios - only one expression, no JOINs.

The goals of this IEP:
1) Extract partitions from complex expressions
2) Support common JOIN scenarios
3) Allow calculation of target partitions on thin client to allow more
efficient request routing
4) Introduce monitoring capabilities to let user know whether optimization
is applicable to specific query or not

IEP covers several complex architecture questions, which will be finalized
during actual implementation:
1) Rules for partition extraction from complex AND and OR expressions, as
well as from "IN (...)", "BETWEEN ... AND ...", and range expressions
2) Rules for partition extraction from JOINs
3) Several subquery rewrite rules which will allow to apply optimization to
certain subqueries.

Also this optimization will introduce some basic building blocks
("co-location tree") for further improvements of our distributed joins.

Will appreciate your review and comments.

Vladimir.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-24%3A+SQL+Partition+Pruning

[dmagda]

Vladimir, thanks for the extensive description. Everything is clear for me
from a user perspective.

Hope Sergi, Taras, and other SQL experts would share their feedback.

--
Denis

On Mon, Nov 26, 2018 at 6:33 AM Vladimir Ozerov <[hidden email]>
wrote:

> Igniters,
>
> I prepared and IEP-24 [1] for so-called "partition pruning" optimization
> for our SQL engine, which will allow us to determine target nodes
> containing query data prior to query execution. We already use this
> optimization for very simple scenarios - only one expression, no JOINs.
>
> The goals of this IEP:
> 1) Extract partitions from complex expressions
> 2) Support common JOIN scenarios
> 3) Allow calculation of target partitions on thin client to allow more
> efficient request routing
> 4) Introduce monitoring capabilities to let user know whether optimization
> is applicable to specific query or not
>
> IEP covers several complex architecture questions, which will be finalized
> during actual implementation:
> 1) Rules for partition extraction from complex AND and OR expressions, as
> well as from "IN (...)", "BETWEEN ... AND ...", and range expressions
> 2) Rules for partition extraction from JOINs
> 3) Several subquery rewrite rules which will allow to apply optimization to
> certain subqueries.
>
> Also this optimization will introduce some basic building blocks
> ("co-location tree") for further improvements of our distributed joins.
>
> Will appreciate your review and comments.
>
> Vladimir.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-24%3A+SQL+Partition+Pruning
>