Re: Different behavior when saving date from Dataframe API and RDD API

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: Different behavior when saving date from Dataframe API and RDD API

Nikolay Izhikov-2
Hell, ray.

> Please advise is this behavior expected?

I think this behavior is expected.
Because it's more efficient to query specific affinity key value.
Anyway, I'm not an expert in SQL engine, so I send your question to the dev-list.

Igniters,

I think this user question is related to the SQL engine, not the Data Frame integration.
So can some of SQL engine experts take a look.

In the first case SQL table will be created as follows `CREATE TABLE table_name(....) PRIMARY KEY a,b,c,d WITH "template=partitioned,affinitykey=a"`

В Пт, 23/03/2018 в 00:48 -0700, Ray пишет:

> I was trying out one of Ignite 2.4's new features - saving data from
> dataframe.
> But I found some inconsistency between the Dataframe API and RDD API.
>
> This is the code from saving dataframe to Ignite.
> DF.write
> .format(FORMAT_IGNITE)
> .mode(SaveMode.Append)
> .option(OPTION_CONFIG_FILE, CONFIG)
> .option(OPTION_TABLE, "table_name")
> .option(OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS, "a,b,c,d")
> .option(OPTION_CREATE_TABLE_PARAMETERS,
> "template=partitioned,affinitykey=a")
> .option(OPTION_STREAMER_ALLOW_OVERWRITE, "true")
> .save()
> After data finished saving, I ran this command to create an index on field
> a.
> CREATE INDEX IF NOT EXISTS idx ON table_name (a);
> Then I run this query to see if the index is working.
>
> explain select a from table_name where a = '303';
> PLAN  SELECT
>     __Z0.a AS __C0_0
> FROM PUBLIC.table_name __Z0
>     /* PUBLIC.AFFINITY_KEY: a = '303' */
> WHERE __Z0.a = '303'
>
> But when I try query the data I insert in the old RDD way, the result is
> explain select a from table_name where a = '303';
> PLAN  SELECT
>     __Z0.a AS __C0_0
> FROM PUBLIC.table_name __Z0
>     /* PUBLIC.table_name_IDX: a = '303' */WHERE __Z0.a = '303'
>
> The result shows with affinity key, the index created is not effective.
> I tried creating index on other non affinity key field, the index is
> working.
> Please advise is this behavior expected?
>
> THanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

signature.asc (465 bytes) Download Attachment