Schema in CacheConfig is not updated after DDL commands(Add/drop column, Create/drop index)

[Ray]

When user performs column and index modification operation in SQL(ex create
index, drop index, add column, drop column),  QueryEntity in
CacheConfiguration for the modified cache is not updated.

Here's my analysis

QueryEntity in QuerySchema is a local copy of the original QueryEntity, so
the original QueryEntity is not updated when modification happens.

I have created a ticket for this issue
https://issues.apache.org/jira/browse/IGNITE-10314

But as Vlad said in the comments "public configuration is immutable, it
represents initial cache parameters. So it is expected that configuration
will not be updated after DDL commands. Real changes are accumulated in
separate query entity which is hidden from user and used internally"

But I think it's only reasonable to return the newest QueryEntity to user.

For example, a user adds a column to a table then he reads data using Spark
data frame API which currently relies on QueryEntity to construct data frame
schema, so user will get wrong schema.

What do you guys think?






--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Vladimir Ozerov]

Hi,

In this case Spark integration should be fixed. as we never stated that DDL
updates will be reflected in IgniteCache.getConfiguration().


On Mon, Nov 19, 2018 at 11:58 AM Ray <[hidden email]> wrote:

> When user performs column and index modification operation in SQL(ex create
> index, drop index, add column, drop column),  QueryEntity in
> CacheConfiguration for the modified cache is not updated.
>
> Here's my analysis
>
> QueryEntity in QuerySchema is a local copy of the original QueryEntity, so
> the original QueryEntity is not updated when modification happens.
>
> I have created a ticket for this issue
> https://issues.apache.org/jira/browse/IGNITE-10314
>
> But as Vlad said in the comments "public configuration is immutable, it
> represents initial cache parameters. So it is expected that configuration
> will not be updated after DDL commands. Real changes are accumulated in
> separate query entity which is hidden from user and used internally"
>
> But I think it's only reasonable to return the newest QueryEntity to user.
>
> For example, a user adds a column to a table then he reads data using Spark
> data frame API which currently relies on QueryEntity to construct data
> frame
> schema, so user will get wrong schema.
>
> What do you guys think?
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[Valentin Kulichenko]

Vladimir,

What is the best way to get current schema information (list of tables,
columns, etc.)?

-Val

On Mon, Nov 19, 2018 at 1:21 AM Vladimir Ozerov <[hidden email]>
wrote:

> Hi,
>
> In this case Spark integration should be fixed. as we never stated that DDL
> updates will be reflected in IgniteCache.getConfiguration().
>
>
> On Mon, Nov 19, 2018 at 11:58 AM Ray <[hidden email]> wrote:
>
> > When user performs column and index modification operation in SQL(ex
> create
> > index, drop index, add column, drop column),  QueryEntity in
> > CacheConfiguration for the modified cache is not updated.
> >
> > Here's my analysis
> >
> > QueryEntity in QuerySchema is a local copy of the original QueryEntity,
> so
> > the original QueryEntity is not updated when modification happens.
> >
> > I have created a ticket for this issue
> > https://issues.apache.org/jira/browse/IGNITE-10314
> >
> > But as Vlad said in the comments "public configuration is immutable, it
> > represents initial cache parameters. So it is expected that configuration
> > will not be updated after DDL commands. Real changes are accumulated in
> > separate query entity which is hidden from user and used internally"
> >
> > But I think it's only reasonable to return the newest QueryEntity to
> user.
> >
> > For example, a user adds a column to a table then he reads data using
> Spark
> > data frame API which currently relies on QueryEntity to construct data
> > frame
> > schema, so user will get wrong schema.
> >
> > What do you guys think?
> >
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

[Vladimir Ozerov]

There is no public API currently. This information can be extracted using
JDBC metadata requests.

On Mon, Nov 19, 2018 at 6:50 PM Valentin Kulichenko <
[hidden email]> wrote:

> Vladimir,
>
> What is the best way to get current schema information (list of tables,
> columns, etc.)?
>
> -Val
>
> On Mon, Nov 19, 2018 at 1:21 AM Vladimir Ozerov <[hidden email]>
> wrote:
>
> > Hi,
> >
> > In this case Spark integration should be fixed. as we never stated that
> DDL
> > updates will be reflected in IgniteCache.getConfiguration().
> >
> >
> > On Mon, Nov 19, 2018 at 11:58 AM Ray <[hidden email]> wrote:
> >
> > > When user performs column and index modification operation in SQL(ex
> > create
> > > index, drop index, add column, drop column),  QueryEntity in
> > > CacheConfiguration for the modified cache is not updated.
> > >
> > > Here's my analysis
> > >
> > > QueryEntity in QuerySchema is a local copy of the original QueryEntity,
> > so
> > > the original QueryEntity is not updated when modification happens.
> > >
> > > I have created a ticket for this issue
> > > https://issues.apache.org/jira/browse/IGNITE-10314
> > >
> > > But as Vlad said in the comments "public configuration is immutable, it
> > > represents initial cache parameters. So it is expected that
> configuration
> > > will not be updated after DDL commands. Real changes are accumulated in
> > > separate query entity which is hidden from user and used internally"
> > >
> > > But I think it's only reasonable to return the newest QueryEntity to
> > user.
> > >
> > > For example, a user adds a column to a table then he reads data using
> > Spark
> > > data frame API which currently relies on QueryEntity to construct data
> > > frame
> > > schema, so user will get wrong schema.
> > >
> > > What do you guys think?
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>

[Nikolay Izhikov-2]

Vladimir.

Since we are talking about internal component(Spark Data Frame integration)
can we use same technique as JdbcRequestHandler to obtain columns metadata?
Seems, code above will fetch up to date columns info:

org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler#getColumnsMeta
```

            for (String cacheName : ctx.cache().publicCacheNames()) {
                for (GridQueryTypeDescriptor table :
ctx.query().types(cacheName)) {
                    if (!matches(table.schemaName(), req.schemaName()))
                        continue;

                    if (!matches(table.tableName(), req.tableName()))
                        continue;
        //Process column here

```

пн, 19 нояб. 2018 г. в 18:54, Vladimir Ozerov <[hidden email]>:

> There is no public API currently. This information can be extracted using
> JDBC metadata requests.
>
> On Mon, Nov 19, 2018 at 6:50 PM Valentin Kulichenko <
> [hidden email]> wrote:
>
> > Vladimir,
> >
> > What is the best way to get current schema information (list of tables,
> > columns, etc.)?
> >
> > -Val
> >
> > On Mon, Nov 19, 2018 at 1:21 AM Vladimir Ozerov <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > In this case Spark integration should be fixed. as we never stated that
> > DDL
> > > updates will be reflected in IgniteCache.getConfiguration().
> > >
> > >
> > > On Mon, Nov 19, 2018 at 11:58 AM Ray <[hidden email]> wrote:
> > >
> > > > When user performs column and index modification operation in SQL(ex
> > > create
> > > > index, drop index, add column, drop column),  QueryEntity in
> > > > CacheConfiguration for the modified cache is not updated.
> > > >
> > > > Here's my analysis
> > > >
> > > > QueryEntity in QuerySchema is a local copy of the original
> QueryEntity,
> > > so
> > > > the original QueryEntity is not updated when modification happens.
> > > >
> > > > I have created a ticket for this issue
> > > > https://issues.apache.org/jira/browse/IGNITE-10314
> > > >
> > > > But as Vlad said in the comments "public configuration is immutable,
> it
> > > > represents initial cache parameters. So it is expected that
> > configuration
> > > > will not be updated after DDL commands. Real changes are accumulated
> in
> > > > separate query entity which is hidden from user and used internally"
> > > >
> > > > But I think it's only reasonable to return the newest QueryEntity to
> > > user.
> > > >
> > > > For example, a user adds a column to a table then he reads data using
> > > Spark
> > > > data frame API which currently relies on QueryEntity to construct
> data
> > > > frame
> > > > schema, so user will get wrong schema.
> > > >
> > > > What do you guys think?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > > >
> > >
> >
>

[Vladimir Ozerov]

Hi Nikolay,

I think yes.

On Mon, Dec 10, 2018 at 9:10 PM Nikolay Izhikov <[hidden email]> wrote:

> Vladimir.
>
> Since we are talking about internal component(Spark Data Frame integration)
> can we use same technique as JdbcRequestHandler to obtain columns metadata?
> Seems, code above will fetch up to date columns info:
>
>
> org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler#getColumnsMeta
> ```
>
>             for (String cacheName : ctx.cache().publicCacheNames()) {
>                 for (GridQueryTypeDescriptor table :
> ctx.query().types(cacheName)) {
>                     if (!matches(table.schemaName(), req.schemaName()))
>                         continue;
>
>                     if (!matches(table.tableName(), req.tableName()))
>                         continue;
>         //Process column here
>
> ```
>
> пн, 19 нояб. 2018 г. в 18:54, Vladimir Ozerov <[hidden email]>:
>
> > There is no public API currently. This information can be extracted using
> > JDBC metadata requests.
> >
> > On Mon, Nov 19, 2018 at 6:50 PM Valentin Kulichenko <
> > [hidden email]> wrote:
> >
> > > Vladimir,
> > >
> > > What is the best way to get current schema information (list of tables,
> > > columns, etc.)?
> > >
> > > -Val
> > >
> > > On Mon, Nov 19, 2018 at 1:21 AM Vladimir Ozerov <[hidden email]>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > In this case Spark integration should be fixed. as we never stated
> that
> > > DDL
> > > > updates will be reflected in IgniteCache.getConfiguration().
> > > >
> > > >
> > > > On Mon, Nov 19, 2018 at 11:58 AM Ray <[hidden email]> wrote:
> > > >
> > > > > When user performs column and index modification operation in
> SQL(ex
> > > > create
> > > > > index, drop index, add column, drop column),  QueryEntity in
> > > > > CacheConfiguration for the modified cache is not updated.
> > > > >
> > > > > Here's my analysis
> > > > >
> > > > > QueryEntity in QuerySchema is a local copy of the original
> > QueryEntity,
> > > > so
> > > > > the original QueryEntity is not updated when modification happens.
> > > > >
> > > > > I have created a ticket for this issue
> > > > > https://issues.apache.org/jira/browse/IGNITE-10314
> > > > >
> > > > > But as Vlad said in the comments "public configuration is
> immutable,
> > it
> > > > > represents initial cache parameters. So it is expected that
> > > configuration
> > > > > will not be updated after DDL commands. Real changes are
> accumulated
> > in
> > > > > separate query entity which is hidden from user and used
> internally"
> > > > >
> > > > > But I think it's only reasonable to return the newest QueryEntity
> to
> > > > user.
> > > > >
> > > > > For example, a user adds a column to a table then he reads data
> using
> > > > Spark
> > > > > data frame API which currently relies on QueryEntity to construct
> > data
> > > > > frame
> > > > > schema, so user will get wrong schema.
> > > > >
> > > > > What do you guys think?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > > > >
> > > >
> > >
> >
>