loadcache very very slow

[wt]

CONTENTS DELETED

The author has deleted this message.

[Alexey Kuznetsov]

Hi,

As I see from code of CacheAbstractJdbcStore
it will try to use range query if possible.

see: CacheAbstractJdbcStore#loadCacheRange

As you are using SQL Server make sure that SQLServerDialect was set to
your  store, please provide config generated by Web Console.

Or just set a break point and trace down to
CacheAbstractJdbcStore#loadCacheRange and see what it is doing actually.

Or attach a reproducer here (generate project + sample SQL dump of your
data, do not forget to replace sensitive data by random stuff).

On Mon, Sep 17, 2018 at 3:56 PM wt <[hidden email]> wrote:

> Hi
>
> I have a project that is generated by the web console (2.5) and it is
> loading an integrated db (SQL Server) table that has 4.5 million rows. When
> running the load cache it is loading 1-3 records per second which is
> unacceptable. Looking at the documentation i see examples of the overridden
> load cache mechanism which makes me think that the default loading
> mechanism
> in this project is not optimal for bulk loads. If i run a query on the jdbc
> db driver it is returning 100k of rows per second.
>
> Looking for advice here as i want to maintain the strategy of not needing
> to
> write load code but obviously want something that wont take forever to
> load.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

--
Alexey Kuznetsov

[wt]

hi Alexey

When i debug it is like going down a maize. what i can see is that it enters

class  = IgniteCacheProxyImpl

void = loadCache
   which calls globalLoadCache
   which calls globalLoadCacheAsync

then it goes all into all sorts of classes back and forth and fails because
debugging breaks the connection timeout and the client is disconnected.

here is the example project  Cluster-project(1).zip
<http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/Cluster-project%281%29.zip>  

here is sample data data.csv
<http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/data.csv>  





--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[Ilya Kasnacheev]

Hello!

Please avoid writing questions about running Apache Ignite to developers
list. Developers list is suited for people who write Apache Ignite's code
or plan its features.

With regards to your project. I don't have MS SQL Server, and I especially
don't have your schema used to create tables.
I imagine that you might be missing index on key field in your table (the
one that Ignite will use according to your config) and thus it has to
resort to full scan while loading tables.

Regards,
--
Ilya Kasnacheev


пн, 17 сент. 2018 г. в 12:43, wt <[hidden email]>:

> hi Alexey
>
> When i debug it is like going down a maize. what i can see is that it
> enters
>
> class  = IgniteCacheProxyImpl
>
> void = loadCache
>    which calls globalLoadCache
>    which calls globalLoadCacheAsync
>
> then it goes all into all sorts of classes back and forth and fails because
> debugging breaks the connection timeout and the client is disconnected.
>
> here is the example project  Cluster-project(1).zip
> <
> http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/Cluster-project%281%29.zip>
>
>
> here is sample data data.csv
> <http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/data.csv>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[Alexey Kuznetsov]

Indeed.

You should ask such questions on user list.
Just start a thread there + attach a DDL script of your schema in SQL
Server.

--
Alexey Kuznetsov

[wt]

hi

The reason i am asking here and not on the users forum is to get down to the
code details of how loadcache works and specifically what the developers
recommendation is on using it for bulk load operations. In reading up on the
subject, i discovered that it is suggested to override the method and use a
different implementation to the default. Asking on the users forum is only
going to go into details like the env and server and drivers etc. I can tell
you now that i can query the same jdbc driver in beaver on the same database
and get back throughput i would have expected to see with the generated
project. Unfortunately the generated project from the web console is only
useful for basic configs and the loading aspect to it should ultimately
either be removed or updated to support bulk inserts by default. I have lost
confidence in using the web console and generated project due to a series of
issues including this.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/