in-memory compression

[999.computing]

Hi developers,

We have released TBLignite compression, an Ignite plugin that provides in-memory compression: http://tblcore.com/download/.
Compression rates are similar to those of SQL Server 2016 columnar compression (10-20x) and our testing show significant performance improvements for datasets that are larger than the available amount of memory.

Currently we are at version 0.1, but we are working on a thoroughly tested 1.0 release so everyone can scratch their compression itch. A couple of questions regarding this:
1) Are there any Ignite regression tests that you can recommend for 3rd party software? Basically we are looking for a way to test all the possible page formats that we need to support.
2) Once we release version 1.0, we would like TBLignite to be added to the Ignite "3rd party binary" page. Who decides what gets on this page?
3) And are there any acceptance criteria or tests that need to be passed?

We would like to hear from you.
Pascal Schuchhard
TBLcore

[V.Pyatkov]

Hi,

I looked you report in site and have some questions.

What means the thesis?
*Your dataset is fit for columnar compression, ea. repeating values and/or
a timeseries-like*
*dataset.*
As I undersent, you are overwrite file io factory (TBLigniteFileIoFactory).
But in this lavel available only bytes, not data entries.

What caused the performance drop to 31% in your test?

On Mon, Apr 1, 2019 at 8:54 AM <[hidden email]> wrote:

> Hi developers,
>
> We have released TBLignite compression, an Ignite plugin that provides
> in-memory compression: http://tblcore.com/download/.
> Compression rates are similar to those of SQL Server 2016 columnar
> compression (10-20x) and our testing show significant performance
> improvements for datasets that are larger than the available amount of
> memory.
>
> Currently we are at version 0.1, but we are working on a thoroughly tested
> 1.0 release so everyone can scratch their compression itch. A couple of
> questions regarding this:
> 1) Are there any Ignite regression tests that you can recommend for 3rd
> party software? Basically we are looking for a way to test all the possible
> page formats that we need to support.
> 2) Once we release version 1.0, we would like TBLignite to be added to the
> Ignite "3rd party binary" page. Who decides what gets on this page?
> 3) And are there any acceptance criteria or tests that need to be passed?
>
> We would like to hear from you.
> Pascal Schuchhard
> TBLcore
>
>

--
Vladislav Pyatkov

[Ilya Kasnacheev]

Hello!

You could try running regular tests, such as PDS test suites, on Apache
Ignite TeamCity with this plugin enabled.

See if there are any new failures.

Regards,
--
Ilya Kasnacheev


пн, 1 апр. 2019 г. в 08:54, <[hidden email]>:

> Hi developers,
>
> We have released TBLignite compression, an Ignite plugin that provides
> in-memory compression: http://tblcore.com/download/.
> Compression rates are similar to those of SQL Server 2016 columnar
> compression (10-20x) and our testing show significant performance
> improvements for datasets that are larger than the available amount of
> memory.
>
> Currently we are at version 0.1, but we are working on a thoroughly tested
> 1.0 release so everyone can scratch their compression itch. A couple of
> questions regarding this:
> 1) Are there any Ignite regression tests that you can recommend for 3rd
> party software? Basically we are looking for a way to test all the possible
> page formats that we need to support.
> 2) Once we release version 1.0, we would like TBLignite to be added to the
> Ignite "3rd party binary" page. Who decides what gets on this page?
> 3) And are there any acceptance criteria or tests that need to be passed?
>
> We would like to hear from you.
> Pascal Schuchhard
> TBLcore
>
>

[999.computing]

Hi Vladislav Pyatkov,

> What means the thesis? Your dataset is fit for columnar compression, ea. repeating values and/or a timeseries-like
Columnar compression relies on repeating values. If every row has a different value for a particular column, columnar compression does not give an advantage for that column.

> As I undersent, you are overwrite file io factory (TBLigniteFileIoFactory). But in this lavel available only bytes, not data entries.
We deserialize the Ignite pages ourselves to get the data entries.

> What caused the performance drop to 31% in your test?
That testcase has a 4 cpu node with 4 clients continuously running queries. If a node is cpu starved, compression can not give a performance advantage and it is better to read directly from disk instead.

Pascal


-----Original Message-----
From: Vladislav Pyatkov <[hidden email]>
Sent: Monday, 1 April 2019 10:49
To: [hidden email]
Subject: Re: in-memory compression

Hi,

I looked you report in site and have some questions.

What means the thesis?
*Your dataset is fit for columnar compression, ea. repeating values and/or a timeseries-like*
*dataset.*
As I undersent, you are overwrite file io factory (TBLigniteFileIoFactory).
But in this lavel available only bytes, not data entries.

What caused the performance drop to 31% in your test?

On Mon, Apr 1, 2019 at 8:54 AM <[hidden email]> wrote:

> Hi developers,
>
> We have released TBLignite compression, an Ignite plugin that provides
> in-memory compression: http://tblcore.com/download/.
> Compression rates are similar to those of SQL Server 2016 columnar
> compression (10-20x) and our testing show significant performance
> improvements for datasets that are larger than the available amount of
> memory.
>
> Currently we are at version 0.1, but we are working on a thoroughly
> tested
> 1.0 release so everyone can scratch their compression itch. A couple
> of questions regarding this:
> 1) Are there any Ignite regression tests that you can recommend for
> 3rd party software? Basically we are looking for a way to test all the
> possible page formats that we need to support.
> 2) Once we release version 1.0, we would like TBLignite to be added to
> the Ignite "3rd party binary" page. Who decides what gets on this page?
> 3) And are there any acceptance criteria or tests that need to be passed?
>
> We would like to hear from you.
> Pascal Schuchhard
> TBLcore
>
>

--
Vladislav Pyatkov

[dmagda]

Hello Pascal,

We'll get you added to the 3rd parties list it's not a big deal. It will be
great if you have and can share any production deployments of the plugin,
just to increase confidence in the module.

However, how is your feature different from the default Ignite-compression
to be released soon?
https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite

-
Denis


On Sun, Mar 31, 2019 at 10:54 PM <[hidden email]> wrote:

> Hi developers,
>
> We have released TBLignite compression, an Ignite plugin that provides
> in-memory compression: http://tblcore.com/download/.
> Compression rates are similar to those of SQL Server 2016 columnar
> compression (10-20x) and our testing show significant performance
> improvements for datasets that are larger than the available amount of
> memory.
>
> Currently we are at version 0.1, but we are working on a thoroughly tested
> 1.0 release so everyone can scratch their compression itch. A couple of
> questions regarding this:
> 1) Are there any Ignite regression tests that you can recommend for 3rd
> party software? Basically we are looking for a way to test all the possible
> page formats that we need to support.
> 2) Once we release version 1.0, we would like TBLignite to be added to the
> Ignite "3rd party binary" page. Who decides what gets on this page?
> 3) And are there any acceptance criteria or tests that need to be passed?
>
> We would like to hear from you.
> Pascal Schuchhard
> TBLcore
>
>