Re: Reproducing the benchmark numbers

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Reproducing the benchmark numbers

dsetrakyan
Cross-copying to the dev@ list.

Yakov, can you please explain the environment to the Hazelcast engineer?

I do find it a bit strange that their CEO, Mr. Luck, first accused us of
“faking” [1][2] the result a week ago, and only now is making an attempt to
actually reproduce the results, but nevertheless, we should help.

[1]
http://blog.hazelcast.com/fake-benchmark-corrected-benchmark-shows-hazelcast-3-6-is-much-faster-than-gridgainapache-ignite-1-5/
[2]
https://dzone.com/articles/gridgain-confirms-apache-ignite-performance-2x-fas

D.

On Thu, Feb 25, 2016 at 4:41 AM, Peter Veentjer <[hidden email]>
wrote:

> We (Hazelcast) would like to reproduce the benchmark numbers:
>
> https://ignite.apache.org/benchmarks/ignite-vs-hazelcast.html
>
> We have set up 3 separate environments, to reduce the change of
> configuration errors, according to the exact specifications on the website,
> but have failed to reproduced the numbers.
>
> There are a few points of confusion which might have caused the
> differences in the measurements:
>
> - which AMI was used? We are using the ami-fce3c696 Ubuntu 14.04 LTS (HVM)
> from us-east-1
>
> - can we get the exact properties being used for Ignite and Hazelcast. We
> know the properties are in the github repo; but these can change and have
> changed.
>
Reply | Threaded
Open this post in threaded view
|

Re: Reproducing the benchmark numbers

yzhdanov
Hello!

Here is the info:

* AMI used -  ami-e78b6687 which should be just Ubuntu 14.04 + git + java +
dstat
* Instance c4.2xlarge: RAM 15GiB, vCPU 8
* VM information: Java(TM) SE Runtime Environment 1.7.0_80-b15 Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 24.80-b11
* Zone: us-west-2a
* Config & properties:
  - Ignite config:
https://github.com/apacheignite/yardstick-ignite/blob/master/config/benchmark.properties
  - HZ config:
https://github.com/gridgain/yardstick-hazelcast/blob/master/config/benchmark.properties
* How to run benchmarks:
  - for yardstick-ignite:
https://github.com/apacheignite/yardstick-ignite/blob/master/README.md
  - for yardstick-hazelcast:
https://github.com/gridgain/yardstick-hazelcast/blob/master/README.md

Please let us know if you have more questions.

--Yakov