Apache Ignite Developers - Legacy Mail Archive

Apache Ignite RPM packaging (Stage II)

Classic

List

Threaded

38 messages Options

Max Shonichev

Re: Apache Ignite RPM packaging (Stage II)

1. About packaging/package.sh

1.1. I personally dislike the use of bashism, e.g. like reserved keywords,
double '[[', double '((', but that's ok
for targeted distro (RHEL/Centos). The only requirement is please add more
comments in tough places
to improve code readability and maintanbility.

1.2. few code style comments:

>> name=$(cat /etc/*release | grep ^NAME | sed -r 's|.*"(.*)".*|\1|')

add

| head -n 1

or else it will fail in case more than one file with *release mask in /etc

1.3. rework prepEnv

>> if [ ! -z installCmd ]; then

this will fail miserably, replace with

if [ ! -z $installCmd ]; then

1.3.1. however running package.sh under Ubuntu still would fail due to
$installCmd is unpacked to 'apt --no-install-recommends'
and -z 'apt --no-install-recommends' would always return false.

1.3.2. and still, when apt/yum is installed and rpm/rpm-build is not, the
whole prepEnv would fail to ensure all
build-time requirements are installed, because it only checks for apt/yum.

1.3.3. installing requires root privileges. However, no checks for $EUID nor
running apt/yum with 'sudo' present.
Please, add this check, otherwise running package.sh under unprivileged user
with no installed build-time dependencies
will fail with very undescriptive errors.

1.4. getBin

1.4.1. I don't like detecting target apache ignite package version by
parsing embedded changelog. This might fail
when any package text description would contain '*' in the first char of the
string. Either move changelog to separate
Changelog file and %include it in .spec or parse for, e.g. '^Version:
[0-9\.]+'

>> igniteVersion=$(cat rpm/SPECS/apache-ignite.spec | grep '^*' | head -1 |
>> sed -r 's|.*\ -\ (.*)-.*|\1|')

1.4.2.

>> ls ${binName} && binPreparedFlag=true || true

why not use

if [ ! -f $binName ];

1.4.3. 'curl' is used, but not checked as build-time dependency.

1.4.4. we try to find and download 'apache-ignite-fabric-$igniteVersion',
that's ok for <2.4 releases, but in future
we would need to find and download 'apache-ignite-$igniteVersion' too.
Better add this check as a fallback option, e.g.
try to use curl with both urls in correct order.

if curl --output /dev/null --silent --head --fail "$url"; then
echo "URL exists: $url"
else
echo "URL does not exist: $url"
fi

1.4.5. also strange, but may be ok that we download
apache-ignite-fabric-2.4.0

2. about apache-ignite.spec

2.1. fix typo

>> Apache Ignite's optinal libs and integrations

2.2. fix description

>> C++ files necessary for using Apache Ignite

platform files are not necessary for Ignite, only for integration with .NET,
afaik?

2.3. Tough place with 'Obsoletes: apache-ignite < 2.5.0'

I understand the current logic behind splitting apache-ignite to several
.rpm's and setting obsolete for the old
apache-ignite.

However, for future releases, e.g. if/when we handover this task to another
engineer, it is not quite obvious that one
must (or must not?) update version in 5 different places, including
changelog. Please add a few lines about
'How to package next (new) version of Apache Ignite RPMs' to Confluence.

2.4. firewalld / firewall-cmd is optionally used if installed, but not set
in the dependency. It is OK, but may be we
add this as a post-install note, like we do recommendation messages in node
logs.

3. service scripts : we'll should add a test suite for it, overall looks
pretty solid.

--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

vveider

Re: Apache Ignite RPM packaging (Stage II)

> On 29 Mar 2018, at 12:32, Max Shonichev <[hidden email]> wrote:
>
>
>
> 1. About packaging/package.sh
>
> 1.1. I personally dislike the use of bashism, e.g. like reserved keywords,
> double '[[', double '((', but that's ok
> for targeted distro (RHEL/Centos). The only requirement is please add more
> comments in tough places
> to improve code readability and maintanbility.

Will try to get rid of them as much as possible, but SH is really outdated scripting language and most all of the systems now support BASH, even Windows :)

>
>
> 1.2. few code style comments:
>
>>> name=$(cat /etc/*release | grep ^NAME | sed -r 's|.*"(.*)".*|\1|')
>
> add
>
> | head -n 1
>
> or else it will fail in case more than one file with *release mask in /etc

I disagree. Script will not fail, just cat every file matched by mask and then pass everything as one single stdout.

>
>
> 1.3. rework prepEnv
>
>>> if [ ! -z installCmd ]; then
>
> this will fail miserably, replace with
>
> if [ ! -z $installCmd ]; then

Fixed

>
> 1.3.1. however running package.sh under Ubuntu still would fail due to
> $installCmd is unpacked to 'apt --no-install-recommends'
> and -z 'apt --no-install-recommends' would always return false.

The main purpose of checking that variable ${installCmd} is not empty — because it is instantiated empty and will stay empty if OS detection won’t be able to define OS and pre-build software installation and environment preparation will be skipped.

>
> 1.3.2. and still, when apt/yum is installed and rpm/rpm-build is not, the
> whole prepEnv would fail to ensure all
> build-time requirements are installed, because it only checks for apt/yum.

I suppose that apt / yum is always installed by default because it is default package manager for corresponding OSes.
I did not get where is error here.

>
> 1.3.3. installing requires root privileges. However, no checks for $EUID nor
> running apt/yum with 'sudo' present.
> Please, add this check, otherwise running package.sh under unprivileged user
> with no installed build-time dependencies
> will fail with very undescriptive errors.

Check for ${EUID} will mean that whole build package process will run under root, which is undesirable.
Added INFO message that explains necessity for super user privileges and added sudo command to root-required commands.

>
>
> 1.4. getBin
>
> 1.4.1. I don't like detecting target apache ignite package version by
> parsing embedded changelog. This might fail
> when any package text description would contain '*' in the first char of the
> string. Either move changelog to separate
> Changelog file and %include it in .spec or parse for, e.g. '^Version:
> [0-9\.]+'
>
>>> igniteVersion=$(cat rpm/SPECS/apache-ignite.spec | grep '^*' | head -1 |
>>> sed -r 's|.*\ -\ (.*)-.*|\1|’)

Replaced with parsing of main version

>
> 1.4.2.
>
>>> ls ${binName} && binPreparedFlag=true || true
>
> why not use
>
> if [ ! -f $binName ];

Indeed this variant is much better, fixed.

>
> 1.4.3. 'curl' is used, but not checked as build-time dependency.

Added to build dependencies (via prepEnv function)

>
> 1.4.4. we try to find and download 'apache-ignite-fabric-$igniteVersion',
> that's ok for <2.4 releases, but in future
> we would need to find and download 'apache-ignite-$igniteVersion' too.
> Better add this check as a fallback option, e.g.
> try to use curl with both urls in correct order.
>
> if curl --output /dev/null --silent --head --fail "$url"; then
> echo "URL exists: $url"
> else
> echo "URL does not exist: $url"
> fi

1. Packages did not exist before 2.4.0.
2. Build script does not provide opportunity to build packages of different from .spec versions and is considered to be part of packaging build process (specifications and packaging script are modified together) — a shortcut, thus relying on current (same commit) state for building.

>
> 1.4.5. also strange, but may be ok that we download
> apache-ignite-fabric-2.4.0

Fabric removal task is not yet merged to master.
Will be supported in its reimplementation

>
>
> 2. about apache-ignite.spec
>
> 2.1. fix typo
>
>>> Apache Ignite's optinal libs and integrations

Fixed

>
> 2.2. fix description
>
>>> C++ files necessary for using Apache Ignite
>
> platform files are not necessary for Ignite, only for integration with .NET,
> afaik?

Introduced alternative description

>
> 2.3. Tough place with 'Obsoletes: apache-ignite < 2.5.0'
>
> I understand the current logic behind splitting apache-ignite to several
> .rpm's and setting obsolete for the old
> apache-ignite.
>
> However, for future releases, e.g. if/when we handover this task to another
> engineer, it is not quite obvious that one
> must (or must not?) update version in 5 different places, including
> changelog. Please add a few lines about
> 'How to package next (new) version of Apache Ignite RPMs' to Confluence.

Everything that relies on package version is mapped to %{version} variable and defined only once.
Other versions in dependencies directives will stay the same in future specification updates.
Task for description of RPM specification update process, I guess, is out of scope of current issue. However I agree with necessity of such documentation in future and will create corresponding task in the nearest time.

>
> 2.4. firewalld / firewall-cmd is optionally used if installed, but not set
> in the dependency. It is OK, but may be we
> add this as a post-install note, like we do recommendation messages in node
> logs.

Firewall-CMD rules from package is an override for current installation rather then necessity in some firewall implementations required for service usage.

>
>
> 3. service scripts : we'll should add a test suite for it, overall looks
> pretty solid.

Agree. Will create corresponding task in the nearest time

>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Ilya Kasnacheev

Re: Apache Ignite RPM packaging (Stage II)

In reply to this post by vveider

Hello!

Let me share my idea of how this shoud work. Splitting package into
sub-packages should be dependency-driven.

It means that all Ignite modules without dependencies or with small
dependencies (such as ignite-log4j) should be included in ignite-core. It
doesn't make sense to make a zillion RPM packages.

Critical things like ignite-spring and ignite-indexing should be in
ignite-core of course, even if they have dependencies. Ignite-core should
be fully self-sufficient and feature-complete.

However, e.g. .net API should probably be in a separate package, because it
should depend on mono | net-core. We may also have ignite-devel package
which should include all modules which only make sense for developers who
write code. Such as hibernate integration.

I'm not sure about MR modules. The main question should be, does it have
dependencies? Can it run stand-alone without writing code?

Hope this helps,

--
Ilya Kasnacheev

2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:

> Hi, Igniters!
>
>
> Here are some news on our RPM packages initiative.
>
> 1. I’ve finished preliminary developing of Stage II version of RPM
> packages [1]. Main “new feature” is — split design. Also I’ve added
> package.sh script for automating package building process which will help
> organise corresponding builds in TC as well as simplify process for
> developers who wishes to have custom packages.
> PR#3703 [2] is ready for review. Denis, in order to catch up with Apache
> Ignite 2.5 release, I’d greatly appreciate your help in finding reviewer.
> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4]
> repositories on Apache Bintray. Though they are already prepared for
> hosting RPM and DEB packages respectively, and there is a way of linking
> them to apache.org/dist/ignite page, there is possible alternative in
> storing there only plain directory layout corresponding to each repository
> type (RPM and DEB) and manage this layout (repodata, distributions,
> versions, etc.) by ourselves, having more control over repositories but
> lacking some simplicity of deploying new releases. WDYT? Should we try
> Cassandra approach? They are storing their DEB packages as I described
> above [5].
>
> Also — a question arose while I was working on this issue: which OSes (and
> which versions of each) are we going to support (if we are going) in terms
> of step-by-step list? Currently RPM packages are tested only with latest
> CentOS (and, respectively — RHEL), but there are a lot more RPM-based
> distributives [6] some of which are more o less popular among OS community
> (ALT, Fedora, openSUSE, etc.).
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> [2] https://github.com/apache/ignite/pull/3703
> [3] https://bintray.com/apache/ignite-rpm
> [4] https://bintray.com/apache/ignite-deb
> [5] https://bintray.com/apache/cassandra/debian#files/
> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_distributions
>
>
>
>
> > On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]> wrote:
> >
> > I suppose that most everything if not all from libs/options will go to
> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > More precise lib selection (if something from optional would better to
> have in core package) will be discussed right after preliminary split
> architecture agreement.
> >
> >
> >
> >> On 15 Mar 2018, at 22:11, Dmitry Pavlov <[hidden email]> wrote:
> >>
> >> I like idea of keeping simple system of modules, so +1 from me.
> >>
> >> Where optional libs (e.g Direct IO plugin) would be included, would it
> be
> >> core or optional?
> >>
> >> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]>:
> >>
> >>>>
> >>>>>
> >>>>> How big would be a final core module?
> >>>> Around 30M. Can be shrinked to ~15M if separate Visor and create it’s
> own
> >>>> package.
> >>>
> >>>
> >>> Guys, 30 vs 280M is a huuuuge difference. I would agree with Petr and
> >>> propose the simplest modular system:
> >>>
> >>> - core module that includes basic Ignite capabilities including SQL,
> >>> compute grid, service grid, k/v
> >>> - optional module hosts the rest - ML, streamers integration (kafka,
> >>> flink), kubernetes, etc.
> >>>
> >>> What do you think?
> >>>
> >>> --
> >>> Denis
> >>>
> >>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <[hidden email]>
> wrote:
> >>>
> >>>> *DEB package
> >>>>
> >>>>
> >>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <[hidden email]> wrote:
> >>>>>
> >>>>> Considering that DEV package for now is almost platform independent
> >>> (its
> >>>> a java application more or less), that package will work almost on any
> >>>> DEB-based linux, including but not limited to Ubuntu, Debian, etc.
> >>>>> The only restriction is existence of systemctl (systemd) service
> >>> manager
> >>>> — we are dependent on it.
> >>>>>
> >>>>> Thats why, for instance, our RPM repository is called simply ‘rpm’
> and
> >>>> package has no arch or dist suffix — it will work on CentOS, RHEL,
> >>> Fedora,
> >>>> etc. with presence of aforementioned systemd.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <[hidden email]>
> >>>> wrote:
> >>>>>>
> >>>>>> Will Debian package work for Ubuntu?
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <[hidden email]>
> >>>> wrote:
> >>>>>>
> >>>>>>> Not a problem, rather nuisance. Also, when we will move to official
> >>>>>>> repositories, there can be a problem from OS community.
> >>>>>>>
> >>>>>>> Concerning DEB packages — I plan to use RPM as base for DEB package
> >>>> build
> >>>>>>> (package layout / install scripts) for speeding up things and
> >>> excluding
> >>>>>>> possible duplication and desynchronisation, so its a matter of ’sit
> >>>> and do’
> >>>>>>> rather then some technical research. Thats why I rose discussion
> >>> about
> >>>>>>> future package architecture, so that after agreement I'm be able to
> >>>> pack
> >>>>>>> both RPM and DEB identically.
> >>>>>>>
> >>>>>>> Yet, if you insist, I can create DEB package according to current
> RPM
> >>>>>>> layout in no time.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> [hidden email]>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Peter,
> >>>>>>>>
> >>>>>>>> I don't think the package size of 280M is going to be a problem at
> >>>> all,
> >>>>>>> but
> >>>>>>>> what you are suggesting can be an improvement down the road.
> >>>>>>>>
> >>>>>>>> In the mean time, I think our top priority should be to provide
> >>>> packages
> >>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly enough.
> >>>>>>>>
> >>>>>>>> Agree?
> >>>>>>>>
> >>>>>>>> D.
> >>>>>>>>
> >>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <[hidden email]>
> >>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi, Igniters!
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Release 2.4 is almost there, at least binary part of it, so I'd
> >>> like
> >>>> to
> >>>>>>>>> move
> >>>>>>>>> forward to further improve and widen AI delivery through
> packages.
> >>>>>>>>> As of now, Apache Ignite ships in RPM package weighing about
> 280M+
> >>>> and,
> >>>>>>> to
> >>>>>>>>> improve usability and significantly reduce required download
> >>> sizes, I
> >>>>>>>>> purpose that in 2.5 release we introduce splitted delivery as
> >>>> follows:
> >>>>>>>>> - CORE
> >>>>>>>>> - bin
> >>>>>>>>> - config
> >>>>>>>>> - libs (!optional)
> >>>>>>>>> - OPTIONAL LIBS
> >>>>>>>>> - BENCHMARKS
> >>>>>>>>> - DOCS (?)
> >>>>>>>>> - EXAMPLES
> >>>>>>>>> - .NET PLATFORM FILES
> >>>>>>>>> - C++ PLATFORM FILES
> >>>>>>>>>
> >>>>>>>>> This architecture, as I assume, will add flexibility (no reason
> to
> >>>>>>> download
> >>>>>>>>> all 280M+ of binaries where you are to run only core node
> >>>> functionality)
> >>>>>>>>> and
> >>>>>>>>> maintainability (you are in full control of what is installed on
> >>> your
> >>>>>>>>> system).
> >>>>>>>>>
> >>>>>>>>> After successful architecture choice, same scheme are planned to
> be
> >>>>>>> used in
> >>>>>>>>> DEB packages as well.
> >>>>>>>>>
> >>>>>>>>> WDYT?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.
> com/
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >
>
>

vveider

Re: Apache Ignite RPM packaging (Stage II)

To top new RPM architecture off, update to release process is introduced — [1] [2].

Both tasks (this one and IGNITE-7647) are ready for review and should be merged simultaneously.

[1] https://issues.apache.org/jira/browse/IGNITE-8172
[2] https://github.com/apache/ignite-release/pull/1

> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <[hidden email]> wrote:
>
> Hello!
>
> Let me share my idea of how this shoud work. Splitting package into
> sub-packages should be dependency-driven.
>
> It means that all Ignite modules without dependencies or with small
> dependencies (such as ignite-log4j) should be included in ignite-core. It
> doesn't make sense to make a zillion RPM packages.
>
> Critical things like ignite-spring and ignite-indexing should be in
> ignite-core of course, even if they have dependencies. Ignite-core should
> be fully self-sufficient and feature-complete.
>
> However, e.g. .net API should probably be in a separate package, because it
> should depend on mono | net-core. We may also have ignite-devel package
> which should include all modules which only make sense for developers who
> write code. Such as hibernate integration.
>
> I'm not sure about MR modules. The main question should be, does it have
> dependencies? Can it run stand-alone without writing code?
>
> Hope this helps,
>
> --
> Ilya Kasnacheev
>
> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
>
>> Hi, Igniters!
>>
>>
>> Here are some news on our RPM packages initiative.
>>
>> 1. I’ve finished preliminary developing of Stage II version of RPM
>> packages [1]. Main “new feature” is — split design. Also I’ve added
>> package.sh script for automating package building process which will help
>> organise corresponding builds in TC as well as simplify process for
>> developers who wishes to have custom packages.
>> PR#3703 [2] is ready for review. Denis, in order to catch up with Apache
>> Ignite 2.5 release, I’d greatly appreciate your help in finding reviewer.
>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4]
>> repositories on Apache Bintray. Though they are already prepared for
>> hosting RPM and DEB packages respectively, and there is a way of linking
>> them to apache.org/dist/ignite page, there is possible alternative in
>> storing there only plain directory layout corresponding to each repository
>> type (RPM and DEB) and manage this layout (repodata, distributions,
>> versions, etc.) by ourselves, having more control over repositories but
>> lacking some simplicity of deploying new releases. WDYT? Should we try
>> Cassandra approach? They are storing their DEB packages as I described
>> above [5].
>>
>> Also — a question arose while I was working on this issue: which OSes (and
>> which versions of each) are we going to support (if we are going) in terms
>> of step-by-step list? Currently RPM packages are tested only with latest
>> CentOS (and, respectively — RHEL), but there are a lot more RPM-based
>> distributives [6] some of which are more o less popular among OS community
>> (ALT, Fedora, openSUSE, etc.).
>>
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
>> [2] https://github.com/apache/ignite/pull/3703
>> [3] https://bintray.com/apache/ignite-rpm
>> [4] https://bintray.com/apache/ignite-deb
>> [5] https://bintray.com/apache/cassandra/debian#files/
>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_distributions
>>
>>
>>
>>
>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]> wrote:
>>>
>>> I suppose that most everything if not all from libs/options will go to
>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
>>> More precise lib selection (if something from optional would better to
>> have in core package) will be discussed right after preliminary split
>> architecture agreement.
>>>
>>>
>>>
>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <[hidden email]> wrote:
>>>>
>>>> I like idea of keeping simple system of modules, so +1 from me.
>>>>
>>>> Where optional libs (e.g Direct IO plugin) would be included, would it
>> be
>>>> core or optional?
>>>>
>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]>:
>>>>
>>>>>>
>>>>>>>
>>>>>>> How big would be a final core module?
>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and create it’s
>> own
>>>>>> package.
>>>>>
>>>>>
>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with Petr and
>>>>> propose the simplest modular system:
>>>>>
>>>>> - core module that includes basic Ignite capabilities including SQL,
>>>>> compute grid, service grid, k/v
>>>>> - optional module hosts the rest - ML, streamers integration (kafka,
>>>>> flink), kubernetes, etc.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> --
>>>>> Denis
>>>>>
>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <[hidden email]>
>> wrote:
>>>>>
>>>>>> *DEB package
>>>>>>
>>>>>>
>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <[hidden email]> wrote:
>>>>>>>
>>>>>>> Considering that DEV package for now is almost platform independent
>>>>> (its
>>>>>> a java application more or less), that package will work almost on any
>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian, etc.
>>>>>>> The only restriction is existence of systemctl (systemd) service
>>>>> manager
>>>>>> — we are dependent on it.
>>>>>>>
>>>>>>> Thats why, for instance, our RPM repository is called simply ‘rpm’
>> and
>>>>>> package has no arch or dist suffix — it will work on CentOS, RHEL,
>>>>> Fedora,
>>>>>> etc. with presence of aforementioned systemd.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <[hidden email]>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Will Debian package work for Ubuntu?
>>>>>>>>
>>>>>>>> D.
>>>>>>>>
>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <[hidden email]>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to official
>>>>>>>>> repositories, there can be a problem from OS community.
>>>>>>>>>
>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for DEB package
>>>>>> build
>>>>>>>>> (package layout / install scripts) for speeding up things and
>>>>> excluding
>>>>>>>>> possible duplication and desynchronisation, so its a matter of ’sit
>>>>>> and do’
>>>>>>>>> rather then some technical research. Thats why I rose discussion
>>>>> about
>>>>>>>>> future package architecture, so that after agreement I'm be able to
>>>>>> pack
>>>>>>>>> both RPM and DEB identically.
>>>>>>>>>
>>>>>>>>> Yet, if you insist, I can create DEB package according to current
>> RPM
>>>>>>>>> layout in no time.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
>> [hidden email]>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Peter,
>>>>>>>>>>
>>>>>>>>>> I don't think the package size of 280M is going to be a problem at
>>>>>> all,
>>>>>>>>> but
>>>>>>>>>> what you are suggesting can be an improvement down the road.
>>>>>>>>>>
>>>>>>>>>> In the mean time, I think our top priority should be to provide
>>>>>> packages
>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly enough.
>>>>>>>>>>
>>>>>>>>>> Agree?
>>>>>>>>>>
>>>>>>>>>> D.
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <[hidden email]>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi, Igniters!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it, so I'd
>>>>> like
>>>>>> to
>>>>>>>>>>> move
>>>>>>>>>>> forward to further improve and widen AI delivery through
>> packages.
>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing about
>> 280M+
>>>>>> and,
>>>>>>>>> to
>>>>>>>>>>> improve usability and significantly reduce required download
>>>>> sizes, I
>>>>>>>>>>> purpose that in 2.5 release we introduce splitted delivery as
>>>>>> follows:
>>>>>>>>>>> - CORE
>>>>>>>>>>> - bin
>>>>>>>>>>> - config
>>>>>>>>>>> - libs (!optional)
>>>>>>>>>>> - OPTIONAL LIBS
>>>>>>>>>>> - BENCHMARKS
>>>>>>>>>>> - DOCS (?)
>>>>>>>>>>> - EXAMPLES
>>>>>>>>>>> - .NET PLATFORM FILES
>>>>>>>>>>> - C++ PLATFORM FILES
>>>>>>>>>>>
>>>>>>>>>>> This architecture, as I assume, will add flexibility (no reason
>> to
>>>>>>>>> download
>>>>>>>>>>> all 280M+ of binaries where you are to run only core node
>>>>>> functionality)
>>>>>>>>>>> and
>>>>>>>>>>> maintainability (you are in full control of what is installed on
>>>>> your
>>>>>>>>>>> system).
>>>>>>>>>>>
>>>>>>>>>>> After successful architecture choice, same scheme are planned to
>> be
>>>>>>>>> used in
>>>>>>>>>>> DEB packages as well.
>>>>>>>>>>>
>>>>>>>>>>> WDYT?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.
>> com/
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>
>>

vveider

Re: Apache Ignite RPM packaging (Stage II)

If someone from PMCы or Committers still sees necessity about including these tasks into Apache Ignite 2.5 release, this is the last chance to do so.
Otherwise this task will be moved to at 2.6 release at least, or even moved to backlog indefinitely.

> On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]> wrote:
>
> To top new RPM architecture off, update to release process is introduced — [1] [2].
>
> Both tasks (this one and IGNITE-7647) are ready for review and should be merged simultaneously.
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8172
> [2] https://github.com/apache/ignite-release/pull/1
>
>
>
>
>> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <[hidden email]> wrote:
>>
>> Hello!
>>
>> Let me share my idea of how this shoud work. Splitting package into
>> sub-packages should be dependency-driven.
>>
>> It means that all Ignite modules without dependencies or with small
>> dependencies (such as ignite-log4j) should be included in ignite-core. It
>> doesn't make sense to make a zillion RPM packages.
>>
>> Critical things like ignite-spring and ignite-indexing should be in
>> ignite-core of course, even if they have dependencies. Ignite-core should
>> be fully self-sufficient and feature-complete.
>>
>> However, e.g. .net API should probably be in a separate package, because it
>> should depend on mono | net-core. We may also have ignite-devel package
>> which should include all modules which only make sense for developers who
>> write code. Such as hibernate integration.
>>
>> I'm not sure about MR modules. The main question should be, does it have
>> dependencies? Can it run stand-alone without writing code?
>>
>> Hope this helps,
>>
>> --
>> Ilya Kasnacheev
>>
>> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
>>
>>> Hi, Igniters!
>>>
>>>
>>> Here are some news on our RPM packages initiative.
>>>
>>> 1. I’ve finished preliminary developing of Stage II version of RPM
>>> packages [1]. Main “new feature” is — split design. Also I’ve added
>>> package.sh script for automating package building process which will help
>>> organise corresponding builds in TC as well as simplify process for
>>> developers who wishes to have custom packages.
>>> PR#3703 [2] is ready for review. Denis, in order to catch up with Apache
>>> Ignite 2.5 release, I’d greatly appreciate your help in finding reviewer.
>>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4]
>>> repositories on Apache Bintray. Though they are already prepared for
>>> hosting RPM and DEB packages respectively, and there is a way of linking
>>> them to apache.org/dist/ignite page, there is possible alternative in
>>> storing there only plain directory layout corresponding to each repository
>>> type (RPM and DEB) and manage this layout (repodata, distributions,
>>> versions, etc.) by ourselves, having more control over repositories but
>>> lacking some simplicity of deploying new releases. WDYT? Should we try
>>> Cassandra approach? They are storing their DEB packages as I described
>>> above [5].
>>>
>>> Also — a question arose while I was working on this issue: which OSes (and
>>> which versions of each) are we going to support (if we are going) in terms
>>> of step-by-step list? Currently RPM packages are tested only with latest
>>> CentOS (and, respectively — RHEL), but there are a lot more RPM-based
>>> distributives [6] some of which are more o less popular among OS community
>>> (ALT, Fedora, openSUSE, etc.).
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
>>> [2] https://github.com/apache/ignite/pull/3703
>>> [3] https://bintray.com/apache/ignite-rpm
>>> [4] https://bintray.com/apache/ignite-deb
>>> [5] https://bintray.com/apache/cassandra/debian#files/
>>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_distributions
>>>
>>>
>>>
>>>
>>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]> wrote:
>>>>
>>>> I suppose that most everything if not all from libs/options will go to
>>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
>>>> More precise lib selection (if something from optional would better to
>>> have in core package) will be discussed right after preliminary split
>>> architecture agreement.
>>>>
>>>>
>>>>
>>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <[hidden email]> wrote:
>>>>>
>>>>> I like idea of keeping simple system of modules, so +1 from me.
>>>>>
>>>>> Where optional libs (e.g Direct IO plugin) would be included, would it
>>> be
>>>>> core or optional?
>>>>>
>>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]>:
>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> How big would be a final core module?
>>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and create it’s
>>> own
>>>>>>> package.
>>>>>>
>>>>>>
>>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with Petr and
>>>>>> propose the simplest modular system:
>>>>>>
>>>>>> - core module that includes basic Ignite capabilities including SQL,
>>>>>> compute grid, service grid, k/v
>>>>>> - optional module hosts the rest - ML, streamers integration (kafka,
>>>>>> flink), kubernetes, etc.
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>> --
>>>>>> Denis
>>>>>>
>>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <[hidden email]>
>>> wrote:
>>>>>>
>>>>>>> *DEB package
>>>>>>>
>>>>>>>
>>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> Considering that DEV package for now is almost platform independent
>>>>>> (its
>>>>>>> a java application more or less), that package will work almost on any
>>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian, etc.
>>>>>>>> The only restriction is existence of systemctl (systemd) service
>>>>>> manager
>>>>>>> — we are dependent on it.
>>>>>>>>
>>>>>>>> Thats why, for instance, our RPM repository is called simply ‘rpm’
>>> and
>>>>>>> package has no arch or dist suffix — it will work on CentOS, RHEL,
>>>>>> Fedora,
>>>>>>> etc. with presence of aforementioned systemd.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <[hidden email]>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Will Debian package work for Ubuntu?
>>>>>>>>>
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <[hidden email]>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to official
>>>>>>>>>> repositories, there can be a problem from OS community.
>>>>>>>>>>
>>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for DEB package
>>>>>>> build
>>>>>>>>>> (package layout / install scripts) for speeding up things and
>>>>>> excluding
>>>>>>>>>> possible duplication and desynchronisation, so its a matter of ’sit
>>>>>>> and do’
>>>>>>>>>> rather then some technical research. Thats why I rose discussion
>>>>>> about
>>>>>>>>>> future package architecture, so that after agreement I'm be able to
>>>>>>> pack
>>>>>>>>>> both RPM and DEB identically.
>>>>>>>>>>
>>>>>>>>>> Yet, if you insist, I can create DEB package according to current
>>> RPM
>>>>>>>>>> layout in no time.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
>>> [hidden email]>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Peter,
>>>>>>>>>>>
>>>>>>>>>>> I don't think the package size of 280M is going to be a problem at
>>>>>>> all,
>>>>>>>>>> but
>>>>>>>>>>> what you are suggesting can be an improvement down the road.
>>>>>>>>>>>
>>>>>>>>>>> In the mean time, I think our top priority should be to provide
>>>>>>> packages
>>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly enough.
>>>>>>>>>>>
>>>>>>>>>>> Agree?
>>>>>>>>>>>
>>>>>>>>>>> D.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <[hidden email]>
>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, Igniters!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it, so I'd
>>>>>> like
>>>>>>> to
>>>>>>>>>>>> move
>>>>>>>>>>>> forward to further improve and widen AI delivery through
>>> packages.
>>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing about
>>> 280M+
>>>>>>> and,
>>>>>>>>>> to
>>>>>>>>>>>> improve usability and significantly reduce required download
>>>>>> sizes, I
>>>>>>>>>>>> purpose that in 2.5 release we introduce splitted delivery as
>>>>>>> follows:
>>>>>>>>>>>> - CORE
>>>>>>>>>>>> - bin
>>>>>>>>>>>> - config
>>>>>>>>>>>> - libs (!optional)
>>>>>>>>>>>> - OPTIONAL LIBS
>>>>>>>>>>>> - BENCHMARKS
>>>>>>>>>>>> - DOCS (?)
>>>>>>>>>>>> - EXAMPLES
>>>>>>>>>>>> - .NET PLATFORM FILES
>>>>>>>>>>>> - C++ PLATFORM FILES
>>>>>>>>>>>>
>>>>>>>>>>>> This architecture, as I assume, will add flexibility (no reason
>>> to
>>>>>>>>>> download
>>>>>>>>>>>> all 280M+ of binaries where you are to run only core node
>>>>>>> functionality)
>>>>>>>>>>>> and
>>>>>>>>>>>> maintainability (you are in full control of what is installed on
>>>>>> your
>>>>>>>>>>>> system).
>>>>>>>>>>>>
>>>>>>>>>>>> After successful architecture choice, same scheme are planned to
>>> be
>>>>>>>>>> used in
>>>>>>>>>>>> DEB packages as well.
>>>>>>>>>>>>
>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.
>>> com/
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>

Ilya Kasnacheev

Re: Apache Ignite RPM packaging (Stage II)

Hello! I have tried your RPM scripts.

First of all, I'm not sure that apache-ignite-core is a good name for
package which contains the actual server node kit, and that apache-ignite
is a good name for a package that will install everything (including cpp
bindings). How does other packages solve this naming? I would go with
apache-ignite and apache-ignite-full.

Moreover I did not find a way to start service if default installed JVM is
Java 7 :( I understand it's EOL, still this is something that hit me.

apache-ignite-docs only contains scaladoc for scalar. Who would need a
separate package for that? Why no javadocs? Maybe it's my build problem?

apache-ignite-libs is a totally unexpected package name. apache-ignite core
doesn't depend on it. It doesn't enable anything out of the box. The
package is huge.

Frankly speaking, I'm not sure that improvements over Stage I are enough as
of now. For demo-like activity, we can probably go with one package fits
all.

--
Ilya Kasnacheev

2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:

> If someone from PMCы or Committers still sees necessity about including
> these tasks into Apache Ignite 2.5 release, this is the last chance to do
> so.
> Otherwise this task will be moved to at 2.6 release at least, or even
> moved to backlog indefinitely.
>
>
>
> > On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]> wrote:
> >
> > To top new RPM architecture off, update to release process is introduced
> — [1] [2].
> >
> > Both tasks (this one and IGNITE-7647) are ready for review and should be
> merged simultaneously.
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > [2] https://github.com/apache/ignite-release/pull/1
> >
> >
> >
> >
> >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <[hidden email]>
> wrote:
> >>
> >> Hello!
> >>
> >> Let me share my idea of how this shoud work. Splitting package into
> >> sub-packages should be dependency-driven.
> >>
> >> It means that all Ignite modules without dependencies or with small
> >> dependencies (such as ignite-log4j) should be included in ignite-core.
> It
> >> doesn't make sense to make a zillion RPM packages.
> >>
> >> Critical things like ignite-spring and ignite-indexing should be in
> >> ignite-core of course, even if they have dependencies. Ignite-core
> should
> >> be fully self-sufficient and feature-complete.
> >>
> >> However, e.g. .net API should probably be in a separate package,
> because it
> >> should depend on mono | net-core. We may also have ignite-devel package
> >> which should include all modules which only make sense for developers
> who
> >> write code. Such as hibernate integration.
> >>
> >> I'm not sure about MR modules. The main question should be, does it have
> >> dependencies? Can it run stand-alone without writing code?
> >>
> >> Hope this helps,
> >>
> >> --
> >> Ilya Kasnacheev
> >>
> >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> >>
> >>> Hi, Igniters!
> >>>
> >>>
> >>> Here are some news on our RPM packages initiative.
> >>>
> >>> 1. I’ve finished preliminary developing of Stage II version of RPM
> >>> packages [1]. Main “new feature” is — split design. Also I’ve added
> >>> package.sh script for automating package building process which will
> help
> >>> organise corresponding builds in TC as well as simplify process for
> >>> developers who wishes to have custom packages.
> >>> PR#3703 [2] is ready for review. Denis, in order to catch up with
> Apache
> >>> Ignite 2.5 release, I’d greatly appreciate your help in finding
> reviewer.
> >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4]
> >>> repositories on Apache Bintray. Though they are already prepared for
> >>> hosting RPM and DEB packages respectively, and there is a way of
> linking
> >>> them to apache.org/dist/ignite page, there is possible alternative in
> >>> storing there only plain directory layout corresponding to each
> repository
> >>> type (RPM and DEB) and manage this layout (repodata, distributions,
> >>> versions, etc.) by ourselves, having more control over repositories but
> >>> lacking some simplicity of deploying new releases. WDYT? Should we try
> >>> Cassandra approach? They are storing their DEB packages as I described
> >>> above [5].
> >>>
> >>> Also — a question arose while I was working on this issue: which OSes
> (and
> >>> which versions of each) are we going to support (if we are going) in
> terms
> >>> of step-by-step list? Currently RPM packages are tested only with
> latest
> >>> CentOS (and, respectively — RHEL), but there are a lot more RPM-based
> >>> distributives [6] some of which are more o less popular among OS
> community
> >>> (ALT, Fedora, openSUSE, etc.).
> >>>
> >>>
> >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> >>> [2] https://github.com/apache/ignite/pull/3703
> >>> [3] https://bintray.com/apache/ignite-rpm
> >>> [4] https://bintray.com/apache/ignite-deb
> >>> [5] https://bintray.com/apache/cassandra/debian#files/
> >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> distributions
> >>>
> >>>
> >>>
> >>>
> >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]> wrote:
> >>>>
> >>>> I suppose that most everything if not all from libs/options will go to
> >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> >>>> More precise lib selection (if something from optional would better to
> >>> have in core package) will be discussed right after preliminary split
> >>> architecture agreement.
> >>>>
> >>>>
> >>>>
> >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <[hidden email]>
> wrote:
> >>>>>
> >>>>> I like idea of keeping simple system of modules, so +1 from me.
> >>>>>
> >>>>> Where optional libs (e.g Direct IO plugin) would be included, would
> it
> >>> be
> >>>>> core or optional?
> >>>>>
> >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]>:
> >>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> How big would be a final core module?
> >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and create
> it’s
> >>> own
> >>>>>>> package.
> >>>>>>
> >>>>>>
> >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with Petr
> and
> >>>>>> propose the simplest modular system:
> >>>>>>
> >>>>>> - core module that includes basic Ignite capabilities including SQL,
> >>>>>> compute grid, service grid, k/v
> >>>>>> - optional module hosts the rest - ML, streamers integration (kafka,
> >>>>>> flink), kubernetes, etc.
> >>>>>>
> >>>>>> What do you think?
> >>>>>>
> >>>>>> --
> >>>>>> Denis
> >>>>>>
> >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <[hidden email]>
> >>> wrote:
> >>>>>>
> >>>>>>> *DEB package
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <[hidden email]>
> wrote:
> >>>>>>>>
> >>>>>>>> Considering that DEV package for now is almost platform
> independent
> >>>>>> (its
> >>>>>>> a java application more or less), that package will work almost on
> any
> >>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian, etc.
> >>>>>>>> The only restriction is existence of systemctl (systemd) service
> >>>>>> manager
> >>>>>>> — we are dependent on it.
> >>>>>>>>
> >>>>>>>> Thats why, for instance, our RPM repository is called simply ‘rpm’
> >>> and
> >>>>>>> package has no arch or dist suffix — it will work on CentOS, RHEL,
> >>>>>> Fedora,
> >>>>>>> etc. with presence of aforementioned systemd.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> [hidden email]>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Will Debian package work for Ubuntu?
> >>>>>>>>>
> >>>>>>>>> D.
> >>>>>>>>>
> >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> [hidden email]>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to
> official
> >>>>>>>>>> repositories, there can be a problem from OS community.
> >>>>>>>>>>
> >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for DEB
> package
> >>>>>>> build
> >>>>>>>>>> (package layout / install scripts) for speeding up things and
> >>>>>> excluding
> >>>>>>>>>> possible duplication and desynchronisation, so its a matter of
> ’sit
> >>>>>>> and do’
> >>>>>>>>>> rather then some technical research. Thats why I rose discussion
> >>>>>> about
> >>>>>>>>>> future package architecture, so that after agreement I'm be
> able to
> >>>>>>> pack
> >>>>>>>>>> both RPM and DEB identically.
> >>>>>>>>>>
> >>>>>>>>>> Yet, if you insist, I can create DEB package according to
> current
> >>> RPM
> >>>>>>>>>> layout in no time.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> >>> [hidden email]>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Peter,
> >>>>>>>>>>>
> >>>>>>>>>>> I don't think the package size of 280M is going to be a
> problem at
> >>>>>>> all,
> >>>>>>>>>> but
> >>>>>>>>>>> what you are suggesting can be an improvement down the road.
> >>>>>>>>>>>
> >>>>>>>>>>> In the mean time, I think our top priority should be to provide
> >>>>>>> packages
> >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly enough.
> >>>>>>>>>>>
> >>>>>>>>>>> Agree?
> >>>>>>>>>>>
> >>>>>>>>>>> D.
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <[hidden email]>
> >>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi, Igniters!
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it, so
> I'd
> >>>>>> like
> >>>>>>> to
> >>>>>>>>>>>> move
> >>>>>>>>>>>> forward to further improve and widen AI delivery through
> >>> packages.
> >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing about
> >>> 280M+
> >>>>>>> and,
> >>>>>>>>>> to
> >>>>>>>>>>>> improve usability and significantly reduce required download
> >>>>>> sizes, I
> >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted delivery as
> >>>>>>> follows:
> >>>>>>>>>>>> - CORE
> >>>>>>>>>>>> - bin
> >>>>>>>>>>>> - config
> >>>>>>>>>>>> - libs (!optional)
> >>>>>>>>>>>> - OPTIONAL LIBS
> >>>>>>>>>>>> - BENCHMARKS
> >>>>>>>>>>>> - DOCS (?)
> >>>>>>>>>>>> - EXAMPLES
> >>>>>>>>>>>> - .NET PLATFORM FILES
> >>>>>>>>>>>> - C++ PLATFORM FILES
> >>>>>>>>>>>>
> >>>>>>>>>>>> This architecture, as I assume, will add flexibility (no
> reason
> >>> to
> >>>>>>>>>> download
> >>>>>>>>>>>> all 280M+ of binaries where you are to run only core node
> >>>>>>> functionality)
> >>>>>>>>>>>> and
> >>>>>>>>>>>> maintainability (you are in full control of what is installed
> on
> >>>>>> your
> >>>>>>>>>>>> system).
> >>>>>>>>>>>>
> >>>>>>>>>>>> After successful architecture choice, same scheme are planned
> to
> >>> be
> >>>>>>>>>> used in
> >>>>>>>>>>>> DEB packages as well.
> >>>>>>>>>>>>
> >>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.
> >>> com/
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>>
> >
>
>

dmagda

Re: Apache Ignite RPM packaging (Stage II)

In reply to this post by vveider

Petr,

I wouldn't postpone this until 2.6 that will be out nor earlier than 3
months from now.

*Anton V.*, could review and sign off the changes? Not sure we have a
better person in the community who can do that.

--
Denis

On Thu, Apr 12, 2018 at 9:10 AM, Petr Ivanov <[hidden email]> wrote:

dsetrakyan

Re: Apache Ignite RPM packaging (Stage II)

I am not sure that an email thread of over 20 messages is a good medium to
discuss proposals. In Ignite, we create IEPs. Can you please summarize your
proposal there and send a link there? Please explain not only the change
itself, but the reason why we need it.

D.

On Thu, Apr 12, 2018 at 4:00 PM, Denis Magda <[hidden email]> wrote:

> Petr,
>
> I wouldn't postpone this until 2.6 that will be out nor earlier than 3
> months from now.
>
> *Anton V.*, could review and sign off the changes? Not sure we have a
> better person in the community who can do that.
>
> --
> Denis
>
> On Thu, Apr 12, 2018 at 9:10 AM, Petr Ivanov <[hidden email]> wrote:
>
> > If someone from PMCы or Committers still sees necessity about including
> > these tasks into Apache Ignite 2.5 release, this is the last chance to do
> > so.
> > Otherwise this task will be moved to at 2.6 release at least, or even
> > moved to backlog indefinitely.
> >
> >
> >
> > > On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]> wrote:
> > >
> > > To top new RPM architecture off, update to release process is
> introduced
> > — [1] [2].
> > >
> > > Both tasks (this one and IGNITE-7647) are ready for review and should
> be
> > merged simultaneously.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > > [2] https://github.com/apache/ignite-release/pull/1
> > >
> > >
> > >
> > >
> > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <[hidden email]>
> > wrote:
> > >>
> > >> Hello!
> > >>
> > >> Let me share my idea of how this shoud work. Splitting package into
> > >> sub-packages should be dependency-driven.
> > >>
> > >> It means that all Ignite modules without dependencies or with small
> > >> dependencies (such as ignite-log4j) should be included in ignite-core.
> > It
> > >> doesn't make sense to make a zillion RPM packages.
> > >>
> > >> Critical things like ignite-spring and ignite-indexing should be in
> > >> ignite-core of course, even if they have dependencies. Ignite-core
> > should
> > >> be fully self-sufficient and feature-complete.
> > >>
> > >> However, e.g. .net API should probably be in a separate package,
> > because it
> > >> should depend on mono | net-core. We may also have ignite-devel
> package
> > >> which should include all modules which only make sense for developers
> > who
> > >> write code. Such as hibernate integration.
> > >>
> > >> I'm not sure about MR modules. The main question should be, does it
> have
> > >> dependencies? Can it run stand-alone without writing code?
> > >>
> > >> Hope this helps,
> > >>
> > >> --
> > >> Ilya Kasnacheev
> > >>
> > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > >>
> > >>> Hi, Igniters!
> > >>>
> > >>>
> > >>> Here are some news on our RPM packages initiative.
> > >>>
> > >>> 1. I’ve finished preliminary developing of Stage II version of RPM
> > >>> packages [1]. Main “new feature” is — split design. Also I’ve added
> > >>> package.sh script for automating package building process which will
> > help
> > >>> organise corresponding builds in TC as well as simplify process for
> > >>> developers who wishes to have custom packages.
> > >>> PR#3703 [2] is ready for review. Denis, in order to catch up with
> > Apache
> > >>> Ignite 2.5 release, I’d greatly appreciate your help in finding
> > reviewer.
> > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4]
> > >>> repositories on Apache Bintray. Though they are already prepared for
> > >>> hosting RPM and DEB packages respectively, and there is a way of
> > linking
> > >>> them to apache.org/dist/ignite page, there is possible alternative
> in
> > >>> storing there only plain directory layout corresponding to each
> > repository
> > >>> type (RPM and DEB) and manage this layout (repodata, distributions,
> > >>> versions, etc.) by ourselves, having more control over repositories
> but
> > >>> lacking some simplicity of deploying new releases. WDYT? Should we
> try
> > >>> Cassandra approach? They are storing their DEB packages as I
> described
> > >>> above [5].
> > >>>
> > >>> Also — a question arose while I was working on this issue: which OSes
> > (and
> > >>> which versions of each) are we going to support (if we are going) in
> > terms
> > >>> of step-by-step list? Currently RPM packages are tested only with
> > latest
> > >>> CentOS (and, respectively — RHEL), but there are a lot more RPM-based
> > >>> distributives [6] some of which are more o less popular among OS
> > community
> > >>> (ALT, Fedora, openSUSE, etc.).
> > >>>
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > >>> [2] https://github.com/apache/ignite/pull/3703
> > >>> [3] https://bintray.com/apache/ignite-rpm
> > >>> [4] https://bintray.com/apache/ignite-deb
> > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > distributions
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]> wrote:
> > >>>>
> > >>>> I suppose that most everything if not all from libs/options will go
> to
> > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > >>>> More precise lib selection (if something from optional would better
> to
> > >>> have in core package) will be discussed right after preliminary split
> > >>> architecture agreement.
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <[hidden email]>
> > wrote:
> > >>>>>
> > >>>>> I like idea of keeping simple system of modules, so +1 from me.
> > >>>>>
> > >>>>> Where optional libs (e.g Direct IO plugin) would be included, would
> > it
> > >>> be
> > >>>>> core or optional?
> > >>>>>
> > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]>:
> > >>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> How big would be a final core module?
> > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and create
> > it’s
> > >>> own
> > >>>>>>> package.
> > >>>>>>
> > >>>>>>
> > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with Petr
> > and
> > >>>>>> propose the simplest modular system:
> > >>>>>>
> > >>>>>> - core module that includes basic Ignite capabilities including
> SQL,
> > >>>>>> compute grid, service grid, k/v
> > >>>>>> - optional module hosts the rest - ML, streamers integration
> (kafka,
> > >>>>>> flink), kubernetes, etc.
> > >>>>>>
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> --
> > >>>>>> Denis
> > >>>>>>
> > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> [hidden email]>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> *DEB package
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <[hidden email]>
> > wrote:
> > >>>>>>>>
> > >>>>>>>> Considering that DEV package for now is almost platform
> > independent
> > >>>>>> (its
> > >>>>>>> a java application more or less), that package will work almost
> on
> > any
> > >>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian,
> etc.
> > >>>>>>>> The only restriction is existence of systemctl (systemd) service
> > >>>>>> manager
> > >>>>>>> — we are dependent on it.
> > >>>>>>>>
> > >>>>>>>> Thats why, for instance, our RPM repository is called simply
> ‘rpm’
> > >>> and
> > >>>>>>> package has no arch or dist suffix — it will work on CentOS,
> RHEL,
> > >>>>>> Fedora,
> > >>>>>>> etc. with presence of aforementioned systemd.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> > [hidden email]>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Will Debian package work for Ubuntu?
> > >>>>>>>>>
> > >>>>>>>>> D.
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> > [hidden email]>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to
> > official
> > >>>>>>>>>> repositories, there can be a problem from OS community.
> > >>>>>>>>>>
> > >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for DEB
> > package
> > >>>>>>> build
> > >>>>>>>>>> (package layout / install scripts) for speeding up things and
> > >>>>>> excluding
> > >>>>>>>>>> possible duplication and desynchronisation, so its a matter of
> > ’sit
> > >>>>>>> and do’
> > >>>>>>>>>> rather then some technical research. Thats why I rose
> discussion
> > >>>>>> about
> > >>>>>>>>>> future package architecture, so that after agreement I'm be
> > able to
> > >>>>>>> pack
> > >>>>>>>>>> both RPM and DEB identically.
> > >>>>>>>>>>
> > >>>>>>>>>> Yet, if you insist, I can create DEB package according to
> > current
> > >>> RPM
> > >>>>>>>>>> layout in no time.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> > >>> [hidden email]>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Peter,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I don't think the package size of 280M is going to be a
> > problem at
> > >>>>>>> all,
> > >>>>>>>>>> but
> > >>>>>>>>>>> what you are suggesting can be an improvement down the road.
> > >>>>>>>>>>>
> > >>>>>>>>>>> In the mean time, I think our top priority should be to
> provide
> > >>>>>>> packages
> > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly enough.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Agree?
> > >>>>>>>>>>>
> > >>>>>>>>>>> D.
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> [hidden email]>
> > >>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Igniters!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it, so
> > I'd
> > >>>>>> like
> > >>>>>>> to
> > >>>>>>>>>>>> move
> > >>>>>>>>>>>> forward to further improve and widen AI delivery through
> > >>> packages.
> > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing about
> > >>> 280M+
> > >>>>>>> and,
> > >>>>>>>>>> to
> > >>>>>>>>>>>> improve usability and significantly reduce required download
> > >>>>>> sizes, I
> > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted delivery
> as
> > >>>>>>> follows:
> > >>>>>>>>>>>> - CORE
> > >>>>>>>>>>>> - bin
> > >>>>>>>>>>>> - config
> > >>>>>>>>>>>> - libs (!optional)
> > >>>>>>>>>>>> - OPTIONAL LIBS
> > >>>>>>>>>>>> - BENCHMARKS
> > >>>>>>>>>>>> - DOCS (?)
> > >>>>>>>>>>>> - EXAMPLES
> > >>>>>>>>>>>> - .NET PLATFORM FILES
> > >>>>>>>>>>>> - C++ PLATFORM FILES
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This architecture, as I assume, will add flexibility (no
> > reason
> > >>> to
> > >>>>>>>>>> download
> > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core node
> > >>>>>>> functionality)
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>> maintainability (you are in full control of what is
> installed
> > on
> > >>>>>> your
> > >>>>>>>>>>>> system).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> After successful architecture choice, same scheme are
> planned
> > to
> > >>> be
> > >>>>>>>>>> used in
> > >>>>>>>>>>>> DEB packages as well.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Sent from: http://apache-ignite-
> developers.2346864.n4.nabble.
> > >>> com/
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>>
> > >
> >
> >
>

vveider

Re: Apache Ignite RPM packaging (Stage II)

In reply to this post by Ilya Kasnacheev

On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <[hidden email]>
wrote:

> Hello! I have tried your RPM scripts.
>
> First of all, I'm not sure that apache-ignite-core is a good name for
> package which contains the actual server node kit, and that apache-ignite
> is a good name for a package that will install everything (including cpp
> bindings). How does other packages solve this naming? I would go with
> apache-ignite and apache-ignite-full.

Previous package was named ‘apache-ignite’ and installed everything.
Current package is named ‘apache-ignite’ and installs everything (via
dependencies). I see convenience in this scheme.

‘apache-ignite-core’ name was selected because it is a core - minimum
required files to start the service. But I am open to suggestions
(-service, -node, etc.)

>
> Moreover I did not find a way to start service if default installed JVM is
> Java 7 :( I understand it's EOL, still this is something that hit me.

Apache Ignite >=2.4 does not support Java 7 - it is said in documentation,
DEVNOTES and even in startup scripts.

>
> apache-ignite-docs only contains scaladoc for scalar. Who would need a
> separate package for that? Why no javadocs? Maybe it's my build problem?

‘apache-ignite-docs’ contains both Javadoc and Scalardoc if sources were
built correctly

>
> apache-ignite-libs is a totally unexpected package name. apache-ignite core
> doesn't depend on it. It doesn't enable anything out of the box. The
> package is huge.

‘apache-ignite-libs’ is an aggregation package (for now) for all optional
libs we are delivering. Possibly later they will be split more granular or
even package per lib (like php, perl, python, etc. do for their libs).
This package dependency on ‘apache-ignite-core’ may seem confusing though,
I will try to explain it in IEP at least for current iteration.

Further naming may become clear when we’ll start initiative on including
packages to popular Linux distributions and theirs community will join
naming discussions.

>
> Frankly speaking, I'm not sure that improvements over Stage I are enough as
> of now. For demo-like activity, we can probably go with one package fits
> all.
>

The process of finding the best package architecture is iterative, but
previously community agreed in split design proposed for 2.5 release.

Also, split architecture is half of proposed improvements. The other half -
new process for deploying packages to Bintray (with virtually indefinite
storage capabilities).

>
> --
> Ilya Kasnacheev
>
> 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
>
> > If someone from PMCы or Committers still sees necessity about including
> > these tasks into Apache Ignite 2.5 release, this is the last chance to do
> > so.
> > Otherwise this task will be moved to at 2.6 release at least, or even
> > moved to backlog indefinitely.
> >
> >
> >
> > > On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]> wrote:
> > >
> > > To top new RPM architecture off, update to release process is
> introduced
> > — [1] [2].
> > >
> > > Both tasks (this one and IGNITE-7647) are ready for review and should
> be
> > merged simultaneously.
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > > [2] https://github.com/apache/ignite-release/pull/1
> > >
> > >
> > >
> > >
> > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <[hidden email]>
> > wrote:
> > >>
> > >> Hello!
> > >>
> > >> Let me share my idea of how this shoud work. Splitting package into
> > >> sub-packages should be dependency-driven.
> > >>
> > >> It means that all Ignite modules without dependencies or with small
> > >> dependencies (such as ignite-log4j) should be included in ignite-core.
> > It
> > >> doesn't make sense to make a zillion RPM packages.
> > >>
> > >> Critical things like ignite-spring and ignite-indexing should be in
> > >> ignite-core of course, even if they have dependencies. Ignite-core
> > should
> > >> be fully self-sufficient and feature-complete.
> > >>
> > >> However, e.g. .net API should probably be in a separate package,
> > because it
> > >> should depend on mono | net-core. We may also have ignite-devel
> package
> > >> which should include all modules which only make sense for developers
> > who
> > >> write code. Such as hibernate integration.
> > >>
> > >> I'm not sure about MR modules. The main question should be, does it
> have
> > >> dependencies? Can it run stand-alone without writing code?
> > >>
> > >> Hope this helps,
> > >>
> > >> --
> > >> Ilya Kasnacheev
> > >>
> > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > >>
> > >>> Hi, Igniters!
> > >>>
> > >>>
> > >>> Here are some news on our RPM packages initiative.
> > >>>
> > >>> 1. I’ve finished preliminary developing of Stage II version of RPM
> > >>> packages [1]. Main “new feature” is — split design. Also I’ve added
> > >>> package.sh script for automating package building process which will
> > help
> > >>> organise corresponding builds in TC as well as simplify process for
> > >>> developers who wishes to have custom packages.
> > >>> PR#3703 [2] is ready for review. Denis, in order to catch up with
> > Apache
> > >>> Ignite 2.5 release, I’d greatly appreciate your help in finding
> > reviewer.
> > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4]
> > >>> repositories on Apache Bintray. Though they are already prepared for
> > >>> hosting RPM and DEB packages respectively, and there is a way of
> > linking
> > >>> them to apache.org/dist/ignite page, there is possible alternative
> in
> > >>> storing there only plain directory layout corresponding to each
> > repository
> > >>> type (RPM and DEB) and manage this layout (repodata, distributions,
> > >>> versions, etc.) by ourselves, having more control over repositories
> but
> > >>> lacking some simplicity of deploying new releases. WDYT? Should we
> try
> > >>> Cassandra approach? They are storing their DEB packages as I
> described
> > >>> above [5].
> > >>>
> > >>> Also — a question arose while I was working on this issue: which OSes
> > (and
> > >>> which versions of each) are we going to support (if we are going) in
> > terms
> > >>> of step-by-step list? Currently RPM packages are tested only with
> > latest
> > >>> CentOS (and, respectively — RHEL), but there are a lot more RPM-based
> > >>> distributives [6] some of which are more o less popular among OS
> > community
> > >>> (ALT, Fedora, openSUSE, etc.).
> > >>>
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > >>> [2] https://github.com/apache/ignite/pull/3703
> > >>> [3] https://bintray.com/apache/ignite-rpm
> > >>> [4] https://bintray.com/apache/ignite-deb
> > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > distributions
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]> wrote:
> > >>>>
> > >>>> I suppose that most everything if not all from libs/options will go
> to
> > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > >>>> More precise lib selection (if something from optional would better
> to
> > >>> have in core package) will be discussed right after preliminary split
> > >>> architecture agreement.
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <[hidden email]>
> > wrote:
> > >>>>>
> > >>>>> I like idea of keeping simple system of modules, so +1 from me.
> > >>>>>
> > >>>>> Where optional libs (e.g Direct IO plugin) would be included, would
> > it
> > >>> be
> > >>>>> core or optional?
> > >>>>>
> > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]>:
> > >>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> How big would be a final core module?
> > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and create
> > it’s
> > >>> own
> > >>>>>>> package.
> > >>>>>>
> > >>>>>>
> > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with Petr
> > and
> > >>>>>> propose the simplest modular system:
> > >>>>>>
> > >>>>>> - core module that includes basic Ignite capabilities including
> SQL,
> > >>>>>> compute grid, service grid, k/v
> > >>>>>> - optional module hosts the rest - ML, streamers integration
> (kafka,
> > >>>>>> flink), kubernetes, etc.
> > >>>>>>
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> --
> > >>>>>> Denis
> > >>>>>>
> > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> [hidden email]>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> *DEB package
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <[hidden email]>
> > wrote:
> > >>>>>>>>
> > >>>>>>>> Considering that DEV package for now is almost platform
> > independent
> > >>>>>> (its
> > >>>>>>> a java application more or less), that package will work almost
> on
> > any
> > >>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian,
> etc.
> > >>>>>>>> The only restriction is existence of systemctl (systemd) service
> > >>>>>> manager
> > >>>>>>> — we are dependent on it.
> > >>>>>>>>
> > >>>>>>>> Thats why, for instance, our RPM repository is called simply
> ‘rpm’
> > >>> and
> > >>>>>>> package has no arch or dist suffix — it will work on CentOS,
> RHEL,
> > >>>>>> Fedora,
> > >>>>>>> etc. with presence of aforementioned systemd.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> > [hidden email]>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Will Debian package work for Ubuntu?
> > >>>>>>>>>
> > >>>>>>>>> D.
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> > [hidden email]>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to
> > official
> > >>>>>>>>>> repositories, there can be a problem from OS community.
> > >>>>>>>>>>
> > >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for DEB
> > package
> > >>>>>>> build
> > >>>>>>>>>> (package layout / install scripts) for speeding up things and
> > >>>>>> excluding
> > >>>>>>>>>> possible duplication and desynchronisation, so its a matter of
> > ’sit
> > >>>>>>> and do’
> > >>>>>>>>>> rather then some technical research. Thats why I rose
> discussion
> > >>>>>> about
> > >>>>>>>>>> future package architecture, so that after agreement I'm be
> > able to
> > >>>>>>> pack
> > >>>>>>>>>> both RPM and DEB identically.
> > >>>>>>>>>>
> > >>>>>>>>>> Yet, if you insist, I can create DEB package according to
> > current
> > >>> RPM
> > >>>>>>>>>> layout in no time.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> > >>> [hidden email]>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Peter,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I don't think the package size of 280M is going to be a
> > problem at
> > >>>>>>> all,
> > >>>>>>>>>> but
> > >>>>>>>>>>> what you are suggesting can be an improvement down the road.
> > >>>>>>>>>>>
> > >>>>>>>>>>> In the mean time, I think our top priority should be to
> provide
> > >>>>>>> packages
> > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly enough.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Agree?
> > >>>>>>>>>>>
> > >>>>>>>>>>> D.
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> [hidden email]>
> > >>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Igniters!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it, so
> > I'd
> > >>>>>> like
> > >>>>>>> to
> > >>>>>>>>>>>> move
> > >>>>>>>>>>>> forward to further improve and widen AI delivery through
> > >>> packages.
> > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing about
> > >>> 280M+
> > >>>>>>> and,
> > >>>>>>>>>> to
> > >>>>>>>>>>>> improve usability and significantly reduce required download
> > >>>>>> sizes, I
> > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted delivery
> as
> > >>>>>>> follows:
> > >>>>>>>>>>>> - CORE
> > >>>>>>>>>>>> - bin
> > >>>>>>>>>>>> - config
> > >>>>>>>>>>>> - libs (!optional)
> > >>>>>>>>>>>> - OPTIONAL LIBS
> > >>>>>>>>>>>> - BENCHMARKS
> > >>>>>>>>>>>> - DOCS (?)
> > >>>>>>>>>>>> - EXAMPLES
> > >>>>>>>>>>>> - .NET PLATFORM FILES
> > >>>>>>>>>>>> - C++ PLATFORM FILES
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This architecture, as I assume, will add flexibility (no
> > reason
> > >>> to
> > >>>>>>>>>> download
> > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core node
> > >>>>>>> functionality)
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>> maintainability (you are in full control of what is
> installed
> > on
> > >>>>>> your
> > >>>>>>>>>>>> system).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> After successful architecture choice, same scheme are
> planned
> > to
> > >>> be
> > >>>>>>>>>> used in
> > >>>>>>>>>>>> DEB packages as well.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Sent from:
> http://apache-ignite-developers.2346864.n4.nabble.
> > >>> com/
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>>
> > >
> >
> >
>

Ilya Kasnacheev

Re: Apache Ignite RPM packaging (Stage II)

2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:

> On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <[hidden email]>
> wrote:
>
> >
> > Moreover I did not find a way to start service if default installed JVM
> is
> > Java 7 :( I understand it's EOL, still this is something that hit me.
>
>
> Apache Ignite >=2.4 does not support Java 7 - it is said in documentation,
> DEVNOTES and even in startup scripts.
>
> I have Java 8 too, but I could not get service from package to start

Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
matter). Is it possible to specify it while running packaged Ignite?

>
> >
> > apache-ignite-libs is a totally unexpected package name. apache-ignite
> core
> > doesn't depend on it. It doesn't enable anything out of the box. The
> > package is huge.
>
> ‘apache-ignite-libs’ is an aggregation package (for now) for all optional
> libs we are delivering. Possibly later they will be split more granular or
> even package per lib (like php, perl, python, etc. do for their libs).
> This package dependency on ‘apache-ignite-core’ may seem confusing though,
> I will try to explain it in IEP at least for current iteration.
>

Okay, but how do you add optional libs to be included into Ignite classpath
while being launched by service? Is it even possible? If it isn't, I think
it doesn't make sense to ship apache-ignite-libs at all.

>
> Further naming may become clear when we’ll start initiative on including
> packages to popular Linux distributions and theirs community will join
> naming discussions.
>
Renaming packages once they're deployed widely will be a pain point to out
users. Some things should probably be thought out in advance.

>
>
>
> >
> > Frankly speaking, I'm not sure that improvements over Stage I are enough
> as
> > of now. For demo-like activity, we can probably go with one package fits
> > all.
> >
>
> The process of finding the best package architecture is iterative, but
> previously community agreed in split design proposed for 2.5 release.
>
> Also, split architecture is half of proposed improvements. The other half -
> new process for deploying packages to Bintray (with virtually indefinite
> storage capabilities).
>

I think we could drop the split for now, or at least drop
apache-ignite-libs package at all. Probably also drop apache-ignite-cpp
package and maybe apache-ignite-benchmarks.

The point of a package is to ship something into root file system that can
be used from root file system. If cpp files require compilation we should
not ship them, or ship them to 'examples'. Ditto with benchmarks. If
there's no mechanism to add optional libs to Ignite classpath, we should
not ship optional libs. Moreover, some of 'optional' modules such as yarn
don't make sense here because they're not supposed to be used with
standalone Ignite.

IMO it is not right to try and shove every file from Ignite distribution
into some package. We should only put in packages things that can be used.
If something can't be used without copying it to a different FS location,
it should be in examples or not packaged at all.

In my opinion, it doesn't make sense to implement an underwhelming package
split right now just because we have agreed to have *some* package split in
2.5. Let's aim for happiness.

>
>
>
> >
> > --
> > Ilya Kasnacheev
>

> >
> > 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> >
> > > If someone from PMCы or Committers still sees necessity about including
> > > these tasks into Apache Ignite 2.5 release, this is the last chance to
> do
> > > so.
> > > Otherwise this task will be moved to at 2.6 release at least, or even
> > > moved to backlog indefinitely.
> > >
> > >
> > >
> > > > On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]> wrote:
> > > >
> > > > To top new RPM architecture off, update to release process is
> > introduced
> > > — [1] [2].
> > > >
> > > > Both tasks (this one and IGNITE-7647) are ready for review and should
> > be
> > > merged simultaneously.
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > > > [2] https://github.com/apache/ignite-release/pull/1
> > > >
> > > >
> > > >
> > > >
> > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <[hidden email]
> >
> > > wrote:
> > > >>
> > > >> Hello!
> > > >>
> > > >> Let me share my idea of how this shoud work. Splitting package into
> > > >> sub-packages should be dependency-driven.
> > > >>
> > > >> It means that all Ignite modules without dependencies or with small
> > > >> dependencies (such as ignite-log4j) should be included in
> ignite-core.
> > > It
> > > >> doesn't make sense to make a zillion RPM packages.
> > > >>
> > > >> Critical things like ignite-spring and ignite-indexing should be in
> > > >> ignite-core of course, even if they have dependencies. Ignite-core
> > > should
> > > >> be fully self-sufficient and feature-complete.
> > > >>
> > > >> However, e.g. .net API should probably be in a separate package,
> > > because it
> > > >> should depend on mono | net-core. We may also have ignite-devel
> > package
> > > >> which should include all modules which only make sense for
> developers
> > > who
> > > >> write code. Such as hibernate integration.
> > > >>
> > > >> I'm not sure about MR modules. The main question should be, does it
> > have
> > > >> dependencies? Can it run stand-alone without writing code?
> > > >>
> > > >> Hope this helps,
> > > >>
> > > >> --
> > > >> Ilya Kasnacheev
> > > >>
> > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > > >>
> > > >>> Hi, Igniters!
> > > >>>
> > > >>>
> > > >>> Here are some news on our RPM packages initiative.
> > > >>>
> > > >>> 1. I’ve finished preliminary developing of Stage II version of RPM
> > > >>> packages [1]. Main “new feature” is — split design. Also I’ve added
> > > >>> package.sh script for automating package building process which
> will
> > > help
> > > >>> organise corresponding builds in TC as well as simplify process for
> > > >>> developers who wishes to have custom packages.
> > > >>> PR#3703 [2] is ready for review. Denis, in order to catch up with
> > > Apache
> > > >>> Ignite 2.5 release, I’d greatly appreciate your help in finding
> > > reviewer.
> > > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4]
> > > >>> repositories on Apache Bintray. Though they are already prepared
> for
> > > >>> hosting RPM and DEB packages respectively, and there is a way of
> > > linking
> > > >>> them to apache.org/dist/ignite page, there is possible alternative
> > in
> > > >>> storing there only plain directory layout corresponding to each
> > > repository
> > > >>> type (RPM and DEB) and manage this layout (repodata, distributions,
> > > >>> versions, etc.) by ourselves, having more control over repositories
> > but
> > > >>> lacking some simplicity of deploying new releases. WDYT? Should we
> > try
> > > >>> Cassandra approach? They are storing their DEB packages as I
> > described
> > > >>> above [5].
> > > >>>
> > > >>> Also — a question arose while I was working on this issue: which
> OSes
> > > (and
> > > >>> which versions of each) are we going to support (if we are going)
> in
> > > terms
> > > >>> of step-by-step list? Currently RPM packages are tested only with
> > > latest
> > > >>> CentOS (and, respectively — RHEL), but there are a lot more
> RPM-based
> > > >>> distributives [6] some of which are more o less popular among OS
> > > community
> > > >>> (ALT, Fedora, openSUSE, etc.).
> > > >>>
> > > >>>
> > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > > >>> [2] https://github.com/apache/ignite/pull/3703
> > > >>> [3] https://bintray.com/apache/ignite-rpm
> > > >>> [4] https://bintray.com/apache/ignite-deb
> > > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > > distributions
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]>
> wrote:
> > > >>>>
> > > >>>> I suppose that most everything if not all from libs/options will
> go
> > to
> > > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > > >>>> More precise lib selection (if something from optional would
> better
> > to
> > > >>> have in core package) will be discussed right after preliminary
> split
> > > >>> architecture agreement.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <[hidden email]>
> > > wrote:
> > > >>>>>
> > > >>>>> I like idea of keeping simple system of modules, so +1 from me.
> > > >>>>>
> > > >>>>> Where optional libs (e.g Direct IO plugin) would be included,
> would
> > > it
> > > >>> be
> > > >>>>> core or optional?
> > > >>>>>
> > > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]>:
> > > >>>>>
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> How big would be a final core module?
> > > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and
> create
> > > it’s
> > > >>> own
> > > >>>>>>> package.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with
> Petr
> > > and
> > > >>>>>> propose the simplest modular system:
> > > >>>>>>
> > > >>>>>> - core module that includes basic Ignite capabilities including
> > SQL,
> > > >>>>>> compute grid, service grid, k/v
> > > >>>>>> - optional module hosts the rest - ML, streamers integration
> > (kafka,
> > > >>>>>> flink), kubernetes, etc.
> > > >>>>>>
> > > >>>>>> What do you think?
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Denis
> > > >>>>>>
> > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> > [hidden email]>
> > > >>> wrote:
> > > >>>>>>
> > > >>>>>>> *DEB package
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <[hidden email]>
> > > wrote:
> > > >>>>>>>>
> > > >>>>>>>> Considering that DEV package for now is almost platform
> > > independent
> > > >>>>>> (its
> > > >>>>>>> a java application more or less), that package will work almost
> > on
> > > any
> > > >>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian,
> > etc.
> > > >>>>>>>> The only restriction is existence of systemctl (systemd)
> service
> > > >>>>>> manager
> > > >>>>>>> — we are dependent on it.
> > > >>>>>>>>
> > > >>>>>>>> Thats why, for instance, our RPM repository is called simply
> > ‘rpm’
> > > >>> and
> > > >>>>>>> package has no arch or dist suffix — it will work on CentOS,
> > RHEL,
> > > >>>>>> Fedora,
> > > >>>>>>> etc. with presence of aforementioned systemd.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> > > [hidden email]>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> Will Debian package work for Ubuntu?
> > > >>>>>>>>>
> > > >>>>>>>>> D.
> > > >>>>>>>>>
> > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> > > [hidden email]>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to
> > > official
> > > >>>>>>>>>> repositories, there can be a problem from OS community.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for DEB
> > > package
> > > >>>>>>> build
> > > >>>>>>>>>> (package layout / install scripts) for speeding up things
> and
> > > >>>>>> excluding
> > > >>>>>>>>>> possible duplication and desynchronisation, so its a matter
> of
> > > ’sit
> > > >>>>>>> and do’
> > > >>>>>>>>>> rather then some technical research. Thats why I rose
> > discussion
> > > >>>>>> about
> > > >>>>>>>>>> future package architecture, so that after agreement I'm be
> > > able to
> > > >>>>>>> pack
> > > >>>>>>>>>> both RPM and DEB identically.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Yet, if you insist, I can create DEB package according to
> > > current
> > > >>> RPM
> > > >>>>>>>>>> layout in no time.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> > > >>> [hidden email]>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Peter,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I don't think the package size of 280M is going to be a
> > > problem at
> > > >>>>>>> all,
> > > >>>>>>>>>> but
> > > >>>>>>>>>>> what you are suggesting can be an improvement down the
> road.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> In the mean time, I think our top priority should be to
> > provide
> > > >>>>>>> packages
> > > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly
> enough.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Agree?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> D.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> > [hidden email]>
> > > >>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi, Igniters!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it,
> so
> > > I'd
> > > >>>>>> like
> > > >>>>>>> to
> > > >>>>>>>>>>>> move
> > > >>>>>>>>>>>> forward to further improve and widen AI delivery through
> > > >>> packages.
> > > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing
> about
> > > >>> 280M+
> > > >>>>>>> and,
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> improve usability and significantly reduce required
> download
> > > >>>>>> sizes, I
> > > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted delivery
> > as
> > > >>>>>>> follows:
> > > >>>>>>>>>>>> - CORE
> > > >>>>>>>>>>>> - bin
> > > >>>>>>>>>>>> - config
> > > >>>>>>>>>>>> - libs (!optional)
> > > >>>>>>>>>>>> - OPTIONAL LIBS
> > > >>>>>>>>>>>> - BENCHMARKS
> > > >>>>>>>>>>>> - DOCS (?)
> > > >>>>>>>>>>>> - EXAMPLES
> > > >>>>>>>>>>>> - .NET PLATFORM FILES
> > > >>>>>>>>>>>> - C++ PLATFORM FILES
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> This architecture, as I assume, will add flexibility (no
> > > reason
> > > >>> to
> > > >>>>>>>>>> download
> > > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core node
> > > >>>>>>> functionality)
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>> maintainability (you are in full control of what is
> > installed
> > > on
> > > >>>>>> your
> > > >>>>>>>>>>>> system).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> After successful architecture choice, same scheme are
> > planned
> > > to
> > > >>> be
> > > >>>>>>>>>> used in
> > > >>>>>>>>>>>> DEB packages as well.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> --
> > > >>>>>>>>>>>> Sent from:
> > http://apache-ignite-developers.2346864.n4.nabble.
> > > >>> com/
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >
> > >
> > >
> >
>

dmagda

Re: Apache Ignite RPM packaging (Stage II)

Ilya,

Thanks for your inputs. The reason why we decided to split Ignite into
several packages mimics the reason why Java community introduced modular
subsystem for JDK. That's all about size. Ignite distribution is too big,
and we're trying to separate it into several components so that people can
install only the features they need.

The point of a package is to ship something into root file system that can
> be used from root file system. If cpp files require compilation we should
> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> there's no mechanism to add optional libs to Ignite classpath, we should
> not ship optional libs. Moreover, some of 'optional' modules such as yarn
> don't make sense here because they're not supposed to be used with
> standalone Ignite.

Agree that we need to ship the code that is ready to be run. As for the
classpath thing, if an optional package is installed into the root (core)
package directory, then its jars have to be added to "ignite/libs" folder.
After that, the one needs to restart a cluster node, nd it will add the
just installed optional libs to the classpath. *Petr*, does it work this
way or can be implemented this way to address Ilya's concerns?

--
Denis

On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <[hidden email]>
wrote:

> 2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:
>
> > On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <[hidden email]
> >
> > wrote:
> >
> > >
> > > Moreover I did not find a way to start service if default installed JVM
> > is
> > > Java 7 :( I understand it's EOL, still this is something that hit me.
> >
> >
> > Apache Ignite >=2.4 does not support Java 7 - it is said in
> documentation,
> > DEVNOTES and even in startup scripts.
> >
> > I have Java 8 too, but I could not get service from package to start
> Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
> matter). Is it possible to specify it while running packaged Ignite?
>
>
> >
> > >
> > > apache-ignite-libs is a totally unexpected package name. apache-ignite
> > core
> > > doesn't depend on it. It doesn't enable anything out of the box. The
> > > package is huge.
> >
> > ‘apache-ignite-libs’ is an aggregation package (for now) for all optional
> > libs we are delivering. Possibly later they will be split more granular
> or
> > even package per lib (like php, perl, python, etc. do for their libs).
> > This package dependency on ‘apache-ignite-core’ may seem confusing
> though,
> > I will try to explain it in IEP at least for current iteration.
> >
>
> Okay, but how do you add optional libs to be included into Ignite classpath
> while being launched by service? Is it even possible? If it isn't, I think
> it doesn't make sense to ship apache-ignite-libs at all.
>
>
> >
> > Further naming may become clear when we’ll start initiative on including
> > packages to popular Linux distributions and theirs community will join
> > naming discussions.
> >
> Renaming packages once they're deployed widely will be a pain point to out
> users. Some things should probably be thought out in advance.
>
>
> >
> >
> >
> > >
> > > Frankly speaking, I'm not sure that improvements over Stage I are
> enough
> > as
> > > of now. For demo-like activity, we can probably go with one package
> fits
> > > all.
> > >
> >
> > The process of finding the best package architecture is iterative, but
> > previously community agreed in split design proposed for 2.5 release.
> >
> > Also, split architecture is half of proposed improvements. The other
> half -
> > new process for deploying packages to Bintray (with virtually indefinite
> > storage capabilities).
> >
> I think we could drop the split for now, or at least drop
> apache-ignite-libs package at all. Probably also drop apache-ignite-cpp
> package and maybe apache-ignite-benchmarks.
>
> The point of a package is to ship something into root file system that can
> be used from root file system. If cpp files require compilation we should
> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> there's no mechanism to add optional libs to Ignite classpath, we should
> not ship optional libs. Moreover, some of 'optional' modules such as yarn
> don't make sense here because they're not supposed to be used with
> standalone Ignite.
>
> IMO it is not right to try and shove every file from Ignite distribution
> into some package. We should only put in packages things that can be used.
> If something can't be used without copying it to a different FS location,
> it should be in examples or not packaged at all.
>
> In my opinion, it doesn't make sense to implement an underwhelming package
> split right now just because we have agreed to have *some* package split in
> 2.5. Let's aim for happiness.
>
>
> >
> >
> >
> > >
> > > --
> > > Ilya Kasnacheev
> >
>
>
>
> > >
> > > 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > >
> > > > If someone from PMCы or Committers still sees necessity about
> including
> > > > these tasks into Apache Ignite 2.5 release, this is the last chance
> to
> > do
> > > > so.
> > > > Otherwise this task will be moved to at 2.6 release at least, or even
> > > > moved to backlog indefinitely.
> > > >
> > > >
> > > >
> > > > > On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]> wrote:
> > > > >
> > > > > To top new RPM architecture off, update to release process is
> > > introduced
> > > > — [1] [2].
> > > > >
> > > > > Both tasks (this one and IGNITE-7647) are ready for review and
> should
> > > be
> > > > merged simultaneously.
> > > > >
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > > > > [2] https://github.com/apache/ignite-release/pull/1
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
> [hidden email]
> > >
> > > > wrote:
> > > > >>
> > > > >> Hello!
> > > > >>
> > > > >> Let me share my idea of how this shoud work. Splitting package
> into
> > > > >> sub-packages should be dependency-driven.
> > > > >>
> > > > >> It means that all Ignite modules without dependencies or with
> small
> > > > >> dependencies (such as ignite-log4j) should be included in
> > ignite-core.
> > > > It
> > > > >> doesn't make sense to make a zillion RPM packages.
> > > > >>
> > > > >> Critical things like ignite-spring and ignite-indexing should be
> in
> > > > >> ignite-core of course, even if they have dependencies. Ignite-core
> > > > should
> > > > >> be fully self-sufficient and feature-complete.
> > > > >>
> > > > >> However, e.g. .net API should probably be in a separate package,
> > > > because it
> > > > >> should depend on mono | net-core. We may also have ignite-devel
> > > package
> > > > >> which should include all modules which only make sense for
> > developers
> > > > who
> > > > >> write code. Such as hibernate integration.
> > > > >>
> > > > >> I'm not sure about MR modules. The main question should be, does
> it
> > > have
> > > > >> dependencies? Can it run stand-alone without writing code?
> > > > >>
> > > > >> Hope this helps,
> > > > >>
> > > > >> --
> > > > >> Ilya Kasnacheev
> > > > >>
> > > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > > > >>
> > > > >>> Hi, Igniters!
> > > > >>>
> > > > >>>
> > > > >>> Here are some news on our RPM packages initiative.
> > > > >>>
> > > > >>> 1. I’ve finished preliminary developing of Stage II version of
> RPM
> > > > >>> packages [1]. Main “new feature” is — split design. Also I’ve
> added
> > > > >>> package.sh script for automating package building process which
> > will
> > > > help
> > > > >>> organise corresponding builds in TC as well as simplify process
> for
> > > > >>> developers who wishes to have custom packages.
> > > > >>> PR#3703 [2] is ready for review. Denis, in order to catch up with
> > > > Apache
> > > > >>> Ignite 2.5 release, I’d greatly appreciate your help in finding
> > > > reviewer.
> > > > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB
> [4]
> > > > >>> repositories on Apache Bintray. Though they are already prepared
> > for
> > > > >>> hosting RPM and DEB packages respectively, and there is a way of
> > > > linking
> > > > >>> them to apache.org/dist/ignite page, there is possible
> alternative
> > > in
> > > > >>> storing there only plain directory layout corresponding to each
> > > > repository
> > > > >>> type (RPM and DEB) and manage this layout (repodata,
> distributions,
> > > > >>> versions, etc.) by ourselves, having more control over
> repositories
> > > but
> > > > >>> lacking some simplicity of deploying new releases. WDYT? Should
> we
> > > try
> > > > >>> Cassandra approach? They are storing their DEB packages as I
> > > described
> > > > >>> above [5].
> > > > >>>
> > > > >>> Also — a question arose while I was working on this issue: which
> > OSes
> > > > (and
> > > > >>> which versions of each) are we going to support (if we are going)
> > in
> > > > terms
> > > > >>> of step-by-step list? Currently RPM packages are tested only with
> > > > latest
> > > > >>> CentOS (and, respectively — RHEL), but there are a lot more
> > RPM-based
> > > > >>> distributives [6] some of which are more o less popular among OS
> > > > community
> > > > >>> (ALT, Fedora, openSUSE, etc.).
> > > > >>>
> > > > >>>
> > > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > > > >>> [2] https://github.com/apache/ignite/pull/3703
> > > > >>> [3] https://bintray.com/apache/ignite-rpm
> > > > >>> [4] https://bintray.com/apache/ignite-deb
> > > > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > > > distributions
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]>
> > wrote:
> > > > >>>>
> > > > >>>> I suppose that most everything if not all from libs/options will
> > go
> > > to
> > > > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > > > >>>> More precise lib selection (if something from optional would
> > better
> > > to
> > > > >>> have in core package) will be discussed right after preliminary
> > split
> > > > >>> architecture agreement.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <[hidden email]
> >
> > > > wrote:
> > > > >>>>>
> > > > >>>>> I like idea of keeping simple system of modules, so +1 from me.
> > > > >>>>>
> > > > >>>>> Where optional libs (e.g Direct IO plugin) would be included,
> > would
> > > > it
> > > > >>> be
> > > > >>>>> core or optional?
> > > > >>>>>
> > > > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]>:
> > > > >>>>>
> > > > >>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> How big would be a final core module?
> > > > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and
> > create
> > > > it’s
> > > > >>> own
> > > > >>>>>>> package.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with
> > Petr
> > > > and
> > > > >>>>>> propose the simplest modular system:
> > > > >>>>>>
> > > > >>>>>> - core module that includes basic Ignite capabilities
> including
> > > SQL,
> > > > >>>>>> compute grid, service grid, k/v
> > > > >>>>>> - optional module hosts the rest - ML, streamers integration
> > > (kafka,
> > > > >>>>>> flink), kubernetes, etc.
> > > > >>>>>>
> > > > >>>>>> What do you think?
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> Denis
> > > > >>>>>>
> > > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> > > [hidden email]>
> > > > >>> wrote:
> > > > >>>>>>
> > > > >>>>>>> *DEB package
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <[hidden email]>
> > > > wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> Considering that DEV package for now is almost platform
> > > > independent
> > > > >>>>>> (its
> > > > >>>>>>> a java application more or less), that package will work
> almost
> > > on
> > > > any
> > > > >>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian,
> > > etc.
> > > > >>>>>>>> The only restriction is existence of systemctl (systemd)
> > service
> > > > >>>>>> manager
> > > > >>>>>>> — we are dependent on it.
> > > > >>>>>>>>
> > > > >>>>>>>> Thats why, for instance, our RPM repository is called simply
> > > ‘rpm’
> > > > >>> and
> > > > >>>>>>> package has no arch or dist suffix — it will work on CentOS,
> > > RHEL,
> > > > >>>>>> Fedora,
> > > > >>>>>>> etc. with presence of aforementioned systemd.
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> > > > [hidden email]>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>> Will Debian package work for Ubuntu?
> > > > >>>>>>>>>
> > > > >>>>>>>>> D.
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> > > > [hidden email]>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to
> > > > official
> > > > >>>>>>>>>> repositories, there can be a problem from OS community.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for
> DEB
> > > > package
> > > > >>>>>>> build
> > > > >>>>>>>>>> (package layout / install scripts) for speeding up things
> > and
> > > > >>>>>> excluding
> > > > >>>>>>>>>> possible duplication and desynchronisation, so its a
> matter
> > of
> > > > ’sit
> > > > >>>>>>> and do’
> > > > >>>>>>>>>> rather then some technical research. Thats why I rose
> > > discussion
> > > > >>>>>> about
> > > > >>>>>>>>>> future package architecture, so that after agreement I'm
> be
> > > > able to
> > > > >>>>>>> pack
> > > > >>>>>>>>>> both RPM and DEB identically.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Yet, if you insist, I can create DEB package according to
> > > > current
> > > > >>> RPM
> > > > >>>>>>>>>> layout in no time.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> > > > >>> [hidden email]>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Peter,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I don't think the package size of 280M is going to be a
> > > > problem at
> > > > >>>>>>> all,
> > > > >>>>>>>>>> but
> > > > >>>>>>>>>>> what you are suggesting can be an improvement down the
> > road.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> In the mean time, I think our top priority should be to
> > > provide
> > > > >>>>>>> packages
> > > > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly
> > enough.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Agree?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> D.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> > > [hidden email]>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi, Igniters!
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it,
> > so
> > > > I'd
> > > > >>>>>> like
> > > > >>>>>>> to
> > > > >>>>>>>>>>>> move
> > > > >>>>>>>>>>>> forward to further improve and widen AI delivery through
> > > > >>> packages.
> > > > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing
> > about
> > > > >>> 280M+
> > > > >>>>>>> and,
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>> improve usability and significantly reduce required
> > download
> > > > >>>>>> sizes, I
> > > > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted
> delivery
> > > as
> > > > >>>>>>> follows:
> > > > >>>>>>>>>>>> - CORE
> > > > >>>>>>>>>>>> - bin
> > > > >>>>>>>>>>>> - config
> > > > >>>>>>>>>>>> - libs (!optional)
> > > > >>>>>>>>>>>> - OPTIONAL LIBS
> > > > >>>>>>>>>>>> - BENCHMARKS
> > > > >>>>>>>>>>>> - DOCS (?)
> > > > >>>>>>>>>>>> - EXAMPLES
> > > > >>>>>>>>>>>> - .NET PLATFORM FILES
> > > > >>>>>>>>>>>> - C++ PLATFORM FILES
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> This architecture, as I assume, will add flexibility (no
> > > > reason
> > > > >>> to
> > > > >>>>>>>>>> download
> > > > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core
> node
> > > > >>>>>>> functionality)
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>> maintainability (you are in full control of what is
> > > installed
> > > > on
> > > > >>>>>> your
> > > > >>>>>>>>>>>> system).
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> After successful architecture choice, same scheme are
> > > planned
> > > > to
> > > > >>> be
> > > > >>>>>>>>>> used in
> > > > >>>>>>>>>>>> DEB packages as well.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> WDYT?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> --
> > > > >>>>>>>>>>>> Sent from:
> > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > >>> com/
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >
> > > >
> > > >
> > >
> >
>

vveider

Re: Apache Ignite RPM packaging (Stage II)

Current packages design (after installation) does not differ from binary
archive - everything works (except necessity to run service instead
ignite.sh) just the same way, including libs/optional.

Also, there can be issues with system JDK version by default, but every
problem will be in journalctl and/or /var/log, and package has strong
dependency on any implementation of JDK 1.8.

I am lacking enough feedback about Apache Ignite from packages from real
users, don’t know real use cases so still "moving in the dark".

On Fri, 13 Apr 2018 at 22:18, Denis Magda <[hidden email]> wrote:

> Ilya,
>
> Thanks for your inputs. The reason why we decided to split Ignite into
> several packages mimics the reason why Java community introduced modular
> subsystem for JDK. That's all about size. Ignite distribution is too big,
> and we're trying to separate it into several components so that people can
> install only the features they need.
>
> The point of a package is to ship something into root file system that can
> > be used from root file system. If cpp files require compilation we should
> > not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> > there's no mechanism to add optional libs to Ignite classpath, we should
> > not ship optional libs. Moreover, some of 'optional' modules such as yarn
> > don't make sense here because they're not supposed to be used with
> > standalone Ignite.
>
>
> Agree that we need to ship the code that is ready to be run. As for the
> classpath thing, if an optional package is installed into the root (core)
> package directory, then its jars have to be added to "ignite/libs" folder.
> After that, the one needs to restart a cluster node, nd it will add the
> just installed optional libs to the classpath. *Petr*, does it work this
> way or can be implemented this way to address Ilya's concerns?
>
> --
> Denis
>
> On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <
> [hidden email]>
> wrote:
>
> > 2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:
> >
> > > On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <
> [hidden email]
> > >
> > > wrote:
> > >
> > > >
> > > > Moreover I did not find a way to start service if default installed
> JVM
> > > is
> > > > Java 7 :( I understand it's EOL, still this is something that hit me.
> > >
> > >
> > > Apache Ignite >=2.4 does not support Java 7 - it is said in
> > documentation,
> > > DEVNOTES and even in startup scripts.
> > >
> > > I have Java 8 too, but I could not get service from package to start
> > Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
> > matter). Is it possible to specify it while running packaged Ignite?
> >
> >
> > >
> > > >
> > > > apache-ignite-libs is a totally unexpected package name.
> apache-ignite
> > > core
> > > > doesn't depend on it. It doesn't enable anything out of the box. The
> > > > package is huge.
> > >
> > > ‘apache-ignite-libs’ is an aggregation package (for now) for all
> optional
> > > libs we are delivering. Possibly later they will be split more granular
> > or
> > > even package per lib (like php, perl, python, etc. do for their libs).
> > > This package dependency on ‘apache-ignite-core’ may seem confusing
> > though,
> > > I will try to explain it in IEP at least for current iteration.
> > >
> >
> > Okay, but how do you add optional libs to be included into Ignite
> classpath
> > while being launched by service? Is it even possible? If it isn't, I
> think
> > it doesn't make sense to ship apache-ignite-libs at all.
> >
> >
> > >
> > > Further naming may become clear when we’ll start initiative on
> including
> > > packages to popular Linux distributions and theirs community will join
> > > naming discussions.
> > >
> > Renaming packages once they're deployed widely will be a pain point to
> out
> > users. Some things should probably be thought out in advance.
> >
> >
> > >
> > >
> > >
> > > >
> > > > Frankly speaking, I'm not sure that improvements over Stage I are
> > enough
> > > as
> > > > of now. For demo-like activity, we can probably go with one package
> > fits
> > > > all.
> > > >
> > >
> > > The process of finding the best package architecture is iterative, but
> > > previously community agreed in split design proposed for 2.5 release.
> > >
> > > Also, split architecture is half of proposed improvements. The other
> > half -
> > > new process for deploying packages to Bintray (with virtually
> indefinite
> > > storage capabilities).
> > >
> > I think we could drop the split for now, or at least drop
> > apache-ignite-libs package at all. Probably also drop apache-ignite-cpp
> > package and maybe apache-ignite-benchmarks.
> >
> > The point of a package is to ship something into root file system that
> can
> > be used from root file system. If cpp files require compilation we should
> > not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> > there's no mechanism to add optional libs to Ignite classpath, we should
> > not ship optional libs. Moreover, some of 'optional' modules such as yarn
> > don't make sense here because they're not supposed to be used with
> > standalone Ignite.
> >
> > IMO it is not right to try and shove every file from Ignite distribution
> > into some package. We should only put in packages things that can be
> used.
> > If something can't be used without copying it to a different FS location,
> > it should be in examples or not packaged at all.
> >
> > In my opinion, it doesn't make sense to implement an underwhelming
> package
> > split right now just because we have agreed to have *some* package split
> in
> > 2.5. Let's aim for happiness.
> >
> >
> > >
> > >
> > >
> > > >
> > > > --
> > > > Ilya Kasnacheev
> > >
> >
> >
> >
> > > >
> > > > 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > > >
> > > > > If someone from PMCы or Committers still sees necessity about
> > including
> > > > > these tasks into Apache Ignite 2.5 release, this is the last chance
> > to
> > > do
> > > > > so.
> > > > > Otherwise this task will be moved to at 2.6 release at least, or
> even
> > > > > moved to backlog indefinitely.
> > > > >
> > > > >
> > > > >
> > > > > > On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]>
> wrote:
> > > > > >
> > > > > > To top new RPM architecture off, update to release process is
> > > > introduced
> > > > > — [1] [2].
> > > > > >
> > > > > > Both tasks (this one and IGNITE-7647) are ready for review and
> > should
> > > > be
> > > > > merged simultaneously.
> > > > > >
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > > > > > [2] https://github.com/apache/ignite-release/pull/1
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > > >>
> > > > > >> Hello!
> > > > > >>
> > > > > >> Let me share my idea of how this shoud work. Splitting package
> > into
> > > > > >> sub-packages should be dependency-driven.
> > > > > >>
> > > > > >> It means that all Ignite modules without dependencies or with
> > small
> > > > > >> dependencies (such as ignite-log4j) should be included in
> > > ignite-core.
> > > > > It
> > > > > >> doesn't make sense to make a zillion RPM packages.
> > > > > >>
> > > > > >> Critical things like ignite-spring and ignite-indexing should be
> > in
> > > > > >> ignite-core of course, even if they have dependencies.
> Ignite-core
> > > > > should
> > > > > >> be fully self-sufficient and feature-complete.
> > > > > >>
> > > > > >> However, e.g. .net API should probably be in a separate package,
> > > > > because it
> > > > > >> should depend on mono | net-core. We may also have ignite-devel
> > > > package
> > > > > >> which should include all modules which only make sense for
> > > developers
> > > > > who
> > > > > >> write code. Such as hibernate integration.
> > > > > >>
> > > > > >> I'm not sure about MR modules. The main question should be, does
> > it
> > > > have
> > > > > >> dependencies? Can it run stand-alone without writing code?
> > > > > >>
> > > > > >> Hope this helps,
> > > > > >>
> > > > > >> --
> > > > > >> Ilya Kasnacheev
> > > > > >>
> > > > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > > > > >>
> > > > > >>> Hi, Igniters!
> > > > > >>>
> > > > > >>>
> > > > > >>> Here are some news on our RPM packages initiative.
> > > > > >>>
> > > > > >>> 1. I’ve finished preliminary developing of Stage II version of
> > RPM
> > > > > >>> packages [1]. Main “new feature” is — split design. Also I’ve
> > added
> > > > > >>> package.sh script for automating package building process which
> > > will
> > > > > help
> > > > > >>> organise corresponding builds in TC as well as simplify process
> > for
> > > > > >>> developers who wishes to have custom packages.
> > > > > >>> PR#3703 [2] is ready for review. Denis, in order to catch up
> with
> > > > > Apache
> > > > > >>> Ignite 2.5 release, I’d greatly appreciate your help in finding
> > > > > reviewer.
> > > > > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB
> > [4]
> > > > > >>> repositories on Apache Bintray. Though they are already
> prepared
> > > for
> > > > > >>> hosting RPM and DEB packages respectively, and there is a way
> of
> > > > > linking
> > > > > >>> them to apache.org/dist/ignite page, there is possible
> > alternative
> > > > in
> > > > > >>> storing there only plain directory layout corresponding to each
> > > > > repository
> > > > > >>> type (RPM and DEB) and manage this layout (repodata,
> > distributions,
> > > > > >>> versions, etc.) by ourselves, having more control over
> > repositories
> > > > but
> > > > > >>> lacking some simplicity of deploying new releases. WDYT? Should
> > we
> > > > try
> > > > > >>> Cassandra approach? They are storing their DEB packages as I
> > > > described
> > > > > >>> above [5].
> > > > > >>>
> > > > > >>> Also — a question arose while I was working on this issue:
> which
> > > OSes
> > > > > (and
> > > > > >>> which versions of each) are we going to support (if we are
> going)
> > > in
> > > > > terms
> > > > > >>> of step-by-step list? Currently RPM packages are tested only
> with
> > > > > latest
> > > > > >>> CentOS (and, respectively — RHEL), but there are a lot more
> > > RPM-based
> > > > > >>> distributives [6] some of which are more o less popular among
> OS
> > > > > community
> > > > > >>> (ALT, Fedora, openSUSE, etc.).
> > > > > >>>
> > > > > >>>
> > > > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > > > > >>> [2] https://github.com/apache/ignite/pull/3703
> > > > > >>> [3] https://bintray.com/apache/ignite-rpm
> > > > > >>> [4] https://bintray.com/apache/ignite-deb
> > > > > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > > > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > > > > distributions
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]>
> > > wrote:
> > > > > >>>>
> > > > > >>>> I suppose that most everything if not all from libs/options
> will
> > > go
> > > > to
> > > > > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > > > > >>>> More precise lib selection (if something from optional would
> > > better
> > > > to
> > > > > >>> have in core package) will be discussed right after preliminary
> > > split
> > > > > >>> architecture agreement.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <
> [hidden email]
> > >
> > > > > wrote:
> > > > > >>>>>
> > > > > >>>>> I like idea of keeping simple system of modules, so +1 from
> me.
> > > > > >>>>>
> > > > > >>>>> Where optional libs (e.g Direct IO plugin) would be included,
> > > would
> > > > > it
> > > > > >>> be
> > > > > >>>>> core or optional?
> > > > > >>>>>
> > > > > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <[hidden email]
> >:
> > > > > >>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> How big would be a final core module?
> > > > > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and
> > > create
> > > > > it’s
> > > > > >>> own
> > > > > >>>>>>> package.
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree
> with
> > > Petr
> > > > > and
> > > > > >>>>>> propose the simplest modular system:
> > > > > >>>>>>
> > > > > >>>>>> - core module that includes basic Ignite capabilities
> > including
> > > > SQL,
> > > > > >>>>>> compute grid, service grid, k/v
> > > > > >>>>>> - optional module hosts the rest - ML, streamers integration
> > > > (kafka,
> > > > > >>>>>> flink), kubernetes, etc.
> > > > > >>>>>>
> > > > > >>>>>> What do you think?
> > > > > >>>>>>
> > > > > >>>>>> --
> > > > > >>>>>> Denis
> > > > > >>>>>>
> > > > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> > > > [hidden email]>
> > > > > >>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> *DEB package
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <
> [hidden email]>
> > > > > wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Considering that DEV package for now is almost platform
> > > > > independent
> > > > > >>>>>> (its
> > > > > >>>>>>> a java application more or less), that package will work
> > almost
> > > > on
> > > > > any
> > > > > >>>>>>> DEB-based linux, including but not limited to Ubuntu,
> Debian,
> > > > etc.
> > > > > >>>>>>>> The only restriction is existence of systemctl (systemd)
> > > service
> > > > > >>>>>> manager
> > > > > >>>>>>> — we are dependent on it.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thats why, for instance, our RPM repository is called
> simply
> > > > ‘rpm’
> > > > > >>> and
> > > > > >>>>>>> package has no arch or dist suffix — it will work on
> CentOS,
> > > > RHEL,
> > > > > >>>>>> Fedora,
> > > > > >>>>>>> etc. with presence of aforementioned systemd.
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> > > > > [hidden email]>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Will Debian package work for Ubuntu?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> D.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> > > > > [hidden email]>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will move
> to
> > > > > official
> > > > > >>>>>>>>>> repositories, there can be a problem from OS community.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for
> > DEB
> > > > > package
> > > > > >>>>>>> build
> > > > > >>>>>>>>>> (package layout / install scripts) for speeding up
> things
> > > and
> > > > > >>>>>> excluding
> > > > > >>>>>>>>>> possible duplication and desynchronisation, so its a
> > matter
> > > of
> > > > > ’sit
> > > > > >>>>>>> and do’
> > > > > >>>>>>>>>> rather then some technical research. Thats why I rose
> > > > discussion
> > > > > >>>>>> about
> > > > > >>>>>>>>>> future package architecture, so that after agreement I'm
> > be
> > > > > able to
> > > > > >>>>>>> pack
> > > > > >>>>>>>>>> both RPM and DEB identically.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Yet, if you insist, I can create DEB package according
> to
> > > > > current
> > > > > >>> RPM
> > > > > >>>>>>>>>> layout in no time.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> > > > > >>> [hidden email]>
> > > > > >>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Peter,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I don't think the package size of 280M is going to be a
> > > > > problem at
> > > > > >>>>>>> all,
> > > > > >>>>>>>>>> but
> > > > > >>>>>>>>>>> what you are suggesting can be an improvement down the
> > > road.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> In the mean time, I think our top priority should be to
> > > > provide
> > > > > >>>>>>> packages
> > > > > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly
> > > enough.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Agree?
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> D.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> > > > [hidden email]>
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Hi, Igniters!
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of
> it,
> > > so
> > > > > I'd
> > > > > >>>>>> like
> > > > > >>>>>>> to
> > > > > >>>>>>>>>>>> move
> > > > > >>>>>>>>>>>> forward to further improve and widen AI delivery
> through
> > > > > >>> packages.
> > > > > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing
> > > about
> > > > > >>> 280M+
> > > > > >>>>>>> and,
> > > > > >>>>>>>>>> to
> > > > > >>>>>>>>>>>> improve usability and significantly reduce required
> > > download
> > > > > >>>>>> sizes, I
> > > > > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted
> > delivery
> > > > as
> > > > > >>>>>>> follows:
> > > > > >>>>>>>>>>>> - CORE
> > > > > >>>>>>>>>>>> - bin
> > > > > >>>>>>>>>>>> - config
> > > > > >>>>>>>>>>>> - libs (!optional)
> > > > > >>>>>>>>>>>> - OPTIONAL LIBS
> > > > > >>>>>>>>>>>> - BENCHMARKS
> > > > > >>>>>>>>>>>> - DOCS (?)
> > > > > >>>>>>>>>>>> - EXAMPLES
> > > > > >>>>>>>>>>>> - .NET PLATFORM FILES
> > > > > >>>>>>>>>>>> - C++ PLATFORM FILES
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> This architecture, as I assume, will add flexibility
> (no
> > > > > reason
> > > > > >>> to
> > > > > >>>>>>>>>> download
> > > > > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core
> > node
> > > > > >>>>>>> functionality)
> > > > > >>>>>>>>>>>> and
> > > > > >>>>>>>>>>>> maintainability (you are in full control of what is
> > > > installed
> > > > > on
> > > > > >>>>>> your
> > > > > >>>>>>>>>>>> system).
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> After successful architecture choice, same scheme are
> > > > planned
> > > > > to
> > > > > >>> be
> > > > > >>>>>>>>>> used in
> > > > > >>>>>>>>>>>> DEB packages as well.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> WDYT?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> --
> > > > > >>>>>>>>>>>> Sent from:
> > > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > > >>> com/
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Ilya Kasnacheev

Re: Apache Ignite RPM packaging (Stage II)

Hello!

With existing binary archive, user can move directories from
apache-ignite-fabric/libs/optional to apache-ignite-fabric/libs to activate
them.

But with RPM, user should not contemplate moving directories from
/usr/lib/apache-ignite/optional to /usr/lib/apache-ignite.

User lacks permissions for that operation and it would defeat the purpose
of having a RPM (imagine trying to upgrade Ignite RPM version with a setup
like that). "Turning distribution into Slackware" should not be an offering.

How it could work: Imagine we create /var/lib/ignite, use it as
IGNITE_HOME, add symlinks from files in /usr/lib to /var/lib/ignite. This
way, we can add and remove symlinks to modules in optional/, and thus
enable and disable them as user sees fit.

This also answers the problem of "what's IGNITE_HOME for visor launched
from /usr/bin"

But obviously this will require dedication and effort.

--
Ilya Kasnacheev

2018-04-14 8:03 GMT+03:00 Peter Ivanov <[hidden email]>:

> Current packages design (after installation) does not differ from binary
> archive - everything works (except necessity to run service instead
> ignite.sh) just the same way, including libs/optional.
>
> Also, there can be issues with system JDK version by default, but every
> problem will be in journalctl and/or /var/log, and package has strong
> dependency on any implementation of JDK 1.8.
>
>
> I am lacking enough feedback about Apache Ignite from packages from real
> users, don’t know real use cases so still "moving in the dark".
>
>
> On Fri, 13 Apr 2018 at 22:18, Denis Magda <[hidden email]> wrote:
>
> > Ilya,
> >
> > Thanks for your inputs. The reason why we decided to split Ignite into
> > several packages mimics the reason why Java community introduced modular
> > subsystem for JDK. That's all about size. Ignite distribution is too big,
> > and we're trying to separate it into several components so that people
> can
> > install only the features they need.
> >
> > The point of a package is to ship something into root file system that
> can
> > > be used from root file system. If cpp files require compilation we
> should
> > > not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> > > there's no mechanism to add optional libs to Ignite classpath, we
> should
> > > not ship optional libs. Moreover, some of 'optional' modules such as
> yarn
> > > don't make sense here because they're not supposed to be used with
> > > standalone Ignite.
> >
> >
> > Agree that we need to ship the code that is ready to be run. As for the
> > classpath thing, if an optional package is installed into the root (core)
> > package directory, then its jars have to be added to "ignite/libs"
> folder.
> > After that, the one needs to restart a cluster node, nd it will add the
> > just installed optional libs to the classpath. *Petr*, does it work this
> > way or can be implemented this way to address Ilya's concerns?
> >
> > --
> > Denis
> >
> > On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <
> > [hidden email]>
> > wrote:
> >
> > > 2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:
> > >
> > > > On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > >
> > > > > Moreover I did not find a way to start service if default installed
> > JVM
> > > > is
> > > > > Java 7 :( I understand it's EOL, still this is something that hit
> me.
> > > >
> > > >
> > > > Apache Ignite >=2.4 does not support Java 7 - it is said in
> > > documentation,
> > > > DEVNOTES and even in startup scripts.
> > > >
> > > > I have Java 8 too, but I could not get service from package to start
> > > Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
> > > matter). Is it possible to specify it while running packaged Ignite?
> > >
> > >
> > > >
> > > > >
> > > > > apache-ignite-libs is a totally unexpected package name.
> > apache-ignite
> > > > core
> > > > > doesn't depend on it. It doesn't enable anything out of the box.
> The
> > > > > package is huge.
> > > >
> > > > ‘apache-ignite-libs’ is an aggregation package (for now) for all
> > optional
> > > > libs we are delivering. Possibly later they will be split more
> granular
> > > or
> > > > even package per lib (like php, perl, python, etc. do for their
> libs).
> > > > This package dependency on ‘apache-ignite-core’ may seem confusing
> > > though,
> > > > I will try to explain it in IEP at least for current iteration.
> > > >
> > >
> > > Okay, but how do you add optional libs to be included into Ignite
> > classpath
> > > while being launched by service? Is it even possible? If it isn't, I
> > think
> > > it doesn't make sense to ship apache-ignite-libs at all.
> > >
> > >
> > > >
> > > > Further naming may become clear when we’ll start initiative on
> > including
> > > > packages to popular Linux distributions and theirs community will
> join
> > > > naming discussions.
> > > >
> > > Renaming packages once they're deployed widely will be a pain point to
> > out
> > > users. Some things should probably be thought out in advance.
> > >
> > >
> > > >
> > > >
> > > >
> > > > >
> > > > > Frankly speaking, I'm not sure that improvements over Stage I are
> > > enough
> > > > as
> > > > > of now. For demo-like activity, we can probably go with one package
> > > fits
> > > > > all.
> > > > >
> > > >
> > > > The process of finding the best package architecture is iterative,
> but
> > > > previously community agreed in split design proposed for 2.5 release.
> > > >
> > > > Also, split architecture is half of proposed improvements. The other
> > > half -
> > > > new process for deploying packages to Bintray (with virtually
> > indefinite
> > > > storage capabilities).
> > > >
> > > I think we could drop the split for now, or at least drop
> > > apache-ignite-libs package at all. Probably also drop apache-ignite-cpp
> > > package and maybe apache-ignite-benchmarks.
> > >
> > > The point of a package is to ship something into root file system that
> > can
> > > be used from root file system. If cpp files require compilation we
> should
> > > not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> > > there's no mechanism to add optional libs to Ignite classpath, we
> should
> > > not ship optional libs. Moreover, some of 'optional' modules such as
> yarn
> > > don't make sense here because they're not supposed to be used with
> > > standalone Ignite.
> > >
> > > IMO it is not right to try and shove every file from Ignite
> distribution
> > > into some package. We should only put in packages things that can be
> > used.
> > > If something can't be used without copying it to a different FS
> location,
> > > it should be in examples or not packaged at all.
> > >
> > > In my opinion, it doesn't make sense to implement an underwhelming
> > package
> > > split right now just because we have agreed to have *some* package
> split
> > in
> > > 2.5. Let's aim for happiness.
> > >
> > >
> > > >
> > > >
> > > >
> > > > >
> > > > > --
> > > > > Ilya Kasnacheev
> > > >
> > >
> > >
> > >
> > > > >
> > > > > 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > > > >
> > > > > > If someone from PMCы or Committers still sees necessity about
> > > including
> > > > > > these tasks into Apache Ignite 2.5 release, this is the last
> chance
> > > to
> > > > do
> > > > > > so.
> > > > > > Otherwise this task will be moved to at 2.6 release at least, or
> > even
> > > > > > moved to backlog indefinitely.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]>
> > wrote:
> > > > > > >
> > > > > > > To top new RPM architecture off, update to release process is
> > > > > introduced
> > > > > > — [1] [2].
> > > > > > >
> > > > > > > Both tasks (this one and IGNITE-7647) are ready for review and
> > > should
> > > > > be
> > > > > > merged simultaneously.
> > > > > > >
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > > > > > > [2] https://github.com/apache/ignite-release/pull/1
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Hello!
> > > > > > >>
> > > > > > >> Let me share my idea of how this shoud work. Splitting package
> > > into
> > > > > > >> sub-packages should be dependency-driven.
> > > > > > >>
> > > > > > >> It means that all Ignite modules without dependencies or with
> > > small
> > > > > > >> dependencies (such as ignite-log4j) should be included in
> > > > ignite-core.
> > > > > > It
> > > > > > >> doesn't make sense to make a zillion RPM packages.
> > > > > > >>
> > > > > > >> Critical things like ignite-spring and ignite-indexing should
> be
> > > in
> > > > > > >> ignite-core of course, even if they have dependencies.
> > Ignite-core
> > > > > > should
> > > > > > >> be fully self-sufficient and feature-complete.
> > > > > > >>
> > > > > > >> However, e.g. .net API should probably be in a separate
> package,
> > > > > > because it
> > > > > > >> should depend on mono | net-core. We may also have
> ignite-devel
> > > > > package
> > > > > > >> which should include all modules which only make sense for
> > > > developers
> > > > > > who
> > > > > > >> write code. Such as hibernate integration.
> > > > > > >>
> > > > > > >> I'm not sure about MR modules. The main question should be,
> does
> > > it
> > > > > have
> > > > > > >> dependencies? Can it run stand-alone without writing code?
> > > > > > >>
> > > > > > >> Hope this helps,
> > > > > > >>
> > > > > > >> --
> > > > > > >> Ilya Kasnacheev
> > > > > > >>
> > > > > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > > > > > >>
> > > > > > >>> Hi, Igniters!
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Here are some news on our RPM packages initiative.
> > > > > > >>>
> > > > > > >>> 1. I’ve finished preliminary developing of Stage II version
> of
> > > RPM
> > > > > > >>> packages [1]. Main “new feature” is — split design. Also I’ve
> > > added
> > > > > > >>> package.sh script for automating package building process
> which
> > > > will
> > > > > > help
> > > > > > >>> organise corresponding builds in TC as well as simplify
> process
> > > for
> > > > > > >>> developers who wishes to have custom packages.
> > > > > > >>> PR#3703 [2] is ready for review. Denis, in order to catch up
> > with
> > > > > > Apache
> > > > > > >>> Ignite 2.5 release, I’d greatly appreciate your help in
> finding
> > > > > > reviewer.
> > > > > > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and
> DEB
> > > [4]
> > > > > > >>> repositories on Apache Bintray. Though they are already
> > prepared
> > > > for
> > > > > > >>> hosting RPM and DEB packages respectively, and there is a way
> > of
> > > > > > linking
> > > > > > >>> them to apache.org/dist/ignite page, there is possible
> > > alternative
> > > > > in
> > > > > > >>> storing there only plain directory layout corresponding to
> each
> > > > > > repository
> > > > > > >>> type (RPM and DEB) and manage this layout (repodata,
> > > distributions,
> > > > > > >>> versions, etc.) by ourselves, having more control over
> > > repositories
> > > > > but
> > > > > > >>> lacking some simplicity of deploying new releases. WDYT?
> Should
> > > we
> > > > > try
> > > > > > >>> Cassandra approach? They are storing their DEB packages as I
> > > > > described
> > > > > > >>> above [5].
> > > > > > >>>
> > > > > > >>> Also — a question arose while I was working on this issue:
> > which
> > > > OSes
> > > > > > (and
> > > > > > >>> which versions of each) are we going to support (if we are
> > going)
> > > > in
> > > > > > terms
> > > > > > >>> of step-by-step list? Currently RPM packages are tested only
> > with
> > > > > > latest
> > > > > > >>> CentOS (and, respectively — RHEL), but there are a lot more
> > > > RPM-based
> > > > > > >>> distributives [6] some of which are more o less popular among
> > OS
> > > > > > community
> > > > > > >>> (ALT, Fedora, openSUSE, etc.).
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > > > > > >>> [2] https://github.com/apache/ignite/pull/3703
> > > > > > >>> [3] https://bintray.com/apache/ignite-rpm
> > > > > > >>> [4] https://bintray.com/apache/ignite-deb
> > > > > > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > > > > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > > > > > distributions
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]>
> > > > wrote:
> > > > > > >>>>
> > > > > > >>>> I suppose that most everything if not all from libs/options
> > will
> > > > go
> > > > > to
> > > > > > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > > > > > >>>> More precise lib selection (if something from optional would
> > > > better
> > > > > to
> > > > > > >>> have in core package) will be discussed right after
> preliminary
> > > > split
> > > > > > >>> architecture agreement.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > >>>>>
> > > > > > >>>>> I like idea of keeping simple system of modules, so +1 from
> > me.
> > > > > > >>>>>
> > > > > > >>>>> Where optional libs (e.g Direct IO plugin) would be
> included,
> > > > would
> > > > > > it
> > > > > > >>> be
> > > > > > >>>>> core or optional?
> > > > > > >>>>>
> > > > > > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <
> [hidden email]
> > >:
> > > > > > >>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> How big would be a final core module?
> > > > > > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and
> > > > create
> > > > > > it’s
> > > > > > >>> own
> > > > > > >>>>>>> package.
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree
> > with
> > > > Petr
> > > > > > and
> > > > > > >>>>>> propose the simplest modular system:
> > > > > > >>>>>>
> > > > > > >>>>>> - core module that includes basic Ignite capabilities
> > > including
> > > > > SQL,
> > > > > > >>>>>> compute grid, service grid, k/v
> > > > > > >>>>>> - optional module hosts the rest - ML, streamers
> integration
> > > > > (kafka,
> > > > > > >>>>>> flink), kubernetes, etc.
> > > > > > >>>>>>
> > > > > > >>>>>> What do you think?
> > > > > > >>>>>>
> > > > > > >>>>>> --
> > > > > > >>>>>> Denis
> > > > > > >>>>>>
> > > > > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> > > > > [hidden email]>
> > > > > > >>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> *DEB package
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <
> > [hidden email]>
> > > > > > wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Considering that DEV package for now is almost platform
> > > > > > independent
> > > > > > >>>>>> (its
> > > > > > >>>>>>> a java application more or less), that package will work
> > > almost
> > > > > on
> > > > > > any
> > > > > > >>>>>>> DEB-based linux, including but not limited to Ubuntu,
> > Debian,
> > > > > etc.
> > > > > > >>>>>>>> The only restriction is existence of systemctl (systemd)
> > > > service
> > > > > > >>>>>> manager
> > > > > > >>>>>>> — we are dependent on it.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thats why, for instance, our RPM repository is called
> > simply
> > > > > ‘rpm’
> > > > > > >>> and
> > > > > > >>>>>>> package has no arch or dist suffix — it will work on
> > CentOS,
> > > > > RHEL,
> > > > > > >>>>>> Fedora,
> > > > > > >>>>>>> etc. with presence of aforementioned systemd.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> > > > > > [hidden email]>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Will Debian package work for Ubuntu?
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> D.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> > > > > > [hidden email]>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will
> move
> > to
> > > > > > official
> > > > > > >>>>>>>>>> repositories, there can be a problem from OS
> community.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base
> for
> > > DEB
> > > > > > package
> > > > > > >>>>>>> build
> > > > > > >>>>>>>>>> (package layout / install scripts) for speeding up
> > things
> > > > and
> > > > > > >>>>>> excluding
> > > > > > >>>>>>>>>> possible duplication and desynchronisation, so its a
> > > matter
> > > > of
> > > > > > ’sit
> > > > > > >>>>>>> and do’
> > > > > > >>>>>>>>>> rather then some technical research. Thats why I rose
> > > > > discussion
> > > > > > >>>>>> about
> > > > > > >>>>>>>>>> future package architecture, so that after agreement
> I'm
> > > be
> > > > > > able to
> > > > > > >>>>>>> pack
> > > > > > >>>>>>>>>> both RPM and DEB identically.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Yet, if you insist, I can create DEB package according
> > to
> > > > > > current
> > > > > > >>> RPM
> > > > > > >>>>>>>>>> layout in no time.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> > > > > > >>> [hidden email]>
> > > > > > >>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Peter,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I don't think the package size of 280M is going to
> be a
> > > > > > problem at
> > > > > > >>>>>>> all,
> > > > > > >>>>>>>>>> but
> > > > > > >>>>>>>>>>> what you are suggesting can be an improvement down
> the
> > > > road.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> In the mean time, I think our top priority should be
> to
> > > > > provide
> > > > > > >>>>>>> packages
> > > > > > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly
> > > > enough.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Agree?
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> D.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> > > > > [hidden email]>
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Hi, Igniters!
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of
> > it,
> > > > so
> > > > > > I'd
> > > > > > >>>>>> like
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>>>>> move
> > > > > > >>>>>>>>>>>> forward to further improve and widen AI delivery
> > through
> > > > > > >>> packages.
> > > > > > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package
> weighing
> > > > about
> > > > > > >>> 280M+
> > > > > > >>>>>>> and,
> > > > > > >>>>>>>>>> to
> > > > > > >>>>>>>>>>>> improve usability and significantly reduce required
> > > > download
> > > > > > >>>>>> sizes, I
> > > > > > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted
> > > delivery
> > > > > as
> > > > > > >>>>>>> follows:
> > > > > > >>>>>>>>>>>> - CORE
> > > > > > >>>>>>>>>>>> - bin
> > > > > > >>>>>>>>>>>> - config
> > > > > > >>>>>>>>>>>> - libs (!optional)
> > > > > > >>>>>>>>>>>> - OPTIONAL LIBS
> > > > > > >>>>>>>>>>>> - BENCHMARKS
> > > > > > >>>>>>>>>>>> - DOCS (?)
> > > > > > >>>>>>>>>>>> - EXAMPLES
> > > > > > >>>>>>>>>>>> - .NET PLATFORM FILES
> > > > > > >>>>>>>>>>>> - C++ PLATFORM FILES
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> This architecture, as I assume, will add flexibility
> > (no
> > > > > > reason
> > > > > > >>> to
> > > > > > >>>>>>>>>> download
> > > > > > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core
> > > node
> > > > > > >>>>>>> functionality)
> > > > > > >>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>> maintainability (you are in full control of what is
> > > > > installed
> > > > > > on
> > > > > > >>>>>> your
> > > > > > >>>>>>>>>>>> system).
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> After successful architecture choice, same scheme
> are
> > > > > planned
> > > > > > to
> > > > > > >>> be
> > > > > > >>>>>>>>>> used in
> > > > > > >>>>>>>>>>>> DEB packages as well.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> WDYT?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> --
> > > > > > >>>>>>>>>>>> Sent from:
> > > > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > > > >>> com/
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

vveider

Re: Apache Ignite RPM packaging (Stage II)

> On 15 Apr 2018, at 10:19, Ilya Kasnacheev <[hidden email]> wrote:
>
> Hello!
>
> With existing binary archive, user can move directories from
> apache-ignite-fabric/libs/optional to apache-ignite-fabric/libs to activate
> them.
>
> But with RPM, user should not contemplate moving directories from
> /usr/lib/apache-ignite/optional to /usr/lib/apache-ignite.

I always thought of that option as COPYING optional libs, not MOVING.

>
>
> User lacks permissions for that operation and it would defeat the purpose
> of having a RPM (imagine trying to upgrade Ignite RPM version with a setup
> like that). "Turning distribution into Slackware" should not be an offering.

Can’t imagine use case where Apache Ignite software is installed by one user, and run by another, without sudo/root permissions.
Yet, ignite user’s login shell is disabled indeed and without sudo/root permissions optional libs cannot be copied.
Also — the symlinks is a good idea, but I’ve already thought of it and I’m planning addition of special utility for enabling / disabling optional libs (like a2enable/a2disable) in next development iteration.

>
> How it could work: Imagine we create /var/lib/ignite, use it as
> IGNITE_HOME, add symlinks from files in /usr/lib to /var/lib/ignite. This
> way, we can add and remove symlinks to modules in optional/, and thus
> enable and disable them as user sees fit.

By linux file hierarchy convention, home dir should be in /usr/lib. /var/lib/ is currently used for working files (MySQL alike).

>
> This also answers the problem of "what's IGNITE_HOME for visor launched
> from /usr/bin”

That is addressed in separate task and will be fixed with minor startup script redesign with universal location-independent solution.

>
> But obviously this will require dedication and effort.

No problems with efforts after final design is discussed an agreed.

>
>
> --
> Ilya Kasnacheev
>
> 2018-04-14 8:03 GMT+03:00 Peter Ivanov <[hidden email]>:
>
>> Current packages design (after installation) does not differ from binary
>> archive - everything works (except necessity to run service instead
>> ignite.sh) just the same way, including libs/optional.
>>
>> Also, there can be issues with system JDK version by default, but every
>> problem will be in journalctl and/or /var/log, and package has strong
>> dependency on any implementation of JDK 1.8.
>>
>>
>> I am lacking enough feedback about Apache Ignite from packages from real
>> users, don’t know real use cases so still "moving in the dark".
>>
>>
>> On Fri, 13 Apr 2018 at 22:18, Denis Magda <[hidden email]> wrote:
>>
>>> Ilya,
>>>
>>> Thanks for your inputs. The reason why we decided to split Ignite into
>>> several packages mimics the reason why Java community introduced modular
>>> subsystem for JDK. That's all about size. Ignite distribution is too big,
>>> and we're trying to separate it into several components so that people
>> can
>>> install only the features they need.
>>>
>>> The point of a package is to ship something into root file system that
>> can
>>>> be used from root file system. If cpp files require compilation we
>> should
>>>> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
>>>> there's no mechanism to add optional libs to Ignite classpath, we
>> should
>>>> not ship optional libs. Moreover, some of 'optional' modules such as
>> yarn
>>>> don't make sense here because they're not supposed to be used with
>>>> standalone Ignite.
>>>
>>>
>>> Agree that we need to ship the code that is ready to be run. As for the
>>> classpath thing, if an optional package is installed into the root (core)
>>> package directory, then its jars have to be added to "ignite/libs"
>> folder.
>>> After that, the one needs to restart a cluster node, nd it will add the
>>> just installed optional libs to the classpath. *Petr*, does it work this
>>> way or can be implemented this way to address Ilya's concerns?
>>>
>>> --
>>> Denis
>>>
>>> On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <
>>> [hidden email]>
>>> wrote:
>>>
>>>> 2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:
>>>>
>>>>> On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <
>>> [hidden email]
>>>>>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Moreover I did not find a way to start service if default installed
>>> JVM
>>>>> is
>>>>>> Java 7 :( I understand it's EOL, still this is something that hit
>> me.
>>>>>
>>>>>
>>>>> Apache Ignite >=2.4 does not support Java 7 - it is said in
>>>> documentation,
>>>>> DEVNOTES and even in startup scripts.
>>>>>
>>>>> I have Java 8 too, but I could not get service from package to start
>>>> Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
>>>> matter). Is it possible to specify it while running packaged Ignite?
>>>>
>>>>
>>>>>
>>>>>>
>>>>>> apache-ignite-libs is a totally unexpected package name.
>>> apache-ignite
>>>>> core
>>>>>> doesn't depend on it. It doesn't enable anything out of the box.
>> The
>>>>>> package is huge.
>>>>>
>>>>> ‘apache-ignite-libs’ is an aggregation package (for now) for all
>>> optional
>>>>> libs we are delivering. Possibly later they will be split more
>> granular
>>>> or
>>>>> even package per lib (like php, perl, python, etc. do for their
>> libs).
>>>>> This package dependency on ‘apache-ignite-core’ may seem confusing
>>>> though,
>>>>> I will try to explain it in IEP at least for current iteration.
>>>>>
>>>>
>>>> Okay, but how do you add optional libs to be included into Ignite
>>> classpath
>>>> while being launched by service? Is it even possible? If it isn't, I
>>> think
>>>> it doesn't make sense to ship apache-ignite-libs at all.
>>>>
>>>>
>>>>>
>>>>> Further naming may become clear when we’ll start initiative on
>>> including
>>>>> packages to popular Linux distributions and theirs community will
>> join
>>>>> naming discussions.
>>>>>
>>>> Renaming packages once they're deployed widely will be a pain point to
>>> out
>>>> users. Some things should probably be thought out in advance.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Frankly speaking, I'm not sure that improvements over Stage I are
>>>> enough
>>>>> as
>>>>>> of now. For demo-like activity, we can probably go with one package
>>>> fits
>>>>>> all.
>>>>>>
>>>>>
>>>>> The process of finding the best package architecture is iterative,
>> but
>>>>> previously community agreed in split design proposed for 2.5 release.
>>>>>
>>>>> Also, split architecture is half of proposed improvements. The other
>>>> half -
>>>>> new process for deploying packages to Bintray (with virtually
>>> indefinite
>>>>> storage capabilities).
>>>>>
>>>> I think we could drop the split for now, or at least drop
>>>> apache-ignite-libs package at all. Probably also drop apache-ignite-cpp
>>>> package and maybe apache-ignite-benchmarks.
>>>>
>>>> The point of a package is to ship something into root file system that
>>> can
>>>> be used from root file system. If cpp files require compilation we
>> should
>>>> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
>>>> there's no mechanism to add optional libs to Ignite classpath, we
>> should
>>>> not ship optional libs. Moreover, some of 'optional' modules such as
>> yarn
>>>> don't make sense here because they're not supposed to be used with
>>>> standalone Ignite.
>>>>
>>>> IMO it is not right to try and shove every file from Ignite
>> distribution
>>>> into some package. We should only put in packages things that can be
>>> used.
>>>> If something can't be used without copying it to a different FS
>> location,
>>>> it should be in examples or not packaged at all.
>>>>
>>>> In my opinion, it doesn't make sense to implement an underwhelming
>>> package
>>>> split right now just because we have agreed to have *some* package
>> split
>>> in
>>>> 2.5. Let's aim for happiness.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>
>>>>
>>>>
>>>>
>>>>>>
>>>>>> 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
>>>>>>
>>>>>>> If someone from PMCы or Committers still sees necessity about
>>>> including
>>>>>>> these tasks into Apache Ignite 2.5 release, this is the last
>> chance
>>>> to
>>>>> do
>>>>>>> so.
>>>>>>> Otherwise this task will be moved to at 2.6 release at least, or
>>> even
>>>>>>> moved to backlog indefinitely.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]>
>>> wrote:
>>>>>>>>
>>>>>>>> To top new RPM architecture off, update to release process is
>>>>>> introduced
>>>>>>> — [1] [2].
>>>>>>>>
>>>>>>>> Both tasks (this one and IGNITE-7647) are ready for review and
>>>> should
>>>>>> be
>>>>>>> merged simultaneously.
>>>>>>>>
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-8172
>>>>>>>> [2] https://github.com/apache/ignite-release/pull/1
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
>>>> [hidden email]
>>>>>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> Let me share my idea of how this shoud work. Splitting package
>>>> into
>>>>>>>>> sub-packages should be dependency-driven.
>>>>>>>>>
>>>>>>>>> It means that all Ignite modules without dependencies or with
>>>> small
>>>>>>>>> dependencies (such as ignite-log4j) should be included in
>>>>> ignite-core.
>>>>>>> It
>>>>>>>>> doesn't make sense to make a zillion RPM packages.
>>>>>>>>>
>>>>>>>>> Critical things like ignite-spring and ignite-indexing should
>> be
>>>> in
>>>>>>>>> ignite-core of course, even if they have dependencies.
>>> Ignite-core
>>>>>>> should
>>>>>>>>> be fully self-sufficient and feature-complete.
>>>>>>>>>
>>>>>>>>> However, e.g. .net API should probably be in a separate
>> package,
>>>>>>> because it
>>>>>>>>> should depend on mono | net-core. We may also have
>> ignite-devel
>>>>>> package
>>>>>>>>> which should include all modules which only make sense for
>>>>> developers
>>>>>>> who
>>>>>>>>> write code. Such as hibernate integration.
>>>>>>>>>
>>>>>>>>> I'm not sure about MR modules. The main question should be,
>> does
>>>> it
>>>>>> have
>>>>>>>>> dependencies? Can it run stand-alone without writing code?
>>>>>>>>>
>>>>>>>>> Hope this helps,
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>
>>>>>>>>> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
>>>>>>>>>
>>>>>>>>>> Hi, Igniters!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Here are some news on our RPM packages initiative.
>>>>>>>>>>
>>>>>>>>>> 1. I’ve finished preliminary developing of Stage II version
>> of
>>>> RPM
>>>>>>>>>> packages [1]. Main “new feature” is — split design. Also I’ve
>>>> added
>>>>>>>>>> package.sh script for automating package building process
>> which
>>>>> will
>>>>>>> help
>>>>>>>>>> organise corresponding builds in TC as well as simplify
>> process
>>>> for
>>>>>>>>>> developers who wishes to have custom packages.
>>>>>>>>>> PR#3703 [2] is ready for review. Denis, in order to catch up
>>> with
>>>>>>> Apache
>>>>>>>>>> Ignite 2.5 release, I’d greatly appreciate your help in
>> finding
>>>>>>> reviewer.
>>>>>>>>>> 2. With the help of ASF INFRA team, we now have RPM [3] and
>> DEB
>>>> [4]
>>>>>>>>>> repositories on Apache Bintray. Though they are already
>>> prepared
>>>>> for
>>>>>>>>>> hosting RPM and DEB packages respectively, and there is a way
>>> of
>>>>>>> linking
>>>>>>>>>> them to apache.org/dist/ignite page, there is possible
>>>> alternative
>>>>>> in
>>>>>>>>>> storing there only plain directory layout corresponding to
>> each
>>>>>>> repository
>>>>>>>>>> type (RPM and DEB) and manage this layout (repodata,
>>>> distributions,
>>>>>>>>>> versions, etc.) by ourselves, having more control over
>>>> repositories
>>>>>> but
>>>>>>>>>> lacking some simplicity of deploying new releases. WDYT?
>> Should
>>>> we
>>>>>> try
>>>>>>>>>> Cassandra approach? They are storing their DEB packages as I
>>>>>> described
>>>>>>>>>> above [5].
>>>>>>>>>>
>>>>>>>>>> Also — a question arose while I was working on this issue:
>>> which
>>>>> OSes
>>>>>>> (and
>>>>>>>>>> which versions of each) are we going to support (if we are
>>> going)
>>>>> in
>>>>>>> terms
>>>>>>>>>> of step-by-step list? Currently RPM packages are tested only
>>> with
>>>>>>> latest
>>>>>>>>>> CentOS (and, respectively — RHEL), but there are a lot more
>>>>> RPM-based
>>>>>>>>>> distributives [6] some of which are more o less popular among
>>> OS
>>>>>>> community
>>>>>>>>>> (ALT, Fedora, openSUSE, etc.).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
>>>>>>>>>> [2] https://github.com/apache/ignite/pull/3703
>>>>>>>>>> [3] https://bintray.com/apache/ignite-rpm
>>>>>>>>>> [4] https://bintray.com/apache/ignite-deb
>>>>>>>>>> [5] https://bintray.com/apache/cassandra/debian#files/
>>>>>>>>>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
>>>>>>> distributions
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]>
>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I suppose that most everything if not all from libs/options
>>> will
>>>>> go
>>>>>> to
>>>>>>>>>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
>>>>>>>>>>> More precise lib selection (if something from optional would
>>>>> better
>>>>>> to
>>>>>>>>>> have in core package) will be discussed right after
>> preliminary
>>>>> split
>>>>>>>>>> architecture agreement.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <
>>> [hidden email]
>>>>>
>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I like idea of keeping simple system of modules, so +1 from
>>> me.
>>>>>>>>>>>>
>>>>>>>>>>>> Where optional libs (e.g Direct IO plugin) would be
>> included,
>>>>> would
>>>>>>> it
>>>>>>>>>> be
>>>>>>>>>>>> core or optional?
>>>>>>>>>>>>
>>>>>>>>>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <
>> [hidden email]
>>>> :
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How big would be a final core module?
>>>>>>>>>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and
>>>>> create
>>>>>>> it’s
>>>>>>>>>> own
>>>>>>>>>>>>>> package.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree
>>> with
>>>>> Petr
>>>>>>> and
>>>>>>>>>>>>> propose the simplest modular system:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - core module that includes basic Ignite capabilities
>>>> including
>>>>>> SQL,
>>>>>>>>>>>>> compute grid, service grid, k/v
>>>>>>>>>>>>> - optional module hosts the rest - ML, streamers
>> integration
>>>>>> (kafka,
>>>>>>>>>>>>> flink), kubernetes, etc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
>>>>>> [hidden email]>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> *DEB package
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <
>>> [hidden email]>
>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Considering that DEV package for now is almost platform
>>>>>>> independent
>>>>>>>>>>>>> (its
>>>>>>>>>>>>>> a java application more or less), that package will work
>>>> almost
>>>>>> on
>>>>>>> any
>>>>>>>>>>>>>> DEB-based linux, including but not limited to Ubuntu,
>>> Debian,
>>>>>> etc.
>>>>>>>>>>>>>>> The only restriction is existence of systemctl (systemd)
>>>>> service
>>>>>>>>>>>>> manager
>>>>>>>>>>>>>> — we are dependent on it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thats why, for instance, our RPM repository is called
>>> simply
>>>>>> ‘rpm’
>>>>>>>>>> and
>>>>>>>>>>>>>> package has no arch or dist suffix — it will work on
>>> CentOS,
>>>>>> RHEL,
>>>>>>>>>>>>> Fedora,
>>>>>>>>>>>>>> etc. with presence of aforementioned systemd.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
>>>>>>> [hidden email]>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Will Debian package work for Ubuntu?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
>>>>>>> [hidden email]>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Not a problem, rather nuisance. Also, when we will
>> move
>>> to
>>>>>>> official
>>>>>>>>>>>>>>>>> repositories, there can be a problem from OS
>> community.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Concerning DEB packages — I plan to use RPM as base
>> for
>>>> DEB
>>>>>>> package
>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>>> (package layout / install scripts) for speeding up
>>> things
>>>>> and
>>>>>>>>>>>>> excluding
>>>>>>>>>>>>>>>>> possible duplication and desynchronisation, so its a
>>>> matter
>>>>> of
>>>>>>> ’sit
>>>>>>>>>>>>>> and do’
>>>>>>>>>>>>>>>>> rather then some technical research. Thats why I rose
>>>>>> discussion
>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>> future package architecture, so that after agreement
>> I'm
>>>> be
>>>>>>> able to
>>>>>>>>>>>>>> pack
>>>>>>>>>>>>>>>>> both RPM and DEB identically.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yet, if you insist, I can create DEB package according
>>> to
>>>>>>> current
>>>>>>>>>> RPM
>>>>>>>>>>>>>>>>> layout in no time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
>>>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Peter,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I don't think the package size of 280M is going to
>> be a
>>>>>>> problem at
>>>>>>>>>>>>>> all,
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> what you are suggesting can be an improvement down
>> the
>>>>> road.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In the mean time, I think our top priority should be
>> to
>>>>>> provide
>>>>>>>>>>>>>> packages
>>>>>>>>>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly
>>>>> enough.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Agree?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
>>>>>> [hidden email]>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi, Igniters!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Release 2.4 is almost there, at least binary part of
>>> it,
>>>>> so
>>>>>>> I'd
>>>>>>>>>>>>> like
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>> forward to further improve and widen AI delivery
>>> through
>>>>>>>>>> packages.
>>>>>>>>>>>>>>>>>>> As of now, Apache Ignite ships in RPM package
>> weighing
>>>>> about
>>>>>>>>>> 280M+
>>>>>>>>>>>>>> and,
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> improve usability and significantly reduce required
>>>>> download
>>>>>>>>>>>>> sizes, I
>>>>>>>>>>>>>>>>>>> purpose that in 2.5 release we introduce splitted
>>>> delivery
>>>>>> as
>>>>>>>>>>>>>> follows:
>>>>>>>>>>>>>>>>>>> - CORE
>>>>>>>>>>>>>>>>>>> - bin
>>>>>>>>>>>>>>>>>>> - config
>>>>>>>>>>>>>>>>>>> - libs (!optional)
>>>>>>>>>>>>>>>>>>> - OPTIONAL LIBS
>>>>>>>>>>>>>>>>>>> - BENCHMARKS
>>>>>>>>>>>>>>>>>>> - DOCS (?)
>>>>>>>>>>>>>>>>>>> - EXAMPLES
>>>>>>>>>>>>>>>>>>> - .NET PLATFORM FILES
>>>>>>>>>>>>>>>>>>> - C++ PLATFORM FILES
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This architecture, as I assume, will add flexibility
>>> (no
>>>>>>> reason
>>>>>>>>>> to
>>>>>>>>>>>>>>>>> download
>>>>>>>>>>>>>>>>>>> all 280M+ of binaries where you are to run only core
>>>> node
>>>>>>>>>>>>>> functionality)
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> maintainability (you are in full control of what is
>>>>>> installed
>>>>>>> on
>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>> system).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> After successful architecture choice, same scheme
>> are
>>>>>> planned
>>>>>>> to
>>>>>>>>>> be
>>>>>>>>>>>>>>>>> used in
>>>>>>>>>>>>>>>>>>> DEB packages as well.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Sent from:
>>>>>> http://apache-ignite-developers.2346864.n4.nabble.
>>>>>>>>>> com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Ilya Kasnacheev

Re: Apache Ignite RPM packaging (Stage II)

Copying anything manually to anywhere /usr (excluding /usr/local) is an
example of slackwarization that package users and creators try to avoid.

> By linux file hierarchy convention, home dir should be in /usr/lib

Citation needed! I bet you're interpreting it wrong, since I've listed a
random dir in /var/lib, and:
~/w/incubator-ignite% sudo ls -al /var/lib/lightdm
drwxr-x--- 6 lightdm lightdm 4096 авг 14 2017 .
drwxr-xr-x 89 root root 4096 мар 10 22:37 ..
drwxrwxr-x 4 lightdm lightdm 4096 июл 19 2017 .cache
drwxr-xr-x 5 lightdm lightdm 4096 июл 19 2017 .config
drwx------ 2 lightdm lightdm 4096 апр 11 18:25 .gconf
drwxr-xr-x 3 lightdm lightdm 4096 июл 19 2017 .local
-rw------- 1 lightdm lightdm 283 апр 11 18:25 .Xauthority

...it definitely looks like a home dir. Which goes with additional benefit
of being writable by end-user.

--
Ilya Kasnacheev

2018-04-16 9:22 GMT+03:00 Petr Ivanov <[hidden email]>:

>
>
> > On 15 Apr 2018, at 10:19, Ilya Kasnacheev <[hidden email]>
> wrote:
> >
> > Hello!
> >
> > With existing binary archive, user can move directories from
> > apache-ignite-fabric/libs/optional to apache-ignite-fabric/libs to
> activate
> > them.
> >
> > But with RPM, user should not contemplate moving directories from
> > /usr/lib/apache-ignite/optional to /usr/lib/apache-ignite.
>
> I always thought of that option as COPYING optional libs, not MOVING.
>
>
> >
> >
> > User lacks permissions for that operation and it would defeat the purpose
> > of having a RPM (imagine trying to upgrade Ignite RPM version with a
> setup
> > like that). "Turning distribution into Slackware" should not be an
> offering.
>
> Can’t imagine use case where Apache Ignite software is installed by one
> user, and run by another, without sudo/root permissions.
> Yet, ignite user’s login shell is disabled indeed and without sudo/root
> permissions optional libs cannot be copied.
> Also — the symlinks is a good idea, but I’ve already thought of it and I’m
> planning addition of special utility for enabling / disabling optional libs
> (like a2enable/a2disable) in next development iteration.
>
>
> >
> > How it could work: Imagine we create /var/lib/ignite, use it as
> > IGNITE_HOME, add symlinks from files in /usr/lib to /var/lib/ignite. This
> > way, we can add and remove symlinks to modules in optional/, and thus
> > enable and disable them as user sees fit.
>
> By linux file hierarchy convention, home dir should be in /usr/lib.
> /var/lib/ is currently used for working files (MySQL alike).
>
>
> >
> > This also answers the problem of "what's IGNITE_HOME for visor launched
> > from /usr/bin”
>
> That is addressed in separate task and will be fixed with minor startup
> script redesign with universal location-independent solution.
>
>
> >
> > But obviously this will require dedication and effort.
>
> No problems with efforts after final design is discussed an agreed.
>
>
> >
> >
> > --
> > Ilya Kasnacheev
> >
> > 2018-04-14 8:03 GMT+03:00 Peter Ivanov <[hidden email]>:
> >
> >> Current packages design (after installation) does not differ from binary
> >> archive - everything works (except necessity to run service instead
> >> ignite.sh) just the same way, including libs/optional.
> >>
> >> Also, there can be issues with system JDK version by default, but every
> >> problem will be in journalctl and/or /var/log, and package has strong
> >> dependency on any implementation of JDK 1.8.
> >>
> >>
> >> I am lacking enough feedback about Apache Ignite from packages from real
> >> users, don’t know real use cases so still "moving in the dark".
> >>
> >>
> >> On Fri, 13 Apr 2018 at 22:18, Denis Magda <[hidden email]> wrote:
> >>
> >>> Ilya,
> >>>
> >>> Thanks for your inputs. The reason why we decided to split Ignite into
> >>> several packages mimics the reason why Java community introduced
> modular
> >>> subsystem for JDK. That's all about size. Ignite distribution is too
> big,
> >>> and we're trying to separate it into several components so that people
> >> can
> >>> install only the features they need.
> >>>
> >>> The point of a package is to ship something into root file system that
> >> can
> >>>> be used from root file system. If cpp files require compilation we
> >> should
> >>>> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> >>>> there's no mechanism to add optional libs to Ignite classpath, we
> >> should
> >>>> not ship optional libs. Moreover, some of 'optional' modules such as
> >> yarn
> >>>> don't make sense here because they're not supposed to be used with
> >>>> standalone Ignite.
> >>>
> >>>
> >>> Agree that we need to ship the code that is ready to be run. As for the
> >>> classpath thing, if an optional package is installed into the root
> (core)
> >>> package directory, then its jars have to be added to "ignite/libs"
> >> folder.
> >>> After that, the one needs to restart a cluster node, nd it will add the
> >>> just installed optional libs to the classpath. *Petr*, does it work
> this
> >>> way or can be implemented this way to address Ilya's concerns?
> >>>
> >>> --
> >>> Denis
> >>>
> >>> On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <
> >>> [hidden email]>
> >>> wrote:
> >>>
> >>>> 2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:
> >>>>
> >>>>> On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <
> >>> [hidden email]
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>>
> >>>>>> Moreover I did not find a way to start service if default installed
> >>> JVM
> >>>>> is
> >>>>>> Java 7 :( I understand it's EOL, still this is something that hit
> >> me.
> >>>>>
> >>>>>
> >>>>> Apache Ignite >=2.4 does not support Java 7 - it is said in
> >>>> documentation,
> >>>>> DEVNOTES and even in startup scripts.
> >>>>>
> >>>>> I have Java 8 too, but I could not get service from package to start
> >>>> Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
> >>>> matter). Is it possible to specify it while running packaged Ignite?
> >>>>
> >>>>
> >>>>>
> >>>>>>
> >>>>>> apache-ignite-libs is a totally unexpected package name.
> >>> apache-ignite
> >>>>> core
> >>>>>> doesn't depend on it. It doesn't enable anything out of the box.
> >> The
> >>>>>> package is huge.
> >>>>>
> >>>>> ‘apache-ignite-libs’ is an aggregation package (for now) for all
> >>> optional
> >>>>> libs we are delivering. Possibly later they will be split more
> >> granular
> >>>> or
> >>>>> even package per lib (like php, perl, python, etc. do for their
> >> libs).
> >>>>> This package dependency on ‘apache-ignite-core’ may seem confusing
> >>>> though,
> >>>>> I will try to explain it in IEP at least for current iteration.
> >>>>>
> >>>>
> >>>> Okay, but how do you add optional libs to be included into Ignite
> >>> classpath
> >>>> while being launched by service? Is it even possible? If it isn't, I
> >>> think
> >>>> it doesn't make sense to ship apache-ignite-libs at all.
> >>>>
> >>>>
> >>>>>
> >>>>> Further naming may become clear when we’ll start initiative on
> >>> including
> >>>>> packages to popular Linux distributions and theirs community will
> >> join
> >>>>> naming discussions.
> >>>>>
> >>>> Renaming packages once they're deployed widely will be a pain point to
> >>> out
> >>>> users. Some things should probably be thought out in advance.
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Frankly speaking, I'm not sure that improvements over Stage I are
> >>>> enough
> >>>>> as
> >>>>>> of now. For demo-like activity, we can probably go with one package
> >>>> fits
> >>>>>> all.
> >>>>>>
> >>>>>
> >>>>> The process of finding the best package architecture is iterative,
> >> but
> >>>>> previously community agreed in split design proposed for 2.5 release.
> >>>>>
> >>>>> Also, split architecture is half of proposed improvements. The other
> >>>> half -
> >>>>> new process for deploying packages to Bintray (with virtually
> >>> indefinite
> >>>>> storage capabilities).
> >>>>>
> >>>> I think we could drop the split for now, or at least drop
> >>>> apache-ignite-libs package at all. Probably also drop
> apache-ignite-cpp
> >>>> package and maybe apache-ignite-benchmarks.
> >>>>
> >>>> The point of a package is to ship something into root file system that
> >>> can
> >>>> be used from root file system. If cpp files require compilation we
> >> should
> >>>> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> >>>> there's no mechanism to add optional libs to Ignite classpath, we
> >> should
> >>>> not ship optional libs. Moreover, some of 'optional' modules such as
> >> yarn
> >>>> don't make sense here because they're not supposed to be used with
> >>>> standalone Ignite.
> >>>>
> >>>> IMO it is not right to try and shove every file from Ignite
> >> distribution
> >>>> into some package. We should only put in packages things that can be
> >>> used.
> >>>> If something can't be used without copying it to a different FS
> >> location,
> >>>> it should be in examples or not packaged at all.
> >>>>
> >>>> In my opinion, it doesn't make sense to implement an underwhelming
> >>> package
> >>>> split right now just because we have agreed to have *some* package
> >> split
> >>> in
> >>>> 2.5. Let's aim for happiness.
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Ilya Kasnacheev
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>>>
> >>>>>> 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> >>>>>>
> >>>>>>> If someone from PMCы or Committers still sees necessity about
> >>>> including
> >>>>>>> these tasks into Apache Ignite 2.5 release, this is the last
> >> chance
> >>>> to
> >>>>> do
> >>>>>>> so.
> >>>>>>> Otherwise this task will be moved to at 2.6 release at least, or
> >>> even
> >>>>>>> moved to backlog indefinitely.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]>
> >>> wrote:
> >>>>>>>>
> >>>>>>>> To top new RPM architecture off, update to release process is
> >>>>>> introduced
> >>>>>>> — [1] [2].
> >>>>>>>>
> >>>>>>>> Both tasks (this one and IGNITE-7647) are ready for review and
> >>>> should
> >>>>>> be
> >>>>>>> merged simultaneously.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-8172
> >>>>>>>> [2] https://github.com/apache/ignite-release/pull/1
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
> >>>> [hidden email]
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hello!
> >>>>>>>>>
> >>>>>>>>> Let me share my idea of how this shoud work. Splitting package
> >>>> into
> >>>>>>>>> sub-packages should be dependency-driven.
> >>>>>>>>>
> >>>>>>>>> It means that all Ignite modules without dependencies or with
> >>>> small
> >>>>>>>>> dependencies (such as ignite-log4j) should be included in
> >>>>> ignite-core.
> >>>>>>> It
> >>>>>>>>> doesn't make sense to make a zillion RPM packages.
> >>>>>>>>>
> >>>>>>>>> Critical things like ignite-spring and ignite-indexing should
> >> be
> >>>> in
> >>>>>>>>> ignite-core of course, even if they have dependencies.
> >>> Ignite-core
> >>>>>>> should
> >>>>>>>>> be fully self-sufficient and feature-complete.
> >>>>>>>>>
> >>>>>>>>> However, e.g. .net API should probably be in a separate
> >> package,
> >>>>>>> because it
> >>>>>>>>> should depend on mono | net-core. We may also have
> >> ignite-devel
> >>>>>> package
> >>>>>>>>> which should include all modules which only make sense for
> >>>>> developers
> >>>>>>> who
> >>>>>>>>> write code. Such as hibernate integration.
> >>>>>>>>>
> >>>>>>>>> I'm not sure about MR modules. The main question should be,
> >> does
> >>>> it
> >>>>>> have
> >>>>>>>>> dependencies? Can it run stand-alone without writing code?
> >>>>>>>>>
> >>>>>>>>> Hope this helps,
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Ilya Kasnacheev
> >>>>>>>>>
> >>>>>>>>> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> >>>>>>>>>
> >>>>>>>>>> Hi, Igniters!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Here are some news on our RPM packages initiative.
> >>>>>>>>>>
> >>>>>>>>>> 1. I’ve finished preliminary developing of Stage II version
> >> of
> >>>> RPM
> >>>>>>>>>> packages [1]. Main “new feature” is — split design. Also I’ve
> >>>> added
> >>>>>>>>>> package.sh script for automating package building process
> >> which
> >>>>> will
> >>>>>>> help
> >>>>>>>>>> organise corresponding builds in TC as well as simplify
> >> process
> >>>> for
> >>>>>>>>>> developers who wishes to have custom packages.
> >>>>>>>>>> PR#3703 [2] is ready for review. Denis, in order to catch up
> >>> with
> >>>>>>> Apache
> >>>>>>>>>> Ignite 2.5 release, I’d greatly appreciate your help in
> >> finding
> >>>>>>> reviewer.
> >>>>>>>>>> 2. With the help of ASF INFRA team, we now have RPM [3] and
> >> DEB
> >>>> [4]
> >>>>>>>>>> repositories on Apache Bintray. Though they are already
> >>> prepared
> >>>>> for
> >>>>>>>>>> hosting RPM and DEB packages respectively, and there is a way
> >>> of
> >>>>>>> linking
> >>>>>>>>>> them to apache.org/dist/ignite page, there is possible
> >>>> alternative
> >>>>>> in
> >>>>>>>>>> storing there only plain directory layout corresponding to
> >> each
> >>>>>>> repository
> >>>>>>>>>> type (RPM and DEB) and manage this layout (repodata,
> >>>> distributions,
> >>>>>>>>>> versions, etc.) by ourselves, having more control over
> >>>> repositories
> >>>>>> but
> >>>>>>>>>> lacking some simplicity of deploying new releases. WDYT?
> >> Should
> >>>> we
> >>>>>> try
> >>>>>>>>>> Cassandra approach? They are storing their DEB packages as I
> >>>>>> described
> >>>>>>>>>> above [5].
> >>>>>>>>>>
> >>>>>>>>>> Also — a question arose while I was working on this issue:
> >>> which
> >>>>> OSes
> >>>>>>> (and
> >>>>>>>>>> which versions of each) are we going to support (if we are
> >>> going)
> >>>>> in
> >>>>>>> terms
> >>>>>>>>>> of step-by-step list? Currently RPM packages are tested only
> >>> with
> >>>>>>> latest
> >>>>>>>>>> CentOS (and, respectively — RHEL), but there are a lot more
> >>>>> RPM-based
> >>>>>>>>>> distributives [6] some of which are more o less popular among
> >>> OS
> >>>>>>> community
> >>>>>>>>>> (ALT, Fedora, openSUSE, etc.).
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> >>>>>>>>>> [2] https://github.com/apache/ignite/pull/3703
> >>>>>>>>>> [3] https://bintray.com/apache/ignite-rpm
> >>>>>>>>>> [4] https://bintray.com/apache/ignite-deb
> >>>>>>>>>> [5] https://bintray.com/apache/cassandra/debian#files/
> >>>>>>>>>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> >>>>>>> distributions
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]>
> >>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> I suppose that most everything if not all from libs/options
> >>> will
> >>>>> go
> >>>>>> to
> >>>>>>>>>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> >>>>>>>>>>> More precise lib selection (if something from optional would
> >>>>> better
> >>>>>> to
> >>>>>>>>>> have in core package) will be discussed right after
> >> preliminary
> >>>>> split
> >>>>>>>>>> architecture agreement.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <
> >>> [hidden email]
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I like idea of keeping simple system of modules, so +1 from
> >>> me.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Where optional libs (e.g Direct IO plugin) would be
> >> included,
> >>>>> would
> >>>>>>> it
> >>>>>>>>>> be
> >>>>>>>>>>>> core or optional?
> >>>>>>>>>>>>
> >>>>>>>>>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <
> >> [hidden email]
> >>>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> How big would be a final core module?
> >>>>>>>>>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and
> >>>>> create
> >>>>>>> it’s
> >>>>>>>>>> own
> >>>>>>>>>>>>>> package.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree
> >>> with
> >>>>> Petr
> >>>>>>> and
> >>>>>>>>>>>>> propose the simplest modular system:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - core module that includes basic Ignite capabilities
> >>>> including
> >>>>>> SQL,
> >>>>>>>>>>>>> compute grid, service grid, k/v
> >>>>>>>>>>>>> - optional module hosts the rest - ML, streamers
> >> integration
> >>>>>> (kafka,
> >>>>>>>>>>>>> flink), kubernetes, etc.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> >>>>>> [hidden email]>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> *DEB package
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <
> >>> [hidden email]>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Considering that DEV package for now is almost platform
> >>>>>>> independent
> >>>>>>>>>>>>> (its
> >>>>>>>>>>>>>> a java application more or less), that package will work
> >>>> almost
> >>>>>> on
> >>>>>>> any
> >>>>>>>>>>>>>> DEB-based linux, including but not limited to Ubuntu,
> >>> Debian,
> >>>>>> etc.
> >>>>>>>>>>>>>>> The only restriction is existence of systemctl (systemd)
> >>>>> service
> >>>>>>>>>>>>> manager
> >>>>>>>>>>>>>> — we are dependent on it.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thats why, for instance, our RPM repository is called
> >>> simply
> >>>>>> ‘rpm’
> >>>>>>>>>> and
> >>>>>>>>>>>>>> package has no arch or dist suffix — it will work on
> >>> CentOS,
> >>>>>> RHEL,
> >>>>>>>>>>>>> Fedora,
> >>>>>>>>>>>>>> etc. with presence of aforementioned systemd.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> >>>>>>> [hidden email]>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Will Debian package work for Ubuntu?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> >>>>>>> [hidden email]>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Not a problem, rather nuisance. Also, when we will
> >> move
> >>> to
> >>>>>>> official
> >>>>>>>>>>>>>>>>> repositories, there can be a problem from OS
> >> community.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Concerning DEB packages — I plan to use RPM as base
> >> for
> >>>> DEB
> >>>>>>> package
> >>>>>>>>>>>>>> build
> >>>>>>>>>>>>>>>>> (package layout / install scripts) for speeding up
> >>> things
> >>>>> and
> >>>>>>>>>>>>> excluding
> >>>>>>>>>>>>>>>>> possible duplication and desynchronisation, so its a
> >>>> matter
> >>>>> of
> >>>>>>> ’sit
> >>>>>>>>>>>>>> and do’
> >>>>>>>>>>>>>>>>> rather then some technical research. Thats why I rose
> >>>>>> discussion
> >>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>> future package architecture, so that after agreement
> >> I'm
> >>>> be
> >>>>>>> able to
> >>>>>>>>>>>>>> pack
> >>>>>>>>>>>>>>>>> both RPM and DEB identically.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Yet, if you insist, I can create DEB package according
> >>> to
> >>>>>>> current
> >>>>>>>>>> RPM
> >>>>>>>>>>>>>>>>> layout in no time.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> >>>>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Peter,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I don't think the package size of 280M is going to
> >> be a
> >>>>>>> problem at
> >>>>>>>>>>>>>> all,
> >>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>> what you are suggesting can be an improvement down
> >> the
> >>>>> road.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> In the mean time, I think our top priority should be
> >> to
> >>>>>> provide
> >>>>>>>>>>>>>> packages
> >>>>>>>>>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly
> >>>>> enough.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Agree?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> >>>>>> [hidden email]>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi, Igniters!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Release 2.4 is almost there, at least binary part of
> >>> it,
> >>>>> so
> >>>>>>> I'd
> >>>>>>>>>>>>> like
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> move
> >>>>>>>>>>>>>>>>>>> forward to further improve and widen AI delivery
> >>> through
> >>>>>>>>>> packages.
> >>>>>>>>>>>>>>>>>>> As of now, Apache Ignite ships in RPM package
> >> weighing
> >>>>> about
> >>>>>>>>>> 280M+
> >>>>>>>>>>>>>> and,
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> improve usability and significantly reduce required
> >>>>> download
> >>>>>>>>>>>>> sizes, I
> >>>>>>>>>>>>>>>>>>> purpose that in 2.5 release we introduce splitted
> >>>> delivery
> >>>>>> as
> >>>>>>>>>>>>>> follows:
> >>>>>>>>>>>>>>>>>>> - CORE
> >>>>>>>>>>>>>>>>>>> - bin
> >>>>>>>>>>>>>>>>>>> - config
> >>>>>>>>>>>>>>>>>>> - libs (!optional)
> >>>>>>>>>>>>>>>>>>> - OPTIONAL LIBS
> >>>>>>>>>>>>>>>>>>> - BENCHMARKS
> >>>>>>>>>>>>>>>>>>> - DOCS (?)
> >>>>>>>>>>>>>>>>>>> - EXAMPLES
> >>>>>>>>>>>>>>>>>>> - .NET PLATFORM FILES
> >>>>>>>>>>>>>>>>>>> - C++ PLATFORM FILES
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> This architecture, as I assume, will add flexibility
> >>> (no
> >>>>>>> reason
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>> download
> >>>>>>>>>>>>>>>>>>> all 280M+ of binaries where you are to run only core
> >>>> node
> >>>>>>>>>>>>>> functionality)
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> maintainability (you are in full control of what is
> >>>>>> installed
> >>>>>>> on
> >>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>> system).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> After successful architecture choice, same scheme
> >> are
> >>>>>> planned
> >>>>>>> to
> >>>>>>>>>> be
> >>>>>>>>>>>>>>>>> used in
> >>>>>>>>>>>>>>>>>>> DEB packages as well.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>> Sent from:
> >>>>>> http://apache-ignite-developers.2346864.n4.nabble.
> >>>>>>>>>> com/
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

vveider

Re: Apache Ignite RPM packaging (Stage II)

There is not much time left for Apache Ignite 2.5 release, so let’s move stage II of packaging architecture implementation (with additional split scheme discussion) to 2.6 scope.

Instead, I’d like to include DEB package to Apache Ignite 2.5 release.
Corresponding PR is already prepared [1].

Also, I’ll try to prepare modifications for release procedure scripts to automate uploading packages to our new Bintray RPM and DEB repositories before the code freeze.

[1] https://github.com/apache/ignite/pull/3869 <https://github.com/apache/ignite/pull/3869>

> On 18 Apr 2018, at 14:44, Ilya Kasnacheev <[hidden email]> wrote:
>
> Copying anything manually to anywhere /usr (excluding /usr/local) is an
> example of slackwarization that package users and creators try to avoid.
>
>> By linux file hierarchy convention, home dir should be in /usr/lib
>
> Citation needed! I bet you're interpreting it wrong, since I've listed a
> random dir in /var/lib, and:
> ~/w/incubator-ignite% sudo ls -al /var/lib/lightdm
> drwxr-x--- 6 lightdm lightdm 4096 авг 14 2017 .
> drwxr-xr-x 89 root root 4096 мар 10 22:37 ..
> drwxrwxr-x 4 lightdm lightdm 4096 июл 19 2017 .cache
> drwxr-xr-x 5 lightdm lightdm 4096 июл 19 2017 .config
> drwx------ 2 lightdm lightdm 4096 апр 11 18:25 .gconf
> drwxr-xr-x 3 lightdm lightdm 4096 июл 19 2017 .local
> -rw------- 1 lightdm lightdm 283 апр 11 18:25 .Xauthority
>
> ...it definitely looks like a home dir. Which goes with additional benefit
> of being writable by end-user.
>
> --
> Ilya Kasnacheev
>
> 2018-04-16 9:22 GMT+03:00 Petr Ivanov <[hidden email]>:
>
>>
>>
>>> On 15 Apr 2018, at 10:19, Ilya Kasnacheev <[hidden email]>
>> wrote:
>>>
>>> Hello!
>>>
>>> With existing binary archive, user can move directories from
>>> apache-ignite-fabric/libs/optional to apache-ignite-fabric/libs to
>> activate
>>> them.
>>>
>>> But with RPM, user should not contemplate moving directories from
>>> /usr/lib/apache-ignite/optional to /usr/lib/apache-ignite.
>>
>> I always thought of that option as COPYING optional libs, not MOVING.
>>
>>
>>>
>>>
>>> User lacks permissions for that operation and it would defeat the purpose
>>> of having a RPM (imagine trying to upgrade Ignite RPM version with a
>> setup
>>> like that). "Turning distribution into Slackware" should not be an
>> offering.
>>
>> Can’t imagine use case where Apache Ignite software is installed by one
>> user, and run by another, without sudo/root permissions.
>> Yet, ignite user’s login shell is disabled indeed and without sudo/root
>> permissions optional libs cannot be copied.
>> Also — the symlinks is a good idea, but I’ve already thought of it and I’m
>> planning addition of special utility for enabling / disabling optional libs
>> (like a2enable/a2disable) in next development iteration.
>>
>>
>>>
>>> How it could work: Imagine we create /var/lib/ignite, use it as
>>> IGNITE_HOME, add symlinks from files in /usr/lib to /var/lib/ignite. This
>>> way, we can add and remove symlinks to modules in optional/, and thus
>>> enable and disable them as user sees fit.
>>
>> By linux file hierarchy convention, home dir should be in /usr/lib.
>> /var/lib/ is currently used for working files (MySQL alike).
>>
>>
>>>
>>> This also answers the problem of "what's IGNITE_HOME for visor launched
>>> from /usr/bin”
>>
>> That is addressed in separate task and will be fixed with minor startup
>> script redesign with universal location-independent solution.
>>
>>
>>>
>>> But obviously this will require dedication and effort.
>>
>> No problems with efforts after final design is discussed an agreed.
>>
>>
>>>
>>>
>>> --
>>> Ilya Kasnacheev
>>>
>>> 2018-04-14 8:03 GMT+03:00 Peter Ivanov <[hidden email]>:
>>>
>>>> Current packages design (after installation) does not differ from binary
>>>> archive - everything works (except necessity to run service instead
>>>> ignite.sh) just the same way, including libs/optional.
>>>>
>>>> Also, there can be issues with system JDK version by default, but every
>>>> problem will be in journalctl and/or /var/log, and package has strong
>>>> dependency on any implementation of JDK 1.8.
>>>>
>>>>
>>>> I am lacking enough feedback about Apache Ignite from packages from real
>>>> users, don’t know real use cases so still "moving in the dark".
>>>>
>>>>
>>>> On Fri, 13 Apr 2018 at 22:18, Denis Magda <[hidden email]> wrote:
>>>>
>>>>> Ilya,
>>>>>
>>>>> Thanks for your inputs. The reason why we decided to split Ignite into
>>>>> several packages mimics the reason why Java community introduced
>> modular
>>>>> subsystem for JDK. That's all about size. Ignite distribution is too
>> big,
>>>>> and we're trying to separate it into several components so that people
>>>> can
>>>>> install only the features they need.
>>>>>
>>>>> The point of a package is to ship something into root file system that
>>>> can
>>>>>> be used from root file system. If cpp files require compilation we
>>>> should
>>>>>> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
>>>>>> there's no mechanism to add optional libs to Ignite classpath, we
>>>> should
>>>>>> not ship optional libs. Moreover, some of 'optional' modules such as
>>>> yarn
>>>>>> don't make sense here because they're not supposed to be used with
>>>>>> standalone Ignite.
>>>>>
>>>>>
>>>>> Agree that we need to ship the code that is ready to be run. As for the
>>>>> classpath thing, if an optional package is installed into the root
>> (core)
>>>>> package directory, then its jars have to be added to "ignite/libs"
>>>> folder.
>>>>> After that, the one needs to restart a cluster node, nd it will add the
>>>>> just installed optional libs to the classpath. *Petr*, does it work
>> this
>>>>> way or can be implemented this way to address Ilya's concerns?
>>>>>
>>>>> --
>>>>> Denis
>>>>>
>>>>> On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <
>>>>> [hidden email]>
>>>>> wrote:
>>>>>
>>>>>> 2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:
>>>>>>
>>>>>>> On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <
>>>>> [hidden email]
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Moreover I did not find a way to start service if default installed
>>>>> JVM
>>>>>>> is
>>>>>>>> Java 7 :( I understand it's EOL, still this is something that hit
>>>> me.
>>>>>>>
>>>>>>>
>>>>>>> Apache Ignite >=2.4 does not support Java 7 - it is said in
>>>>>> documentation,
>>>>>>> DEVNOTES and even in startup scripts.
>>>>>>>
>>>>>>> I have Java 8 too, but I could not get service from package to start
>>>>>> Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
>>>>>> matter). Is it possible to specify it while running packaged Ignite?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> apache-ignite-libs is a totally unexpected package name.
>>>>> apache-ignite
>>>>>>> core
>>>>>>>> doesn't depend on it. It doesn't enable anything out of the box.
>>>> The
>>>>>>>> package is huge.
>>>>>>>
>>>>>>> ‘apache-ignite-libs’ is an aggregation package (for now) for all
>>>>> optional
>>>>>>> libs we are delivering. Possibly later they will be split more
>>>> granular
>>>>>> or
>>>>>>> even package per lib (like php, perl, python, etc. do for their
>>>> libs).
>>>>>>> This package dependency on ‘apache-ignite-core’ may seem confusing
>>>>>> though,
>>>>>>> I will try to explain it in IEP at least for current iteration.
>>>>>>>
>>>>>>
>>>>>> Okay, but how do you add optional libs to be included into Ignite
>>>>> classpath
>>>>>> while being launched by service? Is it even possible? If it isn't, I
>>>>> think
>>>>>> it doesn't make sense to ship apache-ignite-libs at all.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Further naming may become clear when we’ll start initiative on
>>>>> including
>>>>>>> packages to popular Linux distributions and theirs community will
>>>> join
>>>>>>> naming discussions.
>>>>>>>
>>>>>> Renaming packages once they're deployed widely will be a pain point to
>>>>> out
>>>>>> users. Some things should probably be thought out in advance.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Frankly speaking, I'm not sure that improvements over Stage I are
>>>>>> enough
>>>>>>> as
>>>>>>>> of now. For demo-like activity, we can probably go with one package
>>>>>> fits
>>>>>>>> all.
>>>>>>>>
>>>>>>>
>>>>>>> The process of finding the best package architecture is iterative,
>>>> but
>>>>>>> previously community agreed in split design proposed for 2.5 release.
>>>>>>>
>>>>>>> Also, split architecture is half of proposed improvements. The other
>>>>>> half -
>>>>>>> new process for deploying packages to Bintray (with virtually
>>>>> indefinite
>>>>>>> storage capabilities).
>>>>>>>
>>>>>> I think we could drop the split for now, or at least drop
>>>>>> apache-ignite-libs package at all. Probably also drop
>> apache-ignite-cpp
>>>>>> package and maybe apache-ignite-benchmarks.
>>>>>>
>>>>>> The point of a package is to ship something into root file system that
>>>>> can
>>>>>> be used from root file system. If cpp files require compilation we
>>>> should
>>>>>> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
>>>>>> there's no mechanism to add optional libs to Ignite classpath, we
>>>> should
>>>>>> not ship optional libs. Moreover, some of 'optional' modules such as
>>>> yarn
>>>>>> don't make sense here because they're not supposed to be used with
>>>>>> standalone Ignite.
>>>>>>
>>>>>> IMO it is not right to try and shove every file from Ignite
>>>> distribution
>>>>>> into some package. We should only put in packages things that can be
>>>>> used.
>>>>>> If something can't be used without copying it to a different FS
>>>> location,
>>>>>> it should be in examples or not packaged at all.
>>>>>>
>>>>>> In my opinion, it doesn't make sense to implement an underwhelming
>>>>> package
>>>>>> split right now just because we have agreed to have *some* package
>>>> split
>>>>> in
>>>>>> 2.5. Let's aim for happiness.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
>>>>>>>>
>>>>>>>>> If someone from PMCы or Committers still sees necessity about
>>>>>> including
>>>>>>>>> these tasks into Apache Ignite 2.5 release, this is the last
>>>> chance
>>>>>> to
>>>>>>> do
>>>>>>>>> so.
>>>>>>>>> Otherwise this task will be moved to at 2.6 release at least, or
>>>>> even
>>>>>>>>> moved to backlog indefinitely.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> To top new RPM architecture off, update to release process is
>>>>>>>> introduced
>>>>>>>>> — [1] [2].
>>>>>>>>>>
>>>>>>>>>> Both tasks (this one and IGNITE-7647) are ready for review and
>>>>>> should
>>>>>>>> be
>>>>>>>>> merged simultaneously.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-8172
>>>>>>>>>> [2] https://github.com/apache/ignite-release/pull/1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
>>>>>> [hidden email]
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello!
>>>>>>>>>>>
>>>>>>>>>>> Let me share my idea of how this shoud work. Splitting package
>>>>>> into
>>>>>>>>>>> sub-packages should be dependency-driven.
>>>>>>>>>>>
>>>>>>>>>>> It means that all Ignite modules without dependencies or with
>>>>>> small
>>>>>>>>>>> dependencies (such as ignite-log4j) should be included in
>>>>>>> ignite-core.
>>>>>>>>> It
>>>>>>>>>>> doesn't make sense to make a zillion RPM packages.
>>>>>>>>>>>
>>>>>>>>>>> Critical things like ignite-spring and ignite-indexing should
>>>> be
>>>>>> in
>>>>>>>>>>> ignite-core of course, even if they have dependencies.
>>>>> Ignite-core
>>>>>>>>> should
>>>>>>>>>>> be fully self-sufficient and feature-complete.
>>>>>>>>>>>
>>>>>>>>>>> However, e.g. .net API should probably be in a separate
>>>> package,
>>>>>>>>> because it
>>>>>>>>>>> should depend on mono | net-core. We may also have
>>>> ignite-devel
>>>>>>>> package
>>>>>>>>>>> which should include all modules which only make sense for
>>>>>>> developers
>>>>>>>>> who
>>>>>>>>>>> write code. Such as hibernate integration.
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure about MR modules. The main question should be,
>>>> does
>>>>>> it
>>>>>>>> have
>>>>>>>>>>> dependencies? Can it run stand-alone without writing code?
>>>>>>>>>>>
>>>>>>>>>>> Hope this helps,
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>>>
>>>>>>>>>>> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, Igniters!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Here are some news on our RPM packages initiative.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. I’ve finished preliminary developing of Stage II version
>>>> of
>>>>>> RPM
>>>>>>>>>>>> packages [1]. Main “new feature” is — split design. Also I’ve
>>>>>> added
>>>>>>>>>>>> package.sh script for automating package building process
>>>> which
>>>>>>> will
>>>>>>>>> help
>>>>>>>>>>>> organise corresponding builds in TC as well as simplify
>>>> process
>>>>>> for
>>>>>>>>>>>> developers who wishes to have custom packages.
>>>>>>>>>>>> PR#3703 [2] is ready for review. Denis, in order to catch up
>>>>> with
>>>>>>>>> Apache
>>>>>>>>>>>> Ignite 2.5 release, I’d greatly appreciate your help in
>>>> finding
>>>>>>>>> reviewer.
>>>>>>>>>>>> 2. With the help of ASF INFRA team, we now have RPM [3] and
>>>> DEB
>>>>>> [4]
>>>>>>>>>>>> repositories on Apache Bintray. Though they are already
>>>>> prepared
>>>>>>> for
>>>>>>>>>>>> hosting RPM and DEB packages respectively, and there is a way
>>>>> of
>>>>>>>>> linking
>>>>>>>>>>>> them to apache.org/dist/ignite page, there is possible
>>>>>> alternative
>>>>>>>> in
>>>>>>>>>>>> storing there only plain directory layout corresponding to
>>>> each
>>>>>>>>> repository
>>>>>>>>>>>> type (RPM and DEB) and manage this layout (repodata,
>>>>>> distributions,
>>>>>>>>>>>> versions, etc.) by ourselves, having more control over
>>>>>> repositories
>>>>>>>> but
>>>>>>>>>>>> lacking some simplicity of deploying new releases. WDYT?
>>>> Should
>>>>>> we
>>>>>>>> try
>>>>>>>>>>>> Cassandra approach? They are storing their DEB packages as I
>>>>>>>> described
>>>>>>>>>>>> above [5].
>>>>>>>>>>>>
>>>>>>>>>>>> Also — a question arose while I was working on this issue:
>>>>> which
>>>>>>> OSes
>>>>>>>>> (and
>>>>>>>>>>>> which versions of each) are we going to support (if we are
>>>>> going)
>>>>>>> in
>>>>>>>>> terms
>>>>>>>>>>>> of step-by-step list? Currently RPM packages are tested only
>>>>> with
>>>>>>>>> latest
>>>>>>>>>>>> CentOS (and, respectively — RHEL), but there are a lot more
>>>>>>> RPM-based
>>>>>>>>>>>> distributives [6] some of which are more o less popular among
>>>>> OS
>>>>>>>>> community
>>>>>>>>>>>> (ALT, Fedora, openSUSE, etc.).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/3703
>>>>>>>>>>>> [3] https://bintray.com/apache/ignite-rpm
>>>>>>>>>>>> [4] https://bintray.com/apache/ignite-deb
>>>>>>>>>>>> [5] https://bintray.com/apache/cassandra/debian#files/
>>>>>>>>>>>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
>>>>>>>>> distributions
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I suppose that most everything if not all from libs/options
>>>>> will
>>>>>>> go
>>>>>>>> to
>>>>>>>>>>>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
>>>>>>>>>>>>> More precise lib selection (if something from optional would
>>>>>>> better
>>>>>>>> to
>>>>>>>>>>>> have in core package) will be discussed right after
>>>> preliminary
>>>>>>> split
>>>>>>>>>>>> architecture agreement.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <
>>>>> [hidden email]
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I like idea of keeping simple system of modules, so +1 from
>>>>> me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Where optional libs (e.g Direct IO plugin) would be
>>>> included,
>>>>>>> would
>>>>>>>>> it
>>>>>>>>>>>> be
>>>>>>>>>>>>>> core or optional?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <
>>>> [hidden email]
>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How big would be a final core module?
>>>>>>>>>>>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and
>>>>>>> create
>>>>>>>>> it’s
>>>>>>>>>>>> own
>>>>>>>>>>>>>>>> package.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree
>>>>> with
>>>>>>> Petr
>>>>>>>>> and
>>>>>>>>>>>>>>> propose the simplest modular system:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - core module that includes basic Ignite capabilities
>>>>>> including
>>>>>>>> SQL,
>>>>>>>>>>>>>>> compute grid, service grid, k/v
>>>>>>>>>>>>>>> - optional module hosts the rest - ML, streamers
>>>> integration
>>>>>>>> (kafka,
>>>>>>>>>>>>>>> flink), kubernetes, etc.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
>>>>>>>> [hidden email]>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *DEB package
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <
>>>>> [hidden email]>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Considering that DEV package for now is almost platform
>>>>>>>>> independent
>>>>>>>>>>>>>>> (its
>>>>>>>>>>>>>>>> a java application more or less), that package will work
>>>>>> almost
>>>>>>>> on
>>>>>>>>> any
>>>>>>>>>>>>>>>> DEB-based linux, including but not limited to Ubuntu,
>>>>> Debian,
>>>>>>>> etc.
>>>>>>>>>>>>>>>>> The only restriction is existence of systemctl (systemd)
>>>>>>> service
>>>>>>>>>>>>>>> manager
>>>>>>>>>>>>>>>> — we are dependent on it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thats why, for instance, our RPM repository is called
>>>>> simply
>>>>>>>> ‘rpm’
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> package has no arch or dist suffix — it will work on
>>>>> CentOS,
>>>>>>>> RHEL,
>>>>>>>>>>>>>>> Fedora,
>>>>>>>>>>>>>>>> etc. with presence of aforementioned systemd.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
>>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Will Debian package work for Ubuntu?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
>>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Not a problem, rather nuisance. Also, when we will
>>>> move
>>>>> to
>>>>>>>>> official
>>>>>>>>>>>>>>>>>>> repositories, there can be a problem from OS
>>>> community.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Concerning DEB packages — I plan to use RPM as base
>>>> for
>>>>>> DEB
>>>>>>>>> package
>>>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>>>>> (package layout / install scripts) for speeding up
>>>>> things
>>>>>>> and
>>>>>>>>>>>>>>> excluding
>>>>>>>>>>>>>>>>>>> possible duplication and desynchronisation, so its a
>>>>>> matter
>>>>>>> of
>>>>>>>>> ’sit
>>>>>>>>>>>>>>>> and do’
>>>>>>>>>>>>>>>>>>> rather then some technical research. Thats why I rose
>>>>>>>> discussion
>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>> future package architecture, so that after agreement
>>>> I'm
>>>>>> be
>>>>>>>>> able to
>>>>>>>>>>>>>>>> pack
>>>>>>>>>>>>>>>>>>> both RPM and DEB identically.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yet, if you insist, I can create DEB package according
>>>>> to
>>>>>>>>> current
>>>>>>>>>>>> RPM
>>>>>>>>>>>>>>>>>>> layout in no time.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
>>>>>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Peter,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I don't think the package size of 280M is going to
>>>> be a
>>>>>>>>> problem at
>>>>>>>>>>>>>>>> all,
>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> what you are suggesting can be an improvement down
>>>> the
>>>>>>> road.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In the mean time, I think our top priority should be
>>>> to
>>>>>>>> provide
>>>>>>>>>>>>>>>> packages
>>>>>>>>>>>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly
>>>>>>> enough.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Agree?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
>>>>>>>> [hidden email]>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi, Igniters!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Release 2.4 is almost there, at least binary part of
>>>>> it,
>>>>>>> so
>>>>>>>>> I'd
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>>>> forward to further improve and widen AI delivery
>>>>> through
>>>>>>>>>>>> packages.
>>>>>>>>>>>>>>>>>>>>> As of now, Apache Ignite ships in RPM package
>>>> weighing
>>>>>>> about
>>>>>>>>>>>> 280M+
>>>>>>>>>>>>>>>> and,
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> improve usability and significantly reduce required
>>>>>>> download
>>>>>>>>>>>>>>> sizes, I
>>>>>>>>>>>>>>>>>>>>> purpose that in 2.5 release we introduce splitted
>>>>>> delivery
>>>>>>>> as
>>>>>>>>>>>>>>>> follows:
>>>>>>>>>>>>>>>>>>>>> - CORE
>>>>>>>>>>>>>>>>>>>>> - bin
>>>>>>>>>>>>>>>>>>>>> - config
>>>>>>>>>>>>>>>>>>>>> - libs (!optional)
>>>>>>>>>>>>>>>>>>>>> - OPTIONAL LIBS
>>>>>>>>>>>>>>>>>>>>> - BENCHMARKS
>>>>>>>>>>>>>>>>>>>>> - DOCS (?)
>>>>>>>>>>>>>>>>>>>>> - EXAMPLES
>>>>>>>>>>>>>>>>>>>>> - .NET PLATFORM FILES
>>>>>>>>>>>>>>>>>>>>> - C++ PLATFORM FILES
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This architecture, as I assume, will add flexibility
>>>>> (no
>>>>>>>>> reason
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> download
>>>>>>>>>>>>>>>>>>>>> all 280M+ of binaries where you are to run only core
>>>>>> node
>>>>>>>>>>>>>>>> functionality)
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> maintainability (you are in full control of what is
>>>>>>>> installed
>>>>>>>>> on
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>> system).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> After successful architecture choice, same scheme
>>>> are
>>>>>>>> planned
>>>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> used in
>>>>>>>>>>>>>>>>>>>>> DEB packages as well.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Sent from:
>>>>>>>> http://apache-ignite-developers.2346864.n4.nabble.
>>>>>>>>>>>> com/
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>

Ilya Kasnacheev

Re: Apache Ignite RPM packaging (Stage II)

Hello!

I recommend inclusion of this change so that it can make way into 2.5.

Regards,

--
Ilya Kasnacheev

2018-04-19 9:03 GMT+03:00 Petr Ivanov <[hidden email]>:

> There is not much time left for Apache Ignite 2.5 release, so let’s move
> stage II of packaging architecture implementation (with additional split
> scheme discussion) to 2.6 scope.
>
> Instead, I’d like to include DEB package to Apache Ignite 2.5 release.
> Corresponding PR is already prepared [1].
>
> Also, I’ll try to prepare modifications for release procedure scripts to
> automate uploading packages to our new Bintray RPM and DEB repositories
> before the code freeze.
>
>
> [1] https://github.com/apache/ignite/pull/3869 <https://github.com/apache/
> ignite/pull/3869>
>
>
>
> > On 18 Apr 2018, at 14:44, Ilya Kasnacheev <[hidden email]>
> wrote:
> >
> > Copying anything manually to anywhere /usr (excluding /usr/local) is an
> > example of slackwarization that package users and creators try to avoid.
> >
> >> By linux file hierarchy convention, home dir should be in /usr/lib
> >
> > Citation needed! I bet you're interpreting it wrong, since I've listed a
> > random dir in /var/lib, and:
> > ~/w/incubator-ignite% sudo ls -al /var/lib/lightdm
> > drwxr-x--- 6 lightdm lightdm 4096 авг 14 2017 .
> > drwxr-xr-x 89 root root 4096 мар 10 22:37 ..
> > drwxrwxr-x 4 lightdm lightdm 4096 июл 19 2017 .cache
> > drwxr-xr-x 5 lightdm lightdm 4096 июл 19 2017 .config
> > drwx------ 2 lightdm lightdm 4096 апр 11 18:25 .gconf
> > drwxr-xr-x 3 lightdm lightdm 4096 июл 19 2017 .local
> > -rw------- 1 lightdm lightdm 283 апр 11 18:25 .Xauthority
> >
> > ...it definitely looks like a home dir. Which goes with additional
> benefit
> > of being writable by end-user.
> >
> > --
> > Ilya Kasnacheev
> >
> > 2018-04-16 9:22 GMT+03:00 Petr Ivanov <[hidden email]>:
> >
> >>
> >>
> >>> On 15 Apr 2018, at 10:19, Ilya Kasnacheev <[hidden email]>
> >> wrote:
> >>>
> >>> Hello!
> >>>
> >>> With existing binary archive, user can move directories from
> >>> apache-ignite-fabric/libs/optional to apache-ignite-fabric/libs to
> >> activate
> >>> them.
> >>>
> >>> But with RPM, user should not contemplate moving directories from
> >>> /usr/lib/apache-ignite/optional to /usr/lib/apache-ignite.
> >>
> >> I always thought of that option as COPYING optional libs, not MOVING.
> >>
> >>
> >>>
> >>>
> >>> User lacks permissions for that operation and it would defeat the
> purpose
> >>> of having a RPM (imagine trying to upgrade Ignite RPM version with a
> >> setup
> >>> like that). "Turning distribution into Slackware" should not be an
> >> offering.
> >>
> >> Can’t imagine use case where Apache Ignite software is installed by one
> >> user, and run by another, without sudo/root permissions.
> >> Yet, ignite user’s login shell is disabled indeed and without sudo/root
> >> permissions optional libs cannot be copied.
> >> Also — the symlinks is a good idea, but I’ve already thought of it and
> I’m
> >> planning addition of special utility for enabling / disabling optional
> libs
> >> (like a2enable/a2disable) in next development iteration.
> >>
> >>
> >>>
> >>> How it could work: Imagine we create /var/lib/ignite, use it as
> >>> IGNITE_HOME, add symlinks from files in /usr/lib to /var/lib/ignite.
> This
> >>> way, we can add and remove symlinks to modules in optional/, and thus
> >>> enable and disable them as user sees fit.
> >>
> >> By linux file hierarchy convention, home dir should be in /usr/lib.
> >> /var/lib/ is currently used for working files (MySQL alike).
> >>
> >>
> >>>
> >>> This also answers the problem of "what's IGNITE_HOME for visor launched
> >>> from /usr/bin”
> >>
> >> That is addressed in separate task and will be fixed with minor startup
> >> script redesign with universal location-independent solution.
> >>
> >>
> >>>
> >>> But obviously this will require dedication and effort.
> >>
> >> No problems with efforts after final design is discussed an agreed.
> >>
> >>
> >>>
> >>>
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>> 2018-04-14 8:03 GMT+03:00 Peter Ivanov <[hidden email]>:
> >>>
> >>>> Current packages design (after installation) does not differ from
> binary
> >>>> archive - everything works (except necessity to run service instead
> >>>> ignite.sh) just the same way, including libs/optional.
> >>>>
> >>>> Also, there can be issues with system JDK version by default, but
> every
> >>>> problem will be in journalctl and/or /var/log, and package has strong
> >>>> dependency on any implementation of JDK 1.8.
> >>>>
> >>>>
> >>>> I am lacking enough feedback about Apache Ignite from packages from
> real
> >>>> users, don’t know real use cases so still "moving in the dark".
> >>>>
> >>>>
> >>>> On Fri, 13 Apr 2018 at 22:18, Denis Magda <[hidden email]> wrote:
> >>>>
> >>>>> Ilya,
> >>>>>
> >>>>> Thanks for your inputs. The reason why we decided to split Ignite
> into
> >>>>> several packages mimics the reason why Java community introduced
> >> modular
> >>>>> subsystem for JDK. That's all about size. Ignite distribution is too
> >> big,
> >>>>> and we're trying to separate it into several components so that
> people
> >>>> can
> >>>>> install only the features they need.
> >>>>>
> >>>>> The point of a package is to ship something into root file system
> that
> >>>> can
> >>>>>> be used from root file system. If cpp files require compilation we
> >>>> should
> >>>>>> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> >>>>>> there's no mechanism to add optional libs to Ignite classpath, we
> >>>> should
> >>>>>> not ship optional libs. Moreover, some of 'optional' modules such as
> >>>> yarn
> >>>>>> don't make sense here because they're not supposed to be used with
> >>>>>> standalone Ignite.
> >>>>>
> >>>>>
> >>>>> Agree that we need to ship the code that is ready to be run. As for
> the
> >>>>> classpath thing, if an optional package is installed into the root
> >> (core)
> >>>>> package directory, then its jars have to be added to "ignite/libs"
> >>>> folder.
> >>>>> After that, the one needs to restart a cluster node, nd it will add
> the
> >>>>> just installed optional libs to the classpath. *Petr*, does it work
> >> this
> >>>>> way or can be implemented this way to address Ilya's concerns?
> >>>>>
> >>>>> --
> >>>>> Denis
> >>>>>
> >>>>> On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <
> >>>>> [hidden email]>
> >>>>> wrote:
> >>>>>
> >>>>>> 2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:
> >>>>>>
> >>>>>>> On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <
> >>>>> [hidden email]
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Moreover I did not find a way to start service if default
> installed
> >>>>> JVM
> >>>>>>> is
> >>>>>>>> Java 7 :( I understand it's EOL, still this is something that hit
> >>>> me.
> >>>>>>>
> >>>>>>>
> >>>>>>> Apache Ignite >=2.4 does not support Java 7 - it is said in
> >>>>>> documentation,
> >>>>>>> DEVNOTES and even in startup scripts.
> >>>>>>>
> >>>>>>> I have Java 8 too, but I could not get service from package to
> start
> >>>>>> Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that
> >>>>>> matter). Is it possible to specify it while running packaged Ignite?
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> apache-ignite-libs is a totally unexpected package name.
> >>>>> apache-ignite
> >>>>>>> core
> >>>>>>>> doesn't depend on it. It doesn't enable anything out of the box.
> >>>> The
> >>>>>>>> package is huge.
> >>>>>>>
> >>>>>>> ‘apache-ignite-libs’ is an aggregation package (for now) for all
> >>>>> optional
> >>>>>>> libs we are delivering. Possibly later they will be split more
> >>>> granular
> >>>>>> or
> >>>>>>> even package per lib (like php, perl, python, etc. do for their
> >>>> libs).
> >>>>>>> This package dependency on ‘apache-ignite-core’ may seem confusing
> >>>>>> though,
> >>>>>>> I will try to explain it in IEP at least for current iteration.
> >>>>>>>
> >>>>>>
> >>>>>> Okay, but how do you add optional libs to be included into Ignite
> >>>>> classpath
> >>>>>> while being launched by service? Is it even possible? If it isn't, I
> >>>>> think
> >>>>>> it doesn't make sense to ship apache-ignite-libs at all.
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> Further naming may become clear when we’ll start initiative on
> >>>>> including
> >>>>>>> packages to popular Linux distributions and theirs community will
> >>>> join
> >>>>>>> naming discussions.
> >>>>>>>
> >>>>>> Renaming packages once they're deployed widely will be a pain point
> to
> >>>>> out
> >>>>>> users. Some things should probably be thought out in advance.
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Frankly speaking, I'm not sure that improvements over Stage I are
> >>>>>> enough
> >>>>>>> as
> >>>>>>>> of now. For demo-like activity, we can probably go with one
> package
> >>>>>> fits
> >>>>>>>> all.
> >>>>>>>>
> >>>>>>>
> >>>>>>> The process of finding the best package architecture is iterative,
> >>>> but
> >>>>>>> previously community agreed in split design proposed for 2.5
> release.
> >>>>>>>
> >>>>>>> Also, split architecture is half of proposed improvements. The
> other
> >>>>>> half -
> >>>>>>> new process for deploying packages to Bintray (with virtually
> >>>>> indefinite
> >>>>>>> storage capabilities).
> >>>>>>>
> >>>>>> I think we could drop the split for now, or at least drop
> >>>>>> apache-ignite-libs package at all. Probably also drop
> >> apache-ignite-cpp
> >>>>>> package and maybe apache-ignite-benchmarks.
> >>>>>>
> >>>>>> The point of a package is to ship something into root file system
> that
> >>>>> can
> >>>>>> be used from root file system. If cpp files require compilation we
> >>>> should
> >>>>>> not ship them, or ship them to 'examples'. Ditto with benchmarks. If
> >>>>>> there's no mechanism to add optional libs to Ignite classpath, we
> >>>> should
> >>>>>> not ship optional libs. Moreover, some of 'optional' modules such as
> >>>> yarn
> >>>>>> don't make sense here because they're not supposed to be used with
> >>>>>> standalone Ignite.
> >>>>>>
> >>>>>> IMO it is not right to try and shove every file from Ignite
> >>>> distribution
> >>>>>> into some package. We should only put in packages things that can be
> >>>>> used.
> >>>>>> If something can't be used without copying it to a different FS
> >>>> location,
> >>>>>> it should be in examples or not packaged at all.
> >>>>>>
> >>>>>> In my opinion, it doesn't make sense to implement an underwhelming
> >>>>> package
> >>>>>> split right now just because we have agreed to have *some* package
> >>>> split
> >>>>> in
> >>>>>> 2.5. Let's aim for happiness.
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Ilya Kasnacheev
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>> 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> >>>>>>>>
> >>>>>>>>> If someone from PMCы or Committers still sees necessity about
> >>>>>> including
> >>>>>>>>> these tasks into Apache Ignite 2.5 release, this is the last
> >>>> chance
> >>>>>> to
> >>>>>>> do
> >>>>>>>>> so.
> >>>>>>>>> Otherwise this task will be moved to at 2.6 release at least, or
> >>>>> even
> >>>>>>>>> moved to backlog indefinitely.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> To top new RPM architecture off, update to release process is
> >>>>>>>> introduced
> >>>>>>>>> — [1] [2].
> >>>>>>>>>>
> >>>>>>>>>> Both tasks (this one and IGNITE-7647) are ready for review and
> >>>>>> should
> >>>>>>>> be
> >>>>>>>>> merged simultaneously.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-8172
> >>>>>>>>>> [2] https://github.com/apache/ignite-release/pull/1
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
> >>>>>> [hidden email]
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hello!
> >>>>>>>>>>>
> >>>>>>>>>>> Let me share my idea of how this shoud work. Splitting package
> >>>>>> into
> >>>>>>>>>>> sub-packages should be dependency-driven.
> >>>>>>>>>>>
> >>>>>>>>>>> It means that all Ignite modules without dependencies or with
> >>>>>> small
> >>>>>>>>>>> dependencies (such as ignite-log4j) should be included in
> >>>>>>> ignite-core.
> >>>>>>>>> It
> >>>>>>>>>>> doesn't make sense to make a zillion RPM packages.
> >>>>>>>>>>>
> >>>>>>>>>>> Critical things like ignite-spring and ignite-indexing should
> >>>> be
> >>>>>> in
> >>>>>>>>>>> ignite-core of course, even if they have dependencies.
> >>>>> Ignite-core
> >>>>>>>>> should
> >>>>>>>>>>> be fully self-sufficient and feature-complete.
> >>>>>>>>>>>
> >>>>>>>>>>> However, e.g. .net API should probably be in a separate
> >>>> package,
> >>>>>>>>> because it
> >>>>>>>>>>> should depend on mono | net-core. We may also have
> >>>> ignite-devel
> >>>>>>>> package
> >>>>>>>>>>> which should include all modules which only make sense for
> >>>>>>> developers
> >>>>>>>>> who
> >>>>>>>>>>> write code. Such as hibernate integration.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not sure about MR modules. The main question should be,
> >>>> does
> >>>>>> it
> >>>>>>>> have
> >>>>>>>>>>> dependencies? Can it run stand-alone without writing code?
> >>>>>>>>>>>
> >>>>>>>>>>> Hope this helps,
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Ilya Kasnacheev
> >>>>>>>>>>>
> >>>>>>>>>>> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi, Igniters!
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here are some news on our RPM packages initiative.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. I’ve finished preliminary developing of Stage II version
> >>>> of
> >>>>>> RPM
> >>>>>>>>>>>> packages [1]. Main “new feature” is — split design. Also I’ve
> >>>>>> added
> >>>>>>>>>>>> package.sh script for automating package building process
> >>>> which
> >>>>>>> will
> >>>>>>>>> help
> >>>>>>>>>>>> organise corresponding builds in TC as well as simplify
> >>>> process
> >>>>>> for
> >>>>>>>>>>>> developers who wishes to have custom packages.
> >>>>>>>>>>>> PR#3703 [2] is ready for review. Denis, in order to catch up
> >>>>> with
> >>>>>>>>> Apache
> >>>>>>>>>>>> Ignite 2.5 release, I’d greatly appreciate your help in
> >>>> finding
> >>>>>>>>> reviewer.
> >>>>>>>>>>>> 2. With the help of ASF INFRA team, we now have RPM [3] and
> >>>> DEB
> >>>>>> [4]
> >>>>>>>>>>>> repositories on Apache Bintray. Though they are already
> >>>>> prepared
> >>>>>>> for
> >>>>>>>>>>>> hosting RPM and DEB packages respectively, and there is a way
> >>>>> of
> >>>>>>>>> linking
> >>>>>>>>>>>> them to apache.org/dist/ignite page, there is possible
> >>>>>> alternative
> >>>>>>>> in
> >>>>>>>>>>>> storing there only plain directory layout corresponding to
> >>>> each
> >>>>>>>>> repository
> >>>>>>>>>>>> type (RPM and DEB) and manage this layout (repodata,
> >>>>>> distributions,
> >>>>>>>>>>>> versions, etc.) by ourselves, having more control over
> >>>>>> repositories
> >>>>>>>> but
> >>>>>>>>>>>> lacking some simplicity of deploying new releases. WDYT?
> >>>> Should
> >>>>>> we
> >>>>>>>> try
> >>>>>>>>>>>> Cassandra approach? They are storing their DEB packages as I
> >>>>>>>> described
> >>>>>>>>>>>> above [5].
> >>>>>>>>>>>>
> >>>>>>>>>>>> Also — a question arose while I was working on this issue:
> >>>>> which
> >>>>>>> OSes
> >>>>>>>>> (and
> >>>>>>>>>>>> which versions of each) are we going to support (if we are
> >>>>> going)
> >>>>>>> in
> >>>>>>>>> terms
> >>>>>>>>>>>> of step-by-step list? Currently RPM packages are tested only
> >>>>> with
> >>>>>>>>> latest
> >>>>>>>>>>>> CentOS (and, respectively — RHEL), but there are a lot more
> >>>>>>> RPM-based
> >>>>>>>>>>>> distributives [6] some of which are more o less popular among
> >>>>> OS
> >>>>>>>>> community
> >>>>>>>>>>>> (ALT, Fedora, openSUSE, etc.).
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> >>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/3703
> >>>>>>>>>>>> [3] https://bintray.com/apache/ignite-rpm
> >>>>>>>>>>>> [4] https://bintray.com/apache/ignite-deb
> >>>>>>>>>>>> [5] https://bintray.com/apache/cassandra/debian#files/
> >>>>>>>>>>>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> >>>>>>>>> distributions
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]>
> >>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I suppose that most everything if not all from libs/options
> >>>>> will
> >>>>>>> go
> >>>>>>>> to
> >>>>>>>>>>>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> >>>>>>>>>>>>> More precise lib selection (if something from optional would
> >>>>>>> better
> >>>>>>>> to
> >>>>>>>>>>>> have in core package) will be discussed right after
> >>>> preliminary
> >>>>>>> split
> >>>>>>>>>>>> architecture agreement.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <
> >>>>> [hidden email]
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I like idea of keeping simple system of modules, so +1 from
> >>>>> me.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Where optional libs (e.g Direct IO plugin) would be
> >>>> included,
> >>>>>>> would
> >>>>>>>>> it
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>> core or optional?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <
> >>>> [hidden email]
> >>>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> How big would be a final core module?
> >>>>>>>>>>>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and
> >>>>>>> create
> >>>>>>>>> it’s
> >>>>>>>>>>>> own
> >>>>>>>>>>>>>>>> package.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree
> >>>>> with
> >>>>>>> Petr
> >>>>>>>>> and
> >>>>>>>>>>>>>>> propose the simplest modular system:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - core module that includes basic Ignite capabilities
> >>>>>> including
> >>>>>>>> SQL,
> >>>>>>>>>>>>>>> compute grid, service grid, k/v
> >>>>>>>>>>>>>>> - optional module hosts the rest - ML, streamers
> >>>> integration
> >>>>>>>> (kafka,
> >>>>>>>>>>>>>>> flink), kubernetes, etc.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> >>>>>>>> [hidden email]>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *DEB package
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <
> >>>>> [hidden email]>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Considering that DEV package for now is almost platform
> >>>>>>>>> independent
> >>>>>>>>>>>>>>> (its
> >>>>>>>>>>>>>>>> a java application more or less), that package will work
> >>>>>> almost
> >>>>>>>> on
> >>>>>>>>> any
> >>>>>>>>>>>>>>>> DEB-based linux, including but not limited to Ubuntu,
> >>>>> Debian,
> >>>>>>>> etc.
> >>>>>>>>>>>>>>>>> The only restriction is existence of systemctl (systemd)
> >>>>>>> service
> >>>>>>>>>>>>>>> manager
> >>>>>>>>>>>>>>>> — we are dependent on it.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thats why, for instance, our RPM repository is called
> >>>>> simply
> >>>>>>>> ‘rpm’
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> package has no arch or dist suffix — it will work on
> >>>>> CentOS,
> >>>>>>>> RHEL,
> >>>>>>>>>>>>>>> Fedora,
> >>>>>>>>>>>>>>>> etc. with presence of aforementioned systemd.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> >>>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Will Debian package work for Ubuntu?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> >>>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Not a problem, rather nuisance. Also, when we will
> >>>> move
> >>>>> to
> >>>>>>>>> official
> >>>>>>>>>>>>>>>>>>> repositories, there can be a problem from OS
> >>>> community.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Concerning DEB packages — I plan to use RPM as base
> >>>> for
> >>>>>> DEB
> >>>>>>>>> package
> >>>>>>>>>>>>>>>> build
> >>>>>>>>>>>>>>>>>>> (package layout / install scripts) for speeding up
> >>>>> things
> >>>>>>> and
> >>>>>>>>>>>>>>> excluding
> >>>>>>>>>>>>>>>>>>> possible duplication and desynchronisation, so its a
> >>>>>> matter
> >>>>>>> of
> >>>>>>>>> ’sit
> >>>>>>>>>>>>>>>> and do’
> >>>>>>>>>>>>>>>>>>> rather then some technical research. Thats why I rose
> >>>>>>>> discussion
> >>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>> future package architecture, so that after agreement
> >>>> I'm
> >>>>>> be
> >>>>>>>>> able to
> >>>>>>>>>>>>>>>> pack
> >>>>>>>>>>>>>>>>>>> both RPM and DEB identically.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yet, if you insist, I can create DEB package according
> >>>>> to
> >>>>>>>>> current
> >>>>>>>>>>>> RPM
> >>>>>>>>>>>>>>>>>>> layout in no time.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> >>>>>>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Peter,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I don't think the package size of 280M is going to
> >>>> be a
> >>>>>>>>> problem at
> >>>>>>>>>>>>>>>> all,
> >>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>> what you are suggesting can be an improvement down
> >>>> the
> >>>>>>> road.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> In the mean time, I think our top priority should be
> >>>> to
> >>>>>>>> provide
> >>>>>>>>>>>>>>>> packages
> >>>>>>>>>>>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly
> >>>>>>> enough.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Agree?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> >>>>>>>> [hidden email]>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi, Igniters!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Release 2.4 is almost there, at least binary part of
> >>>>> it,
> >>>>>>> so
> >>>>>>>>> I'd
> >>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> move
> >>>>>>>>>>>>>>>>>>>>> forward to further improve and widen AI delivery
> >>>>> through
> >>>>>>>>>>>> packages.
> >>>>>>>>>>>>>>>>>>>>> As of now, Apache Ignite ships in RPM package
> >>>> weighing
> >>>>>>> about
> >>>>>>>>>>>> 280M+
> >>>>>>>>>>>>>>>> and,
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> improve usability and significantly reduce required
> >>>>>>> download
> >>>>>>>>>>>>>>> sizes, I
> >>>>>>>>>>>>>>>>>>>>> purpose that in 2.5 release we introduce splitted
> >>>>>> delivery
> >>>>>>>> as
> >>>>>>>>>>>>>>>> follows:
> >>>>>>>>>>>>>>>>>>>>> - CORE
> >>>>>>>>>>>>>>>>>>>>> - bin
> >>>>>>>>>>>>>>>>>>>>> - config
> >>>>>>>>>>>>>>>>>>>>> - libs (!optional)
> >>>>>>>>>>>>>>>>>>>>> - OPTIONAL LIBS
> >>>>>>>>>>>>>>>>>>>>> - BENCHMARKS
> >>>>>>>>>>>>>>>>>>>>> - DOCS (?)
> >>>>>>>>>>>>>>>>>>>>> - EXAMPLES
> >>>>>>>>>>>>>>>>>>>>> - .NET PLATFORM FILES
> >>>>>>>>>>>>>>>>>>>>> - C++ PLATFORM FILES
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This architecture, as I assume, will add flexibility
> >>>>> (no
> >>>>>>>>> reason
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> download
> >>>>>>>>>>>>>>>>>>>>> all 280M+ of binaries where you are to run only core
> >>>>>> node
> >>>>>>>>>>>>>>>> functionality)
> >>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> maintainability (you are in full control of what is
> >>>>>>>> installed
> >>>>>>>>> on
> >>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>> system).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> After successful architecture choice, same scheme
> >>>> are
> >>>>>>>> planned
> >>>>>>>>> to
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> used in
> >>>>>>>>>>>>>>>>>>>>> DEB packages as well.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>> Sent from:
> >>>>>>>> http://apache-ignite-developers.2346864.n4.nabble.
> >>>>>>>>>>>> com/
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

dsetrakyan

Re: Apache Ignite RPM packaging (Stage II)

+1

On Thu, Apr 19, 2018 at 5:45 AM, Ilya Kasnacheev <[hidden email]>
wrote:

> Hello!
>
> I recommend inclusion of this change so that it can make way into 2.5.
>
> Regards,
>
> --
> Ilya Kasnacheev
>
> 2018-04-19 9:03 GMT+03:00 Petr Ivanov <[hidden email]>:
>
> > There is not much time left for Apache Ignite 2.5 release, so let’s move
> > stage II of packaging architecture implementation (with additional split
> > scheme discussion) to 2.6 scope.
> >
> > Instead, I’d like to include DEB package to Apache Ignite 2.5 release.
> > Corresponding PR is already prepared [1].
> >
> > Also, I’ll try to prepare modifications for release procedure scripts to
> > automate uploading packages to our new Bintray RPM and DEB repositories
> > before the code freeze.
> >
> >
> > [1] https://github.com/apache/ignite/pull/3869 <
> https://github.com/apache/
> > ignite/pull/3869>
> >
> >
> >
> > > On 18 Apr 2018, at 14:44, Ilya Kasnacheev <[hidden email]>
> > wrote:
> > >
> > > Copying anything manually to anywhere /usr (excluding /usr/local) is an
> > > example of slackwarization that package users and creators try to
> avoid.
> > >
> > >> By linux file hierarchy convention, home dir should be in /usr/lib
> > >
> > > Citation needed! I bet you're interpreting it wrong, since I've listed
> a
> > > random dir in /var/lib, and:
> > > ~/w/incubator-ignite% sudo ls -al /var/lib/lightdm
> > > drwxr-x--- 6 lightdm lightdm 4096 авг 14 2017 .
> > > drwxr-xr-x 89 root root 4096 мар 10 22:37 ..
> > > drwxrwxr-x 4 lightdm lightdm 4096 июл 19 2017 .cache
> > > drwxr-xr-x 5 lightdm lightdm 4096 июл 19 2017 .config
> > > drwx------ 2 lightdm lightdm 4096 апр 11 18:25 .gconf
> > > drwxr-xr-x 3 lightdm lightdm 4096 июл 19 2017 .local
> > > -rw------- 1 lightdm lightdm 283 апр 11 18:25 .Xauthority
> > >
> > > ...it definitely looks like a home dir. Which goes with additional
> > benefit
> > > of being writable by end-user.
> > >
> > > --
> > > Ilya Kasnacheev
> > >
> > > 2018-04-16 9:22 GMT+03:00 Petr Ivanov <[hidden email]>:
> > >
> > >>
> > >>
> > >>> On 15 Apr 2018, at 10:19, Ilya Kasnacheev <[hidden email]
> >
> > >> wrote:
> > >>>
> > >>> Hello!
> > >>>
> > >>> With existing binary archive, user can move directories from
> > >>> apache-ignite-fabric/libs/optional to apache-ignite-fabric/libs to
> > >> activate
> > >>> them.
> > >>>
> > >>> But with RPM, user should not contemplate moving directories from
> > >>> /usr/lib/apache-ignite/optional to /usr/lib/apache-ignite.
> > >>
> > >> I always thought of that option as COPYING optional libs, not MOVING.
> > >>
> > >>
> > >>>
> > >>>
> > >>> User lacks permissions for that operation and it would defeat the
> > purpose
> > >>> of having a RPM (imagine trying to upgrade Ignite RPM version with a
> > >> setup
> > >>> like that). "Turning distribution into Slackware" should not be an
> > >> offering.
> > >>
> > >> Can’t imagine use case where Apache Ignite software is installed by
> one
> > >> user, and run by another, without sudo/root permissions.
> > >> Yet, ignite user’s login shell is disabled indeed and without
> sudo/root
> > >> permissions optional libs cannot be copied.
> > >> Also — the symlinks is a good idea, but I’ve already thought of it and
> > I’m
> > >> planning addition of special utility for enabling / disabling optional
> > libs
> > >> (like a2enable/a2disable) in next development iteration.
> > >>
> > >>
> > >>>
> > >>> How it could work: Imagine we create /var/lib/ignite, use it as
> > >>> IGNITE_HOME, add symlinks from files in /usr/lib to /var/lib/ignite.
> > This
> > >>> way, we can add and remove symlinks to modules in optional/, and thus
> > >>> enable and disable them as user sees fit.
> > >>
> > >> By linux file hierarchy convention, home dir should be in /usr/lib.
> > >> /var/lib/ is currently used for working files (MySQL alike).
> > >>
> > >>
> > >>>
> > >>> This also answers the problem of "what's IGNITE_HOME for visor
> launched
> > >>> from /usr/bin”
> > >>
> > >> That is addressed in separate task and will be fixed with minor
> startup
> > >> script redesign with universal location-independent solution.
> > >>
> > >>
> > >>>
> > >>> But obviously this will require dedication and effort.
> > >>
> > >> No problems with efforts after final design is discussed an agreed.
> > >>
> > >>
> > >>>
> > >>>
> > >>> --
> > >>> Ilya Kasnacheev
> > >>>
> > >>> 2018-04-14 8:03 GMT+03:00 Peter Ivanov <[hidden email]>:
> > >>>
> > >>>> Current packages design (after installation) does not differ from
> > binary
> > >>>> archive - everything works (except necessity to run service instead
> > >>>> ignite.sh) just the same way, including libs/optional.
> > >>>>
> > >>>> Also, there can be issues with system JDK version by default, but
> > every
> > >>>> problem will be in journalctl and/or /var/log, and package has
> strong
> > >>>> dependency on any implementation of JDK 1.8.
> > >>>>
> > >>>>
> > >>>> I am lacking enough feedback about Apache Ignite from packages from
> > real
> > >>>> users, don’t know real use cases so still "moving in the dark".
> > >>>>
> > >>>>
> > >>>> On Fri, 13 Apr 2018 at 22:18, Denis Magda <[hidden email]>
> wrote:
> > >>>>
> > >>>>> Ilya,
> > >>>>>
> > >>>>> Thanks for your inputs. The reason why we decided to split Ignite
> > into
> > >>>>> several packages mimics the reason why Java community introduced
> > >> modular
> > >>>>> subsystem for JDK. That's all about size. Ignite distribution is
> too
> > >> big,
> > >>>>> and we're trying to separate it into several components so that
> > people
> > >>>> can
> > >>>>> install only the features they need.
> > >>>>>
> > >>>>> The point of a package is to ship something into root file system
> > that
> > >>>> can
> > >>>>>> be used from root file system. If cpp files require compilation we
> > >>>> should
> > >>>>>> not ship them, or ship them to 'examples'. Ditto with benchmarks.
> If
> > >>>>>> there's no mechanism to add optional libs to Ignite classpath, we
> > >>>> should
> > >>>>>> not ship optional libs. Moreover, some of 'optional' modules such
> as
> > >>>> yarn
> > >>>>>> don't make sense here because they're not supposed to be used with
> > >>>>>> standalone Ignite.
> > >>>>>
> > >>>>>
> > >>>>> Agree that we need to ship the code that is ready to be run. As for
> > the
> > >>>>> classpath thing, if an optional package is installed into the root
> > >> (core)
> > >>>>> package directory, then its jars have to be added to "ignite/libs"
> > >>>> folder.
> > >>>>> After that, the one needs to restart a cluster node, nd it will add
> > the
> > >>>>> just installed optional libs to the classpath. *Petr*, does it work
> > >> this
> > >>>>> way or can be implemented this way to address Ilya's concerns?
> > >>>>>
> > >>>>> --
> > >>>>> Denis
> > >>>>>
> > >>>>> On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <
> > >>>>> [hidden email]>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> 2018-04-13 7:42 GMT+03:00 Peter Ivanov <[hidden email]>:
> > >>>>>>
> > >>>>>>> On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <
> > >>>>> [hidden email]
> > >>>>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> Moreover I did not find a way to start service if default
> > installed
> > >>>>> JVM
> > >>>>>>> is
> > >>>>>>>> Java 7 :( I understand it's EOL, still this is something that
> hit
> > >>>> me.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Apache Ignite >=2.4 does not support Java 7 - it is said in
> > >>>>>> documentation,
> > >>>>>>> DEVNOTES and even in startup scripts.
> > >>>>>>>
> > >>>>>>> I have Java 8 too, but I could not get service from package to
> > start
> > >>>>>> Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for
> that
> > >>>>>> matter). Is it possible to specify it while running packaged
> Ignite?
> > >>>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> apache-ignite-libs is a totally unexpected package name.
> > >>>>> apache-ignite
> > >>>>>>> core
> > >>>>>>>> doesn't depend on it. It doesn't enable anything out of the box.
> > >>>> The
> > >>>>>>>> package is huge.
> > >>>>>>>
> > >>>>>>> ‘apache-ignite-libs’ is an aggregation package (for now) for all
> > >>>>> optional
> > >>>>>>> libs we are delivering. Possibly later they will be split more
> > >>>> granular
> > >>>>>> or
> > >>>>>>> even package per lib (like php, perl, python, etc. do for their
> > >>>> libs).
> > >>>>>>> This package dependency on ‘apache-ignite-core’ may seem
> confusing
> > >>>>>> though,
> > >>>>>>> I will try to explain it in IEP at least for current iteration.
> > >>>>>>>
> > >>>>>>
> > >>>>>> Okay, but how do you add optional libs to be included into Ignite
> > >>>>> classpath
> > >>>>>> while being launched by service? Is it even possible? If it
> isn't, I
> > >>>>> think
> > >>>>>> it doesn't make sense to ship apache-ignite-libs at all.
> > >>>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>> Further naming may become clear when we’ll start initiative on
> > >>>>> including
> > >>>>>>> packages to popular Linux distributions and theirs community will
> > >>>> join
> > >>>>>>> naming discussions.
> > >>>>>>>
> > >>>>>> Renaming packages once they're deployed widely will be a pain
> point
> > to
> > >>>>> out
> > >>>>>> users. Some things should probably be thought out in advance.
> > >>>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> Frankly speaking, I'm not sure that improvements over Stage I
> are
> > >>>>>> enough
> > >>>>>>> as
> > >>>>>>>> of now. For demo-like activity, we can probably go with one
> > package
> > >>>>>> fits
> > >>>>>>>> all.
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> The process of finding the best package architecture is
> iterative,
> > >>>> but
> > >>>>>>> previously community agreed in split design proposed for 2.5
> > release.
> > >>>>>>>
> > >>>>>>> Also, split architecture is half of proposed improvements. The
> > other
> > >>>>>> half -
> > >>>>>>> new process for deploying packages to Bintray (with virtually
> > >>>>> indefinite
> > >>>>>>> storage capabilities).
> > >>>>>>>
> > >>>>>> I think we could drop the split for now, or at least drop
> > >>>>>> apache-ignite-libs package at all. Probably also drop
> > >> apache-ignite-cpp
> > >>>>>> package and maybe apache-ignite-benchmarks.
> > >>>>>>
> > >>>>>> The point of a package is to ship something into root file system
> > that
> > >>>>> can
> > >>>>>> be used from root file system. If cpp files require compilation we
> > >>>> should
> > >>>>>> not ship them, or ship them to 'examples'. Ditto with benchmarks.
> If
> > >>>>>> there's no mechanism to add optional libs to Ignite classpath, we
> > >>>> should
> > >>>>>> not ship optional libs. Moreover, some of 'optional' modules such
> as
> > >>>> yarn
> > >>>>>> don't make sense here because they're not supposed to be used with
> > >>>>>> standalone Ignite.
> > >>>>>>
> > >>>>>> IMO it is not right to try and shove every file from Ignite
> > >>>> distribution
> > >>>>>> into some package. We should only put in packages things that can
> be
> > >>>>> used.
> > >>>>>> If something can't be used without copying it to a different FS
> > >>>> location,
> > >>>>>> it should be in examples or not packaged at all.
> > >>>>>>
> > >>>>>> In my opinion, it doesn't make sense to implement an underwhelming
> > >>>>> package
> > >>>>>> split right now just because we have agreed to have *some* package
> > >>>> split
> > >>>>> in
> > >>>>>> 2.5. Let's aim for happiness.
> > >>>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Ilya Kasnacheev
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>>>
> > >>>>>>>> 2018-04-12 19:10 GMT+03:00 Petr Ivanov <[hidden email]>:
> > >>>>>>>>
> > >>>>>>>>> If someone from PMCы or Committers still sees necessity about
> > >>>>>> including
> > >>>>>>>>> these tasks into Apache Ignite 2.5 release, this is the last
> > >>>> chance
> > >>>>>> to
> > >>>>>>> do
> > >>>>>>>>> so.
> > >>>>>>>>> Otherwise this task will be moved to at 2.6 release at least,
> or
> > >>>>> even
> > >>>>>>>>> moved to backlog indefinitely.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> On 9 Apr 2018, at 19:08, Petr Ivanov <[hidden email]>
> > >>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> To top new RPM architecture off, update to release process is
> > >>>>>>>> introduced
> > >>>>>>>>> — [1] [2].
> > >>>>>>>>>>
> > >>>>>>>>>> Both tasks (this one and IGNITE-7647) are ready for review and
> > >>>>>> should
> > >>>>>>>> be
> > >>>>>>>>> merged simultaneously.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-8172
> > >>>>>>>>>> [2] https://github.com/apache/ignite-release/pull/1
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> On 2 Apr 2018, at 18:22, Ilya Kasnacheev <
> > >>>>>> [hidden email]
> > >>>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hello!
> > >>>>>>>>>>>
> > >>>>>>>>>>> Let me share my idea of how this shoud work. Splitting
> package
> > >>>>>> into
> > >>>>>>>>>>> sub-packages should be dependency-driven.
> > >>>>>>>>>>>
> > >>>>>>>>>>> It means that all Ignite modules without dependencies or with
> > >>>>>> small
> > >>>>>>>>>>> dependencies (such as ignite-log4j) should be included in
> > >>>>>>> ignite-core.
> > >>>>>>>>> It
> > >>>>>>>>>>> doesn't make sense to make a zillion RPM packages.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Critical things like ignite-spring and ignite-indexing should
> > >>>> be
> > >>>>>> in
> > >>>>>>>>>>> ignite-core of course, even if they have dependencies.
> > >>>>> Ignite-core
> > >>>>>>>>> should
> > >>>>>>>>>>> be fully self-sufficient and feature-complete.
> > >>>>>>>>>>>
> > >>>>>>>>>>> However, e.g. .net API should probably be in a separate
> > >>>> package,
> > >>>>>>>>> because it
> > >>>>>>>>>>> should depend on mono | net-core. We may also have
> > >>>> ignite-devel
> > >>>>>>>> package
> > >>>>>>>>>>> which should include all modules which only make sense for
> > >>>>>>> developers
> > >>>>>>>>> who
> > >>>>>>>>>>> write code. Such as hibernate integration.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm not sure about MR modules. The main question should be,
> > >>>> does
> > >>>>>> it
> > >>>>>>>> have
> > >>>>>>>>>>> dependencies? Can it run stand-alone without writing code?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hope this helps,
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Ilya Kasnacheev
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <[hidden email]
> >:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Igniters!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Here are some news on our RPM packages initiative.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. I’ve finished preliminary developing of Stage II version
> > >>>> of
> > >>>>>> RPM
> > >>>>>>>>>>>> packages [1]. Main “new feature” is — split design. Also
> I’ve
> > >>>>>> added
> > >>>>>>>>>>>> package.sh script for automating package building process
> > >>>> which
> > >>>>>>> will
> > >>>>>>>>> help
> > >>>>>>>>>>>> organise corresponding builds in TC as well as simplify
> > >>>> process
> > >>>>>> for
> > >>>>>>>>>>>> developers who wishes to have custom packages.
> > >>>>>>>>>>>> PR#3703 [2] is ready for review. Denis, in order to catch up
> > >>>>> with
> > >>>>>>>>> Apache
> > >>>>>>>>>>>> Ignite 2.5 release, I’d greatly appreciate your help in
> > >>>> finding
> > >>>>>>>>> reviewer.
> > >>>>>>>>>>>> 2. With the help of ASF INFRA team, we now have RPM [3] and
> > >>>> DEB
> > >>>>>> [4]
> > >>>>>>>>>>>> repositories on Apache Bintray. Though they are already
> > >>>>> prepared
> > >>>>>>> for
> > >>>>>>>>>>>> hosting RPM and DEB packages respectively, and there is a
> way
> > >>>>> of
> > >>>>>>>>> linking
> > >>>>>>>>>>>> them to apache.org/dist/ignite page, there is possible
> > >>>>>> alternative
> > >>>>>>>> in
> > >>>>>>>>>>>> storing there only plain directory layout corresponding to
> > >>>> each
> > >>>>>>>>> repository
> > >>>>>>>>>>>> type (RPM and DEB) and manage this layout (repodata,
> > >>>>>> distributions,
> > >>>>>>>>>>>> versions, etc.) by ourselves, having more control over
> > >>>>>> repositories
> > >>>>>>>> but
> > >>>>>>>>>>>> lacking some simplicity of deploying new releases. WDYT?
> > >>>> Should
> > >>>>>> we
> > >>>>>>>> try
> > >>>>>>>>>>>> Cassandra approach? They are storing their DEB packages as I
> > >>>>>>>> described
> > >>>>>>>>>>>> above [5].
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Also — a question arose while I was working on this issue:
> > >>>>> which
> > >>>>>>> OSes
> > >>>>>>>>> (and
> > >>>>>>>>>>>> which versions of each) are we going to support (if we are
> > >>>>> going)
> > >>>>>>> in
> > >>>>>>>>> terms
> > >>>>>>>>>>>> of step-by-step list? Currently RPM packages are tested only
> > >>>>> with
> > >>>>>>>>> latest
> > >>>>>>>>>>>> CentOS (and, respectively — RHEL), but there are a lot more
> > >>>>>>> RPM-based
> > >>>>>>>>>>>> distributives [6] some of which are more o less popular
> among
> > >>>>> OS
> > >>>>>>>>> community
> > >>>>>>>>>>>> (ALT, Fedora, openSUSE, etc.).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > >>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/3703
> > >>>>>>>>>>>> [3] https://bintray.com/apache/ignite-rpm
> > >>>>>>>>>>>> [4] https://bintray.com/apache/ignite-deb
> > >>>>>>>>>>>> [5] https://bintray.com/apache/cassandra/debian#files/
> > >>>>>>>>>>>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > >>>>>>>>> distributions
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On 15 Mar 2018, at 22:15, Petr Ivanov <[hidden email]
> >
> > >>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I suppose that most everything if not all from libs/options
> > >>>>> will
> > >>>>>>> go
> > >>>>>>>> to
> > >>>>>>>>>>>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > >>>>>>>>>>>>> More precise lib selection (if something from optional
> would
> > >>>>>>> better
> > >>>>>>>> to
> > >>>>>>>>>>>> have in core package) will be discussed right after
> > >>>> preliminary
> > >>>>>>> split
> > >>>>>>>>>>>> architecture agreement.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <
> > >>>>> [hidden email]
> > >>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I like idea of keeping simple system of modules, so +1
> from
> > >>>>> me.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Where optional libs (e.g Direct IO plugin) would be
> > >>>> included,
> > >>>>>>> would
> > >>>>>>>>> it
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>> core or optional?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <
> > >>>> [hidden email]
> > >>>>>> :
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> How big would be a final core module?
> > >>>>>>>>>>>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor
> and
> > >>>>>>> create
> > >>>>>>>>> it’s
> > >>>>>>>>>>>> own
> > >>>>>>>>>>>>>>>> package.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree
> > >>>>> with
> > >>>>>>> Petr
> > >>>>>>>>> and
> > >>>>>>>>>>>>>>> propose the simplest modular system:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> - core module that includes basic Ignite capabilities
> > >>>>>> including
> > >>>>>>>> SQL,
> > >>>>>>>>>>>>>>> compute grid, service grid, k/v
> > >>>>>>>>>>>>>>> - optional module hosts the rest - ML, streamers
> > >>>> integration
> > >>>>>>>> (kafka,
> > >>>>>>>>>>>>>>> flink), kubernetes, etc.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> What do you think?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>> Denis
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> > >>>>>>>> [hidden email]>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> *DEB package
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <
> > >>>>> [hidden email]>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Considering that DEV package for now is almost platform
> > >>>>>>>>> independent
> > >>>>>>>>>>>>>>> (its
> > >>>>>>>>>>>>>>>> a java application more or less), that package will work
> > >>>>>> almost
> > >>>>>>>> on
> > >>>>>>>>> any
> > >>>>>>>>>>>>>>>> DEB-based linux, including but not limited to Ubuntu,
> > >>>>> Debian,
> > >>>>>>>> etc.
> > >>>>>>>>>>>>>>>>> The only restriction is existence of systemctl
> (systemd)
> > >>>>>>> service
> > >>>>>>>>>>>>>>> manager
> > >>>>>>>>>>>>>>>> — we are dependent on it.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thats why, for instance, our RPM repository is called
> > >>>>> simply
> > >>>>>>>> ‘rpm’
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> package has no arch or dist suffix — it will work on
> > >>>>> CentOS,
> > >>>>>>>> RHEL,
> > >>>>>>>>>>>>>>> Fedora,
> > >>>>>>>>>>>>>>>> etc. with presence of aforementioned systemd.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan <
> > >>>>>>>>> [hidden email]>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Will Debian package work for Ubuntu?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> D.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov <
> > >>>>>>>>> [hidden email]>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Not a problem, rather nuisance. Also, when we will
> > >>>> move
> > >>>>> to
> > >>>>>>>>> official
> > >>>>>>>>>>>>>>>>>>> repositories, there can be a problem from OS
> > >>>> community.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Concerning DEB packages — I plan to use RPM as base
> > >>>> for
> > >>>>>> DEB
> > >>>>>>>>> package
> > >>>>>>>>>>>>>>>> build
> > >>>>>>>>>>>>>>>>>>> (package layout / install scripts) for speeding up
> > >>>>> things
> > >>>>>>> and
> > >>>>>>>>>>>>>>> excluding
> > >>>>>>>>>>>>>>>>>>> possible duplication and desynchronisation, so its a
> > >>>>>> matter
> > >>>>>>> of
> > >>>>>>>>> ’sit
> > >>>>>>>>>>>>>>>> and do’
> > >>>>>>>>>>>>>>>>>>> rather then some technical research. Thats why I rose
> > >>>>>>>> discussion
> > >>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>> future package architecture, so that after agreement
> > >>>> I'm
> > >>>>>> be
> > >>>>>>>>> able to
> > >>>>>>>>>>>>>>>> pack
> > >>>>>>>>>>>>>>>>>>> both RPM and DEB identically.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Yet, if you insist, I can create DEB package
> according
> > >>>>> to
> > >>>>>>>>> current
> > >>>>>>>>>>>> RPM
> > >>>>>>>>>>>>>>>>>>> layout in no time.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan <
> > >>>>>>>>>>>> [hidden email]>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Peter,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I don't think the package size of 280M is going to
> > >>>> be a
> > >>>>>>>>> problem at
> > >>>>>>>>>>>>>>>> all,
> > >>>>>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>> what you are suggesting can be an improvement down
> > >>>> the
> > >>>>>>> road.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> In the mean time, I think our top priority should be
> > >>>> to
> > >>>>>>>> provide
> > >>>>>>>>>>>>>>>> packages
> > >>>>>>>>>>>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not
> nearly
> > >>>>>>> enough.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Agree?
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> D.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider <
> > >>>>>>>> [hidden email]>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi, Igniters!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Release 2.4 is almost there, at least binary part
> of
> > >>>>> it,
> > >>>>>>> so
> > >>>>>>>>> I'd
> > >>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> move
> > >>>>>>>>>>>>>>>>>>>>> forward to further improve and widen AI delivery
> > >>>>> through
> > >>>>>>>>>>>> packages.
> > >>>>>>>>>>>>>>>>>>>>> As of now, Apache Ignite ships in RPM package
> > >>>> weighing
> > >>>>>>> about
> > >>>>>>>>>>>> 280M+
> > >>>>>>>>>>>>>>>> and,
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> improve usability and significantly reduce required
> > >>>>>>> download
> > >>>>>>>>>>>>>>> sizes, I
> > >>>>>>>>>>>>>>>>>>>>> purpose that in 2.5 release we introduce splitted
> > >>>>>> delivery
> > >>>>>>>> as
> > >>>>>>>>>>>>>>>> follows:
> > >>>>>>>>>>>>>>>>>>>>> - CORE
> > >>>>>>>>>>>>>>>>>>>>> - bin
> > >>>>>>>>>>>>>>>>>>>>> - config
> > >>>>>>>>>>>>>>>>>>>>> - libs (!optional)
> > >>>>>>>>>>>>>>>>>>>>> - OPTIONAL LIBS
> > >>>>>>>>>>>>>>>>>>>>> - BENCHMARKS
> > >>>>>>>>>>>>>>>>>>>>> - DOCS (?)
> > >>>>>>>>>>>>>>>>>>>>> - EXAMPLES
> > >>>>>>>>>>>>>>>>>>>>> - .NET PLATFORM FILES
> > >>>>>>>>>>>>>>>>>>>>> - C++ PLATFORM FILES
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> This architecture, as I assume, will add
> flexibility
> > >>>>> (no
> > >>>>>>>>> reason
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> download
> > >>>>>>>>>>>>>>>>>>>>> all 280M+ of binaries where you are to run only
> core
> > >>>>>> node
> > >>>>>>>>>>>>>>>> functionality)
> > >>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> maintainability (you are in full control of what is
> > >>>>>>>> installed
> > >>>>>>>>> on
> > >>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>>> system).
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> After successful architecture choice, same scheme
> > >>>> are
> > >>>>>>>> planned
> > >>>>>>>>> to
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>> used in
> > >>>>>>>>>>>>>>>>>>>>> DEB packages as well.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>>>>> Sent from:
> > >>>>>>>> http://apache-ignite-developers.2346864.n4.nabble.
> > >>>>>>>>>>>> com/
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>