IEP-18: Transparent Data Encryption
Locked
IEP-18: Transparent Data Encryption
 
|
Hello, Igniters.
Based on previous discussion [1] we've created "IEP-18: Transparent Data Encryption" [2] I've planned to start implementation of TDE in few weeks. I will create JIRA ticket for each piece of implementation. So, please, see IEP-18 and give us feedback. Dima Ryabov, huge thanks for pushing TDE IEP forward. [1] http://apache-ignite-developers.2346864.n4.nabble.com/Transparent-Data-Encryption-TDE-in-Apache-Ignite-td18957.html [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75979078 |
signature.asc (465 bytes)
Re: IEP-18: Transparent Data Encryption
|
Re: IEP-18: Transparent Data Encryption
|
|
Re: IEP-18: Transparent Data Encryption
|
|
Re: IEP-18: Transparent Data Encryption
|
|
Hello, Vladimir.
> 1) Why do you propose to store CEK in separate cache? All CEKs data should be available on all cluster nodes. We want to use system cache to get data synchronization feature "for free". > We consider storing any metadata in system caches as antipattern from our previous experience. Very interesting! Can you share your experience? I didn't know that. I think we can use any convinient storage for CEK. Current design doesn't couple to system caches, so we can change this part at any moment. > 2) I do not think that decryption process should require any "administrator" role and passwords. > Instead, we can have a kind of pluggable interface which will provide decrypted MEK on demand when it is needed. > This should be pre-configured in advance on server node(s). > AFAIK this is how a number of other vendors work. Can't agree with you, for now. Can you please send me some info about implementation you refer to? We study following papers to make current design: 1. ORACLE [1] To enable encryption one has to execute following statement: `ALTER SYSTEM SET ENCRYPTION KEY IDENTIFIED BY *clear_text_password*` Or after database server restart - `ALTER SYSTEM SET WALLET OPEN IDENTIFIED BY *clear_text_password*` So yes, administrator is involved into server restart. And no, it's not all automatic. 2. MSSQL [4] - AFAIK uses some Windows OS internal key storage to write and read master key. DB server process should have administrator(system) priveleges to access that storage. > Instead, we can have a kind of pluggable interface which will provide decrypted MEK on demand when it is needed. We, for sure, will have pluggable interface for providing MEK. But, from my point of view, default implementation should use some default java features. I think we should use java key store. It requires to have a clear text password to be loaded [3] [1] https://docs.oracle.com/cd/B19306_01/network.102/b14268/asotrans.htm#BABJJAIG [2] https://technet.microsoft.com/en-us/library/bb934049(v=sql.110).aspx [3] https://docs.oracle.com/javase/8/docs/api/java/security/KeyStore.html#load-java.io.InputStream-char:A- [4] https://technet.microsoft.com/en-us/library/bb934049(v=sql.110).aspx > 3) CEK decryption should not be tied to MEK decryption. MEK will be available from the moment it obtained to the node stop. So, we can decrypt existing or encrypt newly created CEK whenever it necessary. > 4) I do not think that SSL should be a strict requirement. It is up to the> user to asses the risks. I'm fully OK with it. Let's remove that restriction. В Пн, 09/04/2018 в 11:35 +0300, Vladimir Ozerov пишет: > Hi Nikolay, > > First of all thank you for excellent summary. Two-tiered key management is > well respected technique and makes perfect sense to me. However, several > questions regarding architecture arises: > 3) CEK decryption should not be tied to MEK decryption. Main reason - CEK > could be required during dynamic cache/table creation. So there should be > no coupling between activation and CEK processing. > 4) I do not think that SSL should be a strict requirement. It is up to the > user to asses the risks. > > On Fri, Apr 6, 2018 at 9:59 PM, Denis Magda <[hidden email]> wrote: > > > Nikolay, Dmitriy R., > > > > Thanks for the research and for writing down a summary in the IEP form. > > > > Please answer several high-level questions: > > > > - Is it necessary to have CEP keys for every cache? Not sure how all the > > keys will be managed if the user wants to encrypt 10-100 caches. Is it > > possible to use a single CEP by default with an option of having a > > unique > > one for a cache with more sensitive information? > > - It's not written, but I guess it would be up to me which caches to > > encrypt, right? In practice, you don't need to have all the data > > encrypted. > > Usually, companies look to hide personal, payments history, etc. > > - Should we think of procedures of CEP keys regeneration? A key can be > > lost or stolen. > > - Similar question goes for MEP key. > > > > -- > > Denis > > > > On Thu, Apr 5, 2018 at 2:15 PM, Dmitriy Setrakyan <[hidden email]> > > wrote: > > > > > Here is a correct link to IEP: > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > 18%3A+Transparent+Data+Encryption > > > > > > On Thu, Apr 5, 2018 at 12:01 PM, Nikolay Izhikov <[hidden email]> > > > wrote: > > > > > > > Hello, Igniters. > > > > > > > > Based on previous discussion [1] we've created "IEP-18: Transparent > > > > Data > > > > Encryption" [2] > > > > I've planned to start implementation of TDE in few weeks. > > > > I will create JIRA ticket for each piece of implementation. > > > > > > > > So, please, see IEP-18 and give us feedback. > > > > > > > > Dima Ryabov, huge thanks for pushing TDE IEP forward. > > > > > > > > [1] http://apache-ignite-developers.2346864.n4.nabble. > > > > com/Transparent-Data-Encryption-TDE-in-Apache-Ignite-td18957.html > > > > [2] https://cwiki.apache.org/confluence/pages/viewpage. > > > > action?pageId=75979078 > > > > |
Re: IEP-18: Transparent Data Encryption
|
|
In reply to this post by dmagda
Hello, Denis
> Is it necessary to have CEP keys for every cache? With current design, It's necessary to have difference CEK for every encrypted cache. I don't this it's an issue because CEK should be generated automatically and stored internally in Ignite. Cluster administrator should manage MEK to have > I guess it would be up to me which caches to encrypt, right? Yes. User can enable TDE for any caches he want. Other caches will work without any changes. > Should we think of procedures of CEP keys regeneration? > Similar question goes for MEP key. Yes, we should! Good catch, thank you. I think, it questionable, should we provide such feature in the first release? Will be added to IEP, anyway. В Пт, 06/04/2018 в 11:59 -0700, Denis Magda пишет: > Nikolay, Dmitriy R., > > Thanks for the research and for writing down a summary in the IEP form. > > Please answer several high-level questions: > > - Is it necessary to have CEP keys for every cache? Not sure how all the > keys will be managed if the user wants to encrypt 10-100 caches. Is it > possible to use a single CEP by default with an option of having a unique > one for a cache with more sensitive information? > - It's not written, but I guess it would be up to me which caches to > encrypt, right? In practice, you don't need to have all the data encrypted. > Usually, companies look to hide personal, payments history, etc. > - Should we think of procedures of CEP keys regeneration? A key can be > lost or stolen. > - Similar question goes for MEP key. > > -- > Denis > > On Thu, Apr 5, 2018 at 2:15 PM, Dmitriy Setrakyan <[hidden email]> > wrote: > > > Here is a correct link to IEP: > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > 18%3A+Transparent+Data+Encryption > > > > On Thu, Apr 5, 2018 at 12:01 PM, Nikolay Izhikov <[hidden email]> > > wrote: > > > > > Hello, Igniters. > > > > > > Based on previous discussion [1] we've created "IEP-18: Transparent Data > > > Encryption" [2] > > > I've planned to start implementation of TDE in few weeks. > > > I will create JIRA ticket for each piece of implementation. > > > > > > So, please, see IEP-18 and give us feedback. > > > > > > Dima Ryabov, huge thanks for pushing TDE IEP forward. > > > > > > [1] http://apache-ignite-developers.2346864.n4.nabble. > > > com/Transparent-Data-Encryption-TDE-in-Apache-Ignite-td18957.html > > > [2] https://cwiki.apache.org/confluence/pages/viewpage. > > > action?pageId=75979078 > > > |
Re: IEP-18: Transparent Data Encryption
|
|
Hello, Igniters.
I'm going to create tickets and start implementation of TDE in a few days. Any objections? В Пн, 09/04/2018 в 17:55 +0300, Nikolay Izhikov пишет: > Hello, Denis > > > Is it necessary to have CEP keys for every cache? > > With current design, It's necessary to have difference CEK for every encrypted cache. > I don't this it's an issue because CEK should be generated automatically and stored internally in Ignite. > Cluster administrator should manage MEK to have > > > I guess it would be up to me which caches to encrypt, right? > > Yes. User can enable TDE for any caches he want. > Other caches will work without any changes. > > > Should we think of procedures of CEP keys regeneration? > > Similar question goes for MEP key. > > Yes, we should! Good catch, thank you. > I think, it questionable, should we provide such feature in the first release? > Will be added to IEP, anyway. > > > В Пт, 06/04/2018 в 11:59 -0700, Denis Magda пишет: > > Nikolay, Dmitriy R., > > > > Thanks for the research and for writing down a summary in the IEP form. > > > > Please answer several high-level questions: > > > > - Is it necessary to have CEP keys for every cache? Not sure how all the > > keys will be managed if the user wants to encrypt 10-100 caches. Is it > > possible to use a single CEP by default with an option of having a unique > > one for a cache with more sensitive information? > > - It's not written, but I guess it would be up to me which caches to > > encrypt, right? In practice, you don't need to have all the data encrypted. > > Usually, companies look to hide personal, payments history, etc. > > - Should we think of procedures of CEP keys regeneration? A key can be > > lost or stolen. > > - Similar question goes for MEP key. > > > > -- > > Denis > > > > On Thu, Apr 5, 2018 at 2:15 PM, Dmitriy Setrakyan <[hidden email]> > > wrote: > > > > > Here is a correct link to IEP: > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > 18%3A+Transparent+Data+Encryption > > > > > > On Thu, Apr 5, 2018 at 12:01 PM, Nikolay Izhikov <[hidden email]> > > > wrote: > > > > > > > Hello, Igniters. > > > > > > > > Based on previous discussion [1] we've created "IEP-18: Transparent Data > > > > Encryption" [2] > > > > I've planned to start implementation of TDE in few weeks. > > > > I will create JIRA ticket for each piece of implementation. > > > > > > > > So, please, see IEP-18 and give us feedback. > > > > > > > > Dima Ryabov, huge thanks for pushing TDE IEP forward. > > > > > > > > [1] http://apache-ignite-developers.2346864.n4.nabble. > > > > com/Transparent-Data-Encryption-TDE-in-Apache-Ignite-td18957.html > > > > [2] https://cwiki.apache.org/confluence/pages/viewpage. > > > > action?pageId=75979078 |
Re: IEP-18: Transparent Data Encryption
|
Re: IEP-18: Transparent Data Encryption
|
|
Re: IEP-18: Transparent Data Encryption
|
|
Re: IEP-18: Transparent Data Encryption
|
|
Hello, Vladimir.
> community is not aware of concrete architecture and proposed public API Concrete architecture is described in IEP-18 [1]. Please, tell me, what else you want to be written. I think answers to all questions have to be addressed(and discussed with community!) when we crack into implementation. Personally, I don't believe in any kind of more detailed design from eagle height. So I want to move from one part of this feature to another and make concrete choices just before implementation. Thoughts? Public API proposition will be part of an implementation and, of course, will be discussed. I don't think we have to do specific choices at the moment. > We need to continue discussion and come up with detailed design. Can you give me the check-list of detailed design? As far as I can see many of other IEP(already implemented) doesn't include any details you mentioned here [2], [3]. Anyway, let's make this IEP some kind of "reference implementation" for other IEP's :). > 1) How encryption is configured - public API is needed Similarly to other SPI. > 2) How encrypted cluster is started I study links you provide. I asked DB administrators I know. Looks like we can implement automatic TDE activation. > 3) How and where encrypted cache keys are stored I want to follow your advice and store CEK's near cache. > 4) Protocol of key's exchange between nodes > 5) Detailed description of encryption flow - what algorithm is used? will> we use CBC? > 6) What changes would be required to WAL logic (at the very least -> enumerate affected types of records)? I don't have specific answers to this questions. Do you think we must answer this questions before starting any implementation? > 7) Would it be possible to enable/disable encryption on specific> cache/table in runtime? No, for the first implementation. Cache has to be created as encrypted. If cache created as decrypted it can't become encrypted. In the future we can extend encryption mechanism and add this feature to the product. > 8) Would these changes affect compression design anyhow? No. [1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-18%3A+Transparent+Data+Encryption [2] https://cwiki.apache.org/confluence/display/IGNITE/IEP-4+Baseline+topology+for+caches [3] https://cwiki.apache.org/confluence/display/IGNITE/IEP-7%3A+Ignite+internal+problems+detection В Пн, 16/04/2018 в 13:28 +0300, Vladimir Ozerov пишет: > Hi Nikolay, > > I noticed that some tickets have been created for this feature. Hopefully, > you haven't started implementation yet, because at this point community is > not aware of concrete architecture and proposed public API. We need to > continue discussion and come up with detailed design. Of most importance we > need to understand the following: > 1) How encryption is configured - public API is needed (how to setup key > providers, how to enable encryption for specific caches, etc) > 2) How encrypted cluster is started - normally, I do not expect any > additional manual actions from user once everything is configured. > Especially, in presence of auto-activation capability. > 3) How and where encrypted cache keys are stored > 4) Protocol of key's exchange between nodes > 5) Detailed description of encryption flow - what algorithm is used? will > we use CBC? > 6) What changes would be required to WAL logic (at the very least - > enumerate affected types of records)? > 7) Would it be possible to enable/disable encryption on specific > cache/table in runtime? > 8) Would these changes affect compression design anyhow? > > Vladimir. > > On Tue, Apr 10, 2018 at 5:30 PM, Vladimir Ozerov <[hidden email]> > wrote: > > > Hi NIkolay, > > > > Regarding system caches, rule of thumb here - do not use them. Keys should > > be stored near cache. > > > > As far as password: > > 1) Oracle auto-login wallet [1] > > 2) MySQL- password may be set inside configuration [2] > > > > I do not think that any kind of prompts are needed here out of the box. > > May be we could consider them in future, but at the moment it looks > > redundant to me. > > > > [1] https://docs.oracle.com/cd/E11882_01/network.112/ > > e40393/asotrans.htm#CHDCCHBH > > [2] https://dev.mysql.com/doc/refman/5.7/en/keyring- > > encrypted-file-plugin.html > > > > On Mon, Apr 9, 2018 at 5:42 PM, Nikolay Izhikov <[hidden email]> > > wrote: > > > > > Hello, Vladimir. > > > > > > > 1) Why do you propose to store CEK in separate cache? > > > > > > All CEKs data should be available on all cluster nodes. > > > We want to use system cache to get data synchronization feature "for > > > free". > > > > > > > We consider storing any metadata in system caches as antipattern from > > > > > > our previous experience. > > > > > > Very interesting! > > > Can you share your experience? > > > I didn't know that. > > > I think we can use any convinient storage for CEK. > > > Current design doesn't couple to system caches, so we can change this > > > part at any moment. > > > > > > > 2) I do not think that decryption process should require any > > > > > > "administrator" role and passwords. > > > > Instead, we can have a kind of pluggable interface which will provide > > > > > > decrypted MEK on demand when it is needed. > > > > This should be pre-configured in advance on server node(s). > > > > AFAIK this is how a number of other vendors work. > > > > > > Can't agree with you, for now. > > > Can you please send me some info about implementation you refer to? > > > We study following papers to make current design: > > > > > > 1. ORACLE [1] > > > > > > To enable encryption one has to execute following statement: > > > > > > `ALTER SYSTEM SET ENCRYPTION KEY IDENTIFIED BY *clear_text_password*` > > > Or after database server restart - `ALTER SYSTEM SET WALLET OPEN > > > IDENTIFIED BY *clear_text_password*` > > > So yes, administrator is involved into server restart. > > > And no, it's not all automatic. > > > > > > 2. MSSQL [4] - AFAIK uses some Windows OS internal key storage to write > > > and read master key. > > > DB server process should have administrator(system) priveleges to access > > > that storage. > > > > > > > Instead, we can have a kind of pluggable interface which will provide > > > > > > decrypted MEK on demand when it is needed. > > > > > > We, for sure, will have pluggable interface for providing MEK. > > > But, from my point of view, default implementation should use some > > > default java features. > > > I think we should use java key store. > > > It requires to have a clear text password to be loaded [3] > > > > > > [1] https://docs.oracle.com/cd/B19306_01/network.102/b14268/asot > > > rans.htm#BABJJAIG > > > [2] https://technet.microsoft.com/en-us/library/bb934049(v=sql.110).aspx > > > [3] https://docs.oracle.com/javase/8/docs/api/java/security/ > > > KeyStore.html#load-java.io.InputStream-char:A- > > > [4] https://technet.microsoft.com/en-us/library/bb934049(v=sql.110).aspx > > > > > > > 3) CEK decryption should not be tied to MEK decryption. > > > > > > MEK will be available from the moment it obtained to the node stop. > > > So, we can decrypt existing or encrypt newly created CEK whenever it > > > necessary. > > > > > > > 4) I do not think that SSL should be a strict requirement. It is up to > > > > > > the> user to asses the risks. > > > > > > I'm fully OK with it. > > > Let's remove that restriction. > > > > > > В Пн, 09/04/2018 в 11:35 +0300, Vladimir Ozerov пишет: > > > > Hi Nikolay, > > > > > > > > First of all thank you for excellent summary. Two-tiered key management > > > > > > is > > > > well respected technique and makes perfect sense to me. However, several > > > > questions regarding architecture arises: > > > > 3) CEK decryption should not be tied to MEK decryption. Main reason - > > > > > > CEK > > > > could be required during dynamic cache/table creation. So there should > > > > > > be > > > > no coupling between activation and CEK processing. > > > > 4) I do not think that SSL should be a strict requirement. It is up to > > > > > > the > > > > user to asses the risks. > > > > > > > > On Fri, Apr 6, 2018 at 9:59 PM, Denis Magda <[hidden email]> wrote: > > > > > > > > > Nikolay, Dmitriy R., > > > > > > > > > > Thanks for the research and for writing down a summary in the IEP > > > > > > form. > > > > > > > > > > Please answer several high-level questions: > > > > > > > > > > - Is it necessary to have CEP keys for every cache? Not sure how > > > > > > all the > > > > > keys will be managed if the user wants to encrypt 10-100 caches. > > > > > > Is it > > > > > possible to use a single CEP by default with an option of having a > > > > > unique > > > > > one for a cache with more sensitive information? > > > > > - It's not written, but I guess it would be up to me which caches > > > > > > to > > > > > encrypt, right? In practice, you don't need to have all the data > > > > > encrypted. > > > > > Usually, companies look to hide personal, payments history, etc. > > > > > - Should we think of procedures of CEP keys regeneration? A key > > > > > > can be > > > > > lost or stolen. > > > > > - Similar question goes for MEP key. > > > > > > > > > > -- > > > > > Denis > > > > > > > > > > On Thu, Apr 5, 2018 at 2:15 PM, Dmitriy Setrakyan < > > > > > > [hidden email]> > > > > > wrote: > > > > > > > > > > > Here is a correct link to IEP: > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > > > > 18%3A+Transparent+Data+Encryption > > > > > > > > > > > > On Thu, Apr 5, 2018 at 12:01 PM, Nikolay Izhikov < > > > > > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > Hello, Igniters. > > > > > > > > > > > > > > Based on previous discussion [1] we've created "IEP-18: > > > > > > Transparent > > > > > > > > > > Data > > > > > > > Encryption" [2] > > > > > > > I've planned to start implementation of TDE in few weeks. > > > > > > > I will create JIRA ticket for each piece of implementation. > > > > > > > > > > > > > > So, please, see IEP-18 and give us feedback. > > > > > > > > > > > > > > Dima Ryabov, huge thanks for pushing TDE IEP forward. > > > > > > > > > > > > > > [1] http://apache-ignite-developers.2346864.n4.nabble. > > > > > > > com/Transparent-Data-Encryption-TDE-in-Apache-Ignite-td18957.html > > > > > > > [2] https://cwiki.apache.org/confluence/pages/viewpage. > > > > > > > action?pageId=75979078 > > > > > > > > > > > |
Re: IEP-18: Transparent Data Encryption
|
|
| Free forum by Nabble | Edit this page |