[DISCUSSION] API to KILL any user started computation
Locked
12
12
[DISCUSSION] API to KILL any user started computation
 
|
Hello, Igniters.
As you may know, we put a lot of effort to improve Ignite metric and diagnostic API. We have created the following API: * Metric manager * System view manager As far as I know, we would have tracing capabilities soon. I think it's time to take the next step. We should provide to the administrator the API to cancel(stop, kill) arbitrary user started computation. Right now we have API to do it for: * transactions `TransactionsMXBean#getActiveTransactions`. * SQL queries: `KILL QUERY` sql command and visor task - `VisorQueryCancelTask` Please, note, these features are implemented via different API. I think we should introduce uniform Cancel API for the following computations: * service. * specific service method execution. * compute job(compute task). * query(scan, continuous, text). We should also rework kill transaction and kill SQL queries API to make them similar to each other. I have plans to write an IEP for it and implement it. What do you think? |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Nikolaj,
We already have cancellation possibilities for almost every user computation. Transactions are cancelled using tx.rollback() Queries are cancelled using query.close() Task is cancellable through ComputeTaskSession What is uniform cancel API ? Why do we need it ? ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email]>: > Hello, Igniters. > > As you may know, we put a lot of effort to improve Ignite metric and > diagnostic API. > We have created the following API: > * Metric manager > * System view manager > As far as I know, we would have tracing capabilities soon. > > I think it's time to take the next step. > We should provide to the administrator the API to cancel(stop, kill) > arbitrary user started computation. > > Right now we have API to do it for: > * transactions `TransactionsMXBean#getActiveTransactions`. > * SQL queries: `KILL QUERY` sql command and visor task - > `VisorQueryCancelTask` > > Please, note, these features are implemented via different API. > > I think we should introduce uniform Cancel API for the following > computations: > > * service. > * specific service method execution. > * compute job(compute task). > * query(scan, continuous, text). > > We should also rework kill transaction and kill SQL queries API to make > them similar to each other. > I have plans to write an IEP for it and implement it. > What do you think? > > -- Best regards, Alexei Scherbakov |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Transactions can be also rolled back using tx.close where close is
java.lang.AutoCloseable#close It looks for me to the definition of uniform cancel API. чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov <[hidden email] >: > Nikolaj, > > We already have cancellation possibilities for almost every user > computation. > Transactions are cancelled using tx.rollback() > Queries are cancelled using query.close() > Task is cancellable through ComputeTaskSession > > What is uniform cancel API ? Why do we need it ? > > > > ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email]>: > >> Hello, Igniters. >> >> As you may know, we put a lot of effort to improve Ignite metric and >> diagnostic API. >> We have created the following API: >> * Metric manager >> * System view manager >> As far as I know, we would have tracing capabilities soon. >> >> I think it's time to take the next step. >> We should provide to the administrator the API to cancel(stop, kill) >> arbitrary user started computation. >> >> Right now we have API to do it for: >> * transactions `TransactionsMXBean#getActiveTransactions`. >> * SQL queries: `KILL QUERY` sql command and visor task - >> `VisorQueryCancelTask` >> >> Please, note, these features are implemented via different API. >> >> I think we should introduce uniform Cancel API for the following >> computations: >> >> * service. >> * specific service method execution. >> * compute job(compute task). >> * query(scan, continuous, text). >> >> We should also rework kill transaction and kill SQL queries API to make >> them similar to each other. >> I have plans to write an IEP for it and implement it. >> What do you think? >> >> > > -- > > Best regards, > Alexei Scherbakov > -- Best regards, Alexei Scherbakov |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Hello, Alexey.
I’m talking about *administrator* API. For example, the User has a cluster that is used by several applications. Some application starts buggy compute tasks that consume all CPU resources. Right now, administrator doesn’t have the ability to kill this task. This can lead to instability of the whole cluster. > 16 янв. 2020 г., в 13:42, Alexei Scherbakov <[hidden email]> написал(а): > > Transactions can be also rolled back using tx.close where close is > java.lang.AutoCloseable#close > It looks for me to the definition of uniform cancel API. > > > > чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov <[hidden email] >> : > >> Nikolaj, >> >> We already have cancellation possibilities for almost every user >> computation. >> Transactions are cancelled using tx.rollback() >> Queries are cancelled using query.close() >> Task is cancellable through ComputeTaskSession >> >> What is uniform cancel API ? Why do we need it ? >> >> >> >> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email]>: >> >>> Hello, Igniters. >>> >>> As you may know, we put a lot of effort to improve Ignite metric and >>> diagnostic API. >>> We have created the following API: >>> * Metric manager >>> * System view manager >>> As far as I know, we would have tracing capabilities soon. >>> >>> I think it's time to take the next step. >>> We should provide to the administrator the API to cancel(stop, kill) >>> arbitrary user started computation. >>> >>> Right now we have API to do it for: >>> * transactions `TransactionsMXBean#getActiveTransactions`. >>> * SQL queries: `KILL QUERY` sql command and visor task - >>> `VisorQueryCancelTask` >>> >>> Please, note, these features are implemented via different API. >>> >>> I think we should introduce uniform Cancel API for the following >>> computations: >>> >>> * service. >>> * specific service method execution. >>> * compute job(compute task). >>> * query(scan, continuous, text). >>> >>> We should also rework kill transaction and kill SQL queries API to make >>> them similar to each other. >>> I have plans to write an IEP for it and implement it. >>> What do you think? >>> >>> >> >> -- >> >> Best regards, >> Alexei Scherbakov >> > > > -- > > Best regards, > Alexei Scherbakov |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Nikolaj,
Can we use system views instead of implementing something new ? Each user operation has an unique ID. It's possible to introduce universal SQL kill something like: kill transaction ID where id is taken from system view. чт, 16 янв. 2020 г. в 14:19, Николай Ижиков <[hidden email]>: > Hello, Alexey. > > I’m talking about *administrator* API. > > For example, the User has a cluster that is used by several applications. > Some application starts buggy compute tasks that consume all CPU resources. > Right now, administrator doesn’t have the ability to kill this task. > > This can lead to instability of the whole cluster. > > > > 16 янв. 2020 г., в 13:42, Alexei Scherbakov < > [hidden email]> написал(а): > > > > Transactions can be also rolled back using tx.close where close is > > java.lang.AutoCloseable#close > > It looks for me to the definition of uniform cancel API. > > > > > > > > чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov < > [hidden email] > >> : > > > >> Nikolaj, > >> > >> We already have cancellation possibilities for almost every user > >> computation. > >> Transactions are cancelled using tx.rollback() > >> Queries are cancelled using query.close() > >> Task is cancellable through ComputeTaskSession > >> > >> What is uniform cancel API ? Why do we need it ? > >> > >> > >> > >> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email]>: > >> > >>> Hello, Igniters. > >>> > >>> As you may know, we put a lot of effort to improve Ignite metric and > >>> diagnostic API. > >>> We have created the following API: > >>> * Metric manager > >>> * System view manager > >>> As far as I know, we would have tracing capabilities soon. > >>> > >>> I think it's time to take the next step. > >>> We should provide to the administrator the API to cancel(stop, kill) > >>> arbitrary user started computation. > >>> > >>> Right now we have API to do it for: > >>> * transactions `TransactionsMXBean#getActiveTransactions`. > >>> * SQL queries: `KILL QUERY` sql command and visor task - > >>> `VisorQueryCancelTask` > >>> > >>> Please, note, these features are implemented via different API. > >>> > >>> I think we should introduce uniform Cancel API for the following > >>> computations: > >>> > >>> * service. > >>> * specific service method execution. > >>> * compute job(compute task). > >>> * query(scan, continuous, text). > >>> > >>> We should also rework kill transaction and kill SQL queries API to > make > >>> them similar to each other. > >>> I have plans to write an IEP for it and implement it. > >>> What do you think? > >>> > >>> > >> > >> -- > >> > >> Best regards, > >> Alexei Scherbakov > >> > > > > > > -- > > > > Best regards, > > Alexei Scherbakov > > -- Best regards, Alexei Scherbakov |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Alexey.
I think, yes. We certainly should be able to use system view data for the new KILL API. I think we should support both SQL and Java(JMX) API for this KILL command. > 16 янв. 2020 г., в 15:28, Alexei Scherbakov <[hidden email]> написал(а): > > Nikolaj, > > Can we use system views instead of implementing something new ? > > Each user operation has an unique ID. > > It's possible to introduce universal SQL kill something like: > > kill transaction ID > > where id is taken from system view. > > > чт, 16 янв. 2020 г. в 14:19, Николай Ижиков <[hidden email]>: > >> Hello, Alexey. >> >> I’m talking about *administrator* API. >> >> For example, the User has a cluster that is used by several applications. >> Some application starts buggy compute tasks that consume all CPU resources. >> Right now, administrator doesn’t have the ability to kill this task. >> >> This can lead to instability of the whole cluster. >> >> >>> 16 янв. 2020 г., в 13:42, Alexei Scherbakov < >> [hidden email]> написал(а): >>> >>> Transactions can be also rolled back using tx.close where close is >>> java.lang.AutoCloseable#close >>> It looks for me to the definition of uniform cancel API. >>> >>> >>> >>> чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov < >> [hidden email] >>>> : >>> >>>> Nikolaj, >>>> >>>> We already have cancellation possibilities for almost every user >>>> computation. >>>> Transactions are cancelled using tx.rollback() >>>> Queries are cancelled using query.close() >>>> Task is cancellable through ComputeTaskSession >>>> >>>> What is uniform cancel API ? Why do we need it ? >>>> >>>> >>>> >>>> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email]>: >>>> >>>>> Hello, Igniters. >>>>> >>>>> As you may know, we put a lot of effort to improve Ignite metric and >>>>> diagnostic API. >>>>> We have created the following API: >>>>> * Metric manager >>>>> * System view manager >>>>> As far as I know, we would have tracing capabilities soon. >>>>> >>>>> I think it's time to take the next step. >>>>> We should provide to the administrator the API to cancel(stop, kill) >>>>> arbitrary user started computation. >>>>> >>>>> Right now we have API to do it for: >>>>> * transactions `TransactionsMXBean#getActiveTransactions`. >>>>> * SQL queries: `KILL QUERY` sql command and visor task - >>>>> `VisorQueryCancelTask` >>>>> >>>>> Please, note, these features are implemented via different API. >>>>> >>>>> I think we should introduce uniform Cancel API for the following >>>>> computations: >>>>> >>>>> * service. >>>>> * specific service method execution. >>>>> * compute job(compute task). >>>>> * query(scan, continuous, text). >>>>> >>>>> We should also rework kill transaction and kill SQL queries API to >> make >>>>> them similar to each other. >>>>> I have plans to write an IEP for it and implement it. >>>>> What do you think? >>>>> >>>>> >>>> >>>> -- >>>> >>>> Best regards, >>>> Alexei Scherbakov >>>> >>> >>> >>> -- >>> >>> Best regards, >>> Alexei Scherbakov >> >> > > -- > > Best regards, > Alexei Scherbakov |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Sounds good.
JMX API should be very similar: accept operation type and ID. чт, 16 янв. 2020 г. в 15:46, Николай Ижиков <[hidden email]>: > Alexey. > > I think, yes. > We certainly should be able to use system view data for the new KILL API. > > I think we should support both SQL and Java(JMX) API for this KILL command. > > > > 16 янв. 2020 г., в 15:28, Alexei Scherbakov < > [hidden email]> написал(а): > > > > Nikolaj, > > > > Can we use system views instead of implementing something new ? > > > > Each user operation has an unique ID. > > > > It's possible to introduce universal SQL kill something like: > > > > kill transaction ID > > > > where id is taken from system view. > > > > > > чт, 16 янв. 2020 г. в 14:19, Николай Ижиков <[hidden email]>: > > > >> Hello, Alexey. > >> > >> I’m talking about *administrator* API. > >> > >> For example, the User has a cluster that is used by several > applications. > >> Some application starts buggy compute tasks that consume all CPU > resources. > >> Right now, administrator doesn’t have the ability to kill this task. > >> > >> This can lead to instability of the whole cluster. > >> > >> > >>> 16 янв. 2020 г., в 13:42, Alexei Scherbakov < > >> [hidden email]> написал(а): > >>> > >>> Transactions can be also rolled back using tx.close where close is > >>> java.lang.AutoCloseable#close > >>> It looks for me to the definition of uniform cancel API. > >>> > >>> > >>> > >>> чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov < > >> [hidden email] > >>>> : > >>> > >>>> Nikolaj, > >>>> > >>>> We already have cancellation possibilities for almost every user > >>>> computation. > >>>> Transactions are cancelled using tx.rollback() > >>>> Queries are cancelled using query.close() > >>>> Task is cancellable through ComputeTaskSession > >>>> > >>>> What is uniform cancel API ? Why do we need it ? > >>>> > >>>> > >>>> > >>>> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email]>: > >>>> > >>>>> Hello, Igniters. > >>>>> > >>>>> As you may know, we put a lot of effort to improve Ignite metric and > >>>>> diagnostic API. > >>>>> We have created the following API: > >>>>> * Metric manager > >>>>> * System view manager > >>>>> As far as I know, we would have tracing capabilities soon. > >>>>> > >>>>> I think it's time to take the next step. > >>>>> We should provide to the administrator the API to cancel(stop, kill) > >>>>> arbitrary user started computation. > >>>>> > >>>>> Right now we have API to do it for: > >>>>> * transactions `TransactionsMXBean#getActiveTransactions`. > >>>>> * SQL queries: `KILL QUERY` sql command and visor task - > >>>>> `VisorQueryCancelTask` > >>>>> > >>>>> Please, note, these features are implemented via different API. > >>>>> > >>>>> I think we should introduce uniform Cancel API for the following > >>>>> computations: > >>>>> > >>>>> * service. > >>>>> * specific service method execution. > >>>>> * compute job(compute task). > >>>>> * query(scan, continuous, text). > >>>>> > >>>>> We should also rework kill transaction and kill SQL queries API to > >> make > >>>>> them similar to each other. > >>>>> I have plans to write an IEP for it and implement it. > >>>>> What do you think? > >>>>> > >>>>> > >>>> > >>>> -- > >>>> > >>>> Best regards, > >>>> Alexei Scherbakov > >>>> > >>> > >>> > >>> -- > >>> > >>> Best regards, > >>> Alexei Scherbakov > >> > >> > > > > -- > > > > Best regards, > > Alexei Scherbakov > > -- Best regards, Alexei Scherbakov |
Re: [DISCUSSION] API to KILL any user started computation
|
|
In reply to this post by Nikolay Izhikov-2
Hello Nikolay,
I'm not sure we need JMX API for this case. If I’m not mistaken, it’s about the administrator’s ability to cancel any task/query in the cluster, and in my opinion, such action must require administrator permission. Could you please clarify how it can be done via JMX? I mean permission check and etc. Perhaps, I'm missing something obvious... Thanks, S. чт, 16 янв. 2020 г. в 15:46, Николай Ижиков <[hidden email]>: > Alexey. > > I think, yes. > We certainly should be able to use system view data for the new KILL API. > > I think we should support both SQL and Java(JMX) API for this KILL command. > > > > 16 янв. 2020 г., в 15:28, Alexei Scherbakov < > [hidden email]> написал(а): > > > > Nikolaj, > > > > Can we use system views instead of implementing something new ? > > > > Each user operation has an unique ID. > > > > It's possible to introduce universal SQL kill something like: > > > > kill transaction ID > > > > where id is taken from system view. > > > > > > чт, 16 янв. 2020 г. в 14:19, Николай Ижиков <[hidden email]>: > > > >> Hello, Alexey. > >> > >> I’m talking about *administrator* API. > >> > >> For example, the User has a cluster that is used by several > applications. > >> Some application starts buggy compute tasks that consume all CPU > resources. > >> Right now, administrator doesn’t have the ability to kill this task. > >> > >> This can lead to instability of the whole cluster. > >> > >> > >>> 16 янв. 2020 г., в 13:42, Alexei Scherbakov < > >> [hidden email]> написал(а): > >>> > >>> Transactions can be also rolled back using tx.close where close is > >>> java.lang.AutoCloseable#close > >>> It looks for me to the definition of uniform cancel API. > >>> > >>> > >>> > >>> чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov < > >> [hidden email] > >>>> : > >>> > >>>> Nikolaj, > >>>> > >>>> We already have cancellation possibilities for almost every user > >>>> computation. > >>>> Transactions are cancelled using tx.rollback() > >>>> Queries are cancelled using query.close() > >>>> Task is cancellable through ComputeTaskSession > >>>> > >>>> What is uniform cancel API ? Why do we need it ? > >>>> > >>>> > >>>> > >>>> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email]>: > >>>> > >>>>> Hello, Igniters. > >>>>> > >>>>> As you may know, we put a lot of effort to improve Ignite metric and > >>>>> diagnostic API. > >>>>> We have created the following API: > >>>>> * Metric manager > >>>>> * System view manager > >>>>> As far as I know, we would have tracing capabilities soon. > >>>>> > >>>>> I think it's time to take the next step. > >>>>> We should provide to the administrator the API to cancel(stop, kill) > >>>>> arbitrary user started computation. > >>>>> > >>>>> Right now we have API to do it for: > >>>>> * transactions `TransactionsMXBean#getActiveTransactions`. > >>>>> * SQL queries: `KILL QUERY` sql command and visor task - > >>>>> `VisorQueryCancelTask` > >>>>> > >>>>> Please, note, these features are implemented via different API. > >>>>> > >>>>> I think we should introduce uniform Cancel API for the following > >>>>> computations: > >>>>> > >>>>> * service. > >>>>> * specific service method execution. > >>>>> * compute job(compute task). > >>>>> * query(scan, continuous, text). > >>>>> > >>>>> We should also rework kill transaction and kill SQL queries API to > >> make > >>>>> them similar to each other. > >>>>> I have plans to write an IEP for it and implement it. > >>>>> What do you think? > >>>>> > >>>>> > >>>> > >>>> -- > >>>> > >>>> Best regards, > >>>> Alexei Scherbakov > >>>> > >>> > >>> > >>> -- > >>> > >>> Best regards, > >>> Alexei Scherbakov > >> > >> > > > > -- > > > > Best regards, > > Alexei Scherbakov > > |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Slava,
Not sure I've understand a question. JMX supports security via password protection and secure channel. пт, 17 янв. 2020 г. в 14:30, Вячеслав Коптилин <[hidden email]>: > Hello Nikolay, > > I'm not sure we need JMX API for this case. If I’m not mistaken, it’s about > the administrator’s ability to cancel any task/query in the cluster, and in > my opinion, such action must require administrator permission. > Could you please clarify how it can be done via JMX? I mean permission > check and etc. Perhaps, I'm missing something obvious... > > Thanks, > S. > > чт, 16 янв. 2020 г. в 15:46, Николай Ижиков <[hidden email]>: > > > Alexey. > > > > I think, yes. > > We certainly should be able to use system view data for the new KILL API. > > > > I think we should support both SQL and Java(JMX) API for this KILL > command. > > > > > > > 16 янв. 2020 г., в 15:28, Alexei Scherbakov < > > [hidden email]> написал(а): > > > > > > Nikolaj, > > > > > > Can we use system views instead of implementing something new ? > > > > > > Each user operation has an unique ID. > > > > > > It's possible to introduce universal SQL kill something like: > > > > > > kill transaction ID > > > > > > where id is taken from system view. > > > > > > > > > чт, 16 янв. 2020 г. в 14:19, Николай Ижиков <[hidden email]>: > > > > > >> Hello, Alexey. > > >> > > >> I’m talking about *administrator* API. > > >> > > >> For example, the User has a cluster that is used by several > > applications. > > >> Some application starts buggy compute tasks that consume all CPU > > resources. > > >> Right now, administrator doesn’t have the ability to kill this task. > > >> > > >> This can lead to instability of the whole cluster. > > >> > > >> > > >>> 16 янв. 2020 г., в 13:42, Alexei Scherbakov < > > >> [hidden email]> написал(а): > > >>> > > >>> Transactions can be also rolled back using tx.close where close is > > >>> java.lang.AutoCloseable#close > > >>> It looks for me to the definition of uniform cancel API. > > >>> > > >>> > > >>> > > >>> чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov < > > >> [hidden email] > > >>>> : > > >>> > > >>>> Nikolaj, > > >>>> > > >>>> We already have cancellation possibilities for almost every user > > >>>> computation. > > >>>> Transactions are cancelled using tx.rollback() > > >>>> Queries are cancelled using query.close() > > >>>> Task is cancellable through ComputeTaskSession > > >>>> > > >>>> What is uniform cancel API ? Why do we need it ? > > >>>> > > >>>> > > >>>> > > >>>> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email]>: > > >>>> > > >>>>> Hello, Igniters. > > >>>>> > > >>>>> As you may know, we put a lot of effort to improve Ignite metric > and > > >>>>> diagnostic API. > > >>>>> We have created the following API: > > >>>>> * Metric manager > > >>>>> * System view manager > > >>>>> As far as I know, we would have tracing capabilities soon. > > >>>>> > > >>>>> I think it's time to take the next step. > > >>>>> We should provide to the administrator the API to cancel(stop, > kill) > > >>>>> arbitrary user started computation. > > >>>>> > > >>>>> Right now we have API to do it for: > > >>>>> * transactions `TransactionsMXBean#getActiveTransactions`. > > >>>>> * SQL queries: `KILL QUERY` sql command and visor task - > > >>>>> `VisorQueryCancelTask` > > >>>>> > > >>>>> Please, note, these features are implemented via different API. > > >>>>> > > >>>>> I think we should introduce uniform Cancel API for the following > > >>>>> computations: > > >>>>> > > >>>>> * service. > > >>>>> * specific service method execution. > > >>>>> * compute job(compute task). > > >>>>> * query(scan, continuous, text). > > >>>>> > > >>>>> We should also rework kill transaction and kill SQL queries API to > > >> make > > >>>>> them similar to each other. > > >>>>> I have plans to write an IEP for it and implement it. > > >>>>> What do you think? > > >>>>> > > >>>>> > > >>>> > > >>>> -- > > >>>> > > >>>> Best regards, > > >>>> Alexei Scherbakov > > >>>> > > >>> > > >>> > > >>> -- > > >>> > > >>> Best regards, > > >>> Alexei Scherbakov > > >> > > >> > > > > > > -- > > > > > > Best regards, > > > Alexei Scherbakov > > > > > -- Best regards, Alexei Scherbakov |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Hi Alexei,
> Not sure I've understand a question. JMX supports security via password protection and secure channel. That is the obvious thing I missed! :) Thank you! Thanks, S. пт, 17 янв. 2020 г. в 14:45, Alexei Scherbakov <[hidden email] >: > Slava, > > Not sure I've understand a question. JMX supports security via password > protection and secure channel. > > пт, 17 янв. 2020 г. в 14:30, Вячеслав Коптилин <[hidden email]>: > > > Hello Nikolay, > > > > I'm not sure we need JMX API for this case. If I’m not mistaken, it’s > about > > the administrator’s ability to cancel any task/query in the cluster, and > in > > my opinion, such action must require administrator permission. > > Could you please clarify how it can be done via JMX? I mean permission > > check and etc. Perhaps, I'm missing something obvious... > > > > Thanks, > > S. > > > > чт, 16 янв. 2020 г. в 15:46, Николай Ижиков <[hidden email]>: > > > > > Alexey. > > > > > > I think, yes. > > > We certainly should be able to use system view data for the new KILL > API. > > > > > > I think we should support both SQL and Java(JMX) API for this KILL > > command. > > > > > > > > > > 16 янв. 2020 г., в 15:28, Alexei Scherbakov < > > > [hidden email]> написал(а): > > > > > > > > Nikolaj, > > > > > > > > Can we use system views instead of implementing something new ? > > > > > > > > Each user operation has an unique ID. > > > > > > > > It's possible to introduce universal SQL kill something like: > > > > > > > > kill transaction ID > > > > > > > > where id is taken from system view. > > > > > > > > > > > > чт, 16 янв. 2020 г. в 14:19, Николай Ижиков <[hidden email]>: > > > > > > > >> Hello, Alexey. > > > >> > > > >> I’m talking about *administrator* API. > > > >> > > > >> For example, the User has a cluster that is used by several > > > applications. > > > >> Some application starts buggy compute tasks that consume all CPU > > > resources. > > > >> Right now, administrator doesn’t have the ability to kill this task. > > > >> > > > >> This can lead to instability of the whole cluster. > > > >> > > > >> > > > >>> 16 янв. 2020 г., в 13:42, Alexei Scherbakov < > > > >> [hidden email]> написал(а): > > > >>> > > > >>> Transactions can be also rolled back using tx.close where close is > > > >>> java.lang.AutoCloseable#close > > > >>> It looks for me to the definition of uniform cancel API. > > > >>> > > > >>> > > > >>> > > > >>> чт, 16 янв. 2020 г. в 13:37, Alexei Scherbakov < > > > >> [hidden email] > > > >>>> : > > > >>> > > > >>>> Nikolaj, > > > >>>> > > > >>>> We already have cancellation possibilities for almost every user > > > >>>> computation. > > > >>>> Transactions are cancelled using tx.rollback() > > > >>>> Queries are cancelled using query.close() > > > >>>> Task is cancellable through ComputeTaskSession > > > >>>> > > > >>>> What is uniform cancel API ? Why do we need it ? > > > >>>> > > > >>>> > > > >>>> > > > >>>> ср, 15 янв. 2020 г. в 21:30, Николай Ижиков <[hidden email] > >: > > > >>>> > > > >>>>> Hello, Igniters. > > > >>>>> > > > >>>>> As you may know, we put a lot of effort to improve Ignite metric > > and > > > >>>>> diagnostic API. > > > >>>>> We have created the following API: > > > >>>>> * Metric manager > > > >>>>> * System view manager > > > >>>>> As far as I know, we would have tracing capabilities soon. > > > >>>>> > > > >>>>> I think it's time to take the next step. > > > >>>>> We should provide to the administrator the API to cancel(stop, > > kill) > > > >>>>> arbitrary user started computation. > > > >>>>> > > > >>>>> Right now we have API to do it for: > > > >>>>> * transactions `TransactionsMXBean#getActiveTransactions`. > > > >>>>> * SQL queries: `KILL QUERY` sql command and visor task - > > > >>>>> `VisorQueryCancelTask` > > > >>>>> > > > >>>>> Please, note, these features are implemented via different API. > > > >>>>> > > > >>>>> I think we should introduce uniform Cancel API for the following > > > >>>>> computations: > > > >>>>> > > > >>>>> * service. > > > >>>>> * specific service method execution. > > > >>>>> * compute job(compute task). > > > >>>>> * query(scan, continuous, text). > > > >>>>> > > > >>>>> We should also rework kill transaction and kill SQL queries API > to > > > >> make > > > >>>>> them similar to each other. > > > >>>>> I have plans to write an IEP for it and implement it. > > > >>>>> What do you think? > > > >>>>> > > > >>>>> > > > >>>> > > > >>>> -- > > > >>>> > > > >>>> Best regards, > > > >>>> Alexei Scherbakov > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> > > > >>> Best regards, > > > >>> Alexei Scherbakov > > > >> > > > >> > > > > > > > > -- > > > > > > > > Best regards, > > > > Alexei Scherbakov > > > > > > > > > > > -- > > Best regards, > Alexei Scherbakov > |
|
|
In reply to this post by Nikolay Izhikov-2
It would be grate.
Only one comment: we can't cancel service or service's method execution because service grid API doesn't imply any requirement to service implementation. So if user do something like infinite cycle or blocking but non-interruptible operation it's impossible to interrupt it. Also service method invocation itself isn't cluster wide or local node specific operation. This invocations not register anywhere and can't be tracked and killed. Unfortunately, the same is valid for some jobs in sense of infinite or non-interruptible operations. On Wed, Jan 15, 2020 at 9:30 PM Николай Ижиков <[hidden email]> wrote: > > Hello, Igniters. > > As you may know, we put a lot of effort to improve Ignite metric and diagnostic API. > We have created the following API: > * Metric manager > * System view manager > As far as I know, we would have tracing capabilities soon. > > I think it's time to take the next step. > We should provide to the administrator the API to cancel(stop, kill) arbitrary user started computation. > > Right now we have API to do it for: > * transactions `TransactionsMXBean#getActiveTransactions`. > * SQL queries: `KILL QUERY` sql command and visor task - `VisorQueryCancelTask` > > Please, note, these features are implemented via different API. > > I think we should introduce uniform Cancel API for the following computations: > > * service. > * specific service method execution. > * compute job(compute task). > * query(scan, continuous, text). > > We should also rework kill transaction and kill SQL queries API to make them similar to each other. > I have plans to write an IEP for it and implement it. > What do you think? > |
Re: [DISCUSSION] API to KILL any user started computation
|
|
Hello, Andrey.
Thanks for positive feedback. Appreciate it. > we can't cancel service or service's method I understand it. Seems, we have several obvious options here: * Try to do it and if fail answer to the user: «I’ve tried and fail. Sorry» * Kill hanging thread. > This invocations not register anywhere and can't be tracked and killed. Yes. So we should invent tracking for those invocation to be able to kill it :) > 17 янв. 2020 г., в 21:03, Andrey Gura <[hidden email]> написал(а): > > It would be grate. > > Only one comment: we can't cancel service or service's method > execution because service grid API doesn't imply any requirement to > service implementation. So if user do something like infinite cycle or > blocking but non-interruptible operation it's impossible to interrupt > it. Also service method invocation itself isn't cluster wide or local > node specific operation. This invocations not register anywhere and > can't be tracked and killed. > > Unfortunately, the same is valid for some jobs in sense of infinite or > non-interruptible operations. > > On Wed, Jan 15, 2020 at 9:30 PM Николай Ижиков <[hidden email]> wrote: >> >> Hello, Igniters. >> >> As you may know, we put a lot of effort to improve Ignite metric and diagnostic API. >> We have created the following API: >> * Metric manager >> * System view manager >> As far as I know, we would have tracing capabilities soon. >> >> I think it's time to take the next step. >> We should provide to the administrator the API to cancel(stop, kill) arbitrary user started computation. >> >> Right now we have API to do it for: >> * transactions `TransactionsMXBean#getActiveTransactions`. >> * SQL queries: `KILL QUERY` sql command and visor task - `VisorQueryCancelTask` >> >> Please, note, these features are implemented via different API. >> >> I think we should introduce uniform Cancel API for the following computations: >> >> * service. >> * specific service method execution. >> * compute job(compute task). >> * query(scan, continuous, text). >> >> We should also rework kill transaction and kill SQL queries API to make them similar to each other. >> I have plans to write an IEP for it and implement it. >> What do you think? >> |
[DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
|
|
Hello, Igniters.
I propose to create management API to dance user provided tasks and queries. Below my proposal in the IEP [1] form. Please, share your feedback. Motivation: Ignite provides many API to deploy and execute user-provided code on the server nodes inside the sam JVM as Ignite process runs. We also has a many APIs that allocate many resources on the server nodes: In case of some buggy code that consumes many system resources(CPU, RAM, flood network) or heavy query the whole cluster can becomes instable. We should provide to the cluster administator the ability to stop any user deployed task. Description: JMX beans to cancel listed tasks should be introduced: * Compute task * Service * Continuous query * Transactions * Queries(SQL, Scan, Text) In the scope of IEP-35 System view API introduced. New API should use the same identifier that is used in corresponding System View. Public API changes: * Cancel of Scan(text) queries should work similar to the SQL. * Cancel of compute task should invoke `ComputeTaskInternalFuture#cancel` for the corresponding task. * Cancel of service should be the same as `IgniteServices#cancel` * Cancel of transaction should works the same as in `TransactionMXBean#getActiveTransaction(kill=true)`. * Cancel of a continuous query should work the same as `QueryCursor#close` execution. ``` QueryMXBean { boolean cancelSQL(Long queryId); //Already implemented in IGNITE-4436. boolean cancelScan(Long queryId); boolean cancelText(Long queryId); } ComputeMXBean { boolean cancel(IgniteUuid id); } ServiceMXBean { boolean cancel(IgniteUuid id); } TransactionMXBean { boolean cancel(IgniteUuid xid); } interface ContinuousQueryMXBean { boolean cancel(UUID id); } ``` [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=145724615 > 17 янв. 2020 г., в 22:43, Николай Ижиков <[hidden email]> написал(а): > > Hello, Andrey. > > Thanks for positive feedback. > Appreciate it. > >> we can't cancel service or service's method > > I understand it. > Seems, we have several obvious options here: > > * Try to do it and if fail answer to the user: «I’ve tried and fail. Sorry» > * Kill hanging thread. > >> This invocations not register anywhere and can't be tracked and killed. > > Yes. > So we should invent tracking for those invocation to be able to kill it :) > > >> 17 янв. 2020 г., в 21:03, Andrey Gura <[hidden email]> написал(а): >> >> It would be grate. >> >> Only one comment: we can't cancel service or service's method >> execution because service grid API doesn't imply any requirement to >> service implementation. So if user do something like infinite cycle or >> blocking but non-interruptible operation it's impossible to interrupt >> it. Also service method invocation itself isn't cluster wide or local >> node specific operation. This invocations not register anywhere and >> can't be tracked and killed. >> >> Unfortunately, the same is valid for some jobs in sense of infinite or >> non-interruptible operations. >> >> On Wed, Jan 15, 2020 at 9:30 PM Николай Ижиков <[hidden email]> wrote: >>> >>> Hello, Igniters. >>> >>> As you may know, we put a lot of effort to improve Ignite metric and diagnostic API. >>> We have created the following API: >>> * Metric manager >>> * System view manager >>> As far as I know, we would have tracing capabilities soon. >>> >>> I think it's time to take the next step. >>> We should provide to the administrator the API to cancel(stop, kill) arbitrary user started computation. >>> >>> Right now we have API to do it for: >>> * transactions `TransactionsMXBean#getActiveTransactions`. >>> * SQL queries: `KILL QUERY` sql command and visor task - `VisorQueryCancelTask` >>> >>> Please, note, these features are implemented via different API. >>> >>> I think we should introduce uniform Cancel API for the following computations: >>> >>> * service. >>> * specific service method execution. >>> * compute job(compute task). >>> * query(scan, continuous, text). >>> >>> We should also rework kill transaction and kill SQL queries API to make them similar to each other. >>> I have plans to write an IEP for it and implement it. >>> What do you think? >>> > |
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
|
|
Nick
Can we retain only single cancel(long queryId) method in QueryMXBean ? I think query type can be determined by queryId. I also think we need similar "cancellation" API without the need to use mx beans. вт, 4 февр. 2020 г. в 18:07, Nikolay Izhikov <[hidden email]>: > Hello, Igniters. > > I propose to create management API to dance user provided tasks and > queries. > Below my proposal in the IEP [1] form. > > Please, share your feedback. > > > Motivation: > > Ignite provides many API to deploy and execute user-provided code on > the server nodes > inside the sam JVM as Ignite process runs. > > We also has a many APIs that allocate many resources on the server > nodes: > > In case of some buggy code that consumes many system resources(CPU, > RAM, flood network) or heavy query > the whole cluster can becomes instable. > We should provide to the cluster administator the ability to stop any > user deployed task. > > Description: > JMX beans to cancel listed tasks should be introduced: > * Compute task > * Service > * Continuous query > * Transactions > * Queries(SQL, Scan, Text) > > In the scope of IEP-35 System view API introduced. > New API should use the same identifier that is used in corresponding > System View. > > Public API changes: > * Cancel of Scan(text) queries should work similar to the SQL. > * Cancel of compute task should invoke > `ComputeTaskInternalFuture#cancel` for the corresponding task. > * Cancel of service should be the same as `IgniteServices#cancel` > * Cancel of transaction should works the same as in > `TransactionMXBean#getActiveTransaction(kill=true)`. > * Cancel of a continuous query should work the same as > `QueryCursor#close` execution. > > ``` > QueryMXBean { > boolean cancelSQL(Long queryId); //Already implemented in > IGNITE-4436. > > boolean cancelScan(Long queryId); > > boolean cancelText(Long queryId); > } > > ComputeMXBean { > boolean cancel(IgniteUuid id); > } > > ServiceMXBean { > boolean cancel(IgniteUuid id); > } > > TransactionMXBean { > boolean cancel(IgniteUuid xid); > } > > interface ContinuousQueryMXBean { > boolean cancel(UUID id); > } > ``` > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=145724615 > > > > > 17 янв. 2020 г., в 22:43, Николай Ижиков <[hidden email]> > написал(а): > > > > Hello, Andrey. > > > > Thanks for positive feedback. > > Appreciate it. > > > >> we can't cancel service or service's method > > > > I understand it. > > Seems, we have several obvious options here: > > > > * Try to do it and if fail answer to the user: «I’ve tried and fail. > Sorry» > > * Kill hanging thread. > > > >> This invocations not register anywhere and can't be tracked and killed. > > > > Yes. > > So we should invent tracking for those invocation to be able to kill it > :) > > > > > >> 17 янв. 2020 г., в 21:03, Andrey Gura <[hidden email]> написал(а): > >> > >> It would be grate. > >> > >> Only one comment: we can't cancel service or service's method > >> execution because service grid API doesn't imply any requirement to > >> service implementation. So if user do something like infinite cycle or > >> blocking but non-interruptible operation it's impossible to interrupt > >> it. Also service method invocation itself isn't cluster wide or local > >> node specific operation. This invocations not register anywhere and > >> can't be tracked and killed. > >> > >> Unfortunately, the same is valid for some jobs in sense of infinite or > >> non-interruptible operations. > >> > >> On Wed, Jan 15, 2020 at 9:30 PM Николай Ижиков <[hidden email]> > wrote: > >>> > >>> Hello, Igniters. > >>> > >>> As you may know, we put a lot of effort to improve Ignite metric and > diagnostic API. > >>> We have created the following API: > >>> * Metric manager > >>> * System view manager > >>> As far as I know, we would have tracing capabilities soon. > >>> > >>> I think it's time to take the next step. > >>> We should provide to the administrator the API to cancel(stop, kill) > arbitrary user started computation. > >>> > >>> Right now we have API to do it for: > >>> * transactions `TransactionsMXBean#getActiveTransactions`. > >>> * SQL queries: `KILL QUERY` sql command and visor task - > `VisorQueryCancelTask` > >>> > >>> Please, note, these features are implemented via different API. > >>> > >>> I think we should introduce uniform Cancel API for the following > computations: > >>> > >>> * service. > >>> * specific service method execution. > >>> * compute job(compute task). > >>> * query(scan, continuous, text). > >>> > >>> We should also rework kill transaction and kill SQL queries API to > make them similar to each other. > >>> I have plans to write an IEP for it and implement it. > >>> What do you think? > >>> > > > > -- Best regards, Alexei Scherbakov |
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
|
|
Folks,
Shouldn't we unify identifiers for all ongoing tasks in Ignite and move cancel to a single method in a single bean? Right now different types of IDs (long, UUID, IgniteUuid) look confusing and messy. I understand that such API is enforced by the IDs implementation in the corresponding Ignite subsystems. However, given that lately we developed a unified approach to metrics and system views with Nikolay's help, shouldn't we take one more step forward and unify this part as well? |
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
|
|
Igniters, thanks for the feedback.
> Can we retain only single cancel(long queryId) method in QueryMXBean ? > I think query type can be determined by queryId. It seems the answer is no we can’t do it for now. Different types of queries are handled by different managers and have different sets of identifiers. Please, take a look, into `ScanQueryView` [1], `RunningQueryManager` [2] > Right now different types of IDs (long, UUID, IgniteUuid) look confusing and messy. Yes. Let’s, fix it. > Shouldn't we unify identifiers for all ongoing tasks in Ignite and move cancel to a single method in a single bean? I think we should: 1. create separate beans for each API - TransactionMXBean, ComputeMXBean, etc. * The single method will be very error-prone: Consider copy-pasting wrong id from log to its parameters(killing not the buggy compute task, but *VERY* important transaction. How users even know about this type of error with the single method approach? * When the user wants to kill some task(from a script or by hand) he knows the type of task for sure. 2. Implement IEP-39 with the current identifiers. 3. Improve our API with unified identifier types. Bringing our API to the unified identifiers types approach is a very hard task to do and it will take a long time to implement. We shouldn’t leave our users without the proposed API until this moment. [1] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/spi/systemview/view/ScanQueryView.java [2] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/query/RunningQueryManager.java > 5 февр. 2020 г., в 12:07, Alexey Goncharuk <[hidden email]> написал(а): > > Folks, > > Shouldn't we unify identifiers for all ongoing tasks in Ignite and move > cancel to a single method in a single bean? > > Right now different types of IDs (long, UUID, IgniteUuid) look confusing > and messy. I understand that such API is enforced by the IDs implementation > in the corresponding Ignite subsystems. However, given that lately we > developed a unified approach to metrics and system views with Nikolay's > help, shouldn't we take one more step forward and unify this part as well? |
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
|
|
Nikolay,
> Consider copy-pasting wrong id from log to its > parameters(killing not the buggy compute task, but *VERY* important > transaction. > How users even know about this type of error with the > single method approach? > > I thought that the identifiers would never intersect (meaning that a transaction and a task would never share the same ID) I agree that change ID types for all objects would be a hard task, so probably it's worth discussing a single cancel entry on phase 3. |
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
|
|
Alexey.
I’m talking the following scenario: 1. Consider we have unified bean to kill tasks: CancelMXBean { public void cancel(long id); } 2. And we have the following log: ``` Transaction with ID=42 started. Compute task with ID=43 started. ``` 3. We want to kill compute task and by mistake executing: cancelMxBean.cancel(42); //This will kill transaction not compute task. The user doesn’t have a chance to know, what type of object he is killing. I think we should prevent this type of error by the API design. > 5 февр. 2020 г., в 14:43, Alexey Goncharuk <[hidden email]> написал(а): > > Nikolay, > > >> Consider copy-pasting wrong id from log to its >> parameters(killing not the buggy compute task, but *VERY* important >> transaction. >> How users even know about this type of error with the >> single method approach? >> >> I thought that the identifiers would never intersect (meaning that a > transaction and a task would never share the same ID) > > I agree that change ID types for all objects would be a hard task, so > probably it's worth discussing a single cancel entry on phase 3. |
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
|
|
Ticket [1] created.
[1] https://issues.apache.org/jira/browse/IGNITE-12632 > 5 февр. 2020 г., в 15:36, Nikolay Izhikov <[hidden email]> написал(а): > > Alexey. > > I’m talking the following scenario: > > 1. Consider we have unified bean to kill tasks: > > CancelMXBean { > public void cancel(long id); > } > > 2. And we have the following log: > > ``` > Transaction with ID=42 started. > Compute task with ID=43 started. > ``` > > 3. We want to kill compute task and by mistake executing: > > cancelMxBean.cancel(42); //This will kill transaction not compute task. > > The user doesn’t have a chance to know, what type of object he is killing. > I think we should prevent this type of error by the API design. > > >> 5 февр. 2020 г., в 14:43, Alexey Goncharuk <[hidden email]> написал(а): >> >> Nikolay, >> >> >>> Consider copy-pasting wrong id from log to its >>> parameters(killing not the buggy compute task, but *VERY* important >>> transaction. >>> How users even know about this type of error with the >>> single method approach? >>> >>> I thought that the identifiers would never intersect (meaning that a >> transaction and a task would never share the same ID) >> >> I agree that change ID types for all objects would be a hard task, so >> probably it's worth discussing a single cancel entry on phase 3. > |
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
|
|
Hello,
It seems to me we missed API that should be introduced into control utility. Nikolay, could you please note this requirement on the IEP page? Thanks, S. чт, 6 февр. 2020 г. в 15:29, Nikolay Izhikov <[hidden email]>: > Ticket [1] created. > > [1] https://issues.apache.org/jira/browse/IGNITE-12632 > > > 5 февр. 2020 г., в 15:36, Nikolay Izhikov <[hidden email]> > написал(а): > > > > Alexey. > > > > I’m talking the following scenario: > > > > 1. Consider we have unified bean to kill tasks: > > > > CancelMXBean { > > public void cancel(long id); > > } > > > > 2. And we have the following log: > > > > ``` > > Transaction with ID=42 started. > > Compute task with ID=43 started. > > ``` > > > > 3. We want to kill compute task and by mistake executing: > > > > cancelMxBean.cancel(42); //This will kill transaction not compute task. > > > > The user doesn’t have a chance to know, what type of object he is > killing. > > I think we should prevent this type of error by the API design. > > > > > >> 5 февр. 2020 г., в 14:43, Alexey Goncharuk <[hidden email]> > написал(а): > >> > >> Nikolay, > >> > >> > >>> Consider copy-pasting wrong id from log to its > >>> parameters(killing not the buggy compute task, but *VERY* important > >>> transaction. > >>> How users even know about this type of error with the > >>> single method approach? > >>> > >>> I thought that the identifiers would never intersect (meaning that a > >> transaction and a task would never share the same ID) > >> > >> I agree that change ID types for all objects would be a hard task, so > >> probably it's worth discussing a single cancel entry on phase 3. > > > > |
«
Return to Apache Ignite Developers - Legacy Mail Archive
|
1 view|%1 views
| Free forum by Nabble | Edit this page |