[zeromq-dev] Router/Dealer configuration monopolizes CPU while idle

[Mazzaroth M.]

I have a Router/Dealer setup which is polling on a PULL socket to listen
for jobs. It forks 10 Dealer/Worker processes. When I run it with the
script below it will use about 20% cpu after about 5 min and then about a
minute later use 100% cpu on a ubuntu 16.04 setup. This is all while idle.
I'm not sure of the root cause.

Controller
https://gist.github.com/mazz/f50affc65bbf04a6140e257354b16898

Worker
https://gist.github.com/mazz/34ba3d98296bf5d013037b6ad48b6cb6

import sys
from pushpull.models import homedirs

sys.path[0:0] = homedirs
import actors.controller

if __name__ == '__main__':
sys.exit(actors.controller.run())

[Michel Pelletier]

I just did a brief scan of your code, but it looks like your controller is
using poll(0) in two places in a loop. poll(0) will return immediately and
your code will busy loop around the 'while True' if idle. I think a better
approach would be to register the router socket with the poller and use
poll(None) always processing the router socket messages first.

-Michel

Post by Mazzaroth M.
I have a Router/Dealer setup which is polling on a PULL socket to listen
for jobs. It forks 10 Dealer/Worker processes. When I run it with the
script below it will use about 20% cpu after about 5 min and then about a
minute later use 100% cpu on a ubuntu 16.04 setup. This is all while idle.
I'm not sure of the root cause.
Controller
https://gist.github.com/mazz/f50affc65bbf04a6140e257354b16898
Worker
https://gist.github.com/mazz/34ba3d98296bf5d013037b6ad48b6cb6
import sys
from pushpull.models import homedirs
sys.path[0:0] = homedirs
import actors.controller
sys.exit(actors.controller.run())
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Mazzaroth M.]

yes I think know what you're driving at. Didn't know None/-1 was an option
for poll(). Since I know the root cause I will tinker and see if I can have
it -1 wait on the poller for the router. I guess what you're saying is when
the Router receives something I can drop out of the nested While and poll()
on the PULL socket.

On Sun, Nov 20, 2016 at 1:38 PM, Michel Pelletier <

Post by Michel Pelletier
I just did a brief scan of your code, but it looks like your controller is
using poll(0) in two places in a loop. poll(0) will return immediately and
your code will busy loop around the 'while True' if idle. I think a better
approach would be to register the router socket with the poller and use
poll(None) always processing the router socket messages first.
-Michel

Post by Mazzaroth M.
I have a Router/Dealer setup which is polling on a PULL socket to listen
for jobs. It forks 10 Dealer/Worker processes. When I run it with the
script below it will use about 20% cpu after about 5 min and then about a
minute later use 100% cpu on a ubuntu 16.04 setup. This is all while idle.
I'm not sure of the root cause.
Controller
https://gist.github.com/mazz/f50affc65bbf04a6140e257354b16898
Worker
https://gist.github.com/mazz/34ba3d98296bf5d013037b6ad48b6cb6
import sys
from pushpull.models import homedirs
sys.path[0:0] = homedirs
import actors.controller
sys.exit(actors.controller.run())
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Auer, Jens]

In addition, it is usually better to read more than one message when poll signals activity on a socket. This reduces the number of poll calls, and these are quite expensive.

Cheers,
Jens
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
***@cgi.com<mailto:***@cgi.com>
Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf Of Mazzaroth M.
Sent: 20 November 2016 23:48
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Router/Dealer configuration monopolizes CPU while idle

yes I think know what you're driving at. Didn't know None/-1 was an option for poll(). Since I know the root cause I will tinker and see if I can have it -1 wait on the poller for the router. I guess what you're saying is when the Router receives something I can drop out of the nested While and poll() on the PULL socket.

On Sun, Nov 20, 2016 at 1:38 PM, Michel Pelletier <***@gmail.com<mailto:***@gmail.com>> wrote:
I just did a brief scan of your code, but it looks like your controller is using poll(0) in two places in a loop. poll(0) will return immediately and your code will busy loop around the 'while True' if idle. I think a better approach would be to register the router socket with the poller and use poll(None) always processing the router socket messages first.

-Michel

On Sun, Nov 20, 2016 at 9:41 AM, Mazzaroth M. <***@gmail.com<mailto:***@gmail.com>> wrote:
I have a Router/Dealer setup which is polling on a PULL socket to listen for jobs. It forks 10 Dealer/Worker processes. When I run it with the script below it will use about 20% cpu after about 5 min and then about a minute later use 100% cpu on a ubuntu 16.04 setup. This is all while idle. I'm not sure of the root cause.

Controller
https://gist.github.com/mazz/f50affc65bbf04a6140e257354b16898
Worker
https://gist.github.com/mazz/34ba3d98296bf5d013037b6ad48b6cb6

import sys
from pushpull.models import homedirs

sys.path[0:0] = homedirs
import actors.controller

if __name__ == '__main__':
sys.exit(actors.controller.run())

_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org<mailto:zeromq-***@lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org<mailto:zeromq-***@lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Kevin Sapper]

@Jens that might be a useful pattern that you could consider contributing
to the cookbook (https://github.com/zeromq/cookbook) or the zguide.

Post by Auer, Jens
In addition, it is usually better to read more than one message when poll
signals activity on a socket. This reduces the number of poll calls, and
these are quite expensive.
Cheers,
Jens
*--*
*Dr. Jens Auer *| CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
unter de.cgi.com/pflichtangaben.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
CGI Group Inc. and its affiliates may be contained in this message. If you
are not a recipient indicated or intended in this message (or responsible
for delivery of this message to such person), or you think for any reason
that this message may have been addressed to you in error, you may not use
or copy or deliver this message to anyone else. In such case, you should
destroy this message and are asked to notify the sender by reply e-mail.
Behalf Of *Mazzaroth M.
*Sent:* 20 November 2016 23:48
*To:* ZeroMQ development list
*Subject:* Re: [zeromq-dev] Router/Dealer configuration monopolizes CPU
while idle
yes I think know what you're driving at. Didn't know None/-1 was an option
for poll(). Since I know the root cause I will tinker and see if I can have
it -1 wait on the poller for the router. I guess what you're saying is when
the Router receives something I can drop out of the nested While and poll()
on the PULL socket.
On Sun, Nov 20, 2016 at 1:38 PM, Michel Pelletier <
I just did a brief scan of your code, but it looks like your controller is
using poll(0) in two places in a loop. poll(0) will return immediately and
your code will busy loop around the 'while True' if idle. I think a better
approach would be to register the router socket with the poller and use
poll(None) always processing the router socket messages first.
-Michel
I have a Router/Dealer setup which is polling on a PULL socket to listen
for jobs. It forks 10 Dealer/Worker processes. When I run it with the
script below it will use about 20% cpu after about 5 min and then about a
minute later use 100% cpu on a ubuntu 16.04 setup. This is all while idle.
I'm not sure of the root cause.
Controller
https://gist.github.com/mazz/f50affc65bbf04a6140e257354b16898
Worker
https://gist.github.com/mazz/34ba3d98296bf5d013037b6ad48b6cb6
import sys
from pushpull.models import homedirs
sys.path[0:0] = homedirs
import actors.controller
sys.exit(actors.controller.run())
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev