[zeromq-dev] Assertion failed: !more (src/fq.cpp:117)

Discussion:

Ranjeet Kumar

2016-09-10 13:17:35 UTC

Hi,

I am using zeromq 4.1.5 and czmq 3.0.2. I am using the server-client as
router dealer. I am getting this *Assertion failed: !more (src/fq.cpp:117*)
issue while 50k clients starts making connecting with the server.

stack trace is below:
Assertion failed: !more (src/fq.cpp:117)

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff490d700 (LWP 24885)]
0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-106.el7_2.6.x86_64 libgcc-4.8.5-4.el7.x86_64
libstdc++-4.8.5-4.el7.x86_64
(gdb) bt
#0 0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff68f2ce8 in abort () from /lib64/libc.so.6
#2 0x00007ffff7b8f769 in zmq::zmq_abort (errmsg_=***@entry=0x7ffff7bc1eca
"!more") at src/err.cpp:83
#3 0x00007ffff7b8fccd in zmq::fq_t::recvpipe (this=***@entry=0x60bdc0,
msg_=***@entry=0x7ffff490ce60, pipe_=***@entry=0x7ffff490cd08) at
src/fq.cpp:117
#4 0x00007ffff7ba3423 in zmq::router_t::xrecv (this=0x60b8d0,
msg_=0x7ffff490ce60) at src/router.cpp:301
#5 0x00007ffff7ba878b in zmq::socket_base_t::recv (this=***@entry=0x60b8d0,
msg_=***@entry=0x7ffff490ce60, flags_=***@entry=0) at
src/socket_base.cpp:910
#6 0x00007ffff7ba1015 in forward (from_=***@entry=0x60b8d0,
to_=***@entry=0x60c0b0,
capture_=***@entry=0x0, msg_=...) at src/proxy.cpp:81
#7 0x00007ffff7ba11e7 in zmq::proxy (frontend_=0x60b8d0,
backend_=0x60c0b0, capture_=0x0, control_=0x0) at src/proxy.cpp:188
#8 0x0000000000403fcf in broker_task (args=0x0) at push_server.cpp:1348
#9 0x00007ffff791978f in s_thread_shim (args=0x60d590) at src/zthread.c:84
#10 0x00007ffff76b6dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff69b2ced in clone () from /lib64/libc.so.6

Any suggestion to fix this issue.

Thanks
(Ranjeet Kumar)

Martin Hurton

2016-09-15 18:48:18 UTC

Permalink

Hi Ranjeet,

Is this easy to reproduce? Thanks.

- Martin

Post by Ranjeet Kumar
Hi,
I am using zeromq 4.1.5 and czmq 3.0.2. I am using the server-client as
router dealer. I am getting this *Assertion failed: !more
(src/fq.cpp:117*) issue while 50k clients starts making connecting with
the server.
Assertion failed: !more (src/fq.cpp:117)
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff490d700 (LWP 24885)]
0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-106.el7_2.6.x86_64 libgcc-4.8.5-4.el7.x86_64
libstdc++-4.8.5-4.el7.x86_64
(gdb) bt
#0 0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff68f2ce8 in abort () from /lib64/libc.so.6
"!more") at src/err.cpp:83
src/fq.cpp:117
#4 0x00007ffff7ba3423 in zmq::router_t::xrecv (this=0x60b8d0,
msg_=0x7ffff490ce60) at src/router.cpp:301
src/socket_base.cpp:910
src/proxy.cpp:81
#7 0x00007ffff7ba11e7 in zmq::proxy (frontend_=0x60b8d0,
backend_=0x60c0b0, capture_=0x0, control_=0x0) at src/proxy.cpp:188
#8 0x0000000000403fcf in broker_task (args=0x0) at push_server.cpp:1348
#9 0x00007ffff791978f in s_thread_shim (args=0x60d590) at src/zthread.c:84
#10 0x00007ffff76b6dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff69b2ced in clone () from /lib64/libc.so.6
Any suggestion to fix this issue.
Thanks
(Ranjeet Kumar)
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Ranjeet Kumar

2016-09-16 08:13:29 UTC

Permalink

Hi Martin,

Yes, it is very often. I am having 100k clients connecting with my server.
So in order to run the application i had commented the line from fq.cpp 117
zmq_assert (!more);

And now it is working for me.

Thanks
(Ranjeet Kumar)

Post by Martin Hurton
Hi Ranjeet,
Is this easy to reproduce? Thanks.
- Martin

Post by Ranjeet Kumar
Hi,
I am using zeromq 4.1.5 and czmq 3.0.2. I am using the server-client as
router dealer. I am getting this *Assertion failed: !more
(src/fq.cpp:117*) issue while 50k clients starts making connecting with
the server.
Assertion failed: !more (src/fq.cpp:117)
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff490d700 (LWP 24885)]
0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-106.el7_2.6.x86_64 libgcc-4.8.5-4.el7.x86_64
libstdc++-4.8.5-4.el7.x86_64
(gdb) bt
#0 0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff68f2ce8 in abort () from /lib64/libc.so.6
"!more") at src/err.cpp:83
src/fq.cpp:117
#4 0x00007ffff7ba3423 in zmq::router_t::xrecv (this=0x60b8d0,
msg_=0x7ffff490ce60) at src/router.cpp:301
src/socket_base.cpp:910
src/proxy.cpp:81
#7 0x00007ffff7ba11e7 in zmq::proxy (frontend_=0x60b8d0,
backend_=0x60c0b0, capture_=0x0, control_=0x0) at src/proxy.cpp:188
#8 0x0000000000403fcf in broker_task (args=0x0) at push_server.cpp:1348
#9 0x00007ffff791978f in s_thread_shim (args=0x60d590) at
src/zthread.c:84
#10 0x00007ffff76b6dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff69b2ced in clone () from /lib64/libc.so.6
Any suggestion to fix this issue.
Thanks
(Ranjeet Kumar)
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Doron Somech

2016-10-20 13:51:37 UTC

Permalink

Is the handover option enabled?

Post by Ranjeet Kumar
Hi Martin,
Yes, it is very often. I am having 100k clients connecting with my server.
So in order to run the application i had commented the line from fq.cpp 117
zmq_assert (!more);
And now it is working for me.
Thanks
(Ranjeet Kumar)

Post by Martin Hurton
Hi Ranjeet,
Is this easy to reproduce? Thanks.
- Martin

Post by Ranjeet Kumar
Hi,
I am using zeromq 4.1.5 and czmq 3.0.2. I am using the server-client as
router dealer. I am getting this Assertion failed: !more (src/fq.cpp:117)
issue while 50k clients starts making connecting with the server.
Assertion failed: !more (src/fq.cpp:117)
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff490d700 (LWP 24885)]
0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-106.el7_2.6.x86_64 libgcc-4.8.5-4.el7.x86_64
libstdc++-4.8.5-4.el7.x86_64
(gdb) bt
#0 0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff68f2ce8 in abort () from /lib64/libc.so.6
#2 0x00007ffff7b8f769 in zmq::zmq_abort
src/fq.cpp:117
#4 0x00007ffff7ba3423 in zmq::router_t::xrecv (this=0x60b8d0,
msg_=0x7ffff490ce60) at src/router.cpp:301
#5 0x00007ffff7ba878b in zmq::socket_base_t::recv
src/proxy.cpp:81
#7 0x00007ffff7ba11e7 in zmq::proxy (frontend_=0x60b8d0,
backend_=0x60c0b0, capture_=0x0, control_=0x0) at src/proxy.cpp:188
#8 0x0000000000403fcf in broker_task (args=0x0) at push_server.cpp:1348
#9 0x00007ffff791978f in s_thread_shim (args=0x60d590) at
src/zthread.c:84
#10 0x00007ffff76b6dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff69b2ced in clone () from /lib64/libc.so.6
Any suggestion to fix this issue.
Thanks
(Ranjeet Kumar)
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Ranjeet Kumar

2016-10-20 16:28:04 UTC

Permalink

Yes,
I had used
zsocket_set_router_handover( self->router, 1);

in my code.

Post by Doron Somech
Is the handover option enabled?

Post by Ranjeet Kumar
Hi Martin,
Yes, it is very often. I am having 100k clients connecting with my

server.

Post by Ranjeet Kumar
So in order to run the application i had commented the line from fq.cpp

117

Post by Ranjeet Kumar
zmq_assert (!more);
And now it is working for me.
Thanks
(Ranjeet Kumar)

Post by Martin Hurton
Hi Ranjeet,
Is this easy to reproduce? Thanks.
- Martin

Post by Ranjeet Kumar
Hi,
I am using zeromq 4.1.5 and czmq 3.0.2. I am using the server-client as
router dealer. I am getting this Assertion failed: !more

(src/fq.cpp:117)

Post by Ranjeet Kumar

Post by Martin Hurton

Post by Ranjeet Kumar
issue while 50k clients starts making connecting with the server.
Assertion failed: !more (src/fq.cpp:117)
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff490d700 (LWP 24885)]
0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-106.el7_2.6.x86_64 libgcc-4.8.5-4.el7.x86_64
libstdc++-4.8.5-4.el7.x86_64
(gdb) bt
#0 0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff68f2ce8 in abort () from /lib64/libc.so.6
#2 0x00007ffff7b8f769 in zmq::zmq_abort

=0x60bdc0,

Post by Ranjeet Kumar

Post by Martin Hurton

Post by Ranjeet Kumar
src/fq.cpp:117
#4 0x00007ffff7ba3423 in zmq::router_t::xrecv (this=0x60b8d0,
msg_=0x7ffff490ce60) at src/router.cpp:301
#5 0x00007ffff7ba878b in zmq::socket_base_t::recv
src/proxy.cpp:81
#7 0x00007ffff7ba11e7 in zmq::proxy (frontend_=0x60b8d0,
backend_=0x60c0b0, capture_=0x0, control_=0x0) at src/proxy.cpp:188
#8 0x0000000000403fcf in broker_task (args=0x0) at

push_server.cpp:1348

Post by Ranjeet Kumar

Post by Martin Hurton

Post by Ranjeet Kumar
#9 0x00007ffff791978f in s_thread_shim (args=0x60d590) at
src/zthread.c:84
#10 0x00007ffff76b6dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff69b2ced in clone () from /lib64/libc.so.6
Any suggestion to fix this issue.
Thanks
(Ranjeet Kumar)
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Doron Somech

2016-10-21 07:07:42 UTC

Permalink

Try to apply following commit, I think it should fix it:

https://github.com/somdoron/libzmq/commit/83e93f9ea45ff789d9fa9c92e01335a4c0386be0

Martin, what do you think?

Post by Ranjeet Kumar
Yes,
I had used
zsocket_set_router_handover( self->router, 1);
in my code.

Post by Doron Somech
Is the handover option enabled?

Post by Martin Hurton
Hi Ranjeet,
Is this easy to reproduce? Thanks.
- Martin

Post by Ranjeet Kumar
Hi,
I am using zeromq 4.1.5 and czmq 3.0.2. I am using the server-client as
router dealer. I am getting this Assertion failed: !more
(src/fq.cpp:117)
issue while 50k clients starts making connecting with the server.
Assertion failed: !more (src/fq.cpp:117)
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff490d700 (LWP 24885)]
0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-106.el7_2.6.x86_64 libgcc-4.8.5-4.el7.x86_64
libstdc++-4.8.5-4.el7.x86_64
(gdb) bt
#0 0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff68f2ce8 in abort () from /lib64/libc.so.6
#2 0x00007ffff7b8f769 in zmq::zmq_abort
#3 0x00007ffff7b8fccd in zmq::fq_t::recvpipe
src/fq.cpp:117
#4 0x00007ffff7ba3423 in zmq::router_t::xrecv (this=0x60b8d0,
msg_=0x7ffff490ce60) at src/router.cpp:301
#5 0x00007ffff7ba878b in zmq::socket_base_t::recv
src/proxy.cpp:81
#7 0x00007ffff7ba11e7 in zmq::proxy (frontend_=0x60b8d0,
backend_=0x60c0b0, capture_=0x0, control_=0x0) at src/proxy.cpp:188
#8 0x0000000000403fcf in broker_task (args=0x0) at
push_server.cpp:1348
#9 0x00007ffff791978f in s_thread_shim (args=0x60d590) at src/zthread.c:84
#10 0x00007ffff76b6dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff69b2ced in clone () from /lib64/libc.so.6
Any suggestion to fix this issue.
Thanks
(Ranjeet Kumar)
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Doron Somech

2016-10-21 08:12:31 UTC

Permalink

Sorry, this one should be more accurate:

https://github.com/somdoron/libzmq/commit/5b0c7144df423b22a75c25e5e3c0d83ff70eb006

Post by Doron Somech
https://github.com/somdoron/libzmq/commit/83e93f9ea45ff789d9fa9c92e01335a4c0386be0
Martin, what do you think?

Post by Ranjeet Kumar
Yes,
I had used
zsocket_set_router_handover( self->router, 1);
in my code.

Post by Doron Somech
Is the handover option enabled?

Post by Martin Hurton
Hi Ranjeet,
Is this easy to reproduce? Thanks.
- Martin

Post by Ranjeet Kumar
Hi,
I am using zeromq 4.1.5 and czmq 3.0.2. I am using the server-client as
router dealer. I am getting this Assertion failed: !more (src/fq.cpp:117)
issue while 50k clients starts making connecting with the server.
Assertion failed: !more (src/fq.cpp:117)
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff490d700 (LWP 24885)]
0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-106.el7_2.6.x86_64 libgcc-4.8.5-4.el7.x86_64
libstdc++-4.8.5-4.el7.x86_64
(gdb) bt
#0 0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff68f2ce8 in abort () from /lib64/libc.so.6
#2 0x00007ffff7b8f769 in zmq::zmq_abort
#3 0x00007ffff7b8fccd in zmq::fq_t::recvpipe
src/fq.cpp:117
#4 0x00007ffff7ba3423 in zmq::router_t::xrecv (this=0x60b8d0,
msg_=0x7ffff490ce60) at src/router.cpp:301
#5 0x00007ffff7ba878b in zmq::socket_base_t::recv
src/proxy.cpp:81
#7 0x00007ffff7ba11e7 in zmq::proxy (frontend_=0x60b8d0,
backend_=0x60c0b0, capture_=0x0, control_=0x0) at src/proxy.cpp:188
#8 0x0000000000403fcf in broker_task (args=0x0) at
push_server.cpp:1348
#9 0x00007ffff791978f in s_thread_shim (args=0x60d590) at src/zthread.c:84
#10 0x00007ffff76b6dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff69b2ced in clone () from /lib64/libc.so.6
Any suggestion to fix this issue.
Thanks
(Ranjeet Kumar)
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

zmqdev

2016-10-21 10:23:35 UTC

Permalink

Hi Doron,

I tried out your latest repo

https://github.com/somdoron/libzmq/commit/3775d0853a8c1f1c3854a94c7fe12e78046faeca

with the changes to src/socket_base.cpp, src/pipe.cpp and src/pipe.hpp.

I confirm that the problem reported at

https://github.com/zeromq/libzmq/issues/2163

is solved: the assertion in src/fq.cpp:118 is not triggered anymore.

Thanks!

There is one point I'm unsure about. The documentation of zmq_disconnect
states

Any outstanding messages physically received from the network but not
yet received by the application with _zmq_recv()_ shall be discarded.

https://github.com/zeromq/libzmq/blob/master/doc/zmq_disconnect.txt#L19-L20

In the PUB->SUB test case, the message is not discarded by the
zmq_disconnect, and is received (sans abort) with zmq_msg_recv.

Doesn't this behavior contradict the doc?

Doron Somech

2016-10-21 10:56:08 UTC

Permalink

Not really, only if you are in the middle of receiving multipart
message you will continue to get it. If not, you won't get any pending
messages.

Post by zmqdev
Hi Doron,
I tried out your latest repo
https://github.com/somdoron/libzmq/commit/3775d0853a8c1f1c3854a94c7fe12e78046faeca
with the changes to src/socket_base.cpp, src/pipe.cpp and src/pipe.hpp.
I confirm that the problem reported at
https://github.com/zeromq/libzmq/issues/2163
is solved: the assertion in src/fq.cpp:118 is not triggered anymore.
Thanks!
There is one point I'm unsure about. The documentation of zmq_disconnect
states
Any outstanding messages physically received from the network but not
yet received by the application with _zmq_recv()_ shall be discarded.
https://github.com/zeromq/libzmq/blob/master/doc/zmq_disconnect.txt#L19-L20
In the PUB->SUB test case, the message is not discarded by the
zmq_disconnect, and is received (sans abort) with zmq_msg_recv.
Doesn't this behavior contradict the doc?
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Doron Somech

2016-10-21 12:43:17 UTC

Permalink

I actually fixed this bug almost a year ago and totally forgot about
it, however it was not ported to 4.1.

here is the fix a year ago:
https://github.com/zeromq/libzmq/commit/804bce8294daf243d14ac15f426243c568c574a6

Post by Doron Somech
https://github.com/somdoron/libzmq/commit/5b0c7144df423b22a75c25e5e3c0d83ff70eb006

Post by Doron Somech
https://github.com/somdoron/libzmq/commit/83e93f9ea45ff789d9fa9c92e01335a4c0386be0
Martin, what do you think?

Post by Ranjeet Kumar
Yes,
I had used
zsocket_set_router_handover( self->router, 1);
in my code.

Post by Doron Somech
Is the handover option enabled?

Post by Martin Hurton
Hi Ranjeet,
Is this easy to reproduce? Thanks.
- Martin

Post by Ranjeet Kumar
Hi,
I am using zeromq 4.1.5 and czmq 3.0.2. I am using the server-client as
router dealer. I am getting this Assertion failed: !more (src/fq.cpp:117)
issue while 50k clients starts making connecting with the server.
Assertion failed: !more (src/fq.cpp:117)
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff490d700 (LWP 24885)]
0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-106.el7_2.6.x86_64 libgcc-4.8.5-4.el7.x86_64
libstdc++-4.8.5-4.el7.x86_64
(gdb) bt
#0 0x00007ffff68f15f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff68f2ce8 in abort () from /lib64/libc.so.6
#2 0x00007ffff7b8f769 in zmq::zmq_abort
#3 0x00007ffff7b8fccd in zmq::fq_t::recvpipe
src/fq.cpp:117
#4 0x00007ffff7ba3423 in zmq::router_t::xrecv (this=0x60b8d0,
msg_=0x7ffff490ce60) at src/router.cpp:301
#5 0x00007ffff7ba878b in zmq::socket_base_t::recv
src/proxy.cpp:81
#7 0x00007ffff7ba11e7 in zmq::proxy (frontend_=0x60b8d0,
backend_=0x60c0b0, capture_=0x0, control_=0x0) at src/proxy.cpp:188
#8 0x0000000000403fcf in broker_task (args=0x0) at
push_server.cpp:1348
#9 0x00007ffff791978f in s_thread_shim (args=0x60d590) at src/zthread.c:84
#10 0x00007ffff76b6dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff69b2ced in clone () from /lib64/libc.so.6
Any suggestion to fix this issue.
Thanks
(Ranjeet Kumar)
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev