[zeromq-dev] Zmq 4.2.0 aligned memory

[Emmanuel Taurel]

Hello all,

We are using zeromq since years now without troubles. We have recently
tried our software using Zmq 4.2.0 (on linux hosts).
For our application, we are using multipart messages with 4 parts in
publish/subscribe mode.
With Zmq 4.0.5, on the subscriber side, when we get the last message
part, the received buffer was memory aligned
(at least on 0x4 border). Unfortunately, with Zmq 4.2.0, the buffer is
not aligned any more.
For instance with Zmq 4.0.5, the buffer was at address xxx08 while with
Zmq 4.2.0, it is at address xxx23.

I don't know if it is relevant but our messages are relatively small
messages (few tens of bytes)
This is a problem for us in decoding the buffer content.

Is there something to be done to have memory aligned buffers?

Thank's in advance for your answers

Emmanuel

[Luca Boccassi]

Post by Emmanuel Taurel
Hello all,
We are using zeromq since years now without troubles. We have recently
tried our software using Zmq 4.2.0 (on linux hosts).
For our application, we are using multipart messages with 4 parts in
publish/subscribe mode.
With Zmq 4.0.5, on the subscriber side, when we get the last message
part, the received buffer was memory aligned
(at least on 0x4 border). Unfortunately, with Zmq 4.2.0, the buffer is
not aligned any more.
For instance with Zmq 4.0.5, the buffer was at address xxx08 while with
Zmq 4.2.0, it is at address xxx23.
I don't know if it is relevant but our messages are relatively small
messages (few tens of bytes)
This is a problem for us in decoding the buffer content.
Is there something to be done to have memory aligned buffers?
Thank's in advance for your answers
Emmanuel

Hi,

Since 4.0.5 a couple things have changed externally:

- zmq_msg_t has increased to 64 bytes from 32 bytes

- zmq_msg_t is now aligned to pointer size, to fix SIGBUS crashes on
some architectures

Exactly how large are your payloads?

It's possible that before they didn't fit in the inline buffer, since it
was really small, but they do now and that means they are not aligned
anymore since the inline buffer is not the first element of the struct:

struct {
metadata_t *metadata;
unsigned char data [max_vsm_size];
unsigned char size;
unsigned char type;
unsigned char flags;
char group [16];
uint32_t routing_id;
} vsm;

Although an xxx23 address is a bit strange. In theory it should be
aligned to ptr size + 8.

If they are bigger than the very small size and they end up on the heap,
then the content_t is not aligned at all:

struct content_t
{
void *data;
size_t size;
msg_free_fn *ffn;
void *hint;
zmq::atomic_counter_t refcnt;
};

You could try to add the alignment attribute there in src/msg.hpp and
see if it makes a difference for you.

[Jens Auer]

Hi,

I think I have an idea why you are seeing unaligned messages, but this only applies to messages where the payload is not stored in the msg_t object itself. I think the threshold for this is 64 bytes. In ZeroMQ 4.1, receiving messages was done by first receiving from the socket into a static 8kb buffer, and then a new message object was created that allocated memory externally by calling malloc. The payload was then copied from the receive buffer to the message buffer. The malloced message buffer was aligned probably.

In ZeroMQ 4.2, this is changed to reduce the number of malloc calls and copy operations. The receive buffer is now dynamically allocated as a 8kb block, and messages are constructed as zero-copy messages using the part of the receive buffer containing the payload. This saves malloc calls and copy operations and increases performance. However, the payload may now start at basically arbitrary addresses. As an example, let's assume that we receive a small message of 10 bytes and a large message of 1kb, both received in a single call to recv on the socket. The engine allocates a new buffer of 8kb, calls recv(socket, buffer) and the data is written to the buffer. A small message is then created which contains the data from byte 2-11 in the msg_t, byte 1 contains the header. At byte 12 starts the header of the next message, and at byte 22(?) starts the payload. The large message is created as a zero-copy message using the pointer to byte 22 as storage. This is not aligned to a 4-byte address.

Could you provide some more information about the sizes of the messages that you receive? How do you decode the buffer content?

Best wishes,
Jens

-----Ursprüngliche Nachricht-----
Von: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] Im Auftrag von Emmanuel Taurel
Gesendet: Montag, 14. November 2016 16:49
An: zeromq-***@lists.zeromq.org
Betreff: [zeromq-dev] Zmq 4.2.0 aligned memory

Hello all,

We are using zeromq since years now without troubles. We have recently tried our software using Zmq 4.2.0 (on linux hosts).
For our application, we are using multipart messages with 4 parts in publish/subscribe mode.
With Zmq 4.0.5, on the subscriber side, when we get the last message part, the received buffer was memory aligned (at least on 0x4 border). Unfortunately, with Zmq 4.2.0, the buffer is not aligned any more.
For instance with Zmq 4.0.5, the buffer was at address xxx08 while with Zmq 4.2.0, it is at address xxx23.

I don't know if it is relevant but our messages are relatively small messages (few tens of bytes) This is a problem for us in decoding the buffer content.

Is there something to be done to have memory aligned buffers?

Thank's in advance for your answers

Emmanuel
_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

[KIU Shueng Chuan]

A common use case for me is sending an array of floats.

First message part is some small metadata. Second message part is the float
array.

On reception, zmq_msg_data is cast to float* and accessed directly.

This non-alignment would be problematic.
Or perhaps there never was any alignment guarantee?

Post by Jens Auer
Hi,
I think I have an idea why you are seeing unaligned messages, but this
only applies to messages where the payload is not stored in the msg_t
object itself. I think the threshold for this is 64 bytes. In ZeroMQ 4.1,
receiving messages was done by first receiving from the socket into a
static 8kb buffer, and then a new message object was created that allocated
memory externally by calling malloc. The payload was then copied from the
receive buffer to the message buffer. The malloced message buffer was
aligned probably.
In ZeroMQ 4.2, this is changed to reduce the number of malloc calls and
copy operations. The receive buffer is now dynamically allocated as a 8kb
block, and messages are constructed as zero-copy messages using the part of
the receive buffer containing the payload. This saves malloc calls and copy
operations and increases performance. However, the payload may now start at
basically arbitrary addresses. As an example, let's assume that we receive
a small message of 10 bytes and a large message of 1kb, both received in a
single call to recv on the socket. The engine allocates a new buffer of
8kb, calls recv(socket, buffer) and the data is written to the buffer. A
small message is then created which contains the data from byte 2-11 in the
msg_t, byte 1 contains the header. At byte 12 starts the header of the next
message, and at byte 22(?) starts the payload. The large message is created
as a zero-copy message using the pointer to byte 22 as storage. This is not
aligned to a 4-byte address.
Could you provide some more information about the sizes of the messages
that you receive? How do you decode the buffer content?
Best wishes,
Jens
-----UrsprÃŒngliche Nachricht-----
Gesendet: Montag, 14. November 2016 16:49
Betreff: [zeromq-dev] Zmq 4.2.0 aligned memory
Hello all,
We are using zeromq since years now without troubles. We have recently
tried our software using Zmq 4.2.0 (on linux hosts).
For our application, we are using multipart messages with 4 parts in
publish/subscribe mode.
With Zmq 4.0.5, on the subscriber side, when we get the last message part,
the received buffer was memory aligned (at least on 0x4 border).
Unfortunately, with Zmq 4.2.0, the buffer is not aligned any more.
For instance with Zmq 4.0.5, the buffer was at address xxx08 while with
Zmq 4.2.0, it is at address xxx23.
I don't know if it is relevant but our messages are relatively small
messages (few tens of bytes) This is a problem for us in decoding the
buffer content.
Is there something to be done to have memory aligned buffers?
Thank's in advance for your answers
Emmanuel
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprÃŒft.
https://www.avast.com/antivirus
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Auer, Jens]

Hi,

I don’t think there was an explicit description or guarantee of alignment of the message buffer. The implementation guaranteed this because every payload was malloced for large messages and thus aligned. However, for me a void* indicates that it is binary blob of n bytes with unknown alignment.

Cheers,
Jens
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
***@cgi.com<mailto:***@cgi.com>
Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf Of KIU Shueng Chuan
Sent: 15 November 2016 03:57
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Zmq 4.2.0 aligned memory


A common use case for me is sending an array of floats.

First message part is some small metadata. Second message part is the float array.

On reception, zmq_msg_data is cast to float* and accessed directly.

This non-alignment would be problematic.
Or perhaps there never was any alignment guarantee?

On 15 Nov 2016 3:34 a.m., "Jens Auer" <***@betaversion.net<mailto:***@betaversion.net>> wrote:
Hi,

I think I have an idea why you are seeing unaligned messages, but this only applies to messages where the payload is not stored in the msg_t object itself. I think the threshold for this is 64 bytes. In ZeroMQ 4.1, receiving messages was done by first receiving from the socket into a static 8kb buffer, and then a new message object was created that allocated memory externally by calling malloc. The payload was then copied from the receive buffer to the message buffer. The malloced message buffer was aligned probably.

In ZeroMQ 4.2, this is changed to reduce the number of malloc calls and copy operations. The receive buffer is now dynamically allocated as a 8kb block, and messages are constructed as zero-copy messages using the part of the receive buffer containing the payload. This saves malloc calls and copy operations and increases performance. However, the payload may now start at basically arbitrary addresses. As an example, let's assume that we receive a small message of 10 bytes and a large message of 1kb, both received in a single call to recv on the socket. The engine allocates a new buffer of 8kb, calls recv(socket, buffer) and the data is written to the buffer. A small message is then created which contains the data from byte 2-11 in the msg_t, byte 1 contains the header. At byte 12 starts the header of the next message, and at byte 22(?) starts the payload. The large message is created as a zero-copy message using the pointer to byte 22 as storage. This is not aligned to a 4-byte address.

Could you provide some more information about the sizes of the messages that you receive? How do you decode the buffer content?

Best wishes,
Jens

-----UrsprÃŒngliche Nachricht-----
Von: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org<mailto:zeromq-dev-***@lists.zeromq.org>] Im Auftrag von Emmanuel Taurel
Gesendet: Montag, 14. November 2016 16:49
An: zeromq-***@lists.zeromq.org<mailto:zeromq-***@lists.zeromq.org>
Betreff: [zeromq-dev] Zmq 4.2.0 aligned memory

Hello all,

We are using zeromq since years now without troubles. We have recently tried our software using Zmq 4.2.0 (on linux hosts).
For our application, we are using multipart messages with 4 parts in publish/subscribe mode.
With Zmq 4.0.5, on the subscriber side, when we get the last message part, the received buffer was memory aligned (at least on 0x4 border). Unfortunately, with Zmq 4.2.0, the buffer is not aligned any more.
For instance with Zmq 4.0.5, the buffer was at address xxx08 while with Zmq 4.2.0, it is at address xxx23.

I don't know if it is relevant but our messages are relatively small messages (few tens of bytes) This is a problem for us in decoding the buffer content.

Is there something to be done to have memory aligned buffers?

Thank's in advance for your answers

Emmanuel
_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org<mailto:zeromq-***@lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprÃŒft.
https://www.avast.com/antivirus

_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org<mailto:zeromq-***@lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Luca Boccassi]

Hi,

I agree, I am not aware of any promise in the documentation about
alignment of receive buffers (happy to be disproved of course). Proof of
this is the fact that there was no explicit alignment set, and it was
left to chance.

IMHO any application that relied on this internal undocumented side
effect should be fixed. I'm sorry if this causes troubles for the
involved developers/maintainers, but we had no way to know.

Post by Auer, Jens
Hi,
I don’t think there was an explicit description or guarantee of alignment of the message buffer. The implementation guaranteed this because every payload was malloced for large messages and thus aligned. However, for me a void* indicates that it is binary blob of n bytes with unknown alignment.
Cheers,
Jens
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.
Sent: 15 November 2016 03:57
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Zmq 4.2.0 aligned memory
A common use case for me is sending an array of floats.
First message part is some small metadata. Second message part is the float array.
On reception, zmq_msg_data is cast to float* and accessed directly.
This non-alignment would be problematic.
Or perhaps there never was any alignment guarantee?
Hi,
I think I have an idea why you are seeing unaligned messages, but this only applies to messages where the payload is not stored in the msg_t object itself. I think the threshold for this is 64 bytes. In ZeroMQ 4.1, receiving messages was done by first receiving from the socket into a static 8kb buffer, and then a new message object was created that allocated memory externally by calling malloc. The payload was then copied from the receive buffer to the message buffer. The malloced message buffer was aligned probably.
In ZeroMQ 4.2, this is changed to reduce the number of malloc calls and copy operations. The receive buffer is now dynamically allocated as a 8kb block, and messages are constructed as zero-copy messages using the part of the receive buffer containing the payload. This saves malloc calls and copy operations and increases performance. However, the payload may now start at basically arbitrary addresses. As an example, let's assume that we receive a small message of 10 bytes and a large message of 1kb, both received in a single call to recv on the socket. The engine allocates a new buffer of 8kb, calls recv(socket, buffer) and the data is written to the buffer. A small message is then created which contains the data from byte 2-11 in the msg_t, byte 1 contains the header. At byte 12 starts the header of the next message, and at byte 22(?) starts the payload. The large message is created as a zero-copy message using the pointer to byte 22 as storage. This is not aligned to a 4-byte address.
Could you provide some more information about the sizes of the messages that you receive? How do you decode the buffer content?
Best wishes,
Jens
-----UrsprÃŒngliche Nachricht-----
Gesendet: Montag, 14. November 2016 16:49
Betreff: [zeromq-dev] Zmq 4.2.0 aligned memory
Hello all,
We are using zeromq since years now without troubles. We have recently tried our software using Zmq 4.2.0 (on linux hosts).
For our application, we are using multipart messages with 4 parts in publish/subscribe mode.
With Zmq 4.0.5, on the subscriber side, when we get the last message part, the received buffer was memory aligned (at least on 0x4 border). Unfortunately, with Zmq 4.2.0, the buffer is not aligned any more.
For instance with Zmq 4.0.5, the buffer was at address xxx08 while with Zmq 4.2.0, it is at address xxx23.
I don't know if it is relevant but our messages are relatively small messages (few tens of bytes) This is a problem for us in decoding the buffer content.
Is there something to be done to have memory aligned buffers?
Thank's in advance for your answers
Emmanuel
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprÃŒft.
https://www.avast.com/antivirus
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Emmanuel Taurel]

Hello gurus,

Thank's very much for your answers.
Here are some more info related to the case I am debugging.
As I said, we are using a multipart message with 4 parts.
On the subscriber side, I have:

Message part 1: Size = 41 bytes - Data ptr = xxxa986 -- zmq_msg_t
instance adr = xxx9c90
Message part 2: Size = 1 bytes - Data ptr = xxx9cd8 -- zmq_msg_t
instance adr = xxx9cd0
Message part 3: Size = 15 bytes - Data ptr = xxx9d18 -- zmq_msg_t
instance adr = xxx9d10
Message part 4: Size = 44 bytes - Data ptr = xxxa9e3 --zmq_msg_t
instance adr == xxx9d50

From my understanding of what happens, it seems that part 2 and 3 are
"vsm" and we can see that their buffer is part of the zmq_msg_t instances.
But parts 1 and 4 (size 41 or 44 bytes) are not "vsm" and their buffers
are in other memory chunk that their zmq_msg_t instances.
But those two buffers are not aligned. This generates troubles in our
buffer decoding system!

From Jens answer, I understand that this is related to changes done for
release 4.2
Is there a way to retrieve aligned buffers?

Sincerely

Emmanuel

Post by Jens Auer
Hi,
I think I have an idea why you are seeing unaligned messages, but this only applies to messages where the payload is not stored in the msg_t object itself. I think the threshold for this is 64 bytes. In ZeroMQ 4.1, receiving messages was done by first receiving from the socket into a static 8kb buffer, and then a new message object was created that allocated memory externally by calling malloc. The payload was then copied from the receive buffer to the message buffer. The malloced message buffer was aligned probably.
In ZeroMQ 4.2, this is changed to reduce the number of malloc calls and copy operations. The receive buffer is now dynamically allocated as a 8kb block, and messages are constructed as zero-copy messages using the part of the receive buffer containing the payload. This saves malloc calls and copy operations and increases performance. However, the payload may now start at basically arbitrary addresses. As an example, let's assume that we receive a small message of 10 bytes and a large message of 1kb, both received in a single call to recv on the socket. The engine allocates a new buffer of 8kb, calls recv(socket, buffer) and the data is written to the buffer. A small message is then created which contains the data from byte 2-11 in the msg_t, byte 1 contains the header. At byte 12 starts the header of the next message, and at byte 22(?) starts the payload. The large message is created as a zero-copy message using the pointer to byte 22 as storage. This is not aligned to a 4-byte address.
Could you provide some more information about the sizes of the messages that you receive? How do you decode the buffer content?
Best wishes,
Jens
-----Ursprüngliche Nachricht-----
Gesendet: Montag, 14. November 2016 16:49
Betreff: [zeromq-dev] Zmq 4.2.0 aligned memory
Hello all,
We are using zeromq since years now without troubles. We have recently tried our software using Zmq 4.2.0 (on linux hosts).
For our application, we are using multipart messages with 4 parts in publish/subscribe mode.
With Zmq 4.0.5, on the subscriber side, when we get the last message part, the received buffer was memory aligned (at least on 0x4 border). Unfortunately, with Zmq 4.2.0, the buffer is not aligned any more.
For instance with Zmq 4.0.5, the buffer was at address xxx08 while with Zmq 4.2.0, it is at address xxx23.
I don't know if it is relevant but our messages are relatively small messages (few tens of bytes) This is a problem for us in decoding the buffer content.
Is there something to be done to have memory aligned buffers?
Thank's in advance for your answers
Emmanuel
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

[Auer, Jens]

Hi Emmanuel,

The new mechanism cannot provide a way to ensure any alignment of the payload buffer for received messages. At least I don't see a way how this would work because it works directly on the data buffer used to receive the messages. Since you don't know which and how many messages will be received you cannot force alignment of the payload part of the messages.

I don't think that ZeroMQ guaranteed an alignment of the message buffers. It returns a void* which is guaranteed to be byte-aligned but not anything else. Personally, I think that you should serialize/deserialize objects into messages, and this implies copying something into or out of the buffer. I can see that you want to reduce copy overhead, but since the message is a zero-copy message now, the copy operation in the user-application is the only copy you have to do.

Cheers,
Jens
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
***@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

-----Original Message-----
Emmanuel Taurel
Sent: 15 November 2016 09:11
To: 'ZeroMQ development list'
Subject: Re: [zeromq-dev] Zmq 4.2.0 aligned memory
Hello gurus,
Thank's very much for your answers.
Here are some more info related to the case I am debugging.
As I said, we are using a multipart message with 4 parts.
Message part 1: Size = 41 bytes - Data ptr = xxxa986 -- zmq_msg_t instance adr
= xxx9c90 Message part 2: Size = 1 bytes - Data ptr = xxx9cd8 -- zmq_msg_t
instance adr = xxx9cd0 Message part 3: Size = 15 bytes - Data ptr = xxx9d18 --
zmq_msg_t instance adr = xxx9d10 Message part 4: Size = 44 bytes - Data ptr =
xxxa9e3 --zmq_msg_t instance adr == xxx9d50
From my understanding of what happens, it seems that part 2 and 3 are "vsm" and
we can see that their buffer is part of the zmq_msg_t instances.
But parts 1 and 4 (size 41 or 44 bytes) are not "vsm" and their buffers are in other
memory chunk that their zmq_msg_t instances.
But those two buffers are not aligned. This generates troubles in our buffer decoding
system!
From Jens answer, I understand that this is related to changes done for release 4.2
Is there a way to retrieve aligned buffers?
Sincerely
Emmanuel

Post by Jens Auer
Hi,
I think I have an idea why you are seeing unaligned messages, but this only

applies to messages where the payload is not stored in the msg_t object itself. I
think the threshold for this is 64 bytes. In ZeroMQ 4.1, receiving messages was
done by first receiving from the socket into a static 8kb buffer, and then a new
message object was created that allocated memory externally by calling malloc.
The payload was then copied from the receive buffer to the message buffer. The
malloced message buffer was aligned probably.

Post by Jens Auer
In ZeroMQ 4.2, this is changed to reduce the number of malloc calls and copy

operations. The receive buffer is now dynamically allocated as a 8kb block, and
messages are constructed as zero-copy messages using the part of the receive
buffer containing the payload. This saves malloc calls and copy operations and
increases performance. However, the payload may now start at basically arbitrary
addresses. As an example, let's assume that we receive a small message of 10
bytes and a large message of 1kb, both received in a single call to recv on the
socket. The engine allocates a new buffer of 8kb, calls recv(socket, buffer) and the
data is written to the buffer. A small message is then created which contains the
data from byte 2-11 in the msg_t, byte 1 contains the header. At byte 12 starts the
header of the next message, and at byte 22(?) starts the payload. The large
message is created as a zero-copy message using the pointer to byte 22 as storage.
This is not aligned to a 4-byte address.

Post by Jens Auer
Could you provide some more information about the sizes of the messages that

you receive? How do you decode the buffer content?

Post by Jens Auer
Best wishes,
Jens
-----Ursprüngliche Nachricht-----
Auftrag von Emmanuel Taurel
Gesendet: Montag, 14. November 2016 16:49
Betreff: [zeromq-dev] Zmq 4.2.0 aligned memory
Hello all,
We are using zeromq since years now without troubles. We have recently tried our

software using Zmq 4.2.0 (on linux hosts).

Post by Jens Auer
For our application, we are using multipart messages with 4 parts in

publish/subscribe mode.

Post by Jens Auer
With Zmq 4.0.5, on the subscriber side, when we get the last message part, the

received buffer was memory aligned (at least on 0x4 border). Unfortunately, with
Zmq 4.2.0, the buffer is not aligned any more.

Post by Jens Auer
For instance with Zmq 4.0.5, the buffer was at address xxx08 while with Zmq

4.2.0, it is at address xxx23.

Post by Jens Auer
I don't know if it is relevant but our messages are relatively small messages (few

tens of bytes) This is a problem for us in decoding the buffer content.

Post by Jens Auer
Is there something to be done to have memory aligned buffers?
Thank's in advance for your answers
Emmanuel
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Peter Krey]

Assuming you are programming in C or C++, make sure to insert a line of
#pragma pack(1)

Which will insure the compiler does not add any padding.

Check to insure endianness of both the sender and receiver are the same.

Then you will be ready to send a blob of arbitrary bytes.

Post by Auer, Jens
Hi Emmanuel,
The new mechanism cannot provide a way to ensure any alignment of the
payload buffer for received messages. At least I don't see a way how this
would work because it works directly on the data buffer used to receive the
messages. Since you don't know which and how many messages will be received
you cannot force alignment of the payload part of the messages.
I don't think that ZeroMQ guaranteed an alignment of the message buffers.
It returns a void* which is guaranteed to be byte-aligned but not anything
else. Personally, I think that you should serialize/deserialize objects
into messages, and this implies copying something into or out of the
buffer. I can see that you want to reduce copy overhead, but since the
message is a zero-copy message now, the copy operation in the
user-application is the only copy you have to do.
Cheers,
Jens
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
unter de.cgi.com/pflichtangaben.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
CGI Group Inc. and its affiliates may be contained in this message. If you
are not a recipient indicated or intended in this message (or responsible
for delivery of this message to such person), or you think for any reason
that this message may have been addressed to you in error, you may not use
or copy or deliver this message to anyone else. In such case, you should
destroy this message and are asked to notify the sender by reply e-mail.

-----Original Message-----

Of

Emmanuel Taurel
Sent: 15 November 2016 09:11
To: 'ZeroMQ development list'
Subject: Re: [zeromq-dev] Zmq 4.2.0 aligned memory
Hello gurus,
Thank's very much for your answers.
Here are some more info related to the case I am debugging.
As I said, we are using a multipart message with 4 parts.
Message part 1: Size = 41 bytes - Data ptr = xxxa986 -- zmq_msg_t

instance adr

= xxx9c90 Message part 2: Size = 1 bytes - Data ptr = xxx9cd8 --

zmq_msg_t

instance adr = xxx9cd0 Message part 3: Size = 15 bytes - Data ptr =

xxx9d18 --

zmq_msg_t instance adr = xxx9d10 Message part 4: Size = 44 bytes - Data

ptr =

xxxa9e3 --zmq_msg_t instance adr == xxx9d50
From my understanding of what happens, it seems that part 2 and 3 are

"vsm" and

we can see that their buffer is part of the zmq_msg_t instances.
But parts 1 and 4 (size 41 or 44 bytes) are not "vsm" and their buffers

are in other

memory chunk that their zmq_msg_t instances.
But those two buffers are not aligned. This generates troubles in our

buffer decoding

system!
From Jens answer, I understand that this is related to changes done for

release 4.2

Is there a way to retrieve aligned buffers?
Sincerely
Emmanuel

Post by Jens Auer
Hi,
I think I have an idea why you are seeing unaligned messages, but this

only

applies to messages where the payload is not stored in the msg_t object

itself. I

think the threshold for this is 64 bytes. In ZeroMQ 4.1, receiving

messages was

done by first receiving from the socket into a static 8kb buffer, and

then a new

message object was created that allocated memory externally by calling

malloc.

The payload was then copied from the receive buffer to the message

buffer. The

malloced message buffer was aligned probably.

Post by Jens Auer
In ZeroMQ 4.2, this is changed to reduce the number of malloc calls

and copy

operations. The receive buffer is now dynamically allocated as a 8kb

block, and

messages are constructed as zero-copy messages using the part of the

receive

buffer containing the payload. This saves malloc calls and copy

operations and

increases performance. However, the payload may now start at basically

arbitrary

addresses. As an example, let's assume that we receive a small message

of 10

bytes and a large message of 1kb, both received in a single call to recv

on the

socket. The engine allocates a new buffer of 8kb, calls recv(socket,

buffer) and the

data is written to the buffer. A small message is then created which

contains the

data from byte 2-11 in the msg_t, byte 1 contains the header. At byte 12

starts the

header of the next message, and at byte 22(?) starts the payload. The

large

message is created as a zero-copy message using the pointer to byte 22

as storage.

This is not aligned to a 4-byte address.

Post by Jens Auer
Could you provide some more information about the sizes of the

messages that

you receive? How do you decode the buffer content?

Post by Jens Auer
Best wishes,
Jens
-----UrsprÃŒngliche Nachricht-----
Auftrag von Emmanuel Taurel
Gesendet: Montag, 14. November 2016 16:49
Betreff: [zeromq-dev] Zmq 4.2.0 aligned memory
Hello all,
We are using zeromq since years now without troubles. We have recently

tried our

software using Zmq 4.2.0 (on linux hosts).

Post by Jens Auer
For our application, we are using multipart messages with 4 parts in

publish/subscribe mode.

Post by Jens Auer
With Zmq 4.0.5, on the subscriber side, when we get the last message

part, the

received buffer was memory aligned (at least on 0x4 border).

Unfortunately, with

Zmq 4.2.0, the buffer is not aligned any more.

Post by Jens Auer
For instance with Zmq 4.0.5, the buffer was at address xxx08 while

with Zmq

4.2.0, it is at address xxx23.

Post by Jens Auer
I don't know if it is relevant but our messages are relatively small

messages (few

tens of bytes) This is a problem for us in decoding the buffer content.

Post by Jens Auer
Is there something to be done to have memory aligned buffers?
Thank's in advance for your answers
Emmanuel
_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprÃŒft.
https://www.avast.com/antivirus

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
http://lists.zeromq.org/mailman/listinfo/zeromq-dev