[zeromq-dev] On hooking memory allocations

[Petteri Salo]

Hello,

Let me first do a little introduction as I'm new to this list. I'm a
software engineer with 15+ years of experience working on games at a
company called Remedy Entertainment Ltd. We've done games for PC, and
various games consoles over the years. Most recently we did Quantum Break
for Xbox One.

I've now been tasked with evaluating ZeroMQ. One important feature of any
library we use in games is the ability to hook all memory allocations -
this is to allow the use of custom memory allocators and/or for tracking
when and where memory is allocated.

I've searched the libzmq source code and there is about 150 uses of new,
malloc, realloc , etc.

If we were to adopt libzmq we'd like to put in allocation hooks and that
work would then be something that we'd like to contribute back to the
project. Having those hooks in the main repository would then make it
easier for us to adopt future changes to the library.

So, my question is would this kind of change be something that would be
accepted? Of course assuming that coding conventions, proper way of
submitting the patch etc. are followed. I do realize that one would want to
see the actual code before accepting. I'm interested in the principle of
accepting a change such as this, since it would introduce a new "rule" for
working ión libzmq source code : "All allocations shall go through an
allocation hook."

Best Regards,

Petteri Salo

[Auer, Jens]

Hi,

I am just a user, but I would love to see this change. I have thinking about this and I would like to be able to pass a C++ allocator object to ZeroMQ, so a simple function hook is not enough. My idea is to define a struct in the interface

struct allocator_t
{
void* hint;
void* (allocate)(size_t, void*);
void (deallocate)(void*, size_t, void*);
};

and store this in the context object. Since I don't think that this should be changed during runtime, I would create a new zmq_ctx_new overload which takes a parameter of type allocator_t. The default value would be to call malloc/free.

Cheers,
Jens

--
Jens Auer | CGI | Software-Engineer
CGI (Germany) GmbH & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
***@cgi.com<mailto:***@cgi.com>
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.
________________________________
Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von "Petteri Salo [***@gmail.com]
Gesendet: Montag, 28. November 2016 09:40
An: zeromq-***@lists.zeromq.org
Betreff: [zeromq-dev] On hooking memory allocations

Hello,

Let me first do a little introduction as I'm new to this list. I'm a software engineer with 15+ years of experience working on games at a company called Remedy Entertainment Ltd. We've done games for PC, and various games consoles over the years. Most recently we did Quantum Break for Xbox One.

I've now been tasked with evaluating ZeroMQ. One important feature of any library we use in games is the ability to hook all memory allocations - this is to allow the use of custom memory allocators and/or for tracking when and where memory is allocated.

I've searched the libzmq source code and there is about 150 uses of new, malloc, realloc , etc.

If we were to adopt libzmq we'd like to put in allocation hooks and that work would then be something that we'd like to contribute back to the project. Having those hooks in the main repository would then make it easier for us to adopt future changes to the library.

So, my question is would this kind of change be something that would be accepted? Of course assuming that coding conventions, proper way of submitting the patch etc. are followed. I do realize that one would want to see the actual code before accepting. I'm interested in the principle of accepting a change such as this, since it would introduce a new "rule" for working ión libzmq source code : "All allocations shall go through an allocation hook."

Best Regards,

Petteri Salo

[Luca Boccassi]

That would work for an internal API, but given we expose a C API
unfortunately I don't think that would work as a public API :-( And I
think for this use case they would require a public API.

As an external API, a new zmq_ctx_set that takes a callback would have
been ideal, but it only takes int. So perhaps a new
zmq_ctx_set_allocator that takes a callback pointer would be the next
best.

An alternative would be to have a system similar to what we use for the
poll implementation (epoll kqueue select etc), but this would be a
build-time option, and the implementation would have to be checked in,
which I don't think is an option for this case, right?

On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> Hi,
>
> I am just a user, but I would love to see this change. I have thinking about this and I would like to be able to pass a C++ allocator object to ZeroMQ, so a simple function hook is not enough. My idea is to define a struct in the interface
>
> struct allocator_t
> {
> void* hint;
> void* (allocate)(size_t, void*);
> void (deallocate)(void*, size_t, void*);
> };
>
> and store this in the context object. Since I don't think that this should be changed during runtime, I would create a new zmq_ctx_new overload which takes a parameter of type allocator_t. The default value would be to call malloc/free.
>
> Cheers,
> Jens
>
> --
> Jens Auer | CGI | Software-Engineer
> CGI (Germany) GmbH & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: [phone]
> ***@cgi.com<mailto:***@cgi.com>
> Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.
> ________________________________
> Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von "Petteri Salo [***@gmail.com]
> Gesendet: Montag, 28. November 2016 09:40
> An: zeromq-***@lists.zeromq.org
> Betreff: [zeromq-dev] On hooking memory allocations
>
> Hello,
>
> Let me first do a little introduction as I'm new to this list. I'm a software engineer with 15+ years of experience working on games at a company called Remedy Entertainment Ltd. We've done games for PC, and various games consoles over the years. Most recently we did Quantum Break for Xbox One.
>
> I've now been tasked with evaluating ZeroMQ. One important feature of any library we use in games is the ability to hook all memory allocations - this is to allow the use of custom memory allocators and/or for tracking when and where memory is allocated.
>
> I've searched the libzmq source code and there is about 150 uses of new, malloc, realloc , etc.
>
> If we were to adopt libzmq we'd like to put in allocation hooks and that work would then be something that we'd like to contribute back to the project. Having those hooks in the main repository would then make it easier for us to adopt future changes to the library.
>
> So, my question is would this kind of change be something that would be accepted? Of course assuming that coding conventions, proper way of submitting the patch etc. are followed. I do realize that one would want to see the actual code before accepting. I'm interested in the principle of accepting a change such as this, since it would introduce a new "rule" for working ión libzmq source code : "All allocations shall go through an allocation hook."
>
> Best Regards,
>
> Petteri Salo
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Petteri Salo]

Hi,

I haven't done enough analysis (yet) about current allocations in libzmq to
say which would work best for us, compile time or a runtime hooking. But
given that libzmq comes with source code, I would probably look for a
compile time option at first, like a macro. So roughly:

#ifdef USE_DEFAULT_ALLOCS
#define ZMQ_MALLOC ( ... ) malloc( ... )
#define ZMQ_FREE ( ... ) free( ... )
#else
#define ZMQ_MALLOC ( ... ) my_custom_malloc( ... )
#define ZMQ_FREE ( ... ) my_custom_free( ... )
#endif

and then use placement new in those places where the current code uses a
(plain) new.

-Petteri

On Mon, Nov 28, 2016 at 1:29 PM, Luca Boccassi <***@gmail.com>
wrote:

> That would work for an internal API, but given we expose a C API
> unfortunately I don't think that would work as a public API :-( And I
> think for this use case they would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal, but it only takes int. So perhaps a new
> zmq_ctx_set_allocator that takes a callback pointer would be the next
> best.
>
> An alternative would be to have a system similar to what we use for the
> poll implementation (epoll kqueue select etc), but this would be a
> build-time option, and the implementation would have to be checked in,
> which I don't think is an option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> about this and I would like to be able to pass a C++ allocator object to
> ZeroMQ, so a simple function hook is not enough. My idea is to define a
> struct in the interface
> >
> > struct allocator_t
> > {
> > void* hint;
> > void* (allocate)(size_t, void*);
> > void (deallocate)(void*, size_t, void*);
> > };
> >
> > and store this in the context object. Since I don't think that this
> should be changed during runtime, I would create a new zmq_ctx_new overload
> which takes a parameter of type allocator_t. The default value would be to
> call malloc/free.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: [phone]
> > ***@cgi.com<mailto:***@cgi.com>
> > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to CGI Group Inc. and its affiliates may be contained in this message. If
> you are not a recipient indicated or intended in this message (or
> responsible for delivery of this message to such person), or you think for
> any reason that this message may have been addressed to you in error, you
> may not use or copy or deliver this message to anyone else. In such case,
> you should destroy this message and are asked to notify the sender by reply
> e-mail.
> > ________________________________
> > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> "Petteri Salo [***@gmail.com]
> > Gesendet: Montag, 28. November 2016 09:40
> > An: zeromq-***@lists.zeromq.org
> > Betreff: [zeromq-dev] On hooking memory allocations
> >
> > Hello,
> >
> > Let me first do a little introduction as I'm new to this list. I'm a
> software engineer with 15+ years of experience working on games at a
> company called Remedy Entertainment Ltd. We've done games for PC, and
> various games consoles over the years. Most recently we did Quantum Break
> for Xbox One.
> >
> > I've now been tasked with evaluating ZeroMQ. One important feature of
> any library we use in games is the ability to hook all memory allocations -
> this is to allow the use of custom memory allocators and/or for tracking
> when and where memory is allocated.
> >
> > I've searched the libzmq source code and there is about 150 uses of new,
> malloc, realloc , etc.
> >
> > If we were to adopt libzmq we'd like to put in allocation hooks and that
> work would then be something that we'd like to contribute back to the
> project. Having those hooks in the main repository would then make it
> easier for us to adopt future changes to the library.
> >
> > So, my question is would this kind of change be something that would be
> accepted? Of course assuming that coding conventions, proper way of
> submitting the patch etc. are followed. I do realize that one would want to
> see the actual code before accepting. I'm interested in the principle of
> accepting a change such as this, since it would introduce a new "rule" for
> working ión libzmq source code : "All allocations shall go through an
> allocation hook."
> >
> > Best Regards,
> >
> > Petteri Salo
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-***@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

[Auer, Jens]

Hi,

I don't see a big problem with the C API except that C doesn't support overloads. So if the function has a new name, e.g. zmq_ctx_new_with_allocator, everything stays plain C. The default instance would be a

static void* malloc_(size_t n, void*) {return malloc(n);}
static void free_(void* ptr, size_t n, void*) {free(ptr);}

allocator_t alloc{
NULL,
malloc_,
free_
};

context_t then stores the member and gets methods to forward memory allocations to the function pointers, passing the hint pointer as an additional argument.

In my C++ code, I can then use an allocator
static void* allocate(size_t n, void* obj) {return static_cast<std::allocator<char>>(obj)->allocate(n); }
static void free_(void* ptr, size_t n, void*obj) { static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }

std::allocator<char> a;
allocator_t zmqAlloc{
&a,
allocate,
free_
};

void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);

I think this should work?

Best wishes,
Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
***@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.


> -----Original Message-----
> From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf Of
> Luca Boccassi
> Sent: 28 November 2016 12:30
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] On hooking memory allocations
>
> That would work for an internal API, but given we expose a C API unfortunately I
> don't think that would work as a public API :-( And I think for this use case they
> would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have been ideal,
> but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes a callback
> pointer would be the next best.
>
> An alternative would be to have a system similar to what we use for the poll
> implementation (epoll kqueue select etc), but this would be a build-time option,
> and the implementation would have to be checked in, which I don't think is an
> option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> > about this and I would like to be able to pass a C++ allocator object
> > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > define a struct in the interface
> >
> > struct allocator_t
> > {
> > void* hint;
> > void* (allocate)(size_t, void*);
> > void (deallocate)(void*, size_t, void*); };
> >
> > and store this in the context object. Since I don't think that this should be
> changed during runtime, I would create a new zmq_ctx_new overload which takes a
> parameter of type allocator_t. The default value would be to call malloc/free.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: [phone]
> > ***@cgi.com<mailto:***@cgi.com>
> > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
> de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
> Group Inc. and its affiliates may be contained in this message. If you are not a
> recipient indicated or intended in this message (or responsible for delivery of this
> message to such person), or you think for any reason that this message may have
> been addressed to you in error, you may not use or copy or deliver this message to
> anyone else. In such case, you should destroy this message and are asked to notify
> the sender by reply e-mail.
> > ________________________________
> > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> > "Petteri Salo [***@gmail.com]
> > Gesendet: Montag, 28. November 2016 09:40
> > An: zeromq-***@lists.zeromq.org
> > Betreff: [zeromq-dev] On hooking memory allocations
> >
> > Hello,
> >
> > Let me first do a little introduction as I'm new to this list. I'm a software engineer
> with 15+ years of experience working on games at a company called Remedy
> Entertainment Ltd. We've done games for PC, and various games consoles over the
> years. Most recently we did Quantum Break for Xbox One.
> >
> > I've now been tasked with evaluating ZeroMQ. One important feature of any
> library we use in games is the ability to hook all memory allocations - this is to allow
> the use of custom memory allocators and/or for tracking when and where memory is
> allocated.
> >
> > I've searched the libzmq source code and there is about 150 uses of new, malloc,
> realloc , etc.
> >
> > If we were to adopt libzmq we'd like to put in allocation hooks and that work
> would then be something that we'd like to contribute back to the project. Having
> those hooks in the main repository would then make it easier for us to adopt future
> changes to the library.
> >
> > So, my question is would this kind of change be something that would be
> accepted? Of course assuming that coding conventions, proper way of submitting
> the patch etc. are followed. I do realize that one would want to see the actual code
> before accepting. I'm interested in the principle of accepting a change such as this,
> since it would introduce a new "rule" for working ión libzmq source code : "All
> allocations shall go through an allocation hook."
> >
> > Best Regards,
> >
> > Petteri Salo
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-***@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

[Max Kozlovsky]

Hi,

Would not globally overwriting malloc/free with the custom implementation
achieve the desired behavior (instead of providing hooks for installing
malloc overrides in each and every library)?

Max

On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:

> Hi,
>
> I don't see a big problem with the C API except that C doesn't support
> overloads. So if the function has a new name, e.g.
> zmq_ctx_new_with_allocator, everything stays plain C. The default instance
> would be a
>
> static void* malloc_(size_t n, void*) {return malloc(n);}
> static void free_(void* ptr, size_t n, void*) {free(ptr);}
>
> allocator_t alloc{
> NULL,
> malloc_,
> free_
> };
>
> context_t then stores the member and gets methods to forward memory
> allocations to the function pointers, passing the hint pointer as an
> additional argument.
>
> In my C++ code, I can then use an allocator
> static void* allocate(size_t n, void* obj) {return
> static_cast<std::allocator<char>>(obj)->allocate(n); }
> static void free_(void* ptr, size_t n, void*obj) {
> static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
>
> std::allocator<char> a;
> allocator_t zmqAlloc{
> &a,
> allocate,
> free_
> };
>
> void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
>
> I think this should work?
>
> Best wishes,
> Jens
>
> --
> Dr. Jens Auer | CGI | Software Engineer
> CGI Deutschland Ltd. & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: [phone]
> ***@cgi.com
> Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
>
> > -----Original Message-----
> > From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf
> Of
> > Luca Boccassi
> > Sent: 28 November 2016 12:30
> > To: ZeroMQ development list
> > Subject: Re: [zeromq-dev] On hooking memory allocations
> >
> > That would work for an internal API, but given we expose a C API
> unfortunately I
> > don't think that would work as a public API :-( And I think for this use
> case they
> > would require a public API.
> >
> > As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal,
> > but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes
> a callback
> > pointer would be the next best.
> >
> > An alternative would be to have a system similar to what we use for the
> poll
> > implementation (epoll kqueue select etc), but this would be a build-time
> option,
> > and the implementation would have to be checked in, which I don't think
> is an
> > option for this case, right?
> >
> > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > Hi,
> > >
> > > I am just a user, but I would love to see this change. I have thinking
> > > about this and I would like to be able to pass a C++ allocator object
> > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > define a struct in the interface
> > >
> > > struct allocator_t
> > > {
> > > void* hint;
> > > void* (allocate)(size_t, void*);
> > > void (deallocate)(void*, size_t, void*); };
> > >
> > > and store this in the context object. Since I don't think that this
> should be
> > changed during runtime, I would create a new zmq_ctx_new overload which
> takes a
> > parameter of type allocator_t. The default value would be to call
> malloc/free.
> > >
> > > Cheers,
> > > Jens
> > >
> > > --
> > > Jens Auer | CGI | Software-Engineer
> > > CGI (Germany) GmbH & Co. KG
> > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > T: [phone]
> > > ***@cgi.com<mailto:***@cgi.com>
> > > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter
> > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to CGI
> > Group Inc. and its affiliates may be contained in this message. If you
> are not a
> > recipient indicated or intended in this message (or responsible for
> delivery of this
> > message to such person), or you think for any reason that this message
> may have
> > been addressed to you in error, you may not use or copy or deliver this
> message to
> > anyone else. In such case, you should destroy this message and are asked
> to notify
> > the sender by reply e-mail.
> > > ________________________________
> > > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> > > "Petteri Salo [***@gmail.com]
> > > Gesendet: Montag, 28. November 2016 09:40
> > > An: zeromq-***@lists.zeromq.org
> > > Betreff: [zeromq-dev] On hooking memory allocations
> > >
> > > Hello,
> > >
> > > Let me first do a little introduction as I'm new to this list. I'm a
> software engineer
> > with 15+ years of experience working on games at a company called Remedy
> > Entertainment Ltd. We've done games for PC, and various games consoles
> over the
> > years. Most recently we did Quantum Break for Xbox One.
> > >
> > > I've now been tasked with evaluating ZeroMQ. One important feature of
> any
> > library we use in games is the ability to hook all memory allocations -
> this is to allow
> > the use of custom memory allocators and/or for tracking when and where
> memory is
> > allocated.
> > >
> > > I've searched the libzmq source code and there is about 150 uses of
> new, malloc,
> > realloc , etc.
> > >
> > > If we were to adopt libzmq we'd like to put in allocation hooks and
> that work
> > would then be something that we'd like to contribute back to the
> project. Having
> > those hooks in the main repository would then make it easier for us to
> adopt future
> > changes to the library.
> > >
> > > So, my question is would this kind of change be something that would be
> > accepted? Of course assuming that coding conventions, proper way of
> submitting
> > the patch etc. are followed. I do realize that one would want to see the
> actual code
> > before accepting. I'm interested in the principle of accepting a change
> such as this,
> > since it would introduce a new "rule" for working ión libzmq source code
> : "All
> > allocations shall go through an allocation hook."
> > >
> > > Best Regards,
> > >
> > > Petteri Salo
> > >
> > >
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-***@lists.zeromq.org
> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

[Jens Auer]

Hi,



yes and no. If you overwrite it globally at compute-time every program on the system has to use your custom implementation. So if you deliver your ZeroMQ library with your program it will work, but what if my program wants a different custom allocator?



Cheers,

Jens



Von: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] Im Auftrag von Max Kozlovsky
Gesendet: Montag, 28. November 2016 20:36
An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations



Hi,



Would not globally overwriting malloc/free with the custom implementation achieve the desired behavior (instead of providing hooks for installing malloc overrides in each and every library)?



Max



On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:

Hi,

I don't see a big problem with the C API except that C doesn't support overloads. So if the function has a new name, e.g. zmq_ctx_new_with_allocator, everything stays plain C. The default instance would be a

static void* malloc_(size_t n, void*) {return malloc(n);}
static void free_(void* ptr, size_t n, void*) {free(ptr);}

allocator_t alloc{
NULL,
malloc_,
free_
};

context_t then stores the member and gets methods to forward memory allocations to the function pointers, passing the hint pointer as an additional argument.

In my C++ code, I can then use an allocator
static void* allocate(size_t n, void* obj) {return static_cast<std::allocator<char>>(obj)->allocate(n); }
static void free_(void* ptr, size_t n, void*obj) { static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }

std::allocator<char> a;
allocator_t zmqAlloc{
&a,
allocate,
free_
};

void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);

I think this should work?

Best wishes,
Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone] <tel:%2B49%206151%2036860%20154>
***@cgi.com
Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.



> -----Original Message-----
> From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf Of
> Luca Boccassi
> Sent: 28 November 2016 12:30
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] On hooking memory allocations
>
> That would work for an internal API, but given we expose a C API unfortunately I
> don't think that would work as a public API :-( And I think for this use case they
> would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have been ideal,
> but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes a callback
> pointer would be the next best.
>
> An alternative would be to have a system similar to what we use for the poll
> implementation (epoll kqueue select etc), but this would be a build-time option,
> and the implementation would have to be checked in, which I don't think is an
> option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> > about this and I would like to be able to pass a C++ allocator object
> > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > define a struct in the interface
> >
> > struct allocator_t
> > {
> > void* hint;
> > void* (allocate)(size_t, void*);
> > void (deallocate)(void*, size_t, void*); };
> >
> > and store this in the context object. Since I don't think that this should be
> changed during runtime, I would create a new zmq_ctx_new overload which takes a
> parameter of type allocator_t. The default value would be to call malloc/free.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: [phone]
> > ***@cgi.com<mailto:***@cgi.com>
> > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
> de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
> Group Inc. and its affiliates may be contained in this message. If you are not a
> recipient indicated or intended in this message (or responsible for delivery of this
> message to such person), or you think for any reason that this message may have
> been addressed to you in error, you may not use or copy or deliver this message to
> anyone else. In such case, you should destroy this message and are asked to notify
> the sender by reply e-mail.
> > ________________________________
> > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> > "Petteri Salo [***@gmail.com]
> > Gesendet: Montag, 28. November 2016 09:40
> > An: zeromq-***@lists.zeromq.org
> > Betreff: [zeromq-dev] On hooking memory allocations
> >
> > Hello,
> >
> > Let me first do a little introduction as I'm new to this list. I'm a software engineer
> with 15+ years of experience working on games at a company called Remedy
> Entertainment Ltd. We've done games for PC, and various games consoles over the
> years. Most recently we did Quantum Break for Xbox One.
> >
> > I've now been tasked with evaluating ZeroMQ. One important feature of any
> library we use in games is the ability to hook all memory allocations - this is to allow
> the use of custom memory allocators and/or for tracking when and where memory is
> allocated.
> >
> > I've searched the libzmq source code and there is about 150 uses of new, malloc,
> realloc , etc.
> >
> > If we were to adopt libzmq we'd like to put in allocation hooks and that work
> would then be something that we'd like to contribute back to the project. Having
> those hooks in the main repository would then make it easier for us to adopt future
> changes to the library.
> >
> > So, my question is would this kind of change be something that would be
> accepted? Of course assuming that coding conventions, proper way of submitting
> the patch etc. are followed. I do realize that one would want to see the actual code
> before accepting. I'm interested in the principle of accepting a change such as this,
> since it would introduce a new "rule" for working ión libzmq source code : "All
> allocations shall go through an allocation hook."
> >
> > Best Regards,
> >
> > Petteri Salo
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-***@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev



_____

E-Mail ist virenfrei.
Von AVG ÃŒberprÃŒft - www.avg.com
Version: 2016.0.7924 / Virendatenbank: 4728/13496 - Ausgabedatum: 28.11.2016

[Max Kozlovsky]

Each program can be linked with a separate malloc implementation if the
user so desires. Libraries don't need to be aware which implementation it
is. Different malloc implementation can be even substituted at run time on
platforms with dynamic linking support (LD_PRELOAD etc).

On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <***@betaversion.net>
wrote:

> Hi,
>
>
>
> yes and no. If you overwrite it globally at compute-time every program on
> the system has to use your custom implementation. So if you deliver your
> ZeroMQ library with your program it will work, but what if my program wants
> a different custom allocator?
>
>
>
> Cheers,
>
> Jens
>
>
>
> *Von:* zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] *Im
> Auftrag von *Max Kozlovsky
> *Gesendet:* Montag, 28. November 2016 20:36
> *An:* ZeroMQ development list
> *Betreff:* Re: [zeromq-dev] On hooking memory allocations
>
>
>
> Hi,
>
>
>
> Would not globally overwriting malloc/free with the custom implementation
> achieve the desired behavior (instead of providing hooks for installing
> malloc overrides in each and every library)?
>
>
>
> Max
>
>
>
> On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:
>
> Hi,
>
> I don't see a big problem with the C API except that C doesn't support
> overloads. So if the function has a new name, e.g.
> zmq_ctx_new_with_allocator, everything stays plain C. The default instance
> would be a
>
> static void* malloc_(size_t n, void*) {return malloc(n);}
> static void free_(void* ptr, size_t n, void*) {free(ptr);}
>
> allocator_t alloc{
> NULL,
> malloc_,
> free_
> };
>
> context_t then stores the member and gets methods to forward memory
> allocations to the function pointers, passing the hint pointer as an
> additional argument.
>
> In my C++ code, I can then use an allocator
> static void* allocate(size_t n, void* obj) {return
> static_cast<std::allocator<char>>(obj)->allocate(n); }
> static void free_(void* ptr, size_t n, void*obj) {
> static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
>
> std::allocator<char> a;
> allocator_t zmqAlloc{
> &a,
> allocate,
> free_
> };
>
> void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
>
> I think this should work?
>
> Best wishes,
> Jens
>
> --
> Dr. Jens Auer | CGI | Software Engineer
> CGI Deutschland Ltd. & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: [phone]
> ***@cgi.com
> Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
> > -----Original Message-----
> > From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf
> Of
> > Luca Boccassi
> > Sent: 28 November 2016 12:30
> > To: ZeroMQ development list
> > Subject: Re: [zeromq-dev] On hooking memory allocations
> >
> > That would work for an internal API, but given we expose a C API
> unfortunately I
> > don't think that would work as a public API :-( And I think for this use
> case they
> > would require a public API.
> >
> > As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal,
> > but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes
> a callback
> > pointer would be the next best.
> >
> > An alternative would be to have a system similar to what we use for the
> poll
> > implementation (epoll kqueue select etc), but this would be a build-time
> option,
> > and the implementation would have to be checked in, which I don't think
> is an
> > option for this case, right?
> >
> > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > Hi,
> > >
> > > I am just a user, but I would love to see this change. I have thinking
> > > about this and I would like to be able to pass a C++ allocator object
> > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > define a struct in the interface
> > >
> > > struct allocator_t
> > > {
> > > void* hint;
> > > void* (allocate)(size_t, void*);
> > > void (deallocate)(void*, size_t, void*); };
> > >
> > > and store this in the context object. Since I don't think that this
> should be
> > changed during runtime, I would create a new zmq_ctx_new overload which
> takes a
> > parameter of type allocator_t. The default value would be to call
> malloc/free.
> > >
> > > Cheers,
> > > Jens
> > >
> > > --
> > > Jens Auer | CGI | Software-Engineer
> > > CGI (Germany) GmbH & Co. KG
> > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > T: [phone]
> > > ***@cgi.com<mailto:***@cgi.com>
> > > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter
> > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to CGI
> > Group Inc. and its affiliates may be contained in this message. If you
> are not a
> > recipient indicated or intended in this message (or responsible for
> delivery of this
> > message to such person), or you think for any reason that this message
> may have
> > been addressed to you in error, you may not use or copy or deliver this
> message to
> > anyone else. In such case, you should destroy this message and are asked
> to notify
> > the sender by reply e-mail.
> > > ________________________________
> > > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> > > "Petteri Salo [***@gmail.com]
> > > Gesendet: Montag, 28. November 2016 09:40
> > > An: zeromq-***@lists.zeromq.org
> > > Betreff: [zeromq-dev] On hooking memory allocations
> > >
> > > Hello,
> > >
> > > Let me first do a little introduction as I'm new to this list. I'm a
> software engineer
> > with 15+ years of experience working on games at a company called Remedy
> > Entertainment Ltd. We've done games for PC, and various games consoles
> over the
> > years. Most recently we did Quantum Break for Xbox One.
> > >
> > > I've now been tasked with evaluating ZeroMQ. One important feature of
> any
> > library we use in games is the ability to hook all memory allocations -
> this is to allow
> > the use of custom memory allocators and/or for tracking when and where
> memory is
> > allocated.
> > >
> > > I've searched the libzmq source code and there is about 150 uses of
> new, malloc,
> > realloc , etc.
> > >
> > > If we were to adopt libzmq we'd like to put in allocation hooks and
> that work
> > would then be something that we'd like to contribute back to the
> project. Having
> > those hooks in the main repository would then make it easier for us to
> adopt future
> > changes to the library.
> > >
> > > So, my question is would this kind of change be something that would be
> > accepted? Of course assuming that coding conventions, proper way of
> submitting
> > the patch etc. are followed. I do realize that one would want to see the
> actual code
> > before accepting. I'm interested in the principle of accepting a change
> such as this,
> > since it would introduce a new "rule" for working ión libzmq source code
> : "All
> > allocations shall go through an allocation hook."
> > >
> > > Best Regards,
> > >
> > > Petteri Salo
> > >
> > >
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-***@lists.zeromq.org
> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> ------------------------------
>
> E-Mail ist virenfrei.
> Von AVG ÃŒberprÃŒft - www.avg.com
> Version: 2016.0.7924 / Virendatenbank: 4728/13496 - Ausgabedatum:
> 28.11.2016
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

[Jens Auer]

Yeah, I know that. I always find this more a hack than a real design. It is also not so nice to use with a C++-allocator object because I have to hook it with global functions. And being able to set this for ZeroMQ only e.g. gives me the freedom to use a memory pool there and standard allocators everywhere else. I welcome this flexibility.



Cheers,

Jens



Von: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] Im Auftrag von Max Kozlovsky
Gesendet: Montag, 28. November 2016 20:48
An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations



Each program can be linked with a separate malloc implementation if the user so desires. Libraries don't need to be aware which implementation it is. Different malloc implementation can be even substituted at run time on platforms with dynamic linking support (LD_PRELOAD etc).



On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <***@betaversion.net> wrote:

Hi,



yes and no. If you overwrite it globally at compute-time every program on the system has to use your custom implementation. So if you deliver your ZeroMQ library with your program it will work, but what if my program wants a different custom allocator?



Cheers,

Jens



Von: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] Im Auftrag von Max Kozlovsky
Gesendet: Montag, 28. November 2016 20:36
An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations



Hi,



Would not globally overwriting malloc/free with the custom implementation achieve the desired behavior (instead of providing hooks for installing malloc overrides in each and every library)?



Max



On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:

Hi,

I don't see a big problem with the C API except that C doesn't support overloads. So if the function has a new name, e.g. zmq_ctx_new_with_allocator, everything stays plain C. The default instance would be a

static void* malloc_(size_t n, void*) {return malloc(n);}
static void free_(void* ptr, size_t n, void*) {free(ptr);}

allocator_t alloc{
NULL,
malloc_,
free_
};

context_t then stores the member and gets methods to forward memory allocations to the function pointers, passing the hint pointer as an additional argument.

In my C++ code, I can then use an allocator
static void* allocate(size_t n, void* obj) {return static_cast<std::allocator<char>>(obj)->allocate(n); }
static void free_(void* ptr, size_t n, void*obj) { static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }

std::allocator<char> a;
allocator_t zmqAlloc{
&a,
allocate,
free_
};

void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);

I think this should work?

Best wishes,
Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone] <tel:%2B49%206151%2036860%20154>
***@cgi.com
Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

> -----Original Message-----
> From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf Of
> Luca Boccassi
> Sent: 28 November 2016 12:30
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] On hooking memory allocations
>
> That would work for an internal API, but given we expose a C API unfortunately I
> don't think that would work as a public API :-( And I think for this use case they
> would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have been ideal,
> but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes a callback
> pointer would be the next best.
>
> An alternative would be to have a system similar to what we use for the poll
> implementation (epoll kqueue select etc), but this would be a build-time option,
> and the implementation would have to be checked in, which I don't think is an
> option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> > about this and I would like to be able to pass a C++ allocator object
> > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > define a struct in the interface
> >
> > struct allocator_t
> > {
> > void* hint;
> > void* (allocate)(size_t, void*);
> > void (deallocate)(void*, size_t, void*); };
> >
> > and store this in the context object. Since I don't think that this should be
> changed during runtime, I would create a new zmq_ctx_new overload which takes a
> parameter of type allocator_t. The default value would be to call malloc/free.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: [phone] <tel:%2B49%206151%2036860%20154>
> > ***@cgi.com<mailto:***@cgi.com>
> > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
> de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
> Group Inc. and its affiliates may be contained in this message. If you are not a
> recipient indicated or intended in this message (or responsible for delivery of this
> message to such person), or you think for any reason that this message may have
> been addressed to you in error, you may not use or copy or deliver this message to
> anyone else. In such case, you should destroy this message and are asked to notify
> the sender by reply e-mail.
> > ________________________________
> > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> > "Petteri Salo [***@gmail.com]
> > Gesendet: Montag, 28. November 2016 09:40
> > An: zeromq-***@lists.zeromq.org
> > Betreff: [zeromq-dev] On hooking memory allocations
> >
> > Hello,
> >
> > Let me first do a little introduction as I'm new to this list. I'm a software engineer
> with 15+ years of experience working on games at a company called Remedy
> Entertainment Ltd. We've done games for PC, and various games consoles over the
> years. Most recently we did Quantum Break for Xbox One.
> >
> > I've now been tasked with evaluating ZeroMQ. One important feature of any
> library we use in games is the ability to hook all memory allocations - this is to allow
> the use of custom memory allocators and/or for tracking when and where memory is
> allocated.
> >
> > I've searched the libzmq source code and there is about 150 uses of new, malloc,
> realloc , etc.
> >
> > If we were to adopt libzmq we'd like to put in allocation hooks and that work
> would then be something that we'd like to contribute back to the project. Having
> those hooks in the main repository would then make it easier for us to adopt future
> changes to the library.
> >
> > So, my question is would this kind of change be something that would be
> accepted? Of course assuming that coding conventions, proper way of submitting
> the patch etc. are followed. I do realize that one would want to see the actual code
> before accepting. I'm interested in the principle of accepting a change such as this,
> since it would introduce a new "rule" for working ión libzmq source code : "All
> allocations shall go through an allocation hook."
> >
> > Best Regards,
> >
> > Petteri Salo
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-***@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev



_____

E-Mail ist virenfrei.
Von AVG ÃŒberprÃŒft - www.avg.com
Version: 2016.0.7924 / Virendatenbank: 4728/13496 - Ausgabedatum: 28.11.2016


_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev



_____

E-Mail ist virenfrei.
Von AVG ÃŒberprÃŒft - www.avg.com
Version: 2016.0.7924 / Virendatenbank: 4728/13496 - Ausgabedatum: 28.11.2016

[Max Kozlovsky]

Using a memory pool (as in fixed-block allocator) is a somewhat different
requirement from using custom malloc/free implementation. I am not sure if
the library can make use of memory pools. The logical place to use memory
pools is for incoming message buffer management. This is for all practical
purposes the only place where a lot of allocations are happening and faster
allocation might make a difference. However the library interface promises
that message points to a contiguous buffer of message size, which makes it
difficult to integrate with memory pools.


On Mon, Nov 28, 2016 at 2:34 PM, Jens Auer <***@betaversion.net>
wrote:

> Yeah, I know that. I always find this more a hack than a real design. It
> is also not so nice to use with a C++-allocator object because I have to
> hook it with global functions. And being able to set this for ZeroMQ only
> e.g. gives me the freedom to use a memory pool there and standard
> allocators everywhere else. I welcome this flexibility.
>
>
>
> Cheers,
>
> Jens
>
>
>
> *Von:* zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] *Im
> Auftrag von *Max Kozlovsky
> *Gesendet:* Montag, 28. November 2016 20:48
>
> *An:* ZeroMQ development list
> *Betreff:* Re: [zeromq-dev] On hooking memory allocations
>
>
>
> Each program can be linked with a separate malloc implementation if the
> user so desires. Libraries don't need to be aware which implementation it
> is. Different malloc implementation can be even substituted at run time on
> platforms with dynamic linking support (LD_PRELOAD etc).
>
>
>
> On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <***@betaversion.net>
> wrote:
>
> Hi,
>
>
>
> yes and no. If you overwrite it globally at compute-time every program on
> the system has to use your custom implementation. So if you deliver your
> ZeroMQ library with your program it will work, but what if my program wants
> a different custom allocator?
>
>
>
> Cheers,
>
> Jens
>
>
>
> *Von:* zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] *Im
> Auftrag von *Max Kozlovsky
> *Gesendet:* Montag, 28. November 2016 20:36
> *An:* ZeroMQ development list
> *Betreff:* Re: [zeromq-dev] On hooking memory allocations
>
>
>
> Hi,
>
>
>
> Would not globally overwriting malloc/free with the custom implementation
> achieve the desired behavior (instead of providing hooks for installing
> malloc overrides in each and every library)?
>
>
>
> Max
>
>
>
> On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:
>
> Hi,
>
> I don't see a big problem with the C API except that C doesn't support
> overloads. So if the function has a new name, e.g.
> zmq_ctx_new_with_allocator, everything stays plain C. The default instance
> would be a
>
> static void* malloc_(size_t n, void*) {return malloc(n);}
> static void free_(void* ptr, size_t n, void*) {free(ptr);}
>
> allocator_t alloc{
> NULL,
> malloc_,
> free_
> };
>
> context_t then stores the member and gets methods to forward memory
> allocations to the function pointers, passing the hint pointer as an
> additional argument.
>
> In my C++ code, I can then use an allocator
> static void* allocate(size_t n, void* obj) {return
> static_cast<std::allocator<char>>(obj)->allocate(n); }
> static void free_(void* ptr, size_t n, void*obj) {
> static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
>
> std::allocator<char> a;
> allocator_t zmqAlloc{
> &a,
> allocate,
> free_
> };
>
> void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
>
> I think this should work?
>
> Best wishes,
> Jens
>
> --
> Dr. Jens Auer | CGI | Software Engineer
> CGI Deutschland Ltd. & Co. KG
> Rheinstraße 95 | 64295 Darmstadt | Germany
> T: [phone]
> ***@cgi.com
> Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter de.cgi.com/pflichtangaben.
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
> > -----Original Message-----
> > From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf
> Of
> > Luca Boccassi
> > Sent: 28 November 2016 12:30
> > To: ZeroMQ development list
> > Subject: Re: [zeromq-dev] On hooking memory allocations
> >
> > That would work for an internal API, but given we expose a C API
> unfortunately I
> > don't think that would work as a public API :-( And I think for this use
> case they
> > would require a public API.
> >
> > As an external API, a new zmq_ctx_set that takes a callback would have
> been ideal,
> > but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes
> a callback
> > pointer would be the next best.
> >
> > An alternative would be to have a system similar to what we use for the
> poll
> > implementation (epoll kqueue select etc), but this would be a build-time
> option,
> > and the implementation would have to be checked in, which I don't think
> is an
> > option for this case, right?
> >
> > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > Hi,
> > >
> > > I am just a user, but I would love to see this change. I have thinking
> > > about this and I would like to be able to pass a C++ allocator object
> > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > define a struct in the interface
> > >
> > > struct allocator_t
> > > {
> > > void* hint;
> > > void* (allocate)(size_t, void*);
> > > void (deallocate)(void*, size_t, void*); };
> > >
> > > and store this in the context object. Since I don't think that this
> should be
> > changed during runtime, I would create a new zmq_ctx_new overload which
> takes a
> > parameter of type allocator_t. The default value would be to call
> malloc/free.
> > >
> > > Cheers,
> > > Jens
> > >
> > > --
> > > Jens Auer | CGI | Software-Engineer
> > > CGI (Germany) GmbH & Co. KG
> > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > T: [phone]
> > > ***@cgi.com<mailto:***@cgi.com>
> > > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> unter
> > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to CGI
> > Group Inc. and its affiliates may be contained in this message. If you
> are not a
> > recipient indicated or intended in this message (or responsible for
> delivery of this
> > message to such person), or you think for any reason that this message
> may have
> > been addressed to you in error, you may not use or copy or deliver this
> message to
> > anyone else. In such case, you should destroy this message and are asked
> to notify
> > the sender by reply e-mail.
> > > ________________________________
> > > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> > > "Petteri Salo [***@gmail.com]
> > > Gesendet: Montag, 28. November 2016 09:40
> > > An: zeromq-***@lists.zeromq.org
> > > Betreff: [zeromq-dev] On hooking memory allocations
> > >
> > > Hello,
> > >
> > > Let me first do a little introduction as I'm new to this list. I'm a
> software engineer
> > with 15+ years of experience working on games at a company called Remedy
> > Entertainment Ltd. We've done games for PC, and various games consoles
> over the
> > years. Most recently we did Quantum Break for Xbox One.
> > >
> > > I've now been tasked with evaluating ZeroMQ. One important feature of
> any
> > library we use in games is the ability to hook all memory allocations -
> this is to allow
> > the use of custom memory allocators and/or for tracking when and where
> memory is
> > allocated.
> > >
> > > I've searched the libzmq source code and there is about 150 uses of
> new, malloc,
> > realloc , etc.
> > >
> > > If we were to adopt libzmq we'd like to put in allocation hooks and
> that work
> > would then be something that we'd like to contribute back to the
> project. Having
> > those hooks in the main repository would then make it easier for us to
> adopt future
> > changes to the library.
> > >
> > > So, my question is would this kind of change be something that would be
> > accepted? Of course assuming that coding conventions, proper way of
> submitting
> > the patch etc. are followed. I do realize that one would want to see the
> actual code
> > before accepting. I'm interested in the principle of accepting a change
> such as this,
> > since it would introduce a new "rule" for working ión libzmq source code
> : "All
> > allocations shall go through an allocation hook."
> > >
> > > Best Regards,
> > >
> > > Petteri Salo
> > >
> > >
> > > _______________________________________________
> > > zeromq-dev mailing list
> > > zeromq-***@lists.zeromq.org
> > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> ------------------------------
>
> E-Mail ist virenfrei.
> Von AVG ÃŒberprÃŒft - www.avg.com
> Version: 2016.0.7924 / Virendatenbank: 4728/13496 - Ausgabedatum:
> 28.11.2016
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> ------------------------------
>
> E-Mail ist virenfrei.
> Von AVG ÃŒberprÃŒft - www.avg.com
> Version: 2016.0.7924 / Virendatenbank: 4728/13496 - Ausgabedatum:
> 28.11.2016
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

[Auer, Jens]

Hi,

A memory-pool with fixed blocks for incoming messages is exactly what I have in mind. E.g. in my application, I receive messages of ~1kb at a rate of 10000/s. Right now with ZeroMQ 4.1, this is done by receiving from the socket into a fixed 8k buffer and then allocating a message with a buffer of 1kb where the data is copied. I would like to use a memory pool instead of malloc here, but I can’t. There are not so many other places where memory is allocated, so I think it should be possible to use a pool of a few different block sizes for all memory allocations in ZeroMQ. Most of the block sizes will have very few blocks, but the receiving sizes are the interesting ones.

> However the library interface promises that message points to a contiguous buffer of message size, which makes it difficult to integrate with memory pools.
I am not sure I understand this. Are you saying that with a memory pool managing blocks I allocate let's say a block of 1k and get non-continuous memory?

Cheers,
Jens Auer

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
***@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.

From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf Of Max Kozlovsky
Sent: 29 November 2016 01:11
To: ZeroMQ development list
Subject: Re: [zeromq-dev] On hooking memory allocations

Using a memory pool (as in fixed-block allocator) is a somewhat different requirement from using custom malloc/free implementation. I am not sure if the library can make use of memory pools. The logical place to use memory pools is for incoming message buffer management. This is for all practical purposes the only place where a lot of allocations are happening and faster allocation might make a difference. However the library interface promises that message points to a contiguous buffer of message size, which makes it difficult to integrate with memory pools.


On Mon, Nov 28, 2016 at 2:34 PM, Jens Auer <***@betaversion.net> wrote:
Yeah, I know that. I always find this more a hack than a real design. It is also not so nice to use with a C++-allocator object because I have to hook it with global functions. And being able to set this for ZeroMQ only e.g. gives me the freedom to use a memory pool there and standard allocators everywhere else. I welcome this flexibility.
 
Cheers,
Jens
 
Von: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] Im Auftrag von Max Kozlovsky
Gesendet: Montag, 28. November 2016 20:48

An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations
 
Each program can be linked with a separate malloc implementation if the user so desires. Libraries don't need to be aware which implementation it is. Different malloc implementation can be even substituted at run time on platforms with dynamic linking support (LD_PRELOAD etc).
 
On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <***@betaversion.net> wrote:
Hi,
 
yes and no. If you overwrite it globally at compute-time every program on the system has to use your custom implementation. So if you deliver your ZeroMQ library with your program it will work, but what if my program wants a different custom allocator?
 
Cheers,
Jens
 
Von: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] Im Auftrag von Max Kozlovsky
Gesendet: Montag, 28. November 2016 20:36
An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations
 
Hi,
 
Would not globally overwriting malloc/free with the custom implementation achieve the desired behavior (instead of providing hooks for installing malloc overrides in each and every library)?
 
Max
 
On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:
Hi,

I don't see a big problem with the C API except that C doesn't support overloads. So if the function has a new name, e.g. zmq_ctx_new_with_allocator, everything stays plain C. The default instance would be a

static void* malloc_(size_t n, void*) {return malloc(n);}
static void free_(void* ptr, size_t n, void*) {free(ptr);}

allocator_t alloc{
NULL,
malloc_,
free_
};

context_t then stores the member and gets methods to forward memory allocations to the function pointers, passing the hint pointer as an additional argument.

In my C++ code, I can then use an allocator
static void* allocate(size_t n, void* obj) {return static_cast<std::allocator<char>>(obj)->allocate(n); }
static void free_(void* ptr, size_t n, void*obj) { static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }

std::allocator<char> a;
allocator_t zmqAlloc{
   &a,
   allocate,
   free_
};

void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);

I think this should work?

Best wishes,
Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
***@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.
> -----Original Message-----
> From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf Of
> Luca Boccassi
> Sent: 28 November 2016 12:30
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] On hooking memory allocations
>
> That would work for an internal API, but given we expose a C API unfortunately I
> don't think that would work as a public API :-( And I think for this use case they
> would require a public API.
>
> As an external API, a new zmq_ctx_set that takes a callback would have been ideal,
> but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes a callback
> pointer would be the next best.
>
> An alternative would be to have a system similar to what we use for the poll
> implementation (epoll kqueue select etc), but this would be a build-time option,
> and the implementation would have to be checked in, which I don't think is an
> option for this case, right?
>
> On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > Hi,
> >
> > I am just a user, but I would love to see this change. I have thinking
> > about this and I would like to be able to pass a C++ allocator object
> > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > define a struct in the interface
> >
> > struct allocator_t
> > {
> >     void* hint;
> >     void* (allocate)(size_t, void*);
> >     void (deallocate)(void*, size_t, void*); };
> >
> > and store this in the context object. Since I don't think that this should be
> changed during runtime, I would create a new zmq_ctx_new overload which takes a
> parameter of type allocator_t. The default value would be to call malloc/free.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: [phone]
> > ***@cgi.com<mailto:***@cgi.com>
> > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
> de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
> Group Inc. and its affiliates may be contained in this message. If you are not a
> recipient indicated or intended in this message (or responsible for delivery of this
> message to such person), or you think for any reason that this message may have
> been addressed to you in error, you may not use or copy or deliver this message to
> anyone else. In such case, you should destroy this message and are asked to notify
> the sender by reply e-mail.
> > ________________________________
> > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> > "Petteri Salo [***@gmail.com]
> > Gesendet: Montag, 28. November 2016 09:40
> > An: zeromq-***@lists.zeromq.org
> > Betreff: [zeromq-dev] On hooking memory allocations
> >
> > Hello,
> >
> > Let me first do a little introduction as I'm new to this list. I'm a software engineer
> with 15+ years of experience working on games at a company called Remedy
> Entertainment Ltd. We've done games for PC, and various games consoles over the
> years. Most recently we did Quantum Break for Xbox One.
> >
> > I've now been tasked with evaluating ZeroMQ. One important feature of any
> library we use in games is the ability to hook all memory allocations - this is to allow
> the use of custom memory allocators and/or for tracking when and where memory is
> allocated.
> >
> > I've searched the libzmq source code and there is about 150 uses of new, malloc,
> realloc , etc.
> >
> > If we were to adopt libzmq we'd like to put in allocation hooks and that work
> would then be something that we'd like to contribute back to the project. Having
> those hooks in the main repository would then make it easier for us to adopt future
> changes to the library.
> >
> > So, my question is would this kind of change be something that would be
> accepted? Of course assuming that coding conventions, proper way of submitting
> the patch etc. are followed. I do realize that one would want to see the actual code
> before accepting. I'm interested in the principle of accepting a change such as this,
> since it would introduce a new "rule" for working ión libzmq source code : "All
> allocations shall go through an allocation hook."
> >
> > Best Regards,
> >
> > Petteri Salo
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-***@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
________________________________________
E-Mail ist virenfrei.
Von AVG überprüft - www.avg.com
Version: 2016.0.7924 / Virendatenbank: 4728/13496 - Ausgabedatum: 28.11.2016

_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
 
________________________________________
E-Mail ist virenfrei.
Von AVG überprüft - www.avg.com
Version: 2016.0.7924 / Virendatenbank: 4728/13496 - Ausgabedatum: 28.11.2016

_______________________________________________
zeromq-dev mailing list
zeromq-***@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Michel Pelletier]

On Tue, Nov 29, 2016 at 12:17 AM, Auer, Jens <***@cgi.com> wrote:

> Hi,
>
> A memory-pool with fixed blocks for incoming messages is exactly what I
> have in mind. E.g. in my application, I receive messages of ~1kb at a rate
> of 10000/s. Right now with ZeroMQ 4.1, this is done by receiving from the
> socket into a fixed 8k buffer and then allocating a message with a buffer
> of 1kb where the data is copied.


You might want to consider using an allocation management library like the
Boehm garbage collector:

http://www.hboehm.info/gc/

It's very good at pooling and is very efficient. In addition it can be
used as a leak detection tool and can be used as a drop in replacement for
new/malloc/realloc without code changes.

-Michel

[Auer, Jens]

Hi Michel,

thanks for the suggestion. I will definitely gie it a try. I tried tcmalloc and jemalloc once, but the improvement was not so big. If I understood tcmalloc correctly they use a per-thread cache for allocations. I didn't think of gc.

In my system I have a component which basically receives a data stream (260MB/s, one data frame is 1129 bytes) from an external source, extracts a stream identifier and publishes the data again on a PUB socket with the stream id as a topic. Each frame of 1129 is published as a message part. It also does some filtering and integrate control sockets for RPC, but this is negligible performance-wise. When I profile this I see malloc/free as one of the hot spots. I can also improve performance by increasing the recv buffer in ZeroMQ, but malloc still stays of course. The application is well within the requirements and has enough margin, but I am interested in tuning options.

Cheers,
Jens

--
Jens Auer | CGI | Software-Engineer
CGI (Germany) GmbH & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: [phone]
***@cgi.com<mailto:***@cgi.com>
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.
________________________________
Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von "Michel Pelletier [***@gmail.com]
Gesendet: Dienstag, 29. November 2016 20:16
An: ZeroMQ development list
Betreff: Re: [zeromq-dev] On hooking memory allocations

On Tue, Nov 29, 2016 at 12:17 AM, Auer, Jens <***@cgi.com<mailto:***@cgi.com>> wrote:
Hi,

A memory-pool with fixed blocks for incoming messages is exactly what I have in mind. E.g. in my application, I receive messages of ~1kb at a rate of 10000/s. Right now with ZeroMQ 4.1, this is done by receiving from the socket into a fixed 8k buffer and then allocating a message with a buffer of 1kb where the data is copied.

You might want to consider using an allocation management library like the Boehm garbage collector:

http://www.hboehm.info/gc/

It's very good at pooling and is very efficient. In addition it can be used as a leak detection tool and can be used as a drop in replacement for new/malloc/realloc without code changes.

-Michel

[Steven McCoy]

On 2 December 2016 at 04:50, Auer, Jens <***@cgi.com> wrote:

> thanks for the suggestion. I will definitely gie it a try. I tried
> tcmalloc and jemalloc once, but the improvement was not so big. If I
> understood tcmalloc correctly they use a per-thread cache for allocations.
> I didn't think of gc.
>

These are redundant on FreeBSD where jemalloc is already the default
allocator and Linux where ptmalloc3 also implements a per-thread cache
(SLAB). On Windows you can use the Concurrency CRT API for similar
features.

--
Steve-o

[Luca Boccassi]

That works on Linux (and I guess *NIX) yes, but I'm not sure if Windows
has a similar functionality. Even if it does, the use case here (a game
engine/library) is most likely a statically linked binary, so I think it
won't work there :-)

All in all this is an interesting discussion, but let's see some PRs and
take it from there :-)

On Mon, 2016-11-28 at 11:48 -0800, Max Kozlovsky wrote:
> Each program can be linked with a separate malloc implementation if the
> user so desires. Libraries don't need to be aware which implementation it
> is. Different malloc implementation can be even substituted at run time on
> platforms with dynamic linking support (LD_PRELOAD etc).
>
> On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <***@betaversion.net>
> wrote:
>
> > Hi,
> >
> >
> >
> > yes and no. If you overwrite it globally at compute-time every program on
> > the system has to use your custom implementation. So if you deliver your
> > ZeroMQ library with your program it will work, but what if my program wants
> > a different custom allocator?
> >
> >
> >
> > Cheers,
> >
> > Jens
> >
> >
> >
> > *Von:* zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] *Im
> > Auftrag von *Max Kozlovsky
> > *Gesendet:* Montag, 28. November 2016 20:36
> > *An:* ZeroMQ development list
> > *Betreff:* Re: [zeromq-dev] On hooking memory allocations
> >
> >
> >
> > Hi,
> >
> >
> >
> > Would not globally overwriting malloc/free with the custom implementation
> > achieve the desired behavior (instead of providing hooks for installing
> > malloc overrides in each and every library)?
> >
> >
> >
> > Max
> >
> >
> >
> > On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:
> >
> > Hi,
> >
> > I don't see a big problem with the C API except that C doesn't support
> > overloads. So if the function has a new name, e.g.
> > zmq_ctx_new_with_allocator, everything stays plain C. The default instance
> > would be a
> >
> > static void* malloc_(size_t n, void*) {return malloc(n);}
> > static void free_(void* ptr, size_t n, void*) {free(ptr);}
> >
> > allocator_t alloc{
> > NULL,
> > malloc_,
> > free_
> > };
> >
> > context_t then stores the member and gets methods to forward memory
> > allocations to the function pointers, passing the hint pointer as an
> > additional argument.
> >
> > In my C++ code, I can then use an allocator
> > static void* allocate(size_t n, void* obj) {return
> > static_cast<std::allocator<char>>(obj)->allocate(n); }
> > static void free_(void* ptr, size_t n, void*obj) {
> > static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
> >
> > std::allocator<char> a;
> > allocator_t zmqAlloc{
> > &a,
> > allocate,
> > free_
> > };
> >
> > void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
> >
> > I think this should work?
> >
> > Best wishes,
> > Jens
> >
> > --
> > Dr. Jens Auer | CGI | Software Engineer
> > CGI Deutschland Ltd. & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: [phone]
> > ***@cgi.com
> > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> > unter de.cgi.com/pflichtangaben.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to
> > CGI Group Inc. and its affiliates may be contained in this message. If you
> > are not a recipient indicated or intended in this message (or responsible
> > for delivery of this message to such person), or you think for any reason
> > that this message may have been addressed to you in error, you may not use
> > or copy or deliver this message to anyone else. In such case, you should
> > destroy this message and are asked to notify the sender by reply e-mail.
> >
> > > -----Original Message-----
> > > From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On Behalf
> > Of
> > > Luca Boccassi
> > > Sent: 28 November 2016 12:30
> > > To: ZeroMQ development list
> > > Subject: Re: [zeromq-dev] On hooking memory allocations
> > >
> > > That would work for an internal API, but given we expose a C API
> > unfortunately I
> > > don't think that would work as a public API :-( And I think for this use
> > case they
> > > would require a public API.
> > >
> > > As an external API, a new zmq_ctx_set that takes a callback would have
> > been ideal,
> > > but it only takes int. So perhaps a new zmq_ctx_set_allocator that takes
> > a callback
> > > pointer would be the next best.
> > >
> > > An alternative would be to have a system similar to what we use for the
> > poll
> > > implementation (epoll kqueue select etc), but this would be a build-time
> > option,
> > > and the implementation would have to be checked in, which I don't think
> > is an
> > > option for this case, right?
> > >
> > > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > > Hi,
> > > >
> > > > I am just a user, but I would love to see this change. I have thinking
> > > > about this and I would like to be able to pass a C++ allocator object
> > > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > > define a struct in the interface
> > > >
> > > > struct allocator_t
> > > > {
> > > > void* hint;
> > > > void* (allocate)(size_t, void*);
> > > > void (deallocate)(void*, size_t, void*); };
> > > >
> > > > and store this in the context object. Since I don't think that this
> > should be
> > > changed during runtime, I would create a new zmq_ctx_new overload which
> > takes a
> > > parameter of type allocator_t. The default value would be to call
> > malloc/free.
> > > >
> > > > Cheers,
> > > > Jens
> > > >
> > > > --
> > > > Jens Auer | CGI | Software-Engineer
> > > > CGI (Germany) GmbH & Co. KG
> > > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > > T: [phone]
> > > > ***@cgi.com<mailto:***@cgi.com>
> > > > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> > unter
> > > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > > >
> > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> > to CGI
> > > Group Inc. and its affiliates may be contained in this message. If you
> > are not a
> > > recipient indicated or intended in this message (or responsible for
> > delivery of this
> > > message to such person), or you think for any reason that this message
> > may have
> > > been addressed to you in error, you may not use or copy or deliver this
> > message to
> > > anyone else. In such case, you should destroy this message and are asked
> > to notify
> > > the sender by reply e-mail.
> > > > ________________________________
> > > > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag von
> > > > "Petteri Salo [***@gmail.com]
> > > > Gesendet: Montag, 28. November 2016 09:40
> > > > An: zeromq-***@lists.zeromq.org
> > > > Betreff: [zeromq-dev] On hooking memory allocations
> > > >
> > > > Hello,
> > > >
> > > > Let me first do a little introduction as I'm new to this list. I'm a
> > software engineer
> > > with 15+ years of experience working on games at a company called Remedy
> > > Entertainment Ltd. We've done games for PC, and various games consoles
> > over the
> > > years. Most recently we did Quantum Break for Xbox One.
> > > >
> > > > I've now been tasked with evaluating ZeroMQ. One important feature of
> > any
> > > library we use in games is the ability to hook all memory allocations -
> > this is to allow
> > > the use of custom memory allocators and/or for tracking when and where
> > memory is
> > > allocated.
> > > >
> > > > I've searched the libzmq source code and there is about 150 uses of
> > new, malloc,
> > > realloc , etc.
> > > >
> > > > If we were to adopt libzmq we'd like to put in allocation hooks and
> > that work
> > > would then be something that we'd like to contribute back to the
> > project. Having
> > > those hooks in the main repository would then make it easier for us to
> > adopt future
> > > changes to the library.
> > > >
> > > > So, my question is would this kind of change be something that would be
> > > accepted? Of course assuming that coding conventions, proper way of
> > submitting
> > > the patch etc. are followed. I do realize that one would want to see the
> > actual code
> > > before accepting. I'm interested in the principle of accepting a change
> > such as this,
> > > since it would introduce a new "rule" for working ión libzmq source code
> > : "All
> > > allocations shall go through an allocation hook."
> > > >
> > > > Best Regards,
> > > >
> > > > Petteri Salo

[Petteri Salo]

Hi,

Few questions:
1. PR == Pull Request ? git and github are pretty much unknown to me so I'm
just checking my assumptions...
2. From https://rfc.zeromq.org/spec:42/C4/ section 2.3.6, I assume the
"principle target platform" is Linux?

-Petteri

On Tue, Nov 29, 2016 at 12:51 PM, Luca Boccassi <***@gmail.com>
wrote:

> That works on Linux (and I guess *NIX) yes, but I'm not sure if Windows
> has a similar functionality. Even if it does, the use case here (a game
> engine/library) is most likely a statically linked binary, so I think it
> won't work there :-)
>
> All in all this is an interesting discussion, but let's see some PRs and
> take it from there :-)
>
> On Mon, 2016-11-28 at 11:48 -0800, Max Kozlovsky wrote:
> > Each program can be linked with a separate malloc implementation if the
> > user so desires. Libraries don't need to be aware which implementation it
> > is. Different malloc implementation can be even substituted at run time
> on
> > platforms with dynamic linking support (LD_PRELOAD etc).
> >
> > On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <***@betaversion.net>
> > wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > yes and no. If you overwrite it globally at compute-time every program
> on
> > > the system has to use your custom implementation. So if you deliver
> your
> > > ZeroMQ library with your program it will work, but what if my program
> wants
> > > a different custom allocator?
> > >
> > >
> > >
> > > Cheers,
> > >
> > > Jens
> > >
> > >
> > >
> > > *Von:* zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] *Im
> > > Auftrag von *Max Kozlovsky
> > > *Gesendet:* Montag, 28. November 2016 20:36
> > > *An:* ZeroMQ development list
> > > *Betreff:* Re: [zeromq-dev] On hooking memory allocations
> > >
> > >
> > >
> > > Hi,
> > >
> > >
> > >
> > > Would not globally overwriting malloc/free with the custom
> implementation
> > > achieve the desired behavior (instead of providing hooks for installing
> > > malloc overrides in each and every library)?
> > >
> > >
> > >
> > > Max
> > >
> > >
> > >
> > > On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:
> > >
> > > Hi,
> > >
> > > I don't see a big problem with the C API except that C doesn't support
> > > overloads. So if the function has a new name, e.g.
> > > zmq_ctx_new_with_allocator, everything stays plain C. The default
> instance
> > > would be a
> > >
> > > static void* malloc_(size_t n, void*) {return malloc(n);}
> > > static void free_(void* ptr, size_t n, void*) {free(ptr);}
> > >
> > > allocator_t alloc{
> > > NULL,
> > > malloc_,
> > > free_
> > > };
> > >
> > > context_t then stores the member and gets methods to forward memory
> > > allocations to the function pointers, passing the hint pointer as an
> > > additional argument.
> > >
> > > In my C++ code, I can then use an allocator
> > > static void* allocate(size_t n, void* obj) {return
> > > static_cast<std::allocator<char>>(obj)->allocate(n); }
> > > static void free_(void* ptr, size_t n, void*obj) {
> > > static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
> > >
> > > std::allocator<char> a;
> > > allocator_t zmqAlloc{
> > > &a,
> > > allocate,
> > > free_
> > > };
> > >
> > > void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
> > >
> > > I think this should work?
> > >
> > > Best wishes,
> > > Jens
> > >
> > > --
> > > Dr. Jens Auer | CGI | Software Engineer
> > > CGI Deutschland Ltd. & Co. KG
> > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > T: [phone]
> > > ***@cgi.com
> > > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> > > unter de.cgi.com/pflichtangaben.
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> to
> > > CGI Group Inc. and its affiliates may be contained in this message. If
> you
> > > are not a recipient indicated or intended in this message (or
> responsible
> > > for delivery of this message to such person), or you think for any
> reason
> > > that this message may have been addressed to you in error, you may not
> use
> > > or copy or deliver this message to anyone else. In such case, you
> should
> > > destroy this message and are asked to notify the sender by reply
> e-mail.
> > >
> > > > -----Original Message-----
> > > > From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On
> Behalf
> > > Of
> > > > Luca Boccassi
> > > > Sent: 28 November 2016 12:30
> > > > To: ZeroMQ development list
> > > > Subject: Re: [zeromq-dev] On hooking memory allocations
> > > >
> > > > That would work for an internal API, but given we expose a C API
> > > unfortunately I
> > > > don't think that would work as a public API :-( And I think for this
> use
> > > case they
> > > > would require a public API.
> > > >
> > > > As an external API, a new zmq_ctx_set that takes a callback would
> have
> > > been ideal,
> > > > but it only takes int. So perhaps a new zmq_ctx_set_allocator that
> takes
> > > a callback
> > > > pointer would be the next best.
> > > >
> > > > An alternative would be to have a system similar to what we use for
> the
> > > poll
> > > > implementation (epoll kqueue select etc), but this would be a
> build-time
> > > option,
> > > > and the implementation would have to be checked in, which I don't
> think
> > > is an
> > > > option for this case, right?
> > > >
> > > > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > > > Hi,
> > > > >
> > > > > I am just a user, but I would love to see this change. I have
> thinking
> > > > > about this and I would like to be able to pass a C++ allocator
> object
> > > > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > > > define a struct in the interface
> > > > >
> > > > > struct allocator_t
> > > > > {
> > > > > void* hint;
> > > > > void* (allocate)(size_t, void*);
> > > > > void (deallocate)(void*, size_t, void*); };
> > > > >
> > > > > and store this in the context object. Since I don't think that this
> > > should be
> > > > changed during runtime, I would create a new zmq_ctx_new overload
> which
> > > takes a
> > > > parameter of type allocator_t. The default value would be to call
> > > malloc/free.
> > > > >
> > > > > Cheers,
> > > > > Jens
> > > > >
> > > > > --
> > > > > Jens Auer | CGI | Software-Engineer
> > > > > CGI (Germany) GmbH & Co. KG
> > > > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > > > T: [phone]
> > > > > ***@cgi.com<mailto:***@cgi.com>
> > > > > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden
> Sie
> > > unter
> > > > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > > > >
> > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information
> belonging
> > > to CGI
> > > > Group Inc. and its affiliates may be contained in this message. If
> you
> > > are not a
> > > > recipient indicated or intended in this message (or responsible for
> > > delivery of this
> > > > message to such person), or you think for any reason that this
> message
> > > may have
> > > > been addressed to you in error, you may not use or copy or deliver
> this
> > > message to
> > > > anyone else. In such case, you should destroy this message and are
> asked
> > > to notify
> > > > the sender by reply e-mail.
> > > > > ________________________________
> > > > > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag
> von
> > > > > "Petteri Salo [***@gmail.com]
> > > > > Gesendet: Montag, 28. November 2016 09:40
> > > > > An: zeromq-***@lists.zeromq.org
> > > > > Betreff: [zeromq-dev] On hooking memory allocations
> > > > >
> > > > > Hello,
> > > > >
> > > > > Let me first do a little introduction as I'm new to this list. I'm
> a
> > > software engineer
> > > > with 15+ years of experience working on games at a company called
> Remedy
> > > > Entertainment Ltd. We've done games for PC, and various games
> consoles
> > > over the
> > > > years. Most recently we did Quantum Break for Xbox One.
> > > > >
> > > > > I've now been tasked with evaluating ZeroMQ. One important feature
> of
> > > any
> > > > library we use in games is the ability to hook all memory
> allocations -
> > > this is to allow
> > > > the use of custom memory allocators and/or for tracking when and
> where
> > > memory is
> > > > allocated.
> > > > >
> > > > > I've searched the libzmq source code and there is about 150 uses of
> > > new, malloc,
> > > > realloc , etc.
> > > > >
> > > > > If we were to adopt libzmq we'd like to put in allocation hooks and
> > > that work
> > > > would then be something that we'd like to contribute back to the
> > > project. Having
> > > > those hooks in the main repository would then make it easier for us
> to
> > > adopt future
> > > > changes to the library.
> > > > >
> > > > > So, my question is would this kind of change be something that
> would be
> > > > accepted? Of course assuming that coding conventions, proper way of
> > > submitting
> > > > the patch etc. are followed. I do realize that one would want to see
> the
> > > actual code
> > > > before accepting. I'm interested in the principle of accepting a
> change
> > > such as this,
> > > > since it would introduce a new "rule" for working ión libzmq source
> code
> > > : "All
> > > > allocations shall go through an allocation hook."
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Petteri Salo
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>

[Luca Boccassi]

1) yes

2) depends on who you ask :-D

Don't worry too much about it, if it works on your development platform
it's good enough for the initial iteration.

We have automated CI for Linux (Travis), OSX (Travis) and Windows
(appveyor with gcc and jenkins with vsc++). If anything breaks we'll see
it quickly and will ask you to fix it :-)

If you prefer to check it beforehand you can set up Travis and Appveyor
to run from your Github personal fork when you push to your personal
branch, but it's not a hard requirement. I can give you instructions if
you would like that.

On Tue, 2016-11-29 at 13:39 +0200, Petteri Salo wrote:
> Hi,
>
> Few questions:
> 1. PR == Pull Request ? git and github are pretty much unknown to me so I'm
> just checking my assumptions...
> 2. From https://rfc.zeromq.org/spec:42/C4/ section 2.3.6, I assume the
> "principle target platform" is Linux?
>
> -Petteri
>
> On Tue, Nov 29, 2016 at 12:51 PM, Luca Boccassi <***@gmail.com>
> wrote:
>
> > That works on Linux (and I guess *NIX) yes, but I'm not sure if Windows
> > has a similar functionality. Even if it does, the use case here (a game
> > engine/library) is most likely a statically linked binary, so I think it
> > won't work there :-)
> >
> > All in all this is an interesting discussion, but let's see some PRs and
> > take it from there :-)
> >
> > On Mon, 2016-11-28 at 11:48 -0800, Max Kozlovsky wrote:
> > > Each program can be linked with a separate malloc implementation if the
> > > user so desires. Libraries don't need to be aware which implementation it
> > > is. Different malloc implementation can be even substituted at run time
> > on
> > > platforms with dynamic linking support (LD_PRELOAD etc).
> > >
> > > On Mon, Nov 28, 2016 at 11:39 AM, Jens Auer <***@betaversion.net>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > yes and no. If you overwrite it globally at compute-time every program
> > on
> > > > the system has to use your custom implementation. So if you deliver
> > your
> > > > ZeroMQ library with your program it will work, but what if my program
> > wants
> > > > a different custom allocator?
> > > >
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Jens
> > > >
> > > >
> > > >
> > > > *Von:* zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] *Im
> > > > Auftrag von *Max Kozlovsky
> > > > *Gesendet:* Montag, 28. November 2016 20:36
> > > > *An:* ZeroMQ development list
> > > > *Betreff:* Re: [zeromq-dev] On hooking memory allocations
> > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > Would not globally overwriting malloc/free with the custom
> > implementation
> > > > achieve the desired behavior (instead of providing hooks for installing
> > > > malloc overrides in each and every library)?
> > > >
> > > >
> > > >
> > > > Max
> > > >
> > > >
> > > >
> > > > On Mon, Nov 28, 2016 at 5:08 AM, Auer, Jens <***@cgi.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I don't see a big problem with the C API except that C doesn't support
> > > > overloads. So if the function has a new name, e.g.
> > > > zmq_ctx_new_with_allocator, everything stays plain C. The default
> > instance
> > > > would be a
> > > >
> > > > static void* malloc_(size_t n, void*) {return malloc(n);}
> > > > static void free_(void* ptr, size_t n, void*) {free(ptr);}
> > > >
> > > > allocator_t alloc{
> > > > NULL,
> > > > malloc_,
> > > > free_
> > > > };
> > > >
> > > > context_t then stores the member and gets methods to forward memory
> > > > allocations to the function pointers, passing the hint pointer as an
> > > > additional argument.
> > > >
> > > > In my C++ code, I can then use an allocator
> > > > static void* allocate(size_t n, void* obj) {return
> > > > static_cast<std::allocator<char>>(obj)->allocate(n); }
> > > > static void free_(void* ptr, size_t n, void*obj) {
> > > > static_cast<std::allocator<char>>(obj)->deallocate(ptr, n); }
> > > >
> > > > std::allocator<char> a;
> > > > allocator_t zmqAlloc{
> > > > &a,
> > > > allocate,
> > > > free_
> > > > };
> > > >
> > > > void* ctx = zmq_ctx_new_with_allocator(&zmqAlloc);
> > > >
> > > > I think this should work?
> > > >
> > > > Best wishes,
> > > > Jens
> > > >
> > > > --
> > > > Dr. Jens Auer | CGI | Software Engineer
> > > > CGI Deutschland Ltd. & Co. KG
> > > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > > T: [phone]
> > > > ***@cgi.com
> > > > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden Sie
> > > > unter de.cgi.com/pflichtangaben.
> > > >
> > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging
> > to
> > > > CGI Group Inc. and its affiliates may be contained in this message. If
> > you
> > > > are not a recipient indicated or intended in this message (or
> > responsible
> > > > for delivery of this message to such person), or you think for any
> > reason
> > > > that this message may have been addressed to you in error, you may not
> > use
> > > > or copy or deliver this message to anyone else. In such case, you
> > should
> > > > destroy this message and are asked to notify the sender by reply
> > e-mail.
> > > >
> > > > > -----Original Message-----
> > > > > From: zeromq-dev [mailto:zeromq-dev-***@lists.zeromq.org] On
> > Behalf
> > > > Of
> > > > > Luca Boccassi
> > > > > Sent: 28 November 2016 12:30
> > > > > To: ZeroMQ development list
> > > > > Subject: Re: [zeromq-dev] On hooking memory allocations
> > > > >
> > > > > That would work for an internal API, but given we expose a C API
> > > > unfortunately I
> > > > > don't think that would work as a public API :-( And I think for this
> > use
> > > > case they
> > > > > would require a public API.
> > > > >
> > > > > As an external API, a new zmq_ctx_set that takes a callback would
> > have
> > > > been ideal,
> > > > > but it only takes int. So perhaps a new zmq_ctx_set_allocator that
> > takes
> > > > a callback
> > > > > pointer would be the next best.
> > > > >
> > > > > An alternative would be to have a system similar to what we use for
> > the
> > > > poll
> > > > > implementation (epoll kqueue select etc), but this would be a
> > build-time
> > > > option,
> > > > > and the implementation would have to be checked in, which I don't
> > think
> > > > is an
> > > > > option for this case, right?
> > > > >
> > > > > On Mon, 2016-11-28 at 10:51 +0000, Auer, Jens wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I am just a user, but I would love to see this change. I have
> > thinking
> > > > > > about this and I would like to be able to pass a C++ allocator
> > object
> > > > > > to ZeroMQ, so a simple function hook is not enough. My idea is to
> > > > > > define a struct in the interface
> > > > > >
> > > > > > struct allocator_t
> > > > > > {
> > > > > > void* hint;
> > > > > > void* (allocate)(size_t, void*);
> > > > > > void (deallocate)(void*, size_t, void*); };
> > > > > >
> > > > > > and store this in the context object. Since I don't think that this
> > > > should be
> > > > > changed during runtime, I would create a new zmq_ctx_new overload
> > which
> > > > takes a
> > > > > parameter of type allocator_t. The default value would be to call
> > > > malloc/free.
> > > > > >
> > > > > > Cheers,
> > > > > > Jens
> > > > > >
> > > > > > --
> > > > > > Jens Auer | CGI | Software-Engineer
> > > > > > CGI (Germany) GmbH & Co. KG
> > > > > > Rheinstraße 95 | 64295 Darmstadt | Germany
> > > > > > T: [phone]
> > > > > > ***@cgi.com<mailto:***@cgi.com>
> > > > > > Unsere Pflichtangaben gemÀß § 35a GmbHG / §§ 161, 125a HGB finden
> > Sie
> > > > unter
> > > > > de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> > > > > >
> > > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information
> > belonging
> > > > to CGI
> > > > > Group Inc. and its affiliates may be contained in this message. If
> > you
> > > > are not a
> > > > > recipient indicated or intended in this message (or responsible for
> > > > delivery of this
> > > > > message to such person), or you think for any reason that this
> > message
> > > > may have
> > > > > been addressed to you in error, you may not use or copy or deliver
> > this
> > > > message to
> > > > > anyone else. In such case, you should destroy this message and are
> > asked
> > > > to notify
> > > > > the sender by reply e-mail.
> > > > > > ________________________________
> > > > > > Von: zeromq-dev [zeromq-dev-***@lists.zeromq.org]" im Auftrag
> > von
> > > > > > "Petteri Salo [***@gmail.com]
> > > > > > Gesendet: Montag, 28. November 2016 09:40
> > > > > > An: zeromq-***@lists.zeromq.org
> > > > > > Betreff: [zeromq-dev] On hooking memory allocations
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Let me first do a little introduction as I'm new to this list. I'm
> > a
> > > > software engineer
> > > > > with 15+ years of experience working on games at a company called
> > Remedy
> > > > > Entertainment Ltd. We've done games for PC, and various games
> > consoles
> > > > over the
> > > > > years. Most recently we did Quantum Break for Xbox One.
> > > > > >
> > > > > > I've now been tasked with evaluating ZeroMQ. One important feature
> > of
> > > > any
> > > > > library we use in games is the ability to hook all memory
> > allocations -
> > > > this is to allow
> > > > > the use of custom memory allocators and/or for tracking when and
> > where
> > > > memory is
> > > > > allocated.
> > > > > >
> > > > > > I've searched the libzmq source code and there is about 150 uses of
> > > > new, malloc,
> > > > > realloc , etc.
> > > > > >
> > > > > > If we were to adopt libzmq we'd like to put in allocation hooks and
> > > > that work
> > > > > would then be something that we'd like to contribute back to the
> > > > project. Having
> > > > > those hooks in the main repository would then make it easier for us
> > to
> > > > adopt future
> > > > > changes to the library.
> > > > > >
> > > > > > So, my question is would this kind of change be something that
> > would be
> > > > > accepted? Of course assuming that coding conventions, proper way of
> > > > submitting
> > > > > the patch etc. are followed. I do realize that one would want to see
> > the
> > > > actual code
> > > > > before accepting. I'm interested in the principle of accepting a
> > change
> > > > such as this,
> > > > > since it would introduce a new "rule" for working ión libzmq source
> > code
> > > > : "All
> > > > > allocations shall go through an allocation hook."
> > > > > >
> > > > > > Best Regards,
> > > > > >
> > > > > > Petteri Salo
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-***@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[Adam Majer]

On 11/29/2016 12:39 PM, Petteri Salo wrote:
> Few questions:
> 2. From https://rfc.zeromq.org/spec:42/C4/ section 2.3.6, I assume the
> "principle target platform" is Linux?

The important bit is not to break the ABI.

Thanks in advance
- Adam

[Luca Boccassi]

Hello,

First of all, welcome to the community! Sounds like you have a very
interesting use case :-)

So there is precedent for using some sort of a custom allocator,
although it does not go as far as your requirements:

https://github.com/zeromq/czmq/blob/master/include/czmq_prelude.h#L528

So no objections in principle I think.

Regarding what can get merged and what can't, we use the C4 process as
defined here:

https://rfc.zeromq.org/spec:42/C4/

Basically any correct patch that doesn't break (too much) stuff can be
accepted :-)

I would recommend, following C4, to start sending PRs as soon as you
can, with small changes. We have a DRAFT api process, where new stuff is
hidden behind a --enable-drafts configure call. I would recommend making
as much as possible of this new feature hidden behind the flag, as to
avoid disrupting existing users (we know many users build straight from
the master branch head).

For example, and it's up to you so it's just a suggestion of course, you
could start by changing all the mallocs to a common function sort of
like czmq does, which without --enable-drafts is just an inline static
that calls malloc. And then implement the logic, public and private as
draft (this way you can change the public interface as much as you want
until it's table). Then we can iterate from that.

Again it's entirely up to you. But we've used this model in the past
year to successfully introduce big changes without major disruption (eg:
new sockets, new poller, etc).

On Mon, 2016-11-28 at 10:40 +0200, Petteri Salo wrote:
> Hello,
>
> Let me first do a little introduction as I'm new to this list. I'm a
> software engineer with 15+ years of experience working on games at a
> company called Remedy Entertainment Ltd. We've done games for PC, and
> various games consoles over the years. Most recently we did Quantum Break
> for Xbox One.
>
> I've now been tasked with evaluating ZeroMQ. One important feature of any
> library we use in games is the ability to hook all memory allocations -
> this is to allow the use of custom memory allocators and/or for tracking
> when and where memory is allocated.
>
> I've searched the libzmq source code and there is about 150 uses of new,
> malloc, realloc , etc.
>
> If we were to adopt libzmq we'd like to put in allocation hooks and that
> work would then be something that we'd like to contribute back to the
> project. Having those hooks in the main repository would then make it
> easier for us to adopt future changes to the library.
>
> So, my question is would this kind of change be something that would be
> accepted? Of course assuming that coding conventions, proper way of
> submitting the patch etc. are followed. I do realize that one would want to
> see the actual code before accepting. I'm interested in the principle of
> accepting a change such as this, since it would introduce a new "rule" for
> working ión libzmq source code : "All allocations shall go through an
> allocation hook."
>
> Best Regards,
>
> Petteri Salo
> _______________________________________________
> zeromq-dev mailing list
> zeromq-***@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev