I have been assuming that the pointer back to an ML function passed to a foreign C function will still be valid after the C function has returned, so the callback can occur sometime later (from a different C function). Is that a valid assumption?
I made this assumption based on some tests in which one such callback pointer remained valid for the life of the application. It would be nice to know I wasn't just being (un)lucky!
Phil
Phil, Currently, callbacks remain in effect for ever and are never garbage-collected. I've certainly come across cases where I would want a callback to remain in effect after the function that passed it in had returned. For example, the window procedure in RegisterClassEx in Windows. I don't know if there should be some way to delete a callback explicitly.
Regards, David
On 27/01/2012 17:47, Phil Clayton wrote:
I have been assuming that the pointer back to an ML function passed to a foreign C function will still be valid after the C function has returned, so the callback can occur sometime later (from a different C function). Is that a valid assumption?
I made this assumption based on some tests in which one such callback pointer remained valid for the life of the application. It would be nice to know I wasn't just being (un)lucky!
Phil _______________________________________________ polyml mailing list polyml@inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
David,
Thanks, that's good to know.
Thinking about explicitly deleting callbacks led me to ask whether each ML function passed as a callback has its callback reused on subsequent calls. If reuse occurs, the number of call sites in the source code would bound memory usage.
I'm guessing that these callback objects are actually wrappers to invoke compiled ML code and that there is only one instance of the compiled ML, so there would be no problem having only one instance of the callback object.
Phil
On 27/01/12 19:11, David Matthews wrote:
Phil, Currently, callbacks remain in effect for ever and are never garbage-collected. I've certainly come across cases where I would want a callback to remain in effect after the function that passed it in had returned. For example, the window procedure in RegisterClassEx in Windows. I don't know if there should be some way to delete a callback explicitly.
Regards, David
On 27/01/2012 17:47, Phil Clayton wrote:
I have been assuming that the pointer back to an ML function passed to a foreign C function will still be valid after the C function has returned, so the callback can occur sometime later (from a different C function). Is that a valid assumption?
I made this assumption based on some tests in which one such callback pointer remained valid for the life of the application. It would be nice to know I wasn't just being (un)lucky!
Phil _______________________________________________ polyml mailing list polyml@inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
On 27/01/2012 20:15, Phil Clayton wrote:
Thinking about explicitly deleting callbacks led me to ask whether each ML function passed as a callback has its callback reused on subsequent calls. If reuse occurs, the number of call sites in the source code would bound memory usage.
I'm guessing that these callback objects are actually wrappers to invoke compiled ML code and that there is only one instance of the compiled ML, so there would be no problem having only one instance of the callback object.
It would be nice if they could be reused but currently they aren't. I had a look at the code and did some experiments but the problem is that there is quite a lot of ML wrapping going on. That means that the RTS can't detect that the same function has been passed a second time. After the wrapping has been applied the closure that gets passed into the RTS is always different even if the same ML function was used further up.
I sometimes think the whole FFI system should be redesigned to remove some of the layers.
David
On 29/01/12 15:57, David Matthews wrote:
On 27/01/2012 20:15, Phil Clayton wrote:
Thinking about explicitly deleting callbacks led me to ask whether each ML function passed as a callback has its callback reused on subsequent calls. If reuse occurs, the number of call sites in the source code would bound memory usage.
I'm guessing that these callback objects are actually wrappers to invoke compiled ML code and that there is only one instance of the compiled ML, so there would be no problem having only one instance of the callback object.
It would be nice if they could be reused but currently they aren't. I had a look at the code and did some experiments but the problem is that there is quite a lot of ML wrapping going on. That means that the RTS can't detect that the same function has been passed a second time. After the wrapping has been applied the closure that gets passed into the RTS is always different even if the same ML function was used further up.
I sometimes think the whole FFI system should be redesigned to remove some of the layers.
Thanks for investigating. It looks like every callback is leaking something like 0x38 bytes of memory which isn't an issue in the short term while I am only developing prototype GTK applications. (An application would need to run for days, if not weeks, for this to cause problems.)
I suspect that it would be possible to call an explicit delete from C on the callback pointer when no longer required. Is that the sort of mechanism you envisaged for deleting callbacks? However, I am happy if you want to hold off any solution as the issue is not a show-stopper.
Phil
On 01/02/12 10:44, Phil Clayton wrote:
On 29/01/12 15:57, David Matthews wrote:
On 27/01/2012 20:15, Phil Clayton wrote:
Thinking about explicitly deleting callbacks led me to ask whether each ML function passed as a callback has its callback reused on subsequent calls. If reuse occurs, the number of call sites in the source code would bound memory usage.
I'm guessing that these callback objects are actually wrappers to invoke compiled ML code and that there is only one instance of the compiled ML, so there would be no problem having only one instance of the callback object.
It would be nice if they could be reused but currently they aren't. I had a look at the code and did some experiments but the problem is that there is quite a lot of ML wrapping going on. That means that the RTS can't detect that the same function has been passed a second time. After the wrapping has been applied the closure that gets passed into the RTS is always different even if the same ML function was used further up.
I sometimes think the whole FFI system should be redesigned to remove some of the layers.
Thanks for investigating. It looks like every callback is leaking something like 0x38 bytes of memory which isn't an issue in the short term while I am only developing prototype GTK applications. (An application would need to run for days, if not weeks, for this to cause problems.)
This may be more of an issue than I first thought. I am looking at some profiling output which is indicating that finalizers are not being called for certain vols. These vols are precisely those that are referenced by functions passed to C as callbacks. Presumably, because callbacks are never garbage collected, the callback functions are always reachable, as is everything that they reference. This would explain why the finalizers are never called for those vols and, presumably, means that no ML functions used as callbacks are garbage collected. If I understand correctly, this means a lot more memory is effectively being leaked than I previously thought.
I think there definitely needs to be a way to delete a callback explicitly so garbage collection can clean up and call finalizers. Poly/ML just can't know how long a callback needs to be kept - the application must tell it. However, can the deletion occur from the C side given the callback function pointer? (Naturally the C side should have no pointer into the ML heap.)
Phil
-
David.Matthews＠prolingua.co.uk -
phil.clayton＠lineone.net