On 04/08/2013 22:31, Dmitry Popoinikov wrote:

Hi, David

Thank you for response. Yes I 'translate' multithreading C application to Poly/ML (and it crashes in these callbacks) and I thought I do something wrong.

Let me ask you one more question? Whether you recommend to use Poly/ML for programs working in a mode 7/24? (Whether it is possible to be sure of that the memory expense never will exceed --maxheap?)

Hi Dmitry, I can't see there would be any problem with running a Poly/ML program for a long time. Provided your application only requires a constant data size there's no reason why the heap should grow beyond a steady state. There will be periodic garbage collections of course.

I've now added code to the SVN version so that a thread created within foreign code can call an ML callback. I've done some simple tests of it but you may want to test it more exhaustively.

Regards, David

On 08/08/2013 07:07, Dmitry Popoinikov wrote:

???????, 6 ??????? 2013, 13:55 +01:00 ?? David Matthews <David.Matthews at prolingua.co.uk>:

...

On 04/08/2013 22:31, Dmitry Popoinikov wrote:

Hi Dmitry, I can't see there would be any problem with running a Poly/ML program for a long time. Provided your application only requires a constant data size there's no reason why the heap should grow beyond a steady state. There will be periodic garbage collections of course.

I've now added code to the SVN version so that a thread created within foreign code can call an ML callback. I've done some simple tests of it but you may want to test it more exhaustively.

Regards, David

Hi David, My program works well now. Thank you very much!

Hi Dmitry, I discovered a problem with the code that I added and I have had to remove some of it. In the callback code there is a check to see if the thread has ever been in ML code before and if not Poly creates some data structures for it which includes an ML stack. The difficulty is in knowing when to remove this data. Threads created by Poly always call an "exit" function and that removes the data for those threads but it's not possible to use that for threads created by foreign code. I added some code to the "main" thread to try to detect when a thread has exited but that caused crashes. It seems that, unlike with Windows, there is no safe way to discover when a thread created with pthreads has exited. The simplest possibility would be to remove the data and the ML stack as soon as the callback returns, if that callback created the data. If your thread is making many calls to the callback that could be expensive. Would that be a problem for your code?

Regards, David

On 13/08/2013 16:00, Dmitry Popoinikov wrote:

Hi David, I've tested my program with r1833. It seems everything is good.

Thanks to the suggestion from Jerry James I've been able to restore the code that detected when a thread had terminated. The problem with 1833 was that if a thread exited in foreign code, and that included threads that were originally created by the foreign code, the ML stack would never be deleted. Also, because the main thread had not detected the thread termination it might not exit properly when OS.Process.exit was called. That has all been fixed in the latest commit (1835).

I wanted to try and get this right not just for your case of threads being created in foreign code but also because this would be necessary if, in the future, there was a mechanism to export ML code as a shared library that could be linked to from a main program written in some other language.

David

On Tue, Aug 13, 2013 at 6:27 AM, David Matthews <David.Matthews at prolingua.co.uk> wrote:

Hi Dmitry, I discovered a problem with the code that I added and I have had to remove some of it. In the callback code there is a check to see if the thread has ever been in ML code before and if not Poly creates some data structures for it which includes an ML stack. The difficulty is in knowing when to remove this data. Threads created by Poly always call an "exit" function and that removes the data for those threads but it's not possible to use that for threads created by foreign code. I added some code to the "main" thread to try to detect when a thread has exited but that caused crashes. It seems that, unlike with Windows, there is no safe way to discover when a thread created with pthreads has exited. The simplest possibility would be to remove the data and the ML stack as soon as the callback returns, if that callback created the data. If your thread is making many calls to the callback that could be expensive. Would that be a problem for your code?

Regards, David

There is a way to detect pthread exit. Create a pthread_key_t with a non-NULL destructor function, and set the value of that key to something non-NULL when a thread enters ML code for the first time. When that thread exits, the destructor function will be called. I whipped up an overly simplistic example to illustrate the approach. (Don't be too critical; I pumped this out pretty quickly.)

#include <errno.h> #include <pthread.h> #include <stdlib.h> #include <stdio.h> #include <time.h>

static pthread_key_t key; static struct timespec sleep_time;

static void usage (const char *progname) { fprintf (stderr, "Usage: %s num_threads\n\ where num_threads is between 0 and 1000, exclusive\n", progname); }

static void key_destructor (void *value) { printf ("This is thread %zu signing off!\n", (size_t) value); }

static void * child_function (void *arg) { pthread_setspecific (key, arg); printf ("Thread %zu sleeping for 1 second\n", (size_t) arg); nanosleep (&sleep_time, NULL); printf ("Thread %zu exiting\n", (size_t) arg); return NULL; }

int main (int argc, char *argv[]) { pthread_t *children; size_t i, num_children; int err;

if (argc != 2) { usage (argv[0]); return EXIT_FAILURE; }

sleep_time.tv_sec = 1L; sleep_time.tv_nsec = 0L;

num_children = atoi (argv[1]); if (num_children < 1 || num_children > 999) { usage (argv[0]); return EXIT_FAILURE; } children = (pthread_t *) calloc (num_children, sizeof (*children)); if (children == NULL) { fputs ("Out of memory.\n", stderr); return EXIT_FAILURE; }

err = pthread_key_create (&key, key_destructor); if (err != 0) { errno = err; printf ("pthread_key_create: %m\n"); free (children); return EXIT_FAILURE; }

for (i = 0U; i < num_children; i++) pthread_create (&children[i], NULL, child_function, (void *)i); for (i = 0U; i < num_children; i++) pthread_join (children[i], NULL); nanosleep (&sleep_time, NULL);

pthread_key_delete (key); free (children); return EXIT_SUCCESS; }

Regards,

06/08/13 13:55, David Matthews wrote:

On 04/08/2013 22:31, Dmitry Popoinikov wrote:

...

Hi, David

Thank you for response. Yes I 'translate' multithreading C application to Poly/ML (and it crashes in these callbacks) and I thought I do something wrong.

Let me ask you one more question? Whether you recommend to use Poly/ML for programs working in a mode 7/24? (Whether it is possible to be sure of that the memory expense never will exceed --maxheap?)

Hi Dmitry, I can't see there would be any problem with running a Poly/ML program for a long time. Provided your application only requires a constant data size there's no reason why the heap should grow beyond a steady state. There will be periodic garbage collections of course.

There was a previous discussion here starting on 2012-01-27 entitled "Lifespan of pointers to callback functions passed via C FFI" where it was noted that a callback is never garbage collected and a new callback object is created each time the ML callback conversion is evaluated. The previous discussion can be found in the list archives for the following months: http://lists.inf.ed.ac.uk/pipermail/polyml/2012-January/thread.html http://lists.inf.ed.ac.uk/pipermail/polyml/2012-February/thread.html http://lists.inf.ed.ac.uk/pipermail/polyml/2012-August/thread.html

So it appears that the heap could grow arbitrarily large, albeit very slowly, if there is no upper bound on the number of callbacks constructed. I suspect that this can be avoided by writing SML to ensure that only one callback is constructed for each SML function passed via the FFI.

Regards, Phil