I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local val call1 = buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) val call2 = buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) val call3 = buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix_alloc(size1, size2) val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) in ptr end fun gsl_mexp (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix arr val ptr2 = gsl_matrix_alloc(size1, size2) val _ = call3(ptr, ptr2, 0.01) in ptr2 end end; fun main() = let val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; val ptr = gsl_matrix(a); val m = gsl_mexp(a) (* fails *) in () end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error
Hi Mark,
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't define it), you could let the dynamic linker look for it by using an empty string for the library which has the effect of passing RTLD_DEFAULT to dlsym.
val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so \ poly
which could be put into a wrapper script. Or you could build an executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more portable because shared object names and their paths vary across systems. Linking also avoids the problem of specifying the library names because you use flags such as -lgsl and the linker works it out. Linking can be done in an even more portable way using a utility like pkg-config to generate the linker arguments:
pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards, Phil
On 09/09/18 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local val call1 = buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) val call2 = buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) val call3 = buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix_alloc(size1, size2) val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) in ptr end fun gsl_mexp (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix arr val ptr2 = gsl_matrix_alloc(size1, size2) val _ = call3(ptr, ptr2, 0.01) in ptr2 end end; fun main() = let val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; val ptr = gsl_matrix(a); val m = gsl_mexp(a) (* fails *) in () end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
Slight correction: using an empty string for the library has the effect of passing NULL to dlopen which means the handle passed to dlsym is the current program.
On 10/09/18 15:29, Phil Clayton wrote:
Hi Mark,
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't define it), you could let the dynamic linker look for it by using an empty string for the library which has the effect of passing RTLD_DEFAULT to dlsym.
? val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so \ ? poly
which could be put into a wrapper script.? Or you could build an executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability.? The SML code is more portable because shared object names and their paths vary across systems.? Linking also avoids the problem of specifying the library names because you use flags such as -lgsl and the linker works it out. Linking can be done in an even more portable way using a utility like pkg-config to generate the linker arguments:
? pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards, Phil
On 09/09/18 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local ???? val call1 =??? buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) ???? val call2 =??? buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) ???? val call3 =??? buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = ???? let ???? val size1 = Array2.nRows arr ???? val size2 = Array2.nCols arr ???? val ptr = gsl_matrix_alloc(size1, size2) ???? val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) ???? in ???? ptr ???? end fun gsl_mexp (arr : real Array2.array) = ???? let ???? val size1 = Array2.nRows arr ???? val size2 = Array2.nCols arr ???? val ptr = gsl_matrix arr ???? val ptr2 = gsl_matrix_alloc(size1, size2) ???? val _ = call3(ptr, ptr2, 0.01) ???? in ???? ptr2 ???? end end; fun main() = ???? let ???? val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; ???? val ptr = gsl_matrix(a); ???? val m = gsl_mexp(a) (* fails *) ???? in ???? () ???? end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
Hello. What is the difference between (LoadLibrary "") and (loadExecutable ()) calls?
2018-09-10 17:53 GMT+03:00 Phil Clayton <phil.clayton at veonix.com>:
Slight correction: using an empty string for the library has the effect of passing NULL to dlopen which means the handle passed to dlsym is the current program.
On 10/09/18 15:29, Phil Clayton wrote:
Hi Mark,
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't define it), you could let the dynamic linker look for it by using an empty string for the library which has the effect of passing RTLD_DEFAULT to dlsym.
val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so \ poly
which could be put into a wrapper script. Or you could build an executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more portable because shared object names and their paths vary across systems. Linking also avoids the problem of specifying the library names because you use flags such as -lgsl and the linker works it out. Linking can be done in an even more portable way using a utility like pkg-config to generate the linker arguments:
pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards, Phil
On 09/09/18 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local val call1 = buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) val call2 = buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) val call3 = buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix_alloc(size1, size2) val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) in ptr end fun gsl_mexp (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix arr val ptr2 = gsl_matrix_alloc(size1, size2) val _ = call3(ptr, ptr2, 0.01) in ptr2 end end; fun main() = let val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; val ptr = gsl_matrix(a); val m = gsl_mexp(a) (* fails *) in () end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
Looking at polyffi.cpp it actually looks like `loadLibrary ""` ends up calling dlopen ("", ...) whereas `loadExecutable ()` ends up calling dlopen (NULL, ...) A couple of things I have read suggest that these often give the same result, if not always. So for non-Windows platforms `loadLibrary ""` and `loadExecutable ()` look similar. However, the dlopen man page doesn't mention the empty string case, so it seems preferable to use `loadExecutable ()` over `loadLibrary ""`.
For Windows, I don't know what LoadLibrary("") does. So again it seems preferable to use `loadExecutable ()` over `loadLibrary ""`.
Phil
On 11/09/18 04:15, Kostirya wrote:
Hello. What is the difference between (LoadLibrary "") and (loadExecutable ()) calls?
2018-09-10 17:53 GMT+03:00 Phil Clayton <phil.clayton at veonix.com>:
Slight correction: using an empty string for the library has the effect of passing NULL to dlopen which means the handle passed to dlsym is the current program.
On 10/09/18 15:29, Phil Clayton wrote:
Hi Mark,
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't define it), you could let the dynamic linker look for it by using an empty string for the library which has the effect of passing RTLD_DEFAULT to dlsym.
val gsl = loadLibrary "";To make sure these libraries are searched, you would need to start your poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so \ poly
which could be put into a wrapper script. Or you could build an executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more portable because shared object names and their paths vary across systems. Linking also avoids the problem of specifying the library names because you use flags such as -lgsl and the linker works it out. Linking can be done in an even more portable way using a utility like pkg-config to generate the linker arguments:
pkg-config --libs gsl(This usual requires the -devel or -dev package for the library.)
Regards, Phil
On 09/09/18 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local val call1 = buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) val call2 = buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) val call3 = buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix_alloc(size1, size2) val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) in ptr end fun gsl_mexp (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix arr val ptr2 = gsl_matrix_alloc(size1, size2) val _ = call3(ptr, ptr2, 0.01) in ptr2 end end; fun main() = let val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; val ptr = gsl_matrix(a); val m = gsl_mexp(a) (* fails *) in () end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
In fact `loadLibrary ""` doesn't work on Darwin, `loadExecutable ()` is required.
Phil
On 11/09/18 21:50, Phil Clayton wrote:
Looking at polyffi.cpp it actually looks like `loadLibrary ""` ends up calling ? dlopen ("", ...) whereas `loadExecutable ()` ends up calling ? dlopen (NULL, ...) A couple of things I have read suggest that these often give the same result, if not always.? So for non-Windows platforms `loadLibrary ""` and `loadExecutable ()` look similar.? However, the dlopen man page doesn't mention the empty string case, so it seems preferable to use `loadExecutable ()` over `loadLibrary ""`.
For Windows, I don't know what LoadLibrary("") does.? So again it seems preferable to use `loadExecutable ()` over `loadLibrary ""`.
Phil
On 11/09/18 04:15, Kostirya wrote:
Hello. What is the difference between (LoadLibrary "") and (loadExecutable ()) calls?
2018-09-10 17:53 GMT+03:00 Phil Clayton <phil.clayton at veonix.com>:
Slight correction: using an empty string for the library has the effect of passing NULL to dlopen which means the handle passed to dlsym is the current program.
On 10/09/18 15:29, Phil Clayton wrote:
Hi Mark,
? > (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined ? > symbol: cblas_dgemm *)
To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't define it), you could let the dynamic linker look for it by using an empty string for the library which has the effect of passing RTLD_DEFAULT to dlsym.
??? val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so
\ ??? poly
which could be put into a wrapper script.? Or you could build an executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability.? The SML code is more portable because shared object names and their paths vary across systems. Linking also avoids the problem of specifying the library names because you use flags such as -lgsl and the linker works it out. Linking can be done in an even more portable way using a utility like pkg-config to generate the linker arguments:
??? pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards, Phil
On 09/09/18 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local ????? val call1 =??? buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) ????? val call2 =??? buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) ????? val call3 =??? buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = ????? let ????? val size1 = Array2.nRows arr ????? val size2 = Array2.nCols arr ????? val ptr = gsl_matrix_alloc(size1, size2) ????? val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) ????? in ????? ptr ????? end fun gsl_mexp (arr : real Array2.array) = ????? let ????? val size1 = Array2.nRows arr ????? val size2 = Array2.nCols arr ????? val ptr = gsl_matrix arr ????? val ptr2 = gsl_matrix_alloc(size1, size2) ????? val _ = call3(ptr, ptr2, 0.01) ????? in ????? ptr2 ????? end end; fun main() = ????? let ????? val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; ????? val ptr = gsl_matrix(a); ????? val m = gsl_mexp(a) (* fails *) ????? in ????? () ????? end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
loadLibrary "" doesn't work in Windows either. It returns an error 87 - invalid parameter.
David
On 14/10/2018 15:07, Phil Clayton wrote:
In fact `loadLibrary ""` doesn't work on Darwin, `loadExecutable ()` is required.
Phil
On 11/09/18 21:50, Phil Clayton wrote:
Looking at polyffi.cpp it actually looks like `loadLibrary ""` ends up calling ?? dlopen ("", ...) whereas `loadExecutable ()` ends up calling ?? dlopen (NULL, ...) A couple of things I have read suggest that these often give the same result, if not always.? So for non-Windows platforms `loadLibrary ""` and `loadExecutable ()` look similar.? However, the dlopen man page doesn't mention the empty string case, so it seems preferable to use `loadExecutable ()` over `loadLibrary ""`.
For Windows, I don't know what LoadLibrary("") does.? So again it seems preferable to use `loadExecutable ()` over `loadLibrary ""`.
Phil
On 11/09/18 04:15, Kostirya wrote:
Hello. What is the difference between (LoadLibrary "") and (loadExecutable ()) calls?
2018-09-10 17:53 GMT+03:00 Phil Clayton <phil.clayton at veonix.com>:
Slight correction: using an empty string for the library has the effect of passing NULL to dlopen which means the handle passed to dlsym is the current program.
On 10/09/18 15:29, Phil Clayton wrote:
Hi Mark,
? > (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined ? > symbol: cblas_dgemm *)
To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't define it), you could let the dynamic linker look for it by using an empty string for the library which has the effect of passing RTLD_DEFAULT to dlsym.
??? val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so
\ ??? poly
which could be put into a wrapper script.? Or you could build an executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability.? The SML code is more portable because shared object names and their paths vary across systems. Linking also avoids the problem of specifying the library names because you use flags such as -lgsl and the linker works it out. Linking can be done in an even more portable way using a utility like pkg-config to generate the linker arguments:
??? pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards, Phil
On 09/09/18 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local ????? val call1 =??? buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) ????? val call2 =??? buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) ????? val call3 =??? buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = ????? let ????? val size1 = Array2.nRows arr ????? val size2 = Array2.nCols arr ????? val ptr = gsl_matrix_alloc(size1, size2) ????? val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) ????? in ????? ptr ????? end fun gsl_mexp (arr : real Array2.array) = ????? let ????? val size1 = Array2.nRows arr ????? val size2 = Array2.nCols arr ????? val ptr = gsl_matrix arr ????? val ptr2 = gsl_matrix_alloc(size1, size2) ????? val _ = call3(ptr, ptr2, 0.01) ????? in ????? ptr2 ????? end end; fun main() = ????? let ????? val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; ????? val ptr = gsl_matrix(a); ????? val m = gsl_mexp(a) (* fails *) ????? in ????? () ????? end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
Excellent - thank you. I now have a script called poly_gsl:
#!/bin/bash LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so:/usr/lib/x86_64-linux-gnu/libgslcblas.so poly $*
-- Mark
On 09/10/2018 04:37 PM, Phil Clayton wrote:
Hi Mark,
(* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
To avoid looking for the symbol "cblas_dgemm" in libgsl (which doesn't define it), you could let the dynamic linker look for it by using an empty string for the library which has the effect of passing RTLD_DEFAULT to dlsym.
val gsl = loadLibrary "";
To make sure these libraries are searched, you would need to start your poly session with something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgsl.so":"/usr/lib/x86_64-linux-gnu/libgslcblas.so \ poly
which could be put into a wrapper script. Or you could build an executable (which could be a poly REPL) and link to them.
Another advantage of this approach is portability. The SML code is more portable because shared object names and their paths vary across systems. Linking also avoids the problem of specifying the library names because you use flags such as -lgsl and the linker works it out. Linking can be done in an even more portable way using a utility like pkg-config to generate the linker arguments:
pkg-config --libs gsl
(This usual requires the -devel or -dev package for the library.)
Regards, Phil
On 09/09/18 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local val call1 = buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) val call2 = buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) val call3 = buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix_alloc(size1, size2) val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) in ptr end fun gsl_mexp (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix arr val ptr2 = gsl_matrix_alloc(size1, size2) val _ = call3(ptr, ptr2, 0.01) in ptr2 end end; fun main() = let val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; val ptr = gsl_matrix(a); val m = gsl_mexp(a) (* fails *) in () end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
I've been thinking about adding direct external references as an alternative to dynamic loading of libraries. The mechanism is all there now because the Poly/ML run-time library is linked to the exported code through external references.
I've added Foreign.externalFunctionSymbol : string -> symbol This creates an external reference which is filled in when the code is exported. The actual value is resolved by linking with the appropriate library. For your example that means changing it to replace references to getSymbol to externalFunctionSymbol and removing the calls to loadLibrary. i.e. open Foreign;
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local val call1 = buildCall2((externalFunctionSymbol "gsl_matrix_calloc"), (cInt,cInt), cPointer) val call2 = buildCall4((externalFunctionSymbol "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) val call3 = buildCall3((externalFunctionSymbol "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
It seems to handle your example fine when the code is exported using "polyc" and then linked with the libraries.
I've committed this to a separate branch "ExternalBranch" on GitHub rather than "master". I'd like some feedback on how useful this is and whether there are any problems before merging it.
David
On 09/09/2018 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local val call1 = buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) val call2 = buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) val call3 = buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix_alloc(size1, size2) val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) in ptr end fun gsl_mexp (arr : real Array2.array) = let val size1 = Array2.nRows arr val size2 = Array2.nCols arr val ptr = gsl_matrix arr val ptr2 = gsl_matrix_alloc(size1, size2) val _ = call3(ptr, ptr2, 0.01) in ptr2 end end; fun main() = let val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; val ptr = gsl_matrix(a); val m = gsl_mexp(a) (* fails *) in () end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
Hi David,
Thanks - this is useful. At the very least it will point out missing dependencies at link time rather than run time. That is, provided it is used correctly - see the issue below.
I'm glad to see that Poly/ML does not include symbols in the object file that are referenced from ML but not required, e.g. because they occur in code that is never reached. That's a useful property because it allows a library of ML bindings to support more functions than actually provided by the installed version of a binary library and required by the application. Hopefully it will be possible to keep that property.
I have run into a general issue. After redefining a local version of the getSymbol function to use externalFunctionSymbol internally, I found that programs compiled and linked fine but seg faulted before entering the main ML function. This was due to the scope in which a symbol was declared using externalFunctionSymbol. The attached example call_c_test_13.tar.gz gives a simple example of the problem. The external function symbol isn't being declared until run time so this code is inadvertently depending on a feature of dynamic linking. I can see that use of externalFunctionSymbol is inherently more restricted that getSymbol: externalFunctionSymbol must be applied to a symbol name at compile time but the symbol cannot be used until run time, post-linking. Are there any sort of checks that could be done to warn users if they get this wrong? I can see it's not straightforward because run time is itself compile time for an exported function that is subsequently linked. At the very least, could an exception be raised rather than seg faulting if there is an attempt to use an unresolved symbol?
Finally, is there a reason you chose the name "externalFunctionSymbol" rather than, say, "externalSymbol"? It seems to work fine for non-functions too - see attached example call_c_test_14.tar.gz .
Phil
On 12/09/18 21:18, David Matthews wrote:
I've been thinking about adding direct external references as an alternative to dynamic loading of libraries.? The mechanism is all there now because the Poly/ML run-time library is linked to the exported code through external references.
I've added Foreign.externalFunctionSymbol : string -> symbol This creates an external reference which is filled in when the code is exported.? The actual value is resolved by linking with the appropriate library.? For your example that means changing it to replace references to getSymbol to externalFunctionSymbol and removing the calls to loadLibrary.? i.e. open Foreign;
type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local ??? val call1 =??? buildCall2((externalFunctionSymbol "gsl_matrix_calloc"), (cInt,cInt), cPointer) ??? val call2 =??? buildCall4((externalFunctionSymbol "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) ??? val call3 =??? buildCall3((externalFunctionSymbol "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2)
It seems to handle your example fine when the code is exported using "polyc" and then linked with the libraries.
I've committed this to a separate branch "ExternalBranch" on GitHub rather than "master".? I'd like some feedback on how useful this is and whether there are any problems before merging it.
David
On 09/09/2018 21:15, Mark Clements wrote:
I am enjoying using polyml's Foreign structure.
However, in using the GNU Scientific Library (GSL), I have run into a problem with linkage. The GSL library (libgsl.so) calls standard cBLAS matrix functions with unknown symbols to allow the user to link with different cBLAS implementations (NB: GSL provides an unoptimised cBLAS implementation named libgslcblas.so). When I call a GSL function that depends on the cBLAS functions, I get a symbol not defined error.
I have tried unsuccessfully (a) loading the libgslcblas library either before or after loading libgsl, (b) changing the loadLibrary implementation to use dlopen with RTLD_LAZY | RTLD_GLOBAL, and (c) compiling the SML code to an object file and then linking against the libraries.
How could this be made to work?
Sincerely, Mark.
PS test code for (a) is:
open Foreign; val gsl = loadLibrary "/usr/lib/x86_64-linux-gnu/libgsl.so"; val gslcblas = loadLibrary "/usr/lib/x86_64-linux-gnu/libgslcblas.so"; type gsl_matrix_t = Memory.voidStar; (* for demonstration only *) local ???? val call1 =??? buildCall2((getSymbol gsl "gsl_matrix_calloc"), (cInt,cInt), cPointer) ???? val call2 =??? buildCall4((getSymbol gsl "gsl_matrix_set"), (cPointer,cInt,cInt,cDouble), cVoid) ???? val call3 =??? buildCall3((getSymbol gsl "gsl_linalg_exponential_ss"), (cPointer, cPointer, cDouble), cVoid) in fun gsl_matrix_alloc (size1,size2) = call1(size1,size2) fun gsl_matrix_set (ptr : gsl_matrix_t, i, j, x) = call2(ptr, i, j, x) fun gsl_matrix (arr : real Array2.array) = ???? let ???? val size1 = Array2.nRows arr ???? val size2 = Array2.nCols arr ???? val ptr = gsl_matrix_alloc(size1, size2) ???? val _ = Array2.tabulate Array2.RowMajor (size1, size2, fn (i,j) => gsl_matrix_set(ptr, i, j, Array2.sub(arr, i, j))) ???? in ???? ptr ???? end fun gsl_mexp (arr : real Array2.array) = ???? let ???? val size1 = Array2.nRows arr ???? val size2 = Array2.nCols arr ???? val ptr = gsl_matrix arr ???? val ptr2 = gsl_matrix_alloc(size1, size2) ???? val _ = call3(ptr, ptr2, 0.01) ???? in ???? ptr2 ???? end end; fun main() = ???? let ???? val a = Array2.fromList [[1.0,2.0],[3.0,4.0]]; ???? val ptr = gsl_matrix(a); ???? val m = gsl_mexp(a) (* fails *) ???? in ???? () ???? end; (* symbol lookup error: /usr/lib/x86_64-linux-gnu/libgsl.so: undefined symbol: cblas_dgemm *)
For (c):
polyc -c test.sml g++ test.o -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas `pkg-config --libs polyml` ./a.out # same error _______________________________________________ polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
Hi Phil, Thanks for that. I've made some changes and if it all seems to work out I'll merge it into master. I've added some comments below.
On 05/10/2018 09:23, Phil Clayton wrote:
I'm glad to see that Poly/ML does not include symbols in the object file that are referenced from ML but not required, e.g. because they occur in code that is never reached.? That's a useful property because it allows a library of ML bindings to support more functions than actually provided by the installed version of a binary library and required by the application.? Hopefully it will be possible to keep that property.
External symbols are ML cells that are recognised specially by PolyML.export and turned into linker symbols. Since PolyML.export only exports reachable ML data the resulting object file will only contain symbols that are part of the exported code. A consequence of this is that if the object file is then linked with a static library version of the C code the final executable will only contain the C code that is actually required. An executable created using external symbols and a statically linked library will generally be more compact than having a separate dynamic library. It also avoids the need for path specifications such as LD_LIBRARY_PATH.
I have run into a general issue.? After redefining a local version of the getSymbol function to use externalFunctionSymbol internally, I found that programs compiled and linked fine but seg faulted before entering the main ML function.? This was due to the scope in which a symbol was declared using externalFunctionSymbol.? The attached example call_c_test_13.tar.gz gives a simple example of the problem.? The external function symbol isn't being declared until run time so this code is inadvertently depending on a feature of dynamic linking.? I can see that use of externalFunctionSymbol is inherently more restricted that getSymbol: externalFunctionSymbol must be applied to a symbol name at compile time but the symbol cannot be used until run time, post-linking.? Are there any sort of checks that could be done to warn users if they get this wrong??? I can see it's not straightforward because run time is itself compile time for an exported function that is subsequently linked.? At the very least, could an exception be raised rather than seg faulting if there is an attempt to use an unresolved symbol?
I've been thinking about that. The symbol is created with a zero address. The actually address is set by the linker. It now raises an exception if it tries to use a symbol when the value is zero.
Finally, is there a reason you chose the name "externalFunctionSymbol" rather than, say, "externalSymbol"?? It seems to work fine for non-functions too - see attached example call_c_test_14.tar.gz .
On the X86 and most platforms the same linker symbols are used for both function code and data. There are a very few platforms that use different relocations for these. See commit 76b36c9d. I've added externalDataSymbol for data.
Regards, David
Hi David,
Thanks for the explanation and updates. I have tested this, including externalDataSymbol, and all appears to work on my Linux and Darwin systems.
Initially I found that I wasn't seeing the new Foreign exception for an uninitalized external symbol, although there was no seg fault. It turns out that was actually because the exception was occurring in a function registered with PolyML.onEntry. It seems that PolyML.onEntry silently suppresses any exceptions from the functions it runs. Is that intended?
I have been wondering if there is a way to have a local version of external(Function|Data)Symbol that falls back on getSymbol (loadExecutable ()) if the functions aren't available. Could this be achieved by using one of two files depending on the value of PolyML.rtsVersion () ?
Thanks, Phil
On 07/10/18 12:44, David Matthews wrote:
Hi Phil, Thanks for that.? I've made some changes and if it all seems to work out I'll merge it into master.? I've added some comments below.
On 05/10/2018 09:23, Phil Clayton wrote:
I'm glad to see that Poly/ML does not include symbols in the object file that are referenced from ML but not required, e.g. because they occur in code that is never reached.? That's a useful property because it allows a library of ML bindings to support more functions than actually provided by the installed version of a binary library and required by the application.? Hopefully it will be possible to keep that property.
External symbols are ML cells that are recognised specially by PolyML.export and turned into linker symbols.? Since PolyML.export only exports reachable ML data the resulting object file will only contain symbols that are part of the exported code.? A consequence of this is that if the object file is then linked with a static library version of the C code the final executable will only contain the C code that is actually required.? An executable created using external symbols and a statically linked library will generally be more compact than having a separate dynamic library.? It also avoids the need for path specifications such as LD_LIBRARY_PATH.
I have run into a general issue.? After redefining a local version of the getSymbol function to use externalFunctionSymbol internally, I found that programs compiled and linked fine but seg faulted before entering the main ML function.? This was due to the scope in which a symbol was declared using externalFunctionSymbol.? The attached example call_c_test_13.tar.gz gives a simple example of the problem. The external function symbol isn't being declared until run time so this code is inadvertently depending on a feature of dynamic linking. I can see that use of externalFunctionSymbol is inherently more restricted that getSymbol: externalFunctionSymbol must be applied to a symbol name at compile time but the symbol cannot be used until run time, post-linking.? Are there any sort of checks that could be done to warn users if they get this wrong??? I can see it's not straightforward because run time is itself compile time for an exported function that is subsequently linked.? At the very least, could an exception be raised rather than seg faulting if there is an attempt to use an unresolved symbol?
I've been thinking about that.? The symbol is created with a zero address.? The actually address is set by the linker.? It now raises an exception if it tries to use a symbol when the value is zero.
Finally, is there a reason you chose the name "externalFunctionSymbol" rather than, say, "externalSymbol"?? It seems to work fine for non-functions too - see attached example call_c_test_14.tar.gz .
On the X86 and most platforms the same linker symbols are used for both function code and data.? There are a very few platforms that use different relocations for these.? See commit 76b36c9d.? I've added externalDataSymbol for data.
Regards, David
polyml mailing list polyml at inf.ed.ac.uk http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
Hi Phil,
On 14/10/2018 15:03, Phil Clayton wrote:
Hi David,
Thanks for the explanation and updates.? I have tested this, including externalDataSymbol, and all appears to work on my Linux and Darwin systems.
Initially I found that I wasn't seeing the new Foreign exception for an uninitalized external symbol, although there was no seg fault.? It turns out that was actually because the exception was occurring in a function registered with PolyML.onEntry.? It seems that PolyML.onEntry silently suppresses any exceptions from the functions it runs.? Is that intended?
That's how the code works at the moment. I guess it could call OS.Process.exit OS.Process.failure but there's not much else it could do at that point.
I have been wondering if there is a way to have a local version of external(Function|Data)Symbol that falls back on ? getSymbol (loadExecutable ()) if the functions aren't available.? Could this be achieved by using one of two files depending on the value of PolyML.rtsVersion () ?
Using PolyML.rtsVersion is probably the best way. I've updated the RTS version in Git. Usually it only gets done when the new version is released but probably it should be updated at the same time as the compiler version: as soon as something changes in Git.
David
-
David.Matthews＠prolingua.co.uk -
kostirya＠gmail.com -
mark.clements＠ki.se -
phil.clayton＠veonix.com