Poly/ML

polyml@lists.polyml.org

2 participants
698 discussions

FW: Memory Management across multiple threads using Futures Library
by michael.bradley＠hotmail.co.uk 29 Apr '13

29 Apr '13

Dear Lucas and Makarius,I have had some time to look at the debugging results, although I am not sure how to proceed. I have attached various files to show the output during multiple runs of the algorithm. (Each output was generated by a separate execution of the same function call)I believe that the heap's allocation of space adjustments are the key behaviour causing the process to slow down. In scenarios where the performance seems good, the "heapsize" debug report does not print anything. However, for a poor case, the space changes as depicted in the graph (22 times).As the GC reports suggest, the algorithm does produce a substantial amount of "garbage" due to its high rate of turn over (original description below), and I believe that the GC's very successful runs are causing the process to continually:Set the size of the heapRun the GC. Clears most of the heap, so reconsiders the size.The heap fills very quickly again and is forced to grow back.Possibly? I am not sure if this is important, but the GC reports a single stack overflow:...GC: Mark: Stack overflow. Rescan for 0x7f4f9a404b88GC: Mark: Rescanning from 0x7f55103fa6b0 to 0x7f55104ec540GC: Mark: Rescanning from 0x7f5023b68718 to 0x7f5023c48fc8...very early on in the algorithm's execution.Any advice you have is appreciated!Thanks again and best wishes,Michael Dear all, I am experiencing inefficiency while using the Futures library within PolyML. Parallel threads are being forced to 'sleep' for several seconds at a time for no clear reason, and I suspect it may be due to memory management between multiple threads. The test machine has 80GB RAM and 24 available cores. The parallel process works in roughly the same way as the Bag of Tasks Model with Replacements: (I'll refer to the distributor of tasks as the Controller, and other threads as Workers) 1. The Controller constructs an initial list of tasks.2. It sends the tasks to Workers in parallel, distributing across cores appropriately. Tasks are spawned by Future calls. They return a mixture of new tasks and results.3. It synchronises on all of the spawned tasks, organises them and repeats from (2) if any tasks remain. Tasks require a modest amount of memory and significant time to run, but there are 100,000s of them. The memory consumed can rise by 10GB in a few seconds but does not exceed the available RAM, and is held almost entirely by the Controller. The process can take minutes to run, but I see that spawned Futures in the program are required to sleep for the majority (~75%) of the time. The turnover in memory is very high as well, especially in the Controller thread, so calls to the garbage collector can reduce the RAM required dramatically. Initially, as the original problem size grows the speedup via parallelisation rises to a peak of around 6 times, requiring only a few GBs of RAM. However, as the problem size continues to rise and the problem should be easier to parallelise, the speedup falls to the point where a sequential algorithm evaluating tasks one at a time is more efficient. Artificial tests show that when virtually no memory is required, a speedup of over 10 times is quite easy to achieve. What I believe might be causing this is the allocation of segment sizes to threads in the heap. I believe that calls to the common memory pool to change the segment size are causing all threads to sleep unnecessarily. Does each thread currently have its own independent adaptive segment size, or are all threads required to use the same segment size? Do all threads have to pause if one segment size is being changed?I have timed the synchronisation phase above and it requires 10-100s of microseconds typically, so not accounting for the regular sleep times of several seconds. I know the threads are being paused mid-task computation, and I have tried increasing the heap size as well to no effect. Does anyone know what might be causing this behaviour?Best regards,Michael

1 0

Can't build mlton-20100608 with polyml on ARM
by adam＠spicenitz.org 27 Apr '13

27 Apr '13

Hi, I am trying to bootstrap mlton via polyml. This is working well on x86_64, but failing on ARM. I am getting this error: Exception- InternalError: fixupOffset: jump too far (16-bit offset) raised while compiling Exception- Fail "Exception- InternalError: fixupOffset: jump too far (16-bit offset) raised while compiling" raised Exception- Fail "Exception- InternalError: fixupOffset: jump too far (16-bit offset) raised while compiling" raised > # Error-Structure (Main) has not been declared Found near PolyML.shareCommonData Main.mainWrapped PolyML.export ("mlton-polyml", Main.mainWrapped) Error-Type error in function application. Function: PolyML.shareCommonData Main.mainWrapped : unit Argument: PolyML.export : string * (unit -> unit) -> unit Reason: Value being applied does not have a function type Found near PolyML.shareCommonData Main.mainWrapped PolyML.export ("mlton-polyml", Main.mainWrapped) Error-Structure (Main) has not been declared Found near PolyML.shareCommonData Main.mainWrapped PolyML.export ("mlton-polyml", Main.mainWrapped) Static Errors This seems to be a fundamental limitation of the interpreter. Is there any kind of workaround? Could the interpreter be extended to allow for longer jumps? Thanks, Adam

2 7

'Make_unsigned' was not declared in this scope
by mike.maul＠gmail.com 23 Apr '13

23 Apr '13

Getting this from SVN HEAD xwindows.cpp:2423:22: error: 'Make_unsigned' was not declared in this scope

2 1

type variable lexical syntax
by michael.norrish＠nicta.com.au 23 Apr '13

23 Apr '13

Is the following really supposed to be acceptable? Both Poly/ML and mlton accept it without complaining: datatype '+ = C | D; (Moscow ML rejects it. mlton and Poly/ML treat it as the declaration of a unary type operator called +.) and what about datatype '' foo = C | D; Michael

3 5

PolyML.Exception.traceException
by David.Matthews＠prolingua.co.uk 22 Apr '13

22 Apr '13

For some time Makarius has been asking for a way for PolyML.exception_trace to be able to output the trace through an ML function rather than directly to stdout. I've now added PolyML.Exception.traceException. At the same time I've moved the various functions related to exceptions into PolyML.Exception although for the time being they are still available directly in PolyML. traceException has the signature val traceException = fn: (unit -> 'a) * (string list * exn -> 'a) -> 'a The first argument is the function to run and if this returns without an exception the result is returned from traceException. If it raises an exception the second argument is called with the trace and the original exception. This can then reraise the exception after producing the trace or return a default result. As always I can and may change it before the next release. By the way, it requires the updated compiler so it isn't there until "make compiler" has been run. David

1 0

Call for Participation: MMnet'13: Language and Runtime Support for Concurrent Systems, Wednesday 8th May 2013, Heriot-Watt University, Edinburgh
by H.W.Loidl＠hw.ac.uk 19 Apr '13

19 Apr '13

[ (n-1) apologies for n copies; Registration is open (deadline Mon 29.4.); Slots for presentations are still available -- HWL ] 2013 MMnet Workshop MMnet'13: Language and Runtime Support for Concurrent Systems http://www.macs.hw.ac.uk/~dsg/events/mmnet13.html Wednesday 8th May 2013, Heriot-Watt University, Edinburgh Call for participation ---------------------- The interaction between memory management and concurrency poses a particular challenge in exploiting modern, increasingly parallel hardware. Both language support and runtime-system support for concurrency needs to be integrated with highly optimised memory management policies in order to provide a high level programming model to the programmer. This one-day workshop aims to bring together UK academia and industry with the aim of establishing new links and enhancing existing collaboration within the framework of the UK Memory Management Network. The UK Memory Management Network provides researchers from academia and industry with a unique opportunity to meet, to discuss challenges, state-of-the-art techniques and solutions, and to exchange ideas and experiences. MMNet'13 will take place at Heriot-Watt University, Edinburgh, on Wednesday 8th May 2013. Participation is welcomed from both academic and industrial sectors. Topics of interest include but are not limited to: * virtual machine implementation * garbage collection algorithms * performance tuning techniques * empirical studies of dynamic behaviour * abstractions and models for memory management * runtime support for concurrency * language support for concurrency * transactional memory As with previous workshops, the day will include a mixture of presentations and time for discussion of ongoing work and future research directions. Note that since this is an informal workshop, an MMNet presentation will not prevent publication elsewhere. A list of previous workshops gives an indication of topics covered in this series: http://www.mm-net.org.uk/ Workshop venue -------------- When: Wednesday 8th May 2013, 10:30-17:30 Where: Room 1.82 (1st floor, near the School Office), Earl Mountbatten Buildings, School of Mathemtical and Computer Sciences, Heriot-Watt University, Edinburgh EH14 4AS Registration ------------ Attendance of MMnet'13 is free thanks to sponsorship of MMnet by EPSRC. Lunch and light refreshments will be provided. However, participants must make their own travel and accommodation arrangements. You can register for the workshop through the MMnet'13 web page. Talk slots will be allocated on a first-come first-served basis. If you would like to give a talk, please indicate this when registering for the workshop, and indicate whether you want to offer a short (15 min) or normal (30 min) slot. We will collect presentations from the workshop to form an informal online proceedings. We do not consider these to be a formal (citable) publication so that any work in progress presented here may be submitted later for formal publication. Contact details --------------- If you have any questions about the workshop or the MMnet network (http://www.mm-net.org.uk/) please contact: * Hans-Wolfgang Loidl, Heriot-Watt University (http://www.macs.hw.ac.uk/~hwloidl/) * Richard Jones, University of Kent (http://www.cs.kent.ac.uk/people/staff/rej/)

1 0

missing shared libraries
by gbuday＠gmail.com 18 Apr '13

18 Apr '13

Hi, I got the following running Isabelle: ### Cannot execute Poly/ML in 32bit mode (missing shared libraries for C/C++) ### Using bulky 64bit version of Poly/ML instead What are these shared libraries that I need? I have glibc.i686 already. - Gergely

3 2

suppressing compiler output
by gbuday＠gmail.com 18 Apr '13

18 Apr '13

Hi, I would like to feed an sml program into poly from standard input: $ cat hello.sml |poly Poly/ML 5.4.1 Release > # Hello World!val it = (): unit Is it possible to use this so that the compiler itself does not print anything? I have found poly -q which does not print the release message but that still prints all the rest. - Gergely

11 41

The rules of polymorphism
by aleks0＠gmail.com 15 Apr '13

15 Apr '13

This is more of a standard ML question, but I figure folks here will know. I've always found the situations in which poly/ML allows polymorphic values (versus forcing a monotype) a bit puzzling. Take this bit of code, for instance: datatype 'a box = Empty | Box of 'a datatype 'a wrap = Wrap of 'a val box = Box val wrap = Wrap val empty = Empty (* first situation *) val x = Wrap empty (* polymorphic value : 'a box wrap *) val y = wrap empty (* monotype : _a box wrap *) (* second situation *) fun wrapbox1 x = wrap (box x) (* : 'a -> 'a box wrap *) val wrapbox2 = wrap o box (* : _a -> _a box wrap *) In the first situation, its clear that SML treats a type constructor differently than it's associated function, which makes sense. I've always treated the "val foo = Foo" syntax as syntactic sugar for "fun foo x = Foo x". Is this right, or only approximately right? In the second situation, it seems like the expression "val wrapbox2 = wrap o box" is forcing "wrap o box" to be executed "too soon", so the compiler has no choice but to give it a monotype. For instance, changing the expression to "fn () => (wrap o box)" preserves the polymorphism. So, my main question: Is there a reasonable/memorable set of rules for determining when an expression will produce a polymorphic value as opposed to a monotype? Is this written down somewhere (e.g. in the SML 97 spec/commentary)? A second, more specific question: Is there a way for values resulting from a function call to maintain their polymorphism? That is, can I produce a polymorphic empty value of type " 'a box wrap " using just "wrap : 'a -> 'a wrap" and "empty : 'a box"? Where everything is at the top-level, this doesn't seem necessary. However, with a bunch of structures and abstract types around, I've run into this dead end before. Is it possible to get something like "val empty = Foo.mk (Bar.empty, Baz.empty)", with "Bar.empty : 'a bar", and "Baz.empty : 'a baz" and "val empty : ('a bar, 'b baz) foo"? The only solutions I've found are ditching the nice abstract type Foo.T by moving the datatype def to the signature (boo!) or replacing "val empty" with "fun empty () = ..." and needing to write "(empty ())" everywhere (double boo!). Does anyone know a nice way around this?

2 2

suppressing compiler output
by David.Matthews＠prolingua.co.uk 15 Apr '13

15 Apr '13

Alex, I don't know if you intended to send this to the list as well but I'm copying it back there because I think it might be of general interest. While I was playing around with the polyc script I wondered about adding a --usestring option, similar to --use but with the actual ML in there rather than the file name. Similar to the -e option to sed and awk rather than the -f. It would then be possible to include the command to load your saved state on the poly command line. e.g. poly --usestring "PolyML.SaveState.loadState\"$SAVE\";" --use ... Would that work? David On 05/04/2013 13:48, Alex Merry wrote: > On 05/04/13 13:24, David Matthews wrote: >> I've added a "polyc" script that is generated from the build process. >> The idea of this is to provide the similar sort of functionality that >> users of C expect from the "cc" command. It's very simple at the moment >> and is limited to a few options. It compiles an ML source file and >> exports the "main" function. The -o option specifies where the >> executable is to be placed, defaulting to a.out on Unix. > > This sounds really useful. > > One thing I would find useful is an option to load a saved state before > using a file; in Quantomatic, we save the build state, then have some > executables built on top of that (and also have a way of getting a > toplevel with that state). > > Of course, we could just load the state from the file that contains the > main function, but that means either hard-coding the path to the saved > state in those files or having option-parsing boilerplate at the top of > each file to get the path of the saved state. > > Currently, our build scripts have an -l/--load option (the two are > equivalent) to do this. > > Alex >

2 4

← Newer
1
...
29
30
31
32
33
34
35
...
70
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Poly/ML