Now that 5.8.2 has been released I've updated Git master with some changes that have been in the pipeline for some time. These affect a number of issues so they probably need a bit of explanation. These are: a basic code-generator for 64-bit ARM, position-independent executables and a new bootstrap process.

This is quite a long message since each of these require some explanation.

Bootstrap The updated bootstrap process affects all architectures. Because Poly/ML is written in ML there needs to be an ML compiler to compile the source. The solution up to now has been to have pre-built compilers for each architecture. The bootstrap process then compiles the basis library and builds the final binary from that. The problem is that this requires a compiler for each architectures; for 5.8.2 that is seven in all: X86/32, 32-bit interpreted, 64-bit interpreted, X86/64 and X86/64/32, with different version for the last two for Windows and Unix because of the different ABIs. Adding ARM64 would have increased this further.

The solution was to bootstrap from the interpreted version. This requires only two pre-built compilers, for 32- and 64-bits. However building a final native code compiler requires the whole system to be compiled several times. This doesn't take long on reasonable hardware but can be slow on under-powered machines or with debugging turned on. The final binary is as fast as before; it's just the bootstrap that is slow.

A consequence of this is that it is no longer necessary to run "make compiler" when building from Git. Previously the compiler itself was not rebuilt with a simple "make" so changes that involved the compiler itself needed "make compiler" in order to be incorporated. That is no longer the case since the bootstrap process recompiles the compiler. "make compiler" has been retained for compatibility but it may actually be better not to use it, particularly if --enable-intinf-as-int has been included. --enable-intinf-as-int builds the basis library and any subsequent code with int as arbitrary precision. Running "make compiler" would build the compiler itself with arbitrary precision rather than fixed precision which will be bulkier even if it doesn't noticeably affect the speed.

Position-independent code The new version now generates position-independent executables on X86/64 and ARM64. This has been in the pipe-line for a while but was spurred on by the fact that Mac OS requires it for ARM code. What this means is that the code segments in object files created by PolyML.export no longer contain absolute addresses. The "constant area" associated with the code for each function is pulled out and placed in a read-only, non-executable area. It's too complicated to do this X86/32 so 32-bit programs will continue to need special treatment on platforms that have problems with non-PIC but for the majority of code on 64-bits there should no longer be problems in this area. It doesn't apply to Windows, which doesn't need this, or to compact 32-bit or interpreted code where the "code" is not marked as executable.

ARM64 Last but not least there is now a basic code-generator for the ARM64. This has been tested on a wide range of hardware and systems including Windows 10, Debian under Windows-subsystem-for-Linux, Mac OS X, PiOS on various 64-bit Raspberry Pis and even big-endian NetBSD on a Raspberry Pi. It is complete including compiled FFI and compact 32-bit. However, at this stage the code-generator has no optimisation or proper register allocation and treats the machine as essentially single-register plus stack. That greatly simplifies the code-generation at the expense of bulky and slow code. On a Mac Mini X86/64 code translated by Rosetta is still roughly 1.5 to 2 times faster. Rosetta code is reported to be about 80% of the speed of optimised ARM code so things should improve with optimisation.

There is one point about the ARM code-generator that is worth making. The ARM has a weaker memory model than the X86 and that can affect multi-threaded code with shared references. Code that uses mutexes to protect all accesses to shared references is not affected since the mutex access includes the appropriate memory barriers but during testing I came across a problem with futures in Isabelle. Since once a future is evaluated it can never change it is generally possible to access it without a lock and only use a lock if it appears to be unevaluated.

Unlike the X86 the ARM does not guarantee that another thread will see updates to different addresses in the same order as they are made by the thread making the assignments. In this particular case a thread was creating a value on the heap and then assigning the address of this heap cell to a shared reference. On the X86 the update to the heap would always be seen before the update to the shared reference but on the ARM it was possible for another thread to cache that part of the heap and so read values in the heap that were completely random. Since the consequence of this is completely unpredictable behaviour I took the decision to implement '!' and ':=' using instructions that incorporate memory barriers giving read-acquire/store-release semantics. Currently this does not apply to other mutable structures such as arrays but for consistency it probably should. There doesn't seem to be any measurable difference in speed by using these instructions compared with the ones without the memory barriers although the code is slightly longer.

David