Where mvx came from

Why a compiler

When Apple put LLVM at the centre of its developer tools, I thought: that is how you would build Pick BASIC properly. A real compiler, producing real executables. I had always liked the language and had never seen it compiled the way everything else is.

Earlier attempts usually translated BASIC to C and compiled that. It works, but you lose the debugger and pay in build times. mvx goes straight from BASIC to LLVM and on to machine code, and it writes standard DWARF debug information against your BASIC line numbers.

That turned out to matter more than speed. To gdb, lldb, perf, Instruments or valgrind, a compiled BASIC program is just another program. A stock profiler, told nothing but a process id, names the line of Pick BASIC where the time is going. The compiler can also write out its LLVM IR with your variable names intact. That gives future analysis and security tools a starting point that MultiValue has never had.

Proving it was fast

I wanted a benchmark nobody could say I had picked to suit myself, so I used Dave Plummer's prime sieve from the Primes project, where the same program is measured in every language on the same terms. The Pick BASIC version is bench/sieve.b.

Sieving to one million for five seconds, mvx managed 13,660 passes against 14,221 for C on the same machine: about 96% of C, from Pick BASIC. Each system was measured against a C baseline on its own hardware. UniData 8.3 managed 66 passes against C's 13,967. jBASE 6.2.1, which compiles through C, managed 145, or 216 with its optimiser turned up.

jBASE settled the question for me. Compiling alone is not the difference; how values are held is. If every variable stays a string that must be checked and converted before each sum, the C compiler underneath cannot see past it. The gain comes from the front end proving a variable is numeric and emitting a native integer or double.

UniVerse could not run the sieve at all, because it limits an array to 64,000 elements. So I wrote a banked version that stores the flags in dynamic arrays, which every MultiValue system can express. That version exposed a real weakness in mvx.

Before the fix, the fastest system on the flat sieve came third of four. Banking cost UniData a factor of two and mvx a factor of 440, because every dynamic-array operation allocated memory, and replacing one element copied the whole string twice. The profiler put 45% of the time in the memory allocator.

Three changes fixed it. A value reuses its own buffer when nothing else shares it. An element assignment patches the string in place. Long arrays get an index built on demand, so reaching element N no longer means scanning from the start. Writing an element went from 170 to 64 nanoseconds, and on long arrays the cost stopped growing with length. That took mvx to 76 passes on the banked sieve, ahead of both interpreters, and the flat result did not change.

The work carried on from there. The index now records every element boundary and covers short arrays too, it survives when a shared string is copied, and integers are turned into text without printf. Reading or writing an element now takes 5 to 8 nanoseconds at any length and in any order. The banked sieve is at 339 passes, and the flat sieve matches C: 14,419 passes against 14,281.

System	Banked sieve	Flat sieve
mvx today	339	14,419
UniData 8.3	35	68
UniVerse 14.2	35	cannot run
OpenQM (ScarletDME)	19	46
mvx after the first fix (Aug)	76	13,660
mvx before the fix (Aug)	31	13,660
jBASE 6.2.1 (Aug)	7	145

Part of that speed is a choice the older systems could not make. They were designed when a large machine had a few megabytes of memory, so every byte counted and a dynamic array was kept as bare text and scanned to find an element, rather than indexed. mvx is written for today's machines, where memory comes in gigabytes. It spends some of it on purpose: the index costs eight bytes for each element it tracks, and in return reaching any element takes the same time however long the array is.

In September I measured everything again, and added OpenQM through ScarletDME, its open-source descendant, built with optimisation. It managed 46 passes on the flat sieve and 19 on the banked one. Most of that gap is the interpreter. The rest is how it stores values: OpenQM copies the whole string on every element write, scans from the start of a field to reach each value, and formats every number it stores with sprintf.

mvx is not far ahead everywhere. UniData already walks a list in order at a constant cost per element, so for the usual loop over a list the difference is mostly compiled against interpreted. UniVerse inserts and deletes within a factor of two of mvx.

Drivers instead of a database

I never set out to write a database. Durability, replication, backup and monitoring are problems PostgreSQL and its neighbours have spent decades solving. So mvx has storage drivers instead: a file is bound to SQLite, LMDB, PostgreSQL, MongoDB or MySQL, and one account can mix them.

The mapping layer builds real tables from your dictionaries. MultiValue programs read and write records, while anything that speaks SQL reads the same live data as rows. And because a session is a process and a file can live on another machine, the system spreads across as many containers as you give it, instead of needing one bigger server.

The full story is in Git in the PICK world on heydon.consulting.