rest of basic required primitives

array and hash primitives

array, len, hash, keys

len must work for strings too

while, for and foreach in the language

these will be implemented as part of the standard library. Implemented as “standard library” code in Shript itself.

a hasKey primitive.

cry loudly if a variable or argument

has the name of a required primitive

parser errors should show source context, too

cry loudly if trying to access a hash value

via ‘.’ uses a non-existing key; return null if the access is made via ‘[]’.

only catch our errors (improve on the ignore-errors forms in shovel)

comments :-)

FN must also record the number of arguments accepted by the callable.

blocks are expressions :-)

apostrophes for string quotation

NEW-FRAME must also have a ‘debug’ form

where it takes a list of the names of variable present in the frame, not the number of variables.

when saving the return addresses, also save the position in code

for the call, so we can build nice stack traces when an exception is thrown.

keep the names of the variables in the environments and

display them if in case of an exception.

check the number of arguments for primitives and required primitives

add a ‘panic’ primitive that throws an exception.

tests !

record file names in source positions

(so we can show useful line/column information for multiple files).

serialization and deserialization

for the bytecode

for the VM state

the state should be serialized with:

the MD5 of the serialization of the bytecode for the current VM;
the MD5 of the serialization of the source for the current VM;
the Shovel VM version.

This way we can make sure that we throw an error if an attempt is made to restart a virtual machine with another VM’s state (such an attempt would most likely result in an error down the road, but it’s better not to corrupt data and throw an error ASAP).

the state serialization and the bytecode serialization

should start with an MD5 hash of the content to make sure that the data wasn’t messed up;

maybe the representation of instructions should take up less space

don’t record lines and columns everywhere;

record only character indices within files and calculate line/column on demand; store the file names in a special dummy instruction; instruction start/end positions in source code should be only deltas from previous start/end positions;

figure out how interruptible user primitives are written

they return an extra value, which is one of:

:continue or nil, which means keep on going;
:nap-and-retry-on-wake-up, go to sleep and retry the primitive call on wake-up;
:nap, go to sleep and go on from the next instruction after the primitive call with the result returned by the primitive call (which is pushed on the stack before going to sleep).

implement interruptible user primitives

checking of return values for user primitives

to make sure that they return an instance of a shovel type; tests should make sure that required primitives return an instance of a shovel type.

user primitives should handle their exceptions;

exception raising and handling

We don’t have any, but we’ll have non-local exits (document this)

error checking for arrays (check that indices are in range)

push and pop for vectors (make them adjustable)

comparison (== and !=) must work for booleans and when one of the

operands is guest-null; also do reference comparison for arrays and hashes;

results of string slice or indexing should be strings

it should be possible to assign strings of length 1 to string indices

add ‘upper’ and ‘lower’ primitives for strings

maybe environment frames should have contain information about

the location where they were introduced? (file and character position). This would help when debugging.

the VM version should be stored before the payload

for both bytecode and VM state.

VM versions are backward compatible. This way it is possible to check that we’re not trying to decode a version n+m state/bytecode on code that only knows how to handle version n state/bytecode.

add some reserved keywords

class, try, catch, throw, block, block_return

the type for keyword tokens should be changed to

:keyword in ‘tokenize-identifier’ (and I can probably drop ‘token-is-keyword’)

non local exits

reserve and parse keyword ‘block’

reserve and parse keyword ‘block_return’

add VM instruction ‘BLOCK address’

add VM instruction ‘BLOCK_RETURN’

add VM instruction ‘POP_BLOCK’

document new instructions in the README

serialization with named blocks on the stack

eliminate explicit local exits

standard library functions for unique block names

standard library functions for try/catch

a keyword - ‘context’ that pushes on the stack

a hash containing:

under key ‘stack’: the string representation of the current stack trace and
under key ‘environment’: the string representation of the current environment.

short-circuit behavior for && and ||

implement by replacing with ifs;
t1 || t2 - if t1 true else t2;
t1 && t2 - if t1 t2 else false.

fix the ‘string’ and ‘stringRepresentation’ primitives

to avoid infinite loops when stringifying circular data structures;

Should not be able to serialize or restart VMs

that threw an exception caused by a programming error; or have an unhandled exception from a user-defined primitive;

clean up the exported symbols

(the user should only need to use functions exported from the ‘shovel’ package)

the run-vm should allow a parameter ‘vm’ (reuse a vm directly)

there should be a way to wake up a vm (clear ‘should-take-a-nap’)

publish the functions to extract the context from a VM

(the stack and environment); should work even for a crashed VM.

extract errors from a VM

VM execution quotas for ‘run-vm’

offer a way to specify how expensive user-primitives are

in terms of the execution quota

Gaaaah must write a garbage collector anyway -

or find another way to limit Shovel process memory usage. Solution: limit the number of used cells, count allocated cells (via array, hash, push, call, block, const etc.); if the number exceeds the limit, count the actual number of cells used (by running a modified version of the serializer) and crash the VM if the number of really used cells still exceeds the limit.

test that user-defined primitives can

model their memory usage;

test shovel-vm:cells-increment-herald calls

the number of ticks executed so far must be made part of the

serialized VM state;

execution quotas basic

limit integers to a known amount (60 bits?)

unbounded bignums make it very hard to limit memory and CPU usage

implement primitive quotas by function calls

incrementTicks
incrementCells
askPermissionForCells

the ‘pop’ required primitive should set the array item to null before

decrementing the fill pointer (so the CL GC can reclaim the object pointed to by that array element)

public getters for VM execution statistics

provide a public getter for vm-used-ticks

provide a public getter for vm-used-cells

use #-/#+ in the asd file to eliminate real-tests.lisp

move SHRIPT-FILE and SHOVEL-ERROR and errors to a

SHOVEL-PUBLIC-TYPES file that is inside package SHOVEL;

try to optimize step-vm by building a jump table with functions

not worth it

remove references to Shript and rename it ShovelScript

documentation

ShovelScript (language, required primitives)

language reference

required primitives description

VM

VM instructions

VM state

VM Serialization

explain the CPU and RAM bounds for the VM

explain why integers are modulo 2^60.

CL getting started Guide

Web number-guessing game

Version 1

Version 2 - db, multiple users, redirects

The user writes the program

add APIs to access the MD5 of bytecode

(so we can check if the other party already has the bytecode and not send the bytecode over and over again). (also update the CL getting started guide to mention this)

ClGettingStarted.md uses shovel-vm:serialize-vm-state

All public APIs should are moved to package ‘shovel’. Update the document.

test the documentation by reading it and pasting the code

the getting started guide

the number guessing tutorial

API documentation

the documentation should mention that primitives

can’t call a closure (it breaks interruptability)

document the fact that user primitives should handle their exceptions;

When they don’t, the error is stored in the ‘user-primitive-error’ field of the virtual machine and the virtual machine goes to sleep as if :nap-and-retry-on-wake-up had been returned by the primitive as a second value.

Files

TODO.org

Latest commit

History