These primitives are developed in the context of the Mojave Compiler Collection, a multi-language compiler with a relational architecture and the MetaPRL theorem prover. Primitives are implemented as part of a functional intermediate language in the Mojave compiler, which has a formal operational semantics and complete typing rules.
The distributed application problem
Checkpoints in distributed applications
With checkpointing as a primitive language, most of the work of transmitting process state can be automated by the compiler. In case of failure, built-in language primitives can restore the entire process and I/O state using the latest system checkpoint.
Grid computation
When checkpoints are embedded in the language, the compiler is able to optimize checkpoint performance by choosing internal representations of the data that are easy to store and restore. The compiler is well suited for generating checkpoint code because it has a record of the internal representation of each process's state.
Capturing snapshots of a process state
Whole-process migration
- Load balancing
- Active messages
- Fault tolerance in distributed programming
Migration can be applied to passive messages or messages that include executable code and data. So when a node in a distributed network fails, migration can be used to revive all the tasks that were running on the system before the failure.
Speculative execution
- Fault isolation and transaction-like behavior
- Fault tolerance in distributed programming
- Generalization of speculative evaluation
In the traditional case, the programmer must manually undo the state of all changed files if an I/O error occurs. If A is likely to be true and the cost of speculation is low, then it is in the best interest of the process to assume A and back out if the assumption later turns out to be false.
The Mojave Compiler Collection (MCC)
These projects extend the concepts of migration and speculation to the design of persistent storage and distributed communication technologies.
Organization of this thesis
- FIR type system
- FIR statements
- Atoms
- Expressions
- Heap and store values
- Kinds
- Contexts and judgments
Both tuples and arrays are seen as type safe - the compiler guarantees that values in these memory areas have the appropriate types. The atomic level is the name of the speculation to abort (the speculation checkpoints are stored in a list).
FIR operational semantics
- External calls
- Process migration
- Speculations
- Speculations and external state
- Speculations and transactions
- Subscripting operations
- External call typing rule
- Migration typing rule
- Speculation typing rules
- Subscripting typing rules
In the special call expression migrate [j, aptr, aoff] afun(a1, . . . , an), the atoms aptr and aoff specify a string (as a block of raw data and an offset) that describes the migration protocol and target (eg, machine name). Operational evaluation of the expression leads to process migration followed by evaluation of the tail call afun(a1, . . . , an). Operationally, the checkpoint is deleted from the checkpoint context and evaluation continues with a call to the tail function fun(a1, . . . , an).
The migration Γ`specialtailop :t Ty-Special-Call Process is given by the special-call expression migrate[j, aptr, aoff] af(a1, . . . , an), which is described in Section 4.3. Otherwise, calling speculation works exactly like calling tail function af with arguments (aconst, a1, . . . , an).
Preservation
For security, the runtime environment generates a security check at runtime to verify that the value being accessed has a compatible type. Since the reduction only removes checkpoints from the context, the checkpoint context remains well-formed.
Progress
In the first case, there is a line in the full FIR, Red-Atom-Var, that will expand the variable.
Runtime safety checks
- Runtime data structures and invariants
- Data blocks and the heap
- Pointer table
- Function pointers
- Pointer safety
A data value in a heap is called a block, and a heap contains several (preferably non-contiguous) blocks. Immutable: All entries in the function table contain the address of the entry point of the function in the text area. The tableptr pointer table effectively acts as a segment table for the blocks (segments) in the heap.
In the IA32 runtime, the overhead is more than 12 bytes per block, including the pointer table. Similar to block pointers, function pointers in the heap are represented as indexes in the function table.
Compiling FIR to assembly code
For the runtime(Γ| hci[i] :t←h) operation, the runtime invariants guarantee that ift represents a pointer, and that it is then a valid pointer to a block in the heap; if it is a function type, then this is a valid reference to a function header. During conversion from FIR to assembly code, it is important for system migration to maintain a correlation between an instruction in the FIR and an instruction in the assembly code. Currently, for each migrate[i, aptr, aoff]f(a1, . . , am) instruction, an exact correlation is maintained between the FIR and the assembly code.
Currently, the most advanced optimizations are performed on the FIR code, where they can be formally verified using MetaPRL [12], an integrated theorem prover. It's hard to think about MIR because of the weakened type system and semantics, so background optimizations are limited to architecture-specific optimizations.
Process migration
Using process migration in the FIR
The first is the migration protocol, which sends all the state of a process to another machine and terminates the process on the original machine. If the migration fails for any reason, the process will continue to run on the original machine. While a process can indirectly observe the result of a migration by calling external functions, the process is indifferent to the machine it is running on and does not observe a successful migration.
The fact that the process does not observe the result of migration is reflected in the operational semantics. Thesuspendprotocol writes the process state to a file and terminates the process if successfully written.
Runtime support for migration
- The pack and unpack operations
- The migrate operation
Since no data is stored in variables, no data will be stored in hardware specific registers. Since all data is in the stack during migration, all data follows the standard, architecture-independent representation for MCC. There is a performance issue related to saving all log data for a process on the heap.
However, for the checkpoint protocol it means that all log data must be stored on the heap and then immediately loaded back into the logs, with safety checks. Since all pointers are stored on the heap using indexes, the migration must be careful that the indexes on the destination machine match exactly.
Speculative operations
- Using speculations in the FIR
- Speculation state
- Speculation properties and invariants
- Invariants related to the organization of the heap
- Invariants related to pointer difference tables
- Liveness invariant
- Implementation of speculations
- The entry operation
- Copy-on-write faults
- The commit operation
- The rollback operation
- Garbage collector state
- Garbage collection and speculations
Speculation levels use copy-on-write (COW) semantics; when a block in the heap is modified, the block is cloned and the pointer table is updated to point to the new copy of the block, preserving the data in the original block. Note that a copy-on-write error does not affect the liveness of the original block. The pointer table always points to the most recent live version of a block. Therefore, before the copy-on-write error, there was no indexed version of the block at the current speculation level, and the uniqueness of speculation level is preserved.
Whenl= 1,ptr diff[l−1] is discarded — the original copies of blocks that are in error at level 1 are automatically reclaimed during the next garbage collection. This allows the small pile to be collected in isolation, reducing the total cost of the garbage disposal.
Garbage collector main algorithm
By Speculation Level Uniqueness and Speculation Liveness, the first m−1 blocks must remain alive on a garbage collection. Note that blocks that are not indexed during a commitor rollback operation are never listed in any pointer table, and are always collected by the garbage collector. After a large garbage collection, for each block b, no block b0 exists that is at the same speculation level and has the same index.
Speculation is responsible for maintaining the garbage collector state in such a way that the above properties remain valid. Neither operation needs to change the small pile or the smaller roots, as both are conservatively maintained and will still meet the properties.
Mark operation
B's pointer table points to the original field in block A, and the original field in block A points to A's header. Using the index embedded in the header forfield, the algorithm looks up its pointer-table entry, which was previously changed to point to block[mark]. It then traverses the cursor in block[mark] to restore the original block, and then restores the value in block[mark] to field.
To ensure that the algorithm makes progress, it immediately marks each new block it discovers, and sets the least significant bit on the changed pointer table entry to avoid passing over the now invalid pointer table entry. All valid entries in the pointer table are even (blocks are word-aligned), so by setting the least significant bit in the pointer table entry we ensure that references to the current block are ignored.
Minor collection
InPointer-Table-Minor, entries in the pointer table that refer to minor blocks are reset to the parent of the block, if a parent exists. This code is responsible for returning the pointer table entries in case the latest version of the block is dead. For smaller blocks that remain alive, the pointer table entry is restored to the latest version of the block (which undoes the effect of Pointer-Table-Minor).
For small blocks that are dead, the pointer table entry is released if the pointer table points to the dead version of the block. If the pointer table references a parent of the block, the pointer table remains unchanged.
Major collection
At the end of Mark-Major, for each block, the pointer table entry refers to the newest version within levels of uncollected speculation{0.gc-1}, if such a version exists. The effect on the pointer table is similar to the Pointer-Table-Minor collection — if all versions of a block are dead in the collected region, then the pointer table is left referring to the latest version in the uncollected region. As with small collections, the pointer table is updated for each live block encountered, resulting in a pointer table that always refers to the newest version of a block.
If an untagged block is encountered and the pointer table entry points to that version of the block, then the pointer table entry is freed. This means that the mechanism for freeing pointer table entries does not interfere with the mechanism that ensures that the pointer table points to the most recent version of a block.
Future work
The time given is to determine that the pattern *h*e*l*l*o*w*o*r*l*d* occurs in the text of the introduction to this paper. InACM Symposium on Principles of Programming Languages, pages available as part of the MCC distribution), August 2002. available as part of the MCC distribution), July 2002.
Example of grid computation using checkpoints
Fault tolerance using migration in a heterogeneous network
File transfer operation using lightweight transactions
Design of the MCC compiler
FIR base terms
FIR type system
FIR atoms and expressions
FIR operators
Heap and store values
Program contexts and judgments
Data block header format
Pointer table representation
Function header format
Speculation variables
Heap data with multiple speculation levels
GC variables
GC properties in presence of speculative operations
GC main function
Mark operation in garbage collector
Illustration of pointer inversion in mark phase
GC minor collection
GC minor collection, mark phase
GC minor collection, sweep phase
GC major collection
GC major collection, mark phase
GC major collection, sweep phase
