Run the enclave runtime

The enclave runtime is the package voynich-enclave. It is one Node.js process that attests, registers, loads the models it is granted, answers calls over HTTP and settles them on chain. What it needs from the machine is in Hardware.

Install#

Shell
npm install -g voynich voynich-enclave

voynich is the CLI and voynich-enclave is the runtime its enclave commands load. The runtime package also installs a voynich-enclave binary. Both need Node.js 22 or later.

Build#

The runtime also builds from the repository's backend folder with Node.js 22. It is built after the contract and the SDK it bundles.

Shell
npm ci --include=dev
npm run build --workspace @voynich/contract
npm run build --workspace voynich
npm run build --workspace voynich-enclave

The result is one bundle, enclave/dist/runtime.js. Its SHA-384 is the enclave's measurement.

Start#

Shell
voynich enclave run --keypair <operator keypair file>

The container image is built from enclave/Dockerfile with the backend folder as its context. It runs node enclave/dist/runtime.js in /app, sets ENCLAVE_HOST to 0.0.0.0 and exposes port 9707.

The CLI's voynich enclave run, voynich enclave register and voynich enclave status call the installed runtime and read the same environment. The voynich-enclave binary is the installed runtime started on its own: it runs as voynich enclave run does, with OPERATOR_KEYPAIR taken from the environment alone, and voynich-enclave --measurement prints the runtime's measurement and exits. The Node and Container ways run a bundle built from the repository. The command reference is in CLI: enclave.

Configuration#

The runtime reads its configuration from environment variables. It also loads a .env file from the working directory when one exists. A variable set to an empty value counts as unset.

FieldMeaning
SOLANA_RPC_URLURLrequired

The Solana RPC endpoint the runtime reads from and sends its transactions through.

ENCLAVE_RPC_RPSinteger

The runtime's request budget on the RPC, in requests per second. Every RPC request waits for its turn under this rate.

ENCLAVE_SCAN_INTERVAL_MSinteger

The minimum gap in milliseconds between two program account scans. The runtime scans for its grants and for registered enclaves. Both budgets are required when SOLANA_RPC_URL is not a local address. With an RPC on localhost or 127.0.0.1 they may be omitted: the rate is then 50 and the gap 0.

OPERATOR_KEYPAIRfile path or JSON arrayrequired

The operator wallet: a Solana keypair file, or the 64-byte secret as a JSON array. It pays the runtime's fees and rent and receives the operator share. With voynich enclave run and voynich enclave register, --keypair sets it. When neither is given, the CLI uses the keypair of the Solana CLI configuration, then ~/.config/solana/id.json.

DEV_ATTESTATION_ROOT_KEYPAIRfile path or JSON arrayrequired

The keypair of the attest program's configured root (dev_root) when Config.attestation_program points to attest (used at mainnet launch for standard-server dev enclaves as well as on local clusters, until set_config switches to Solana Confidential Compute). The runtime's hardware type dev report is signed with it.

DEV_SEALING_KEY64 hexadecimal charactersrequired

The 32-byte key that encrypts the state file.

ENCLAVE_STATE_PATHfile path

Where the encrypted state file is kept. .local/enclave.sealed.json under the working directory when omitted.

ENCLAVE_ENDPOINTURL

The endpoint registered on chain, where the Voynich API reaches the runtime. It starts with http:// or https://. http://127.0.0.1: followed by the port when omitted.

ENCLAVE_HOSThost

The address the HTTP server binds. 127.0.0.1 when omitted. The container image sets 0.0.0.0.

ENCLAVE_PORTinteger

The port the HTTP server listens on. 9707 when omitted.

ENCLAVE_POLL_MSinteger

The interval in milliseconds between two passes of the runtime's loop: refresh, grants, granting on. 5000 when omitted.

PROGRAM_IDaddress

The Voynich program. The program id bundled with the runtime when omitted.

ATTESTATION_PROGRAM_IDaddress

The attestation program. The program id bundled with the runtime when omitted.

LOOKUP_TABLEaddress

An address lookup table the runtime's transactions use.

What it does on boot#

  1. Keys

    The runtime opens the state file. When there is none it makes the enclave's two keys, an Ed25519 signing key and an X25519 encryption key, and writes the file. The Enclave account's address and the attestation record's address both derive from the signing key.

  2. Chain

    It binds the state file to the cluster's genesis hash and the two program ids, and stops when the file belongs to another chain. It sends again any transaction it had signed and saved before the last stop.

  3. Signing key balance

    It transfers the rent-exempt minimum for an account with no data from the operator wallet to the signing key, when the signing key holds less.

  4. Attestation

    It submits its report to the configured attestation program: the signing key, the hardware type, the measurement, and a recent slot with its hash. When Config.attestation_program points to attest (the initial mainnet deployment and local clusters), the runtime signs the hardware type dev report with DEV_ATTESTATION_ROOT_KEYPAIR. The program writes the attestation record with its expiry slot. The runtime skips this step while its record states its measurement and is further than a third of a record lifetime from expiry.

  5. Registration

    When no Enclave account exists it sends register_enclave with its encryption key and endpoint, signed by the operator wallet and the signing key. When one exists and its expiry slot, endpoint or measurement differs from the record and the configuration, it sends refresh_enclave.

  6. Grants and models

    It reads every Grant made to its Enclave account and loads each granted model in memory. It then grants the models it holds to other registered enclaves. Both are told in Register and get granted.

  7. Calls left open

    It resumes every call it had accepted and had not finished: it answers, or settles the answer it had already signed.

  8. Serve

    It starts the HTTP server, then repeats attestation, registration, grants and resuming every ENCLAVE_POLL_MS. A pass that fails is logged as enclave.sync.failed and the next pass runs after the same interval, while the HTTP server keeps answering.

On a stop signal the runtime stops its loop, finishes the calls it is answering and releases its models.

Keys and state#

Everything the runtime keeps is in one file at ENCLAVE_STATE_PATH, encrypted with AES-256-GCM under DEV_SEALING_KEY. Each write uses a fresh nonce, goes to a temporary file and is renamed into place.

In the state fileUsed for
The signing key's seedThe enclave's identity: its Enclave account, its attestation record, its receipts.
The encryption key's secretUnwrapping grants and decrypting call inputs.
The chain it belongs toThe genesis hash and the two program ids.
The grants it has loadedEach Grant's address, model and wrapped key.
Accepted callsEach call's encrypted input envelope, its hash and its answer.
Signed transactionsEvery transaction is saved signed before its first broadcast, and the same bytes are sent again after a restart.

Model keys and decrypted weights are held in memory only.

Two enclaves on one machine each take their own ENCLAVE_STATE_PATH and port.

Endpoints#

The runtime serves three routes. The Voynich API carries callers' requests to the two call routes at the registered endpoint, as told in HTTP API: Relay. Failed requests answer in the error envelope of HTTP API: Overview.

GET /health#

Returns:

FieldMeaning
statusready.
enclaveThe Enclave account's address.
measurementThe runtime's measurement, 96 hexadecimal characters.
modelsThe addresses of the models loaded in memory.

The Voynich API requests this route at the registered endpoint every 15 seconds. While it answers with the enclave's address and the measurement of its attestation record, the enclave is reachable. The site, the SDK and the CLI send calls only to live, reachable enclaves. The rule is in HTTP API: Reachability.

POST /v1/calls/:call/input#

Takes the encrypted input of an open call. :call is the Call account's address. The body is JSON:

FieldMeaning
envelopebase64required

The input envelope, as described in Calls and receipts.

The runtime reads the Call from chain and accepts the input when the call names this enclave, the SHA-256 of the envelope equals the call's input hash, the call is open and its deadline has not passed. It returns call, the call's address, and status, accepted, then answers in the background. Sending the same envelope again returns the same response.

RefusalCode
No such callCALL_NOT_FOUND
The call names another enclaveINVALID_ENVELOPE
The envelope's hash differs from the call's input hashINPUT_HASH_MISMATCH
The call is settled or refundedCALL_NOT_OPEN
The deadline has passedCALL_DEADLINE_PASSED

GET /v1/calls/:call/answer#

Returns the state of the answer:

FieldMeaning
statuswaiting, answered once the settling transaction is confirmed, or failed.
outputEnvelopeThe encrypted output, base64.
outputHashThe SHA-256 of the output envelope, hexadecimal.
signatureThe enclave's signature over the receipt message, base64.
settleTransactionThe signature of the settling transaction.
receiptAddressThe Receipt account's address.
failureFor a failed call: code, message and refundState.

Every field but status is null until it exists.

Logs#

The runtime writes one JSON line per event. Each line carries event:

EventWritten whenStream
rpcAn RPC request returns: the method, the HTTP status and the milliseconds.standard error
httpAn HTTP request is answered: the method, the route, the status and the milliseconds.standard output
listeningThe HTTP server is bound: the host and the port.standard output
model.loadedA granted model is loaded: its address and name.standard output
call.failedA call could not be answered or settled: the call's address.standard error
enclave.sync.failedA pass of the loop failed: the error's name and its message.standard error

Started with voynich enclave run, every line goes to standard error.

Status#

Shell
voynich enclave status

It reads the state file and the Enclave account with the same environment as the runtime. It sends no transaction, creates no keys, writes no file and takes no keypair. It prints:

FieldMeaning
enclaveThe Enclave account's address, or null when no state file exists.
signingKeyThe enclave's signing key, or null when no state file exists.
measurementThe measurement of the runtime bundle.
registeredWhether the Enclave account exists.
expiresAtSlotThe expiry slot the Enclave account holds, or null before registration.

GET /health on a running runtime adds the models it has loaded. The enclave's page on the site, /enclaves/<key>, shows its record, grants, receipts and earnings.

Refresh#

A running runtime refreshes by itself. On every pass of its loop it compares the current slot with its record's expiry slot. Once the record is within a third of a record lifetime of expiry, it submits a new report bound to a fresh slot, which renews the record in place, and sends refresh_enclave, which copies the new expiry slot into the Enclave account.

What expiry means for grants and calls is in Enclaves and attestation.