Feature Parity Testing

When testing your program, assurances that it will run the same in various clusters is essential to both quality and producing expected outcomes.

Facts

Fact Sheet

Features are capabilities that are introduced to Solana validators and require activation to be used.
Features may be activated in one cluster (e.g. testnet) but not so in another (e.g. mainnet-beta).
However; when running default solana-test-validator locally, all available features in your Solana version are automagically activated. The result is that when testing locally, the capabilities and results of your testing may not be the same when deploying and running in a different cluster!

Scenario

Assume you have a Transaction that contained three (3) instructions and each instruction consumes approximately 100_000 Compute Units (CU). When running in a Solana 1.8.x version, you would observe your instruction CU consumption similar to:

Instruction	Starting CU	Execution	Remaining CU
1	200_000	-100_000	100_000
2	200_000	-100_000	100_000
3	200_000	-100_000	100_000

In Solana 1.9.2 a feature called 'transaction wide compute cap' was introduced where a Transaction, by default, has a 200_000 CU budget and the encapsulated instructions draw down from that Transaction budget. Running the same transaction as noted above would have very different behavior:

Instruction	Starting CU	Execution	Remaining CU
1	200_000	-100_000	100_000
2	100_000	-100_000	0
3	0	FAIL!!!	FAIL!!!

Yikes! If you were unaware of this you'd likely be frustrated as there was no change to your instruction behavior that would cause this. In devnet it worked fine, but locally it was failing?!?

There is the ability to increase the overall Transaction budget, to lets say 300_000 CU, and salvage your sanity but this demonstrates why testing with Feature Parity provides a proactive way to avoid any confusion.

Feature Status

It is fairly easy to check what features are enabled for a particular cluster with the solana feature status command.

solana feature status -ud   // Displays by feature status for devnet
solana feature status -ut   // Displays for testnet
solana feature status -um   // Displays for mainnet-beta
solana feature status -ul   // Displays for local, requires running solana-test-validator

Alternatively, you could use a tool like scfsd to observe all feature state across clusters which would display, partial screen shown here, and does not require solana-test-validator to be running:

Parity Testing

As noted above, the solana-test-validator activates all features automagically. So to answer the question "How can I test locally in an environment that has parity with devnet, testnet or even mainnet-beta?".

Solution: PRs were added to Solana 1.9.6 to allow deactivation of features:

solana-test-validator --deactivate-feature <FEATURE_PUBKEY> ...

Simple Demonstration

Suppose you have a simple program that logs the data it receives in it's entry-point. And you are testing a transaction that adds two (2) instructions for your program.

All features activated

You start the test validator in one terminal:

solana config set -ul
solana-test-validator -l ./ledger --bpf-program target/deploy/PROGNAME.so --reset`

In another terminal you start the log streamer:

solana logs

You then run your transaction. You would see something similar in the log terminal (edited for clarity):

Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc invoke [1]
Program log: process_instruction: PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc: 0 accounts, data=[0]
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc consumed 12843 of 200000 compute units
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc success
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc invoke [1]
Program log: process_instruction: PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc: 0 accounts, data=[1]
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc consumed 12843 of 187157 compute units
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc success[

Because our feature 'transaction wide compute cap' is automatically activated by default, we observe each instruction drawing down CU from the starting Transaction budget of 200_000 CU.

Selective features deactivated

For this run, we want to run so that the CU budget behavior is in parity with what is running in devnet. Using the tool(s) described in Feature Status we isolate the transaction wide compute cap public key and use the --deactivate-feature on the test validator startup

solana-test-validator -l ./ledger --deactivate-feature 5ekBxc8itEnPv4NzGJtr8BVVQLNMQuLMNQQj7pHoLNZ9 --bpf-program target/deploy/PROGNAME.so --reset`

We now see in our logs that our instructions now have their own 200_000 CU budget (edited for clarity) which is currently the state in all upstream clusters:

Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc invoke [1]
Program log: process_instruction: PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc: 0 accounts, data=[0]
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc consumed 12843 of 200000 compute units
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc success
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc invoke [1]
Program log: process_instruction: PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc: 0 accounts, data=[0]
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc consumed 12843 of 200000 compute units
Program PWDnx8LkjJUn9bAVzG6Fp6BuvB41x7DkBZdo9YLMGcc success

Full Parity Testing

You can be in full parity with a specific cluster by identifying each feature that is not yet activated and add a --deactivate-feature <FEATURE_PUBKEY> for each when invoking solana-test-validator:

solana-test-validator --deactivate-feature PUBKEY_1 --deactivate-feature PUBKEY_2 ...

Alternatively, scfsd provides a command switch to output the complete deactivated feature set for a cluster to feed directly into the solana-test-validator startup:

solana-test-validator -l ./.ledger $(scfsd -c devnet -k -t)

If you open another terminal, while the validator is running, and solana feature status you will see features deactivated that were found deactivated in devnet

Full Parity Testing Programmatically

For those who control running the test validator within their test code, modifying the test validator activated/deactivated features is possible using TestValidatorGenesis. With Solana 1.9.6 a function has been added to the validator builder to support this.

At the root of your program folder, create a new folder called tests and add a parity_test.rs file. Here is the boiler place functions (boiler-plate if you will) used by each test

Press </> button to view full source

#[cfg(test)]
mod tests {
    use std::{error, path::PathBuf, str::FromStr};

    // Use gadget-scfs to get interegate feature lists from clusters
    // must have `gadgets-scfs = "0.2.0" in Cargo.toml [dev-dependencies] to use
    use gadgets_scfs::{ScfsCriteria, ScfsMatrix, SCFS_DEVNET};
    use solana_client::rpc_client::RpcClient;
    use solana_program::{instruction::Instruction, message::Message, pubkey::Pubkey};
    use solana_sdk::{
        // Added in Solana 1.9.2
        compute_budget::ComputeBudgetInstruction,
        pubkey,
        signature::{Keypair, Signature},
        signer::Signer,
        transaction::Transaction,
    };
    // Extended in Solana 1.9.6
    use solana_test_validator::{TestValidator, TestValidatorGenesis};

    /// Location/Name of ProgramTestGenesis ledger
    const LEDGER_PATH: &str = "./.ledger";
    /// Path to BPF program (*.so) change if needed
    const PROG_PATH: &str = "target/deploy/";
    /// Program name from program Cargo.toml
    /// FILL IN WITH YOUR PROGRAM_NAME
    const PROG_NAME: &str = "PROGRAM_NAME";
    /// Program public key
    /// FILL IN WITH YOUR PROGRAM'S PUBLIC KEY str
    const PROG_KEY: Pubkey = pubkey!("PROGRAMS_PUBLIC_KEY_BASE58_STRING");
    /// 'transaction wide compute cap' public key
    const TXWIDE_LIMITS: Pubkey = pubkey!("5ekBxc8itEnPv4NzGJtr8BVVQLNMQuLMNQQj7pHoLNZ9");

    /// Setup the test validator passing features
    /// you want to deactivate before running transactions
    pub fn setup_validator(
        invalidate_features: Vec<Pubkey>,
    ) -> Result<(TestValidator, Keypair), Box<dyn error::Error>> {
        // Extend environment variable to include our program location
        std::env::set_var("BPF_OUT_DIR", PROG_PATH);
        // Instantiate the test validator
        let mut test_validator = TestValidatorGenesis::default();
        // Once instantiated, TestValidatorGenesis configuration functions follow
        // a builder pattern enabling chaining of settings function calls
        let (test_validator, kp) = test_validator
            // Set the ledger path and name
            // maps to `solana-test-validator --ledger <DIR>`
            .ledger_path(LEDGER_PATH)
            // Load our program. Ignored if reusing ledger
            // maps to `solana-test-validator --bpf-program <ADDRESS_OR_PATH BPF_PROGRAM.SO>`
            .add_program(PROG_NAME, PROG_KEY)
            // Identify features to deactivate. Ignored if reusing ledger
            // maps to `solana-test-validator --deactivate-feature <FEATURE_PUBKEY>`
            .deactivate_features(&invalidate_features)
            // Start the test validator
            .start();
        Ok((test_validator, kp))
    }

    /// Convenience function to remove existing ledger before TestValidatorGenesis setup
    /// maps to `solana-test-validator ... --reset`
    pub fn clean_ledger_setup_validator(
        invalidate_features: Vec<Pubkey>,
    ) -> Result<(TestValidator, Keypair), Box<dyn error::Error>> {
        if PathBuf::from_str(LEDGER_PATH).unwrap().exists() {
            std::fs::remove_dir_all(LEDGER_PATH).unwrap();
        }
        setup_validator(invalidate_features)
    }

    /// Submits a transaction with programs instruction
    /// Boiler plate
    fn submit_transaction(
        rpc_client: &RpcClient,
        wallet_signer: &dyn Signer,
        instructions: Vec<Instruction>,
    ) -> Result<Signature, Box<dyn std::error::Error>> {
        let mut transaction =
            Transaction::new_unsigned(Message::new(&instructions, Some(&wallet_signer.pubkey())));
        let recent_blockhash = rpc_client
            .get_latest_blockhash()
            .map_err(|err| format!("error: unable to get recent blockhash: {}", err))?;
        transaction
            .try_sign(&vec![wallet_signer], recent_blockhash)
            .map_err(|err| format!("error: failed to sign transaction: {}", err))?;
        let signature = rpc_client
            .send_and_confirm_transaction(&transaction)
            .map_err(|err| format!("error: send transaction: {}", err))?;
        Ok(signature)
    }
    // UNIT TEST FOLLOWS
}

/// Setup the test validator passing features
/// you want to deactivate before running transactions
pub fn setup_validator(
    invalidate_features: Vec<Pubkey>,
) -> Result<(TestValidator, Keypair), Box<dyn error::Error>> {
    // Extend environment variable to include our program location
    std::env::set_var("BPF_OUT_DIR", PROG_PATH);
    // Instantiate the test validator
    let mut test_validator = TestValidatorGenesis::default();
    // Once instantiated, TestValidatorGenesis configuration functions follow
    // a builder pattern enabling chaining of settings function calls
    let (test_validator, kp) = test_validator
        // Set the ledger path and name
        // maps to `solana-test-validator --ledger <DIR>`
        .ledger_path(LEDGER_PATH)
        // Load our program. Ignored if reusing ledger
        // maps to `solana-test-validator --bpf-program <ADDRESS_OR_PATH BPF_PROGRAM.SO>`
        .add_program(PROG_NAME, PROG_KEY)
        // Identify features to deactivate. Ignored if reusing ledger
        // maps to `solana-test-validator --deactivate-feature <FEATURE_PUBKEY>`
        .deactivate_features(&invalidate_features)
        // Start the test validator
        .start();
    Ok((test_validator, kp))
}

/// Convenience function to remove existing ledger before TestValidatorGenesis setup
/// maps to `solana-test-validator ... --reset`
pub fn clean_ledger_setup_validator(
    invalidate_features: Vec<Pubkey>,
) -> Result<(TestValidator, Keypair), Box<dyn error::Error>> {
    if PathBuf::from_str(LEDGER_PATH).unwrap().exists() {
        std::fs::remove_dir_all(LEDGER_PATH).unwrap();
    }
    setup_validator(invalidate_features)
}

/// Submits a transaction with programs instruction
/// Boiler plate
fn submit_transaction(
    rpc_client: &RpcClient,
    wallet_signer: &dyn Signer,
    instructions: Vec<Instruction>,
) -> Result<Signature, Box<dyn std::error::Error>> {
    let mut transaction =
        Transaction::new_unsigned(Message::new(&instructions, Some(&wallet_signer.pubkey())));
    let recent_blockhash = rpc_client
        .get_latest_blockhash()
        .map_err(|err| format!("error: unable to get recent blockhash: {}", err))?;
    transaction
        .try_sign(&vec![wallet_signer], recent_blockhash)
        .map_err(|err| format!("error: failed to sign transaction: {}", err))?;
    let signature = rpc_client
        .send_and_confirm_transaction(&transaction)
        .map_err(|err| format!("error: send transaction: {}", err))?;
    Ok(signature)
}

We can now add test functions in the body of mod test {...} to demonstrate default validator setup (all features enabled) and then disabling the transaction wide compute cap as per previous examples running solana-test-validator from the command line.

#[test]
fn test_base_pass() {
    // Run with all features activated (default for TestValidatorGenesis)
    let inv_feat = vec![];
    // Start validator with clean (new) ledger
    let (test_validator, main_payer) = clean_ledger_setup_validator(inv_feat).unwrap();
    // Get the RpcClient
    let connection = test_validator.get_rpc_client();
    // Capture our programs log statements
    solana_logger::setup_with_default("solana_runtime::message=debug");

    // This example doesn't require sending any accounts to program
    let accounts = &[];
    // Build instruction array and submit transaction
    let txn = submit_transaction(
        &connection,
        &main_payer,
        // Add two (2) instructions to transaction to demonstrate
        // that each instruction CU draws down from default Transaction CU (200_000)
        // Replace with instructions that make sense for your program
        [
            Instruction::new_with_borsh(PROG_KEY, &0u8, accounts.to_vec()),
            Instruction::new_with_borsh(PROG_KEY, &1u8, accounts.to_vec()),
        ]
        .to_vec(),
    );
    assert!(txn.is_ok());
}

#[test]
fn test_deactivate_tx_cu_pass() {
    // Run with all features activated except 'transaction wide compute cap'
    let inv_feat = vec![TXWIDE_LIMITS];
    // Start validator with clean (new) ledger
    let (test_validator, main_payer) = clean_ledger_setup_validator(inv_feat).unwrap();
    // Get the RpcClient
    let connection = test_validator.get_rpc_client();
    // Capture our programs log statements
    solana_logger::setup_with_default("solana_runtime::message=debug");

    // This example doesn't require sending any accounts to program
    let accounts = &[];
    // Build instruction array and submit transaction
    let txn = submit_transaction(
        &connection,
        &main_payer,
        [
            // This instruction adds CU to transaction budget (1.9.2) but does nothing
            // when we deactivate the 'transaction wide compute cap' feature
            ComputeBudgetInstruction::request_units(400_000u32),
            // Add two (2) instructions to transaction
            // Replace with instructions that make sense for your program
            // You will see that each instruction has the 1.8.x 200_000 CU per budget
            Instruction::new_with_borsh(PROG_KEY, &0u8, accounts.to_vec()),
            Instruction::new_with_borsh(PROG_KEY, &1u8, accounts.to_vec()),
        ]
        .to_vec(),
    );
    assert!(txn.is_ok());
}

Alternatively, the scfs engine gadget can produce a full vector of deactivated features for a cluster. The following demonstrates using that engine to get a list of all deactivated features for devnet.

#[test]
fn test_devnet_parity_pass() {
    // Use gadget-scfs to get all deactivated features from devnet
    // must have `gadgets-scfs = "0.2.0" in Cargo.toml to use
    // Here we setup for a run that samples features only
    // from devnet
    let mut my_matrix = ScfsMatrix::new(Some(ScfsCriteria {
        clusters: Some(vec![SCFS_DEVNET.to_string()]),
        ..Default::default()
    }))
    .unwrap();
    // Run the sampler matrix
    assert!(my_matrix.run().is_ok());
    // Get all deactivated features
    let deactivated = my_matrix
        .get_features(Some(&ScfsMatrix::any_inactive))
        .unwrap();
    // Confirm we have them
    assert_ne!(deactivated.len(), 0);
    // Setup test validator and logging while deactivating all
    // features that are deactivated in devnet
    let (test_validator, main_payer) = clean_ledger_setup_validator(deactivated).unwrap();
    let connection = test_validator.get_rpc_client();
    solana_logger::setup_with_default("solana_runtime::message=debug");

    let accounts = &[];
    let txn = submit_transaction(
        &connection,
        &main_payer,
        [
            // Add two (2) instructions to transaction
            // Replace with instructions that make sense for your program
            Instruction::new_with_borsh(PROG_KEY, &0u8, accounts.to_vec()),
            Instruction::new_with_borsh(PROG_KEY, &1u8, accounts.to_vec()),
        ]
        .to_vec(),
    );
    assert!(txn.is_ok());
}

Happy Testing!

Resources

scfsdopen in new window

gadget-scfsopen in new window

# Feature Parity Testing

# Facts

# Scenario

# Feature Status

# Parity Testing

# Simple Demonstration

# All features activated

# Selective features deactivated

# Full Parity Testing

# Full Parity Testing Programmatically

# Resources