Benchmark Tool

The collection of various models or mechanisms to perform the benchmark operation on supra network, it is designed to assess the performance and behavior of the supra network under varying load conditions.

Benchmark Models

1. Burst Model

The bursting mechanism is designed to load the network with N transactions at regular intervals of D seconds over R rounds. The primary objective of this mechanism is to stress-test the network and monitor its behavior under load.

In each round, N accounts will send N transactions to the network. After every round, there will be a cool-down period of D seconds before the next round begins. This process will repeat for a total of [BurstModelArgs::total_rounds] rounds during the entire bursting process.

Once the N transactions in a round are executed, the system will wait for the cool-down duration defined as [BurstModelArgs::cool_down_duration] seconds. After the cool-down period ends, the next round will commence, and the same process will be repeated until all rounds are completed.

Quick start example:

With user input based generated EntryFunction type tx payload:

BIN_PATH="qa_cli bin path"

RUST_BACKTRACE=1 RUST_LOG="off,qa_cli=INFO" $BIN_PATH benchmark burst-model \
    --rpc-url http://localhost:27000/ \
    --total-rounds 10 --burst-size 10 --cool-down-duration 1 \
    --tx-sender-account-store-file ./account_stores.json \
    --tx-sender-start-index 0 --tx-sender-end-index 99 \
    --coin-receiver-account-store-file ./account_stores.json \
    --coin-receiver-start-index 100 --coin-receiver-end-index 199 \
    --max-polling-attempts 5 --polling-wait-time-in-ms 200 \
    --tx-request-retry 5 --wait-before-request-retry-in-ms 100 \
    --total-http-clients-instance 1 \
    --total-class-groups 10 \
    --generate-metrics-file-path local-network-benchmark.json \
    >benchmark.log

# Please run `$BIN_PATH benchmark burst-model --help` to get more information

With AutomationRegistration type tx payload and static EntryFunction as RegistrationParamsV1::automated_function. EntryFunctionArgumentsJSON type is retrieved from static_entry_function_payload file and parsed into EntryFunction, check example.benchmark_static_payload.json:

BIN_PATH="qa_cli bin path"

RUST_BACKTRACE=1 RUST_LOG="off,qa_cli=INFO" $BIN_PATH benchmark burst-model \
    --rpc-url http://localhost:27000/ \
    --total-rounds 10 --burst-size 10 --cool-down-duration 1 \
    --tx-sender-account-store-file ./account_stores.json \
    --tx-sender-start-index 0 --tx-sender-end-index 99 \
    --should-automation-tx-type-payload \
    --static-payload-file-path ./example.benchmark_static_payload.json \
    --automation-task-max-gas-amount 5000 \
    --automation-task-gas-price-cap 100 \
    --automation-fee-cap-for-epoch 500000 \
    --automation-expiration-duration-secs 60 \
    --max-polling-attempts 5 --polling-wait-time-in-ms 200 \
    --tx-request-retry 5 --wait-before-request-retry-in-ms 100 \
    --total-http-clients-instance 1 \
    --total-class-groups 10 \
    --generate-metrics-file-path local-network-benchmark.json \
    >benchmark.log

# Please run `$BIN_PATH benchmark burst-model --help` to get more information

2. Lazy Stream Model

The lazy streaming mechanism is designed to identify the network's lazy constant TPS (the maximum number of transactions the network can lazily execute within D duration).

While this mechanism shares some similarities with the burst mechanism, it is fundamentally different in both purpose and core functionality. However, both mechanisms share a common concept: waiting for each transaction to be executed before proceeding further. In this mechanism, the value of N (number of transactions per round) is dynamically adjusted based on the average end-to-end (e2e) execution time of the current round. The goal is to determine the optimal value of N that the network can handle efficiently within the specified duration.

Mechanism Overview

  1. Transaction Transmission and Adjustment
    In each round, N transactions are sent. After all transactions are transmitted, the e2e execution time of the current round is analyzed. Based on this analysis, the value of N is adjusted, and a new round is started.

  2. Dynamic Adjustment of N

    • The initial value of N is set using [LazyStreamModelArgs::initial_tx_set_size].

    • The value of N increases or decreases based on whether the e2e execution time of transactions in the current round meets the threshold defined by [LazyStreamModelArgs::tx_set_execution_timeout].

Adjustment Logic

  • Increasing N:
    If the e2e execution time is less than or equal to [LazyStreamModelArgs::tx_set_execution_timeout] for a consecutive number of rounds defined by [LazyStreamModelArgs::tx_increase_threshold], then N is increased by [LazyStreamModelArgs::tx_increase_percentage]%.

  • Decreasing N:
    If the e2e execution time exceeds [LazyStreamModelArgs::tx_set_execution_timeout] for a consecutive number of rounds defined by [LazyStreamModelArgs::tx_decrease_threshold], then N is reduced by [LazyStreamModelArgs::tx_decrease_percentage]%.

  • Updating the Upper Bound:
    Each time N is reduced, the minimum upper bound for the transaction set size is updated. This adjustment reflects the network's inability to handle N transactions within [LazyStreamModelArgs::tx_set_execution_timeout]. Consequently, this updated upper bound prevents attempting an N value that the network cannot process efficiently.

Key Properties

  • The upper bound of N will always decrease with each reduction.

  • Once the upper bound stabilizes (i.e., stops changing), the current N is identified as the expected lazy constant TPS.

  • If this stabilized N consistently results in an e2e execution time that is less than or equal to [LazyStreamModelArgs::tx_set_execution_timeout] for [LazyStreamModelArgs::lazy_constant_tps_threshold] consecutive rounds, then this N is finalized as the lazy constant TPS.

Quick start example:

BIN_PATH="qa_cli bin path"

RUST_BACKTRACE=1 RUST_LOG="off,qa_cli=INFO" $BIN_PATH benchmark lazy-stream-model \
    --rpc-url http://localhost:27000/ \
    --initial-tx-set-size 100 --tx-set-execution-timeout 1 \
    --tx-increase-percentage 10 --tx-increase-threshold 10 \
    --tx-decrease-percentage 20 --tx-decrease-threshold 5 \
    --lazy-constant-tps-threshold 20 \
    --tx-sender-account-store-file ./account_stores.json \
    --tx-sender-start-index 0 --tx-sender-end-index 4999 \
    --coin-receiver-account-store-file ./account_stores.json \
    --coin-receiver-start-index 5000 --coin-receiver-end-index 9999 \
    --max-polling-attempts 100 --polling-wait-time-in-ms 100 \
    --tx-request-retry 5 --wait-before-request-retry-in-ms 100 \
    --total-http-clients-instance 100 \
    --total-class-groups 2 \
    --generate-metrics-file-path local-network-benchmark.json \
    >benchmark.log

# Please run `$BIN_PATH benchmark lazy-stream-model --help` to get more information

3. Active Stream Model

The active streaming mechanism identifies the active constant TPS (the number of transactions the network can actively execute within D duration).

This mechanism follows a worker-controller model, where N workers continuously send transactions and report transaction metrics to a metrics aggregator (global storage for transaction data). Based on the total transactions sent by the workers in the current round, the controller determines the value of N for the next round.

Types of Rounds

  1. Normal Round: The primary purpose of the normal round is to identify the efficient worker set size. Normal rounds continue until an optimal number of workers is determined.

    • Each normal round lasts for [ActiveStreamModelArgs::normal_round_duration_in_ms].

  2. Final Threshold Round: Once the efficient worker set size is identified, the system transitions to final threshold rounds. In this phase, transactions are sent continuously using the determined efficient worker set size (N).

    • There are [ActiveStreamModelArgs::active_constant_tps_threshold] consecutive final threshold rounds.

    • Each final threshold round lasts for [ActiveStreamModelArgs::final_threshold_round_duration_in_ms].

Worker Lifecycle

  • Independence from Rounds:
    Workers are round-independent. They continuously send transactions and periodically check whether the controller still requires their services. If no longer needed, workers self-destruct.

  • Dynamic Worker Adjustment:
    After every D duration, based on the total transactions transmitted by workers, the value of N is adjusted for the next round:

    • If more workers are needed, new workers are spawned, and each is assigned a unique sequential Worker ID by the controller.

    • If fewer workers are needed, the workers with the highest Worker IDs self-destruct to save system resources.

    • Example:

      • If N = 8 and W = 5, the controller spawns three additional workers.

      • If N = 5 and W = 8, the workers with IDs [6, 7, 8] self-destruct.

      • This ensures that older workers are prioritized, while newly spawned workers are expected to self-destruct when the worker count exceeds the required N.

Adjusting N (Number of Workers in the Next Normal Round) To Get Efficient Worker Set Size

  • Initialization:
    The initial value of N is set using [ActiveStreamModelArgs::initial_worker_set_size].

  • Increasing N:
    If the total transactions sent by workers are greater than or equal to the total number of workers for [ActiveStreamModelArgs::worker_set_increase_threshold] consecutive normal rounds, N is increased by [ActiveStreamModelArgs::worker_set_increase_percentage]%.

  • Decreasing N:
    If the total transactions sent by workers are less than the total number of workers for [ActiveStreamModelArgs::worker_set_decrease_threshold] consecutive normal rounds, N is reduced by [ActiveStreamModelArgs::worker_set_decrease_percentage]%.

  • Upper Bound:
    When N is reduced, it signifies that the network cannot handle that many workers efficiently. This value is then considered an upper bound for future worker set sizes.

    • The upper bound is always decreasing.
    • Once the upper bound stabilizes, the current N is identified as the efficient worker set size.

Final Threshold Round To Obtain Constant TPS

Once the efficient worker set size is determined, normal rounds stop, and the system transitions to the final threshold rounds.

  • Success Condition:
    If the total transactions sent by workers are greater than or equal to N in [ActiveStreamModelArgs::active_constant_tps_threshold] consecutive final threshold rounds, the active streaming model concludes successfully.

  • Failure Condition:
    If the total transactions sent by workers is less than the number of active workers during any final threshold round, it indicates either network instability or an inefficient worker set size. In this case, the efficient worker set size calculation restarts.

  • Improving Accuracy:

    • Use high values for [ActiveStreamModelArgs::worker_set_increase_threshold] and [ActiveStreamModelArgs::worker_set_decrease_threshold].

    • Use low percentages for [ActiveStreamModelArgs::worker_set_increase_percentage] and [ActiveStreamModelArgs::worker_set_decrease_percentage].

    • This approach minimizes the likelihood of calculating an incorrect worker set size and ensures greater accuracy in determining the active constant TPS.

Quick start example:

BIN_PATH="qa_cli bin path"

RUST_BACKTRACE=1 RUST_LOG="off,qa_cli=DEBUG" $BIN_PATH benchmark active-stream-model \
     --rpc-url http://localhost:27000/ \
    --initial-worker-set-size 50 --normal-round-duration-in-ms 1000 \
    --worker-set-increase-percentage 10 --worker-set-increase-threshold 10 \
    --worker-set-decrease-percentage 20 --worker-set-decrease-threshold 5 \
    --final-threshold-round-duration-in-ms 2000 --active-constant-tps-threshold 20 \
    --tx-sender-account-store-file ./account_stores.json \
    --tx-sender-start-index 0 --tx-sender-end-index 4999 \
    --coin-receiver-account-store-file ./account_stores.json \
    --coin-receiver-start-index 5000 --coin-receiver-end-index 9999 \
    --max-polling-attempts 100 --polling-wait-time-in-ms 100 \
    --tx-request-retry 5 --wait-before-request-retry-in-ms 100 \
    --total-http-clients-instance 100 \
    --total-class-groups 2 \
    --generate-metrics-file-path local-network-benchmark.json \
    >benchmark.log

# Please run `$BIN_PATH benchmark active-stream-model --help` to get more information