What Blocks My Blockchain’s Throughput?
Developing a Generalizable Approach for Identifying Bottlenecks in Permissioned Blockchains

Fabric has risen to prominence as one of the industry’s leading permissioned blockchains (Zheng, 2019; Guggenberger et al., 2022). Compared to other permissioned blockchains, its unique architecture provides novel opportunities for improving performance and eliminates the need for domain-specific and deterministic smart contract (called chaincode in the case of Fabric) programming languages (Androulaki et al., 2018). Fabric’s architecture relies on the execute-order-validate paradigm. Under this approach, nodes first simulate (execute) transactions in any order, which are then batched and ordered in consensus. Finally, nodes validate the individual transactions and ensure that no conflicts have emerged owing to the potentially different ordering in the execution phase (Androulaki et al., 2018). This approach is useful if cross-checking a computation-heavy transaction requires weaker agreement than an honest (super-)majority (Guggenberger et al., 2022). At the ordering and/or validation stages, nodes can verify whether sufficient signatures for agreement, according to previously defined rules, are present.

In Fabric, nodes are grouped into organizations, and a node can take at least one of the roles of a peer node (peer) or an orderer (orderer) (Androulaki et al., 2018). Only peers maintain an append-only ledger, whereas orderers create and broadcast blocks. We detail the characteristics of the execute-order-validate architecture below:

1. (1)

   Execution Phase: A client sends a cryptographically signed transaction proposal to the peers. Upon receiving the transaction proposal, peers simulate the transaction, i.e., they run the required chaincode on their own copy of the ledger without updating their ledger or sending a corresponding update to other peers. After the simulation, peers respond to the client with an endorsement. The endorsement includes the peer’s digital certificate chain, confirming their organization membership, and the peer’s digital signature on the transaction proposal, the transaction’s read-write set at the time of simulation (which details the version numbers but not the values of the variables the transaction interacts with (Hyperledger Fabric Architecture Reference, 2020)), and the outcome of the simulation.

2. (2)

   Ordering Phase: Once a client has collected sufficient endorsements for a proposal to be deemed acceptable by honest peers, it packs these endorsements into a transaction and forwards it to the ordering service. The orderers use a consensus protocol, such as Kafka (Kreps et al., 2011) or Reliable, Replicated, Redundant, And Fault-Tolerant (RAFT) (Ongaro and Ousterhout, 2014), to order the transactions, grouping them into batches (blocks), and signing them without evaluating their validity. Upon reaching consensus on a block, they broadcast it to a subset of peers (one anchor peer for each organization) for validation.

3. (3)

   Validation Phase: Once a peer receives a block – either directly from an orderer or via a gossip protocol from a fellow peer belonging to the same organization – it validates the content in three steps (Foschini et al., 2020):

   1. (a)

      First, every transaction within a block is subjected to parallel verification through the validation system chaincode (VSCC), which ensures that they have accumulated the appropriate number of endorsements as dictated by the endorsement policy. The VSCC also verifies that all executions came to the same result. If a transaction fails any of these tests, VSCC marks it as invalid.

   2. (b)

      Next, valid transactions undergo multi version concurrency control (MVCC) – a sequential check of whether the simulations were conducted on compatible versions of the ledger. Should any variable within the transaction’s read-set have been altered in the peer’s ledger since its endorsement, the transaction is marked as invalid.

   3. (c)

      Once a transaction in a block has passed the VSCC and MVCC checks, the peer updates the corresponding variables in its state database in what is called the commit step. Invalid transactions, while not influencing the ledger’s state, are nevertheless recorded in the ledger.

Quorum is another blockchain that has emerged as a significant player among the industry’s permissioned blockchains (Alastria, 2024; VAKT, 2024). Quorum is based on Geth, an Ethereum client, and, as a result, leverages many of its characteristics. In contrast to Fabric, it utilizes a single, deterministic smart contract language, Solidity. This choice streamlines development, as Solidity is the primary language used in the broader, permissionless blockchain space (DeFiLlama, 2024), making it simpler to transfer applications from the public permissionless Ethereum blockchain to Quorum. Quorum relies on the more common order-execute architecture under which transactions are first ordered and batched into blocks using a consensus mechanism and only subsequently executed.

Quorum supports four different consensus mechanisms: RAFT, Quorum Byzantine Fault Tolerant (QBFT), Istanbul Byzantine Fault Tolerant (IBFT), and Clique (Consensys, 2024). The details around the architecture vary slightly depending on the consensus mechanism used. Under RAFT (the consensus mechanism used later in our analysis), each node can take only one of two roles: leader (minter) or follower (verifier). There is only one leader in each network. Below, we detail the characteristics of the order-execute architecture based on the RAFT consensus mechanism:

1. (1)

   Ordering Phase: Clients send cryptographically signed transactions to the nodes. Subsequently, the nodes perform preliminary validations on the transactions, such as verification of the transactions’ attributes (e.g., nonce, syntax, signatures, etc.). Once verified, the nodes disseminate the pre-validated transactions to other nodes via a gossip protocol, allowing them to be added to each node’s transaction pool (mempool). Once the transaction reaches the leader, it orders valid transactions according to the network’s established priority rules. Finally, the ordered transactions are compiled into a block, which is disseminated across the network to follower nodes.

2. (2)

   Execution Phase: Upon receiving the block, followers attach the block to their version of the blockchain and send a message to the leader to confirm acceptance of the block. After receiving acceptance messages from the majority of nodes, the block is considered valid and becomes the new head of the blockchain.

The DLPS is designed as an end-to-end pipeline in which users can define the benchmarking specifications both for the configuration of the blockchain and client network, as well as the experiment settings (Sedlmeir et al., 2021). All the necessary parameters can be set using a single config file, and we simplified further the deployment of DLPS across different computing environments by dockerizing its launch on local machines. Additionally, we extended its graphical capabilities to allow for a more in-depth analysis. The DLPS framework is composed of three Python packages, BlockchainFormation, DAppFormation, and ChainLab, which automatically set up the blockchain networks’ nodes, smart contracts, and clients and iteratively execute measurements and collect and analyze data based on which they steer the benchmarking process. The benchmarking follows a recursive localization of maximum throughput with growing resolution (for more details, see Sedlmeir et al., 2021).

This process is implemented by gradually increasing the request rate directed at the blockchain to push the network to its throughput limit (Sedlmeir et al., 2021). The first localization run starts by sending asynchronous requests at a specified base rate. The DLPS measures the time at which clients send their transaction requests and receive successful notifications of their execution and determines average request frequency (f${}_{\text{req}}$) and response frequency (f${}_{\text{resp}}$) across all clients. If the network manages to keep up with f${}_{\text{req}}$ for a certain duration, this test concludes, and the next one starts with an increased request rate. If f${}_{\text{resp}}$ falls behind f${}_{\text{req}}$ by more than a predetermined threshold (e.g., 5 %), a specified number of retries is performed. Should all retries fail, the ramping-up sequence is terminated, and the next ramping-up sequence for localization commences, starting from a base rate slightly lower than the maximum throughput achieved previously (e.g., 80 %). Further rampings can follow with smaller increments. This increases the granularity of measurements and data collection around the rate at which the system failed initially, allowing for the determination of a network’s behavior close to maximum sustainable throughput. Once the experiment is complete, the DLPS generates multiple figures summarizing the gathered information. The DLPS also stores the benchmarking data in CSV files, allowing users to perform their own analyses. We make heavy use of these CSV files for the subsequent analysis.

The initial analysis focuses on understanding the impact of incremental increases in the request rate on the resource utilization of each node. This is achieved by plotting a data point for each node every second during a 14-second time-frame within a 20-second experiment (excluding the initial and final 3 seconds where the ramp-up and ramp-down of f${}_{\text{req}}$ may cause temporary effects), corresponding to different request frequencies (Figure 2). This method aims to uncover potential trends and correlations between resource usage and increased f${}_{\text{req}}$. This initial step already indicates that peers exhibit the highest strain in central processing unit (CPU) usage, as demonstrated by the maximum utilization percentage across all cores on each node while ordering and client nodes display significantly lower CPU utilization rates.

In terms of memory usage, both clients and ordering nodes appear largely unaffected by increases in the request rate. However, peers show signs of impact, albeit without a clear pattern. Regarding network and hardware utilization, client nodes maintain stable performance, whereas peer and orderer nodes demonstrate a moderate correlation between increased utilization and f${}_{\text{req}}$, but as before, the exact trend is not clear. Note that at this early stage of the analysis, any anomalies could also suggest inaccuracies in the experimental setup. Therefore, it is crucial to eliminate this possibility before conducting a more detailed analysis to determine the structural causes behind these observations. Upon ruling out experimental errors, a separate analysis is conducted for each component.

In the second part of our analysis, we attempt to find correlations between Fabric’s throughput and the components highlighted as potential bottlenecks in the first part, namely peer CPU utilization and peer network traffic. We start by gaining an overview of how the request rate affects throughput. This is achieved by plotting the throughput as a rolling mean across different window sizes (Figure 11). A one-second window size results in each data point being plotted individually, revealing how unstable the throughput of this Fabric network gets beyond f${}_{\text{req}}$=1200 s^-1, with fluctuations in f${}_{\text{resp}}$ reaching up to 1000 s^-1 within a single run. To see the overall network performance trend, we increase the window size, taking 3 cases at 3 seconds, 8 seconds, and full run. In these cases, the mean is calculated over more data points, and outliers are smoothed out. Here, we see that, on average, the system keeps up with the request rate until reaching approximately f${}_{\text{req}}$=1600 s^-1. Beyond this point, the interrelationship breaks, marking this as the peak throughput observed in the experiment.

To obtain insights into the link between peer CPU utilization and network traffic, we examine Figure 12, which plots the two resources against the network’s throughput employing a rolling average with a 3-second window. We selected this interval as it matches the average duration required for a transaction to be committed to the blockchain under high request rates. This is significant because queuing effects become prominent at elevated f${}_{\text{req}}$, and opting for a shorter time window could underestimate throughput.

For CPU usage, we observe an initial linear increase with throughput until the characteristic plateau is reached. Both variables seem to increase at a similar rate at the beginning, implying that increased throughput proportionally strains the CPU. Additionally, we observe that the instability in CPU utilization starts at around f${}_{\text{resp}}$=1200 s^-1, which is also the point at which the correlation between f${}_{\text{req}}$ and f${}_{\text{resp}}$ starts breaking down (Figure 11a). This indicates that CPU usage is more closely correlated to throughput than to the request rate. Similarly, although network traffic increases with throughput, it does so at a lower rate and begins to exhibit instability already at around f${}_{\text{resp}}$=600 s^-1, where the throughput still manages to keep up with the request rate.

Digging deeper, we focus on the throughput of individual peers (Figure 13). Here, we see that every peer contributes similarly to the overall throughput, with all of them moving in unison with minor variations at high f${}_{\text{req}}$. Given our findings in Section 4.1.2, where we noted significant differences in network traffic between gossip leaders and followers, this outcome suggests that network traffic is not a significant factor in determining throughput compared to CPU utilization. If it were, we would expect noticeable differences in throughput between gossip leaders and followers. Consequently, this leaves us with peer CPU utilization as the primary factor behind throughput leveling off.

The crucial role of peer CPU utilization further supports the findings in Section 4.1 that VSCC and MVCC are the main bottlenecks in Fabric as both depend on peer CPU. Given that validations in VSCC are parallelized and we have noted that the mean core utilization is limited and mean CPU utilization is similar across all peers, it appears that the main bottleneck is VSCC. However, the sequential execution of concurrency controls in MVCC, combined with the fluctuating utilization per core featured in Figure 5, makes it impossible to determine its role as a bottleneck. As a result, while VSCC appears to be the main culprit, we cannot conclusively confirm this with the available data. This also highlights a shortcoming of the DLPS’ resource utilization-based approach, which was chosen to make it blockchain agnostic. However, as a trade-off for this generalized approach, the resulting data are less specific than they could be. Therefore, further research would require taking into account and evaluating more precise monitoring data, e.g., using SoundCloud’s Prometheus that is incorporated in Fabric (Hyperledger Fabric Operations Guides, 2020).

The Quorum analysis, as before, starts by gaining an overview of the four resources in relation to f${}_{\text{req}}$ (Figure 14). To analyze Quorum’s performance in more detail, we refine our analysis by increasing the request rate in increments of 100 requests per second. As with the case of Fabric, even at this early stage, it is apparent that CPU and network utilization are closely correlated with the request rate. Memory and hard drive usage are limited, with I/O operations exhibiting high fluctuations but generally registering less than 1 % usage. Across all resources, client utilization appears to be minimal, and it is either unaffected or grows linearly with the request rate. As a result, in the following sections, we concentrate on the resource utilization of nodes.

The throughput analysis of Quorum starts by examining its correlation with the request rate. We see that even for small window sizes, throughput remains relatively stable but starts to exhibit higher fluctuations at f${}_{\text{req}}$=1800 s^-1. Examining the full run, we observe that throughput plateaus at around f${}_{\text{resp}}$=2100 s^-1, which is significantly lower than the maximum request rate. This suggests that the overall performance of the blockchain started to decline long before reaching the highest request rate. This observation suggests that the performance degradation noted at f${}_{\text{req}}$=1800 s^-1 for both CPU and network utilization may be more critical in identifying the bottleneck.

Examining throughput against the CPU and network utilization (Figure 21), we see similar growth rates and patterns. Both resources keep up with throughput at the beginning and experience higher performance fluctuations at around f${}_{\text{resp}}$=1800 s^-1. Beyond this point, the differences in throughput between request rates become less pronounced, becoming indistinguishable as it approaches f${}_{\text{resp}}$=2100 s^-1.

Since both resources exhibit similar patterns, we search for a common source behind the blockchain’s performance degradation. Considering that block and validated transaction propagation are network-related, if they were the bottleneck, they would mainly impact network traffic. As a result, they are less likely to be the bottleneck. This leaves only transaction pre-validation as the possible constraining factor, which can impact both CPU and network as it represents the first step in processing a transaction.

Looking further into pre-validation, we deduce that the bottleneck is either pre-validation itself, i.e., the nodes do not validate the incoming transactions fast enough, which in turn slows down the network, or the nodes do not receive enough transactions. From Figure 14, we see that the network utilization of clients increases linearly with the request rate, which suggests that they send the proper number of transactions to the nodes. Examining further the interaction of nodes with the incoming transactions, we look into the number of rejected transactions (Figure 22). By the term rejected transactions, we refer to those transactions that nodes decline to propagate through the network, resulting in clients receiving almost immediate (within less than 50ms) notification of transaction failure after submission. Initially, the count of rejected transactions remains minimal but begins to surge at f${}_{\text{req}}$=1800 s^-1, culminating in approximately 7000 rejections by f${}_{\text{req}}$=2500 s^-1. This escalation translates to a rejection rate of 14 % before reaching a plateau. This significant increase in rejections prompts us to investigate further to understand the underlying causes.

In Quorum, transactions are categorized as either executable or non-executable. Executable transactions are the ones that can be immediately included in a block, while non-executable transactions are out of nonce order and must wait for preceding transactions with lower nonce to execute first. Investigating the reasons behind the increasing rejection rates, we identify two main constraints in Quorum’s transaction handling. First, Quorum caps the transaction pool at 4096 executable and 10000 non-executable transactions (Consensys, 2022). However, according to Figure 23a, the transaction pool’s capacity is never fully utilized, suggesting that this is not the cause behind the rejections. The periodic drop in the number of transactions in the pool aligns with the end of each ramp-up phase of DLPS, where nodes commit pending transactions to blocks and clear the pool before the new ramp-up commences.

The second limitation involves the clients, who are limited to 16 executable and 500 non-executable transactions in the pool at any given time (Consensys, 2022). Examining Figure 23b, we see that at f${}_{\text{req}}$=1800 s^-1, transaction submission by six clients begins to exhibit instability, diverging from the previous linear relationship with the request rate, and most clients diverge at f${}_{\text{req}}$=2400 s^-1. This pattern suggests that clients are reaching the limit of executable transactions in the pool, leading to an increased rate of rejections. To answer this definitively, we would have to separate the client transactions into executable and non-executable and examine which type is getting rejected, something that is not possible due to DLPS’ blockchain agnostic nature that does not allow us to distinguish transactions between the different types. However, considering that the number of rejected transactions increases sharply at f${}_{\text{req}}$=1800 s^-1, combined with the observed reduction in CPU and network utilization starting at the same point and the subsequent plateau at f${}_{\text{req}}$=2400 s^-1, strongly suggest that the primary bottleneck is the limit on the number of executable transactions per client.