Server Performance

Updated: 1 Feb, 2022

Sparrow Wallet depends on the Electrum server protocol for retrieving and sending transaction information. This document serves to provide a reasonably up-to-date performance benchmark for different full index Electrum server implementations running on commonly used hardware. It also contains a discussion on two different approaches taken by implementations, based on these findings.

Background

An Electrum server is an example of a Bitcoin address index. In simple terms, you provide it an arbitrary address, and it returns transactions associated with that address. The Bitcoin reference implementation does not (and probably will never) support this functionality. Note that you can also obtain this information using compact block filters, but this method does not support mempool transactions and is much less performant.

Other examples of address indexes exist, but the Electrum server protocol is by far the most widely used Bitcoin address index protocol. This benchmark focuses on full address indexes. A full address index is important privacy-wise because it does not store details about any particular wallet on the server. For true cold storage, all details about the wallet should be contained in the wallet file and not left on the server once the wallet has been closed. This excludes projects such as BWT and EPS which must store the addresses for any wallet provided to them.

Motivation

A previous performance report was written by Jameson Lopp in July 2020, and is worth reading.

This page serves to build on that work in two ways:

A Raspberry Pi 4 is used instead of an AWS server
Up-to-date builds of the projects are used, and retested whenever important changes are made

As such, Sparrow users who are looking to run their own Electrum server on a single board computer may find it useful to compare and contrast different implementations for their own needs.

Hardware

In order to mimic the situation of most privacy-conscious Sparrow users, a single board computer (SBC) is used for all these tests. Currently the most common SBC is a Raspberry Pi 4. This benchmark uses the 8GB iteration of that board running Ubuntu 21.10 64-bit operating system.

The Bitcoin blockchain, and any indexes built off of it, are fairly large in data size (currently around 0.5 TB). For this reason, it is generally recommended to prefer an SSD over a normal HDD for data storage. This test uses a 1TB external USB SSD.

Projects

ElectrumX

ElectrumX is the second iteration of Electrum server implementations after the original Electrum server project was abandoned in favour of it in 2017. After the original author of ElectrumX decided not to support the Bitcoin blockchain, the Electrum developers forked the project. It is now maintained at https://github.com/spesmilo/electrumx.

A notable difficulty in running ElectrumX is initially building the index. Building the index on the hardware used for this test takes around one week. Note however that the index can be built on a more powerful computer and transferred to an SBC.

Current database size: 75 GB
Requires txindex enabled on Bitcoin Core
No binaries provided
Tested version: ElectrumX 1.16

Electrs

While ElectrumX was designed with public server use in mind, Electrs is designed for personal use. As such, it has lower storage requirements (but higher CPU usage as we will see later). It is maintained at https://github.com/romanz/electrs.

In contrast to ElectrumX, building the index takes 12-24 hours on the hardware used for this test. This has led to it being preferred on all prebuilt node packages.

Current database size: 32 GB
Does not require txindex on Bitcoin Core
No binaries provided
Tested version: Electrs 0.9.4

Fulcrum

Fulcrum is a recent implementation written in modern C++. Although it has higher disk space requirements than ElectrumX and Electrs, it is highly performant. It is maintained at https://github.com/cculianu/Fulcrum.

Fulcrum’s indexing can be completed in 2-3 days on the hardware used here, depending on configuration. This puts it between Electrs and ElectrumX on this metric, but the performance it achieves once the index is built is remarkable.

Current database size: 102 GB
Requires txindex enabled on Bitcoin Core
Binaries provided for Linux (x86_64 and arm64) and Windows
Tested version: Fulcrum 1.6.0

Electrs-esplora

This is a fork of the Electrs project which builds a number of additional indexes to improve performance for enterprise use. However given the high data requirements (~800 GB) this implementation was not considered suitable for the hardware used in this test. Note that it can also be run with the --lightmode flag, which halves the disk space requirements. However, even then the capacity of a 1TB drive would be almost exhausted considering the blockchain itself is around 420GB currently.

addrindexrs

This is another fork of the Electrs project used by the Dojo backend to retrieve historical transaction data. There are no significant performance-related changes to warrant inclusion as a separate implementation.

Indexing

Before proceeding to the performance testing it’s important to consider initial indexing in more depth. The three tested implementations range between 1 day and 1 week. Considering each in turn:

ElectrumX is the slowest to index, likely due to it’s use of Python which is single-threaded and slower to parse blocks. However, the index it builds is able to respond to common requests without reaching out to Bitcoin Core, which leads to a more scalable server. Be sure to use a 64-bit operating system with ElectrumX - the indexing performance is far poorer otherwise.

Electrs indexes quickly because it has a smaller less complicated index, and because block retrieval and parsing is highly optimized. It needs to be, because Electrs doesn’t store enough information in its index to respond to most requests - instead, it must reparse blocks on the fly during normal operation. This tradeoff becomes problematic when wallets increase in depth as discussed later.

Fulcrum builds a slightly larger index than ElectrumX, but does so much more quickly. Being highly resource intensive, the indexing performance benefits from the lower level language implementation. Note that indexing performance can be improved through use of the fast sync configuration option. It will become clear that this comprehensive index leads to very impressive performance in operation.

The Test

In order to provide meaningful numbers, a large wallet (~3000 used addresses) was used.

This benchmark serves to test two common (but quite different) server loads from Sparrow:

Initial loading of a wallet. Since an existing Sparrow wallet already contains much of the wallet data (transactions and blocks) which does not need to be re-retrieved, this test focuses on address subscriptions. Address subscriptions allow Sparrow to “subscribe” an address so that the server will provide it with updates whenever transactions for an address change. In addition, a subscription request returns a hash of all of the transaction ids (txids) and block heights affecting that address. This second requirement is key to understanding server performance. The test measures how long subscribing to all the addresses in the wallet takes.
Refreshing a wallet. In ideal circumstances, wallet refreshes should never be necessary - the wallet should only need to be updated incrementally. However, communication and server issues can lead to bad data, which is solved by refreshing the wallet in Sparrow (View menu). This test measures only the time to retrieve all wallet data (transactions and blocks), since addresses are already subscribed to when a refresh takes place.

All servers tested support batching of requests. For this test, a batch page size of 50 was used.

Results

Test 1: Initial load

Test	Cold Start	Run 1	Run 2	Run 3
ElectrumX	52655 ms	40721 ms	54143 ms	49011 ms
Electrs	322386 ms	393303 ms	384036 ms	427722 ms
Fulcrum	2333 ms	1413 ms	1472 ms	1413 ms

Tl;dr: Fulcrum is 22x faster than ElectrumX, ~300x faster than Electrs

Test 2: Wallet refresh

Test	Cold Start	Run 1	Run 2	Run 3
ElectrumX	114466 ms	66175 ms	80133 ms	75489 ms
Electrs	17562 ms	11621 ms	11219 ms	11521 ms
Fulcrum	14152 ms	7854 ms	7382 ms	7442 ms

Tl;dr: Fulcrum is 8x faster than ElectrumX, 1.5x faster than Electrs

Discussion

The results of this test are at first surprising for Test 1. There is a vast difference in performance between Fulcrum and Electrs (1.4 secs vs 6 mins to produce the same data!). This is primarily because Electrs does not store all the data required to find the transactions associated with an address.

As can be seen from the Electrs database schema, only the block height for an address is stored. To be more specific, the ScriptPubKey is hashed and the first 8 bytes of this ‘script hash’ are stored as a key where the value is the confirmed block height for a transaction associated with that script hash. This means that block must be retrieved from Bitcoin Core using the P2P interface and parsed for transactions that have outputs matching the address. For a deep wallet such as that used in this test, this can be a lot of data that must be reparsed for every wallet load. For this particular wallet, 3.5GB of blocks needed to be fetched from Bitcoin Core and parsed every time the wallet is loaded!

This presents a significant CPU load for an SBC, both for the Bitcoin Core and Electrs processes involved. The most significant problem this causes is that Electrs is sometimes so busy it fails to respond to any other requests, or to notify Sparrow when one of its subscribed addresses has new transactions during this period. This can lead to outdated or bad data in any wallet should a loading wallet have significant depth. If you have ever had cause to use the Refresh Wallet function in Sparrow when using Electrs, this is probably the reason.

ElectrumX and Fulcrum on the other hand do store the data required for a full address index lookup. More specifically, when provided with an address they can return all of the transaction IDs associated with that address, along with their block heights. This allows wallets like Sparrow to efficiently check if they already know about all existing transactions affecting the wallet, and to request only the transactions they don’t. When combined with the transaction index (txindex=1) in Bitcoin Core, this approach can be highly efficient as the Fulcrum results show.

Even with this in mind, the performance of Fulcrum is remarkable. It is clear that being written in modern C++17 is particularly advantageous, not only because it is faster, but because performance is consistent across platforms. In the case of ElectrumX, Python appears a less suitable choice for this application. Not only is it slower, but both indexing and query performance varies depending on the underlying server architecture - the previous version of this benchmark using a 32 bit operating system showed faster query performance but much slower indexing. It is worth noting as well that CPU utilisation was comparatively higher using ElectrumX than Fulcrum while serving queries (as well as being much longer in duration).

Conclusion

We can be relatively sure of two trends continuing into the next few years - wallet depths will increase, and storage costs will decrease.

For this reason, Fulcrum emerges as a clear winner in this benchmark. Although it takes slightly longer to index than Electrs, once that indexing is complete the benefits of a scalable and performant server will greatly outweigh this once-off cost, not only in query speed and reliability but in prolonging hardware lifespan.

Although Electrs may have some application when disk space is very limited and wallet depths are small, Fulcrum is recommended as the ideal server to pair with Sparrow.

Technical note

It is interesting to understand how an efficient address index can be built. Let’s consider how Fulcrum and ElectrumX store data. In ElectrumX, similar to Electrs, the history database stores the first 11 bytes of the script hash as a key (this slightly longer key increases data storage but reduces ‘false positive’ incorrect lookups). Fulcrum stores the full 32 bytes of each script hash, which means a bigger index but no false positives at all.

Contrary to Electrs (which maps the script hash to a block height), the value stored against this script hash is a list containing the ordered numbers (called tx_num) of its associated transactions in the blockchain. The tx_num can be described as follows: Each transaction is ordered within a block, so a cumulative number for a transaction can be determined by looking at its order in the entire blockchain. This is effectively a shorthand for the txid, which not only reduces data storage but also allows for fast block height lookups.

For example, the txid is determined from a logical file containing all txids in blockchain order, where the offset in the file is simply the tx_num * 32 bytes (the size in bytes of a SHA256 hash). Similarly, Fulcrum and ElectrumX keep an array in memory where each index corresponds to the blockheight, and the value at that index contains the cumulative number of transactions that existed at that height. This means from the list of tx_nums retrieved from a script hash, they can quickly lookup both the txid and the block height for each tx_num. This is how they achieve such fast performance in comparison to Electrs. The cost to this performance is increased data storage - in the case of ElectrumX, an additional 21GB of storage is needed for the logical file to perform the tx_num to txid lookup. Software engineering is often about tradeoffs, and the tradeoff here leads to dramatic improvements in querying performance.