Lightning Labs Blog - The Official Blog of Lightning Labs

by Bryan Vu

alice Our previous two posts in this series have been about Carol, a future Lightning user. For this post, we’ll be traveling back to the present, where Alice lives with substandard mobile Bitcoin apps. In Alice’s world, smartphones have been the predominant global computing platform for years, but the current generation of Bitcoin apps leave a lot to be desired. The options she’s tried have had major flaws in security, privacy, and/or ease-of-use.

Several of today’s top mobile Bitcoin wallets are built on centralized services that present significant security risks. As a wise man said more than once: “Not your keys, not your Bitcoin.” Centralized services also risk exposure of users’ financial information (i.e. transaction histories). On the other hand, centralized services are faster, easier to use, and work for new users with small amounts of bitcoin. As a long-time, experienced bitcoiner, Alice has quite a few friends who have lost money with centralized services, so she decides to continue on her quest for a wallet that gives her full control of her bitcoins.

grail The other existing option for users like Alice is wallets based on Simplified Payment Verification (BIP 37). SPV wallets don’t have the centralized security risks noted above, but they suffer from privacy problems of their own. In addition, the current generation of SPV wallets tend to be slower and more difficult to use than their centralized counterparts. Alice doesn’t like sluggish apps and she values her financial privacy, so she hasn’t been satisfied with these wallets either.

Incredibly, it’s now been six years since the last improvement proposal directed at mobile platforms was created, despite Bitcoin’s exponential growth in adoption and the ever-increasing ubiquity of mobile devices. To address the various issues described above, Lightning Labs’ Olaoluwa Osuntokun (roasbeef) and Alex Akselrod, along with Jim Posen (formerly of Coinbase), proposed the Neutrino protocol (BIP 157, BIP 158).

Neutrino benefits

The challenge for decentralized mobile Bitcoin apps is to take bitcoin’s global transaction history (currently approx. 200 GB) and quickly extract only the transactions relevant to an individual user. In addition, a successful mobile app must provide security, privacy, and a high-quality user experience. The Neutrino protocol was designed to facilitate this on devices with low-power processors, limited storage, limited bandwidth, intermittent power, and erratic connectivity.

Security - As mentioned above, due to the well-known pitfalls of centralized services, experienced users like Alice, or those with larger amounts of bitcoin, generally prefer to maintain control of their own keys. Neutrino is designed to make apps with user-controlled keys faster, easier to use, and hopefully more compelling. In addition, Neutrino includes new safeguards for resolving conflicting transaction information,¹ which is particularly useful for Lightning users.

privacy Privacy - A key requirement for financial software should be to protect user privacy. Account balances and transaction history shouldn’t be revealed without the consent of the user. SPV/BIP 37 clients leak quite a lot of user information including a probabilistic list of the user’s addresses, which can be used to derive an approximate transaction history. Centralized services have full access to this data and provide no guarantees of privacy either. Neutrino is a much-improved option for mobile users who don’t want to expose their financial history to third parties.²

Scalability and decentralization - The Neutrino protocol requires less computation on the part of Bitcoin full nodes because filters sent to Neutrino clients must be computed only once for all Neutrino users. Under the older SPV/BIP 37 protocol, full nodes compute different results for each client user, which can be a significant computational burden if a full node is serving many mobile clients.

Censorship-resistance - Because Neutrino users can control their own keys and broadcast their own transactions, we hope Neutrino-based apps will enhance the permissionless and censorship-resistant properties of Bitcoin.

Tech details: how Neutrino works

For more technically inclined readers, the following section gives an overview of the Neutrino protocol. For even more detailed information, please see BIP 157 and BIP 158.

GCS filters - At the high-level, Neutrino is based on a chain of “filters” that each correspond to a block in the Bitcoin blockchain. These filters use Golomb-Rice coding to represent the addresses contained in a block. Neutrino filters (called GCS filters, for “Golomb-Coded Set”) are a much more compressed way of representing a block and are ~15KB each, whereas the source blocks can be over 250 times as large (up to ~4MB).

This compression allows low-bandwidth devices to observe the blockchain and determine if newly-mined blocks are relevant to the user’s wallet. The steps involved in this process are as follows:

Step 1 - filter creation: As each new block is mined, full nodes calculate the corresponding GCS filters. These filters are then served to Neutrino light clients.

Step 2 - observe and compare filters: Approximately every ten minutes Neutrino clients receive a filter, which they then compare against the user's wallet to determine if there are any new transactions that are relevant to the user.

Step 3 - download relevant blocks: If a filter indicates that a block does contain relevant transactions, the Neutrino client will request the "stripped" block,³ which contains all of the transaction data in the block, but excludes signatures or "witness data." With the block downloaded, the client can scan the block for the relevant transaction(s).

Step 4 - apply to wallet balance: Wallet balances are updated with the new transaction information, and the Neutrino client is now up-to-date.

Note that with this approach, information that directly identifies a user’s addresses is never sent over the network, and the matching between the blockchain and the user’s wallet is done by the user’s software rather than by an untrusted, remote node as is the case with BIP 37.

Syncing - The way Neutrino nodes process new blocks is described above, but when Alice opens her Neutrino-based app for the first time, her app must sync up to the current state of the blockchain. An abbreviated version of this process also takes place when a Neutrino node has been offline and needs to “catch up.” There are four steps to sync a Neutrino node:

First, like a regular Bitcoin client or BIP 37 client, a Neutrino client will download and validate the chain of block headers, which is used to determine the position of each block in the blockchain (i.e. block height) and verify proof-of-work. Each header is 80 bytes, adding up to roughly 40 MB for the current Bitcoin blockchain.

syncing Second, a Neutrino client will (concurrently) download a chain of “filter headers,” that corresponds to the chain of block headers described above. In the Neutrino protocol, each block in the blockchain has an associated GCS filter. The chain of filter headers gives Neutrino clients a lightweight way to associate each GCS filter with its corresponding block. This process is detailed in BIP 157.

Each filter header is 32 bytes and the total amount of bandwidth required for filter headers is around 20 MB. Headers from blocks prior to the wallet’s creation date are discarded once validated so that on-disk storage is minimal. Headers are downloaded and verified one time each.

The third step is the download of the GCS filter for each block starting from when the wallet is created. Total ongoing bandwidth for filters is around 70 MB/month, and filters need not be stored.

Finally, the Neutrino node will check the filters to determine if there are transactions relevant to the user’s wallet, and if so, the stripped blocks will be downloaded and transactions extracted as described above.

For a new wallet, the initial syncing process takes just a few minutes on a typical mobile device. The total amount of blockchain data stored by a Neutrino node is approximately tens of megabytes, compared to about 200 gigabytes currently for a full Bitcoin node. After the initial synchronization, bandwidth requirements are also quite low for typical Neutrino clients due to GCS compression. Note that while Neutrino nodes require less bandwidth than full nodes, the Neutrino protocol may require more bandwidth than BIP 37/SPV in most cases. The required bandwidth is still well within the capabilities of the vast majority of mobile devices, and we believe the privacy and security benefits are well worth the tradeoff. In addition, Neutrino was designed with batching and parallelism for filter and block data in mind, so even though it requires more bandwidth, the user experience for Neutrino users will generally be snappier and more responsive.

Future developments

Performance - As mentioned above, the Neutrino protocol is designed to work on resource-constrained devices with intermittent connectivity and power. Note however, that the current early-stage Neutrino implementation in lnd hasn’t yet been fully optimized. Further work in this area is ongoing.

Block retrieval - Neutrino clients can download blocks or filters from any source, including sources outside the Bitcoin P2P network. An option to be explored for block retrieval that would further improve privacy is Private Information Retrieval. Currently, Tor is a more feasible option, with new circuits created for each block fetched.

Neutrino serving neutrino - Because the filters used in Neutrino are the same for all nodes (unlike the filters used in BIP 37), Neutrino nodes can be configured to serve filters to other Neutrino nodes. This could enable Neutrino clients to contribute to the network as well, possibly reducing costs for full node operators.

Filter commitments - With a soft fork, the hash for each block’s GCS filter could be added directly to Bitcoin block headers by miners, eliminating the need for the filter header chain and eliminating adversarial cases where a full node could attempt to send an incorrect filter to a Neutrino client. Filters could also be used by other applications that need to quickly scan or search the blockchain.

Developer tools - We’re working on additional APIs and a daemon mode that will allow developers to use Neutrino nodes for general blockchain data queries.

Conclusion

We’re excited to have released the first implementation of BIP 157/158-compliant Neutrino clients in lnd 0.5 and in the Lightning App alpha. While these initial releases are for testnet, mainnet Neutrino is in progress as well. Note also that Neutrino serving support has been added to btcd, and support for BIP 158 has been merged into bitcoind (Bitcoin Core).⁴ In the longer term, we hope to see Neutrino supported by many more node operators and Bitcoin wallet developers so that users can enjoy a better user experience while preserving privacy and mitigating security risks. We believe Neutrino will provide a much improved foundation for the next generation of mobile wallets that will allow Alice to finally succeed in her quest.

Footnotes

If a light client is querying to see if particular funds (i.e. transaction outputs) have been spent, a full node could attempt to incorrectly report that the funds have not been spent. This is an unlikely attack in the case of simple payments, as the parties to the payment can request that the payment be re-sent or can withhold delivery of the good or service. In the case of Lightning, however, the blockchain must be monitored for improper channel close transactions, so omission of transactions could result in a loss of funds.

With Neutrino, if different peers return conflicting filters, the Neutrino client can locate the conflicting block (using the filter header chain), then download the full block. With the full block, the client can automatically determine which peer was correct. BIP 37/SPV clients can alert the user when there are conflicts, but cannot automatically resolve them.
Neutrino clients still leak some information about a user’s wallet, since an adversary can potentially observe which blocks the client is downloading and find addresses that are reused across those downloaded blocks. To mitigate this, we’re exploring the use of Private Information Retrieval for blocks as mentioned above. Nonetheless, we believe Neutrino is still far better for user privacy than other mobile options.
With the advent of Segwit, Bitcoin transaction signatures (witness data) were moved into a separate data structure. Neutrino clients don’t need this additional data, and Segwit allows for the download of “stripped” blocks without witness data, resulting in significant bandwidth savings (potentially 50% or more).
While support for BIP 158 has been merged into bitcoind, support for BIP 157 has not yet been merged, so Neutrino filters can’t yet be served by bitcoind full nodes. Work is ongoing to add full Neutrino support over the coming months.