Blockchain sharding made so simple your dog would understand

Welcome to Hard Fork Basics, a collection of tips, tricks, guides, and advice to keep you up to date in the cryptocurrency and blockchain world.

Scaling blockchain is becoming one of the technology’s great mysteries. Does anyone know how to do it? Can it even be done? Some seem to think so, but of course, it won’t be simple.

There are numerous “solutions” being offered by various developers, researchers, and academics. One such solution being explored is known as “sharding.” In this article Hard Fork is going to take a look under the hood and see what this blockchain sharding is all about.

Sharding is nothing new

The concept of sharding existed long before it became part of the blockchain industry’s lexicon. It’s traditionally used in database management to ensure the database operates at optimum speed and efficiency.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

Traditional sharding does this by separating a database into a series of smaller databases that remain connected, but spread the workload over a great number of servers or nodes on that database network.

Generally speaking, none of these smaller “sharded” databases should have anything that is also replicated in another shard to reduce duplicates of data. This further serves to increase the efficiency and reliability of the database by ensuring every entry is unique.

In a database where records are not interlinked and follow a fairly basic data structure, sharding is a relatively simple process. All system architects need to do is create a database structure that operates securely and efficiently for its given use case.

This might include geographically locating databases on servers in the countries most relevant to the data they store. This is fine when your database is centralized and you have complete control over it, so what happens when you shard the blockchain?

Sharding the blockchain is, umm, complicated

One of the blockchain’s great strengths is decentralization, so doing anything that might compromise this will make the system weaker, and possibly susceptible to attack.

Given that all the transactions stored in a blockchain are linked, it makes sharding this type of database even more complicated. But the principle remains the same, if we can shard the blockchain, we can increase its throughput and increase the number of transactions it can handle every second – which we’re told is a good thing.

How a blockchain can be sharded depends heavily on the blockchain’s underlying consensus mechanism – by its nature a Proof-of-Work-based blockchain is very difficult to shard. As this would involve validating transactions without having access to the entire history of transactions, so new transactions would need to be validated without its history being known. Obviously, this creates a bit of a problem.

However, cryptographer Peter Todd thinks it can be done. It would require another component adding to the functioning of the blockchain, something called a “proofchain.” A proofchain effectively maintains the historical integrity of transactions, without needing to reference the whole database. It can be thought of as an audit trail, one which could be used to punish dishonest miners should the need arise.

In the case of Proof-of-Stake blockchains sharding is a little more attainable. In Proof-of-Stake there are dedicated nodes that take care of validating transactions. These nodes can only validate transactions relational to the value of cryptocurrency they are willing to stake for the opportunity. This encourages decentralization and prevents one node/staker from having too much power.

However, because the nodes only have to validate a portion of the blockchain’s transactions, the database could be sharded and shared amongst them lowering the processing overheard and speeding up the network’s overall throughput.

It’s likely that scaling the blockchain will continue to be one of the industry’s most contentious topics. It’s impossible to dramatically increase the throughput of true blockhain-based systems without altering its underlying structure. In doing so, some of blockchain’s core benefits, like decentralization, might be compromised.