Skip to main content

Scale Estimates

Background​

With the newfound focus on getting the protocol production ready, it would be useful to have some rough estimates of the amount of throughput and storage we should be expecting in the medium term.

We want to be able to handle all the traffic we expect to receive in the next 6-12 months, but don't want to waste energy prematurely optimizing. While it is difficult to predict demand for a product that doesn't exist, there are some heuristics that can be helpful here. Even back-of-the-envelope estimates of different types of traffic can be useful to ensure we are putting our scaling resources in the right place.

Proposed Heuristics​

For the purpose of these calculations, we will divide our potential types of messages into three buckets.

  1. DApp -> individual communication.
  2. Individual <-> individual conversations (with both participants consenting to the conversation)
  3. SPAM

Let's look at what we know about these types of conversation and try and figure out a reasonable upper-bound for what a very successful launch looks like.

DApp -> Individual​

If we look at the pattern of announcements on various NFT community and DeFi community Discord #announcements channels we can get a sense of the type of the communication that will be coming of this type.

Announcements are infrequent (maybe a few a week for an active community), but are expecting to go out to many users. How many?

These are top 10 NFT communities by owner count, according to Rarity Tools

If we expand the list to the top 250, the average owner count is 4443.

Let's say that we can get 50 of the top 250 communities over the next 12 months to use XMTP to send announcements 4X per week (a huge outcome IMO).

The number of messages per day works out to (50 * 4443 * 4) / 7 = 126,942. That's roughly 1.5 messages per second, albeit delivered in a fairly bursty way (most messages will come over the course of a few minutes per day). A full year at that volume would result in us needing to store 46,333,830 messages.

I had a harder time finding breakdowns of Defi tokens by number of wallet addresses, but if someone with a Glassnode account wanted to add them that would be an interesting addition to the conversation.

Individual -> Individual​

Glassnode calculates that the number of addresses active in the past 7 days is somewhere around 550k. The total number of addresses that have ever been active on Ethereum is closer to 4M.

While that seems like a fairly large number, we have a lot of reasons to believe Individual -> Individual messaging rates are going to be quite low.

  1. There are a ton of alternatives for individuals to communicate 1:1 (Telegram, Twitter DMs, Discord DMs, Signal, SMS, etc).
  2. Changing behaviour of a large number of people takes time
  3. Our product is going to be much much worse than the competition for the forseeable future as a texting tool.
  4. It's hard to imagine a lot of use-cases beyond novelty, and highly specific use-cases, where someone would need to communicate with a random address on the blockchain 1:1. It happens (maybe you want to reach out to a whale that is using a particular protocol, or want to make an offer on an ENS domain)

I believe group messaging would change this dynamic quite a bit, but until we have that I would say that even getting 1% of active Ethereum addresses to engage in a conversation would be a real stretch.

Still, let's put an estimate on what that would look like.

550000 * 0.01 = 5500. So, 5,500 people engaging in one conversation in a given week. Let's estimate a generous 10 messages sent per person per conversation. That works out to ((5500 / 2) * 10) / 7 = 3928 messages per day, or < 0.05/second. Even if we are wrong by a factor of 10, that feels like a rounding error and something we can easily mitigate.

SPAM​

Without any economic disincentive to send SPAM messages in place, these numbers could be pretty wild. If a single bad actor chose to message all 4M Ethereum addresses on any given day that would be 4M messages / day (46 /second). This would be extremely bursty traffic.

In all likelihood, we will have many bad actors in the network. Although many of them may choose to do more targeted attacks (send a phishing link to all BAYC holders), the upper bound of SPAM remains incredibly high.

Without effective mitigations, the upper bound for SPAM could easily become many millions of messages per day. Hopefully it's less, but because a small number of bad actors can have such an outsized influence I wouldn't bet against it. About 85% of all email is SPAM.

Bad actors may also decide to DOS the network just to watch the world burn.

How large is a message​

I did some quick estimations of the size of a message on the network with the following process.

const shortMessage = await WakuMessage.fromBytes(
  (await Message.encode(alice.keys, bobContact, 'gm', new Date())).toBytes(),
  buildDirectMessageTopic(alice.address, bob.address),
  {
    timestamp: new Date(),
  }
)

const longMessage = await WakuMessage.fromBytes(
  (
    await Message.encode(
      alice.keys,
      bobContact,
      'Cardano thought the constant orphan, yet Bitcoin Cash expected lots of arbitrage behind few FOMO although Litecoin counted lots of proof of stake. Digitex Futures generated few constant exchange! NFT broadcast a burned chain, yet Tezos formed few reinvested consensus process! Basic Attention Token looked at some dormant testnet after some airdrop because Ontology did the automated shitcoin, nor because blockchain generates some coin, Decred returns lots of safe algorithm of many orphan. Cardano counted some hot hardware wallet although Lightning Network is wary of lots of centralised accidental fork until some instamine! Golem limited a testnet! Digitex Futures proves!',
      new Date()
    )
  ).toBytes(),
  buildDirectMessageTopic(alice.address, bob.address),
  {
    timestamp: new Date(),
  }
)

console.log(shortMessage.encode().byteLength)
console.log(longMessage.encode().byteLength)

The short message works out to 818 bytes, the long message works out to 1496 bytes. Let's call it an even 1kb at today's message payload size.

There are a bunch of other factors that will affect the final size stored. Compression of the payloads will help shrink larger messages. Additional database indices will add to the overall storage size. As will adding new fields to the payload, such as the ContentType. And if we decide to allow content types like images or audio, these numbers are wildly off.

As a rough rule of thumb, we can say that for every million messages we store we need at least 1GB of storage.

Conclusions​

Based on the data above, there are a few patterns worth calling out.

  1. Both Dapp -> Individual and Individual -> Individual messaging are likely to be a manageable size in terms of both network throughput and storage in the near future. Even if they add up to tens of millions of messages per year, using tried and tested techniques like database sharding we should be able to performantly query the datastore. Just old-school DBA work can get us quite far with SQL, or we can always migrate to fancier datastores like Planetscale or Cassandra. Even unsharded SQL can vertically scale to tens of millions of rows of storage and 5-10k TPS with a good managed provider like RDS, a well thought out indexing strategy, and a few thousand dollars a month. It's unclear how far we can get with SQLite, but given how easy it is to migrate to a more scalable SQL data provider it likely makes sense to do that in the short/medium term.

  2. Unmitigated SPAM can get us into scary territory pretty fast. And even if we could manage the storage and throughput, the quality of experience on the network would be incredibly poor if we left those messages unchecked.

  3. Dapp -> Individual messages are likely to be the bulk of legitimate network activity in the near term. This may change with group messaging, where we can start to take over some of the high-volume activity that currently happens in Discord and Telegram. But until then most of the messages on the network are likely to come from a small number of Dapp wallets. In other words, we will likely have many more recipients than we have senders.

Item 3 in particular makes me wonder how far we could get with a per-wallet whitelist for sending messages. If we were to limit message sending to only DApps and friends-of-XMTP through some mechanism, and had the ability to revoke these permissions and delete messages when we detect bad actors, we can likely handle a lot of messages sent without the risk of being overrun with SPAM.

I am much less worried about opening up wide access to reads than writes, since they are both more performant from a database perspective and harder to SPAM. A two-tiered launch where anyone with a wallet could receive messages but only whitelisted individuals could send messages feels a lot safer than even a modest launch where anyone has R/W access to the network.