Skip to main content

Network protocol v0

Status: Complete Height: T-1361 T-1665

Background​

See this PRD for an overview of the broader product vision, use cases, and requirements.

Problem​

The network layer consists of the nodes that user clients can interface with in order to relay, filter, and store messages.

Functional Requirements​

  • System provides the ability to send messages to other users while they're online.
  • System provides the ability to send messages to other users while they're offline.
  • System provides the ability to stream messages being sent to a user while they're online.
  • System provides the ability to retrieve past messages sent to a user, even if they were offline.

Non-functional requirements​

  • Performance, reliability requirements...
    • TBD
  • Security, privacy requirements...
    • TBD
  • The future network will be incentivized and decentralized, where anybody can operate a full node.

Non-goals​

  • Economic spam controls is out of scope for this iteration (v0) of the network.
  • Economic node incentives, and the ability for anybody to operate a full node in the XMTP network, is out of scope for this iteration (v0) of the network.

Solution​

The Waku2 protocol provides a scaffolding for an open messaging protocol that we can leverage.

  • The XMTP network is a deployment of Waku2, where the network-level protocol of XMTP is the Waku2 protocol.
  • All network-level full nodes are initially operated by XMTP Labs.
  • Client-level light nodes are embedded in the XMTP client SDK, with additional protocol layers implemented here for message authentication, encryption, and user data storage.

Message storage and retention​

  • Messages need to be stored in the network so that users can retrieve messages sent to them while they were offline.
  • Waku2/store protocol spec: https://rfc.vac.dev/spec/13/
  • Scalability of storage is not handled in this initial iteration, with all data replicated across all nodes.

Security and privacy​

Performance and reliability​

  • DDoS protection via Cloudflare
  • Spam mitigation via libp2p rate limiting

Plan​

  • Proof-of-concept
  • XMTP-operated T-1665
    • Deployment of multi-node network of go-waku nodes.
      • Peer discovery
      • Storage
    • Observability monitoring, alerting, operational tooling
    • Conformance test suite
      • E2E tests
      • Performance, reliability, security tests
    • Cloudflare DDoS protection
    • Performance and reliability optimizations
    • Security review
    • TBD
  • Decentralized (future)
    • Resource usage accounting and settlement (economic node incentives)
    • Economic spam controls
    • Performance and reliability optimizations
    • Security review/audit
    • TBD

Questions​

  • What is the data retention policy of the network? Are stored messages removed after some time, or after some volume of messages for a user?
  • What are the performance and reliability boundaries of the Waku2 network? How well does it scale and when/where does it break down?
  • How to mitigate DDoS attacks on the network and/or specific nodes in it?
  • How to mitigate spam via high volume of messages to a recipient? Should the network layer, in this iteration, play any role in that?
  • What are the performance and reliability tradeoffs of using JSON-RPC vs TCP via libp2p directly for interfacing with Waku2 nodes?
  • Partitioning the shared relay pubsub topic.
  • How can a receiver or the network validate that the timestamp on a message is accurately representing the time that it was sent?
    • Option described in the Waku2 specs: A robust and verifiable timestamp allows the receiver of a message to verify that a message has been generated prior to the claimed timestamp. One solution is the use of open timestamps e.g., block height in Blockchain-based timestamps. That is, messages contain the most recent block height perceived by their senders at the time of message generation. This proves accuracy within a range of minutes (e.g., in Bitcoin blockchain) or seconds (e.g., in Ethereum 2.0) from the time of origination.

Possible Solutions​

  • Waku2 nodes deployed as the XMTP network
    • Summary
      • XMTP network consists of Waku2 nodes; it's a Waku2 network.
    • Pros
      • Significantly reduced effort to build XMTP clients on top of a Waku2 network.
      • Leverage years of work and learnings from the Vac/Status teams.
    • Cons
      • If we decide to diverge significantly from the functionality of Waku2 and the interface it provides, then clients will need to be updated to match that evolution.
      • Unclear how this performs under load or abusive actors at this time.
  • Waku2 embedded in XMTP nodes with custom API
    • Summary
      • XMTP network consists of custom-built libp2p or RPC/HTTP nodes as a thin wrapper around Waku2 relay, filter, and store interfaces, with the Waku2 node embedded in the XMTP node.
    • Pros
      • Facilitates swapping out the implementation behind the thin interface without the users of that interface being aware or needing to update their clients (in theory; often messier in practice).
    • Cons
      • Additional layers of indirection on top of an existing well-formed protocol, can result in more maintenance, performance, and operational complexity.
  • Custom-built XMTP nodes using libp2p and IPFS
    • Summary
    • Pros
      • Leverage existing IPFS relay, store, and replication.
      • Leverage existing IPFS peer management and clustering.
      • Leverage IPFS content-addressable IDs for validation of message integrity.
    • Cons
      • Unclear how this performs under load or abusive actors at this time.
      • If messages are stored in IPFS logs per receiver in order to faciliate efficient retrieval, then unlinkability is not provided.
      • Unclear how to manage partitioning message data to be pinned across the network.