Conversations Abstraction

Status: Draft
Date: Feb 9, 2022

Background

To make development against the XMTP SDK easier, we need a simple abstraction to represent the concept of "conversations", rather than forcing the developer to understand the intricacies of introduction topics and conversation topics.

Problem

Requirements

There are a few clear use-cases for the conversation abstraction

1. Get a list of all conversations that have been started
2. Listen for new conversations that were started while the user's session is active
3. Get all messages in a 1:1 conversation
4. Listen for new messages in a 1:1 conversation
5. Listen for incoming messages across all conversations

Non-Goals

1. Persistence. While a persistent store of a conversations is desirable, and could be built on top of this abstraction, it is not an immediate goal.
2. 1:many conversations. That's a problem for another day, and may require a rethink of the interface

Proposed solution

The proposed solution is stateless and does not keep a store (even transient) of either the conversation list or message history.

interface Conversations {
    // Return a list of conversation objects for all conversions that have an introduction message.
    // Would dedupe any repeated introductions
    list(): Promise<Conversation[]>;
    // Listen for any new conversations that have been started since the stream has been initiated
    stream(): Stream<Conversation>;
    // Initiate a new conversation. Will throw error if the user is not registered in the XMTP network
    // Does not send any message on its own, but returns a Conversation instance so that you can use conversation.send(...) for an identical iterface for first/subsequent messages.
    newConversation(address: string): Promise<Conversation>;
}

interface Conversation {
    peerAddress: string;
    // Load all available messages for a 1:1 conversation
    messages(): Promise<Message[]>;
    // Stream all incoming messages
    streamMessages(): Stream<Message>;
    // Send a message into the conversation
    send(msg: string): Promise<void>;
}

Open Questions

Is a stateless Conversation interface enough?

I could imagine a stateful store being pretty useful. Imagine if you could just call await Conversations.create(client) and it would fetch any data from LocalStorage, update it with a call from ListMessages, and then watch in the background for all changes. Then you could just have a single list of conversations that was fully up-to-date without having to understand any of the internals.

There are some pretty strong reasons we should consider building a stateful store.

1. Waku messages last at most 30 days. Persisting the conversation list to LocalStorage lets sessions last much longer. And once we have to integrate data that lives in local storage, data that is gathered from querying the store with ListMessages, and data that is streamed in real-time, it is putting a lot of work on the user to handle all of that.
2. Even within 30 days, users of the SDK will have to do a one-time fetch of all conversations and then stream for real-time updates, then integrate that data on their own. While we can do some deduping in the "fetch all" stage, and maybe in the "stream messages in real-time" phase, we can't dedupe them globally and ultimately the client will have to be responsible for figuring out which streamed messages are truly new.

So, this proposal is not really saying "we won't build a stateful store", so much as "it's a less pressing priority right now". Ultimately, the building blocks outlined above will be an important input to that future stateful store. Building the stateful version is much more complicated - and harder to fix if we mess up since we will have to migrate persisted data - so if we are to do it at all, we should do it thoughtfully.

Are async iterators the only way we want to offer access to streaming data

Async Iterators are great, but come with a few important drawbacks.

1. Browser/build support. This is a brand new Javascript feature and not all browsers/bundlers are configured to support for await ... syntax, which would make them harder to use for at least some developers without configuration changes. Anyone can still use a while loop and the next() function, but it's far less slick.
2. Streams that never end are an awkward fit for async iterators. If a caller wants to dispose of a stream they have to either figure out how to break from the loop, or keep a copy of the stream instance available to call stream.return() from somewhere else. Observables and EventEmitters both have better understood methods of stopping early.
3. Hard to integrate with synchronous code. Many developers have lint rules that discourage use of Promises that are not awaited for. That makes it awkward to use these infinite streams without blocking your whole application. Something like an event emitter or Observable would work anywhere, including inside synchronous functions.
4. Performance. Async iterators for large data sets are much slower than other methods of iteration. Every item iterated yields the event loop using process.nextTick

I'm in favour of exposing an Async Iterator interface, but I suspect we will need other ways of accessing streaming data as well. I'm proposing we defer that work for another day.