EdGo Writeup Pt 2.

About

This is pt2 in a writeup about the EdGo app, which is an unofficial app client for the EdDiscussion platform, explaining the technical architecture and decisions.

Tech Stack

Firstly a small background, EdDiscussion is a web-based forum platform with no native apps. Because of this, I chose to write it in React Native (Expo) as it was a great fit for EdDiscussion's API with a good balance between native performance and web API compatibility.
As a side note, I attempted a Sparkling+Lynx stack, but it is very much in beta and it did not compile at all on Windows.

For database operations, the primary DB utilises Expo-Sqlite interfaced through DrizzleORM due to its developer experience, although DrizzleORM has been patched to support asynchronous operations, which will be elaborated on later. Additionally, react-native-mmkv is used as a persistent KV store, and expo-secure-store is used for sensitive data (API Keys).
For styling, I wanted a Tailwind-like experience, and chose uniwind over nativewind as it was far easier to install and integrate, in addition to apparently better performance.
Finally, Effect-ts is used, it handles most business logic and data fetching. This was a mistake as this meant I was learning both Effect-ts and React-Native simultaneously and this made for a cliff of a learning curve.

Local First

The core of the EdGo's technical architecture is that the app should be local-first, where the local DB results are treated as the source of truth, and API calls are used to sync the DB.
In practice, this means a heavy use of the useLiveQuery hook, which returns data from a query and automatically retriggers a rerender when query results change.
So when a user opens a screen, it triggers a API fetch, in the meantime, the screen renders the DB results, when the API fetch is resolved, it updates the db, which triggers a rerender with the updated results.
To prevent extremely large queries, data fetches are paginated, which showed a significant flaw, as when loading a new chunk of data, the UI would be stuck waiting for API queries to resolve despite only fetching data. This didn't occur when the user was offline.
This was eventually root-caused to be a limitation in DrizzleORM itself as its Expo-Sqlite driver is synchronous, and it turns out that the new page fetch was queued behind the previous page sync operation, which was waiting on the API results. When offline, the API call would fail almost immediately which terminated the sync and unblocked the next page sync.
This is still an issue, and my "fix" was adopting a hacky patch to support asynchronous operations, while waiting for the Drizzle driver to be updated to be asynchronous.(Github Issue).

Rendering Threads

One of the other big technical hurdles was rendering threads and comments, as EdDiscussion returns these through a custom HTML-like XML schema which caused a couple of issues.
Firstly, TurboXML (XML parsing library) did not support self-closing tags, which EdDiscussion used for line breaks. TurboXML had to be patched to support this and to lightly restructure the returned data to be more easily parsed.
Secondly, the largest difficulty with the rendering is the text node merging. React-Native does not support copying text from across different Text nodes, hence a core part of the XML rendering was to collect all consecutive text nodes and merge them into a single Text node.
Additionally, as part of the XML schema, each text formatting (bold, italics, underline etc.) had its own node.
To solve this, the root XML tree firstly runs as expected until a paragraph node is hit, as EdDiscussion used that tag as a container for all text-based nodes.
Then each child is recursively searched for text children until it hits a leaf, before moving back up the tree merging the text nodes as we go.
Finally by the time we get back to the root paragraph node, it is parsed as a single Text node, with subnodes for text formatting.

Storage Layer

For storage, it is quite simple, the database is used as the source of truth, while a react-native-mmkv keystore acts as a caching layer, particular for XML parsed threads and comments.

Conclusion

The creation of EdGo was a really good learning opportunity.
After this experience, given the choice I would use React-Native (Expo) again, Effect-ts if an application fit its usecase, but likely not DrizzleORM on mobile, until its Expo-Sqlite driver properly supports asynchronous operations.

Thanks

A special thanks to smartspot2 (Alec Li) for his work documenting most of the Ed Discussion API here.

Addendum: Additional API documentation

As an aside, I had to reverse engineer more of the Ed API to get some features to work.

Region (GET)

https://edstem.org/api/region
This returns country_code, default_region, which are both 2 character country codes e.g. (us, au)

Voting (POST)

https://edstem.org/api/threads/{threadId}/upvote
https://edstem.org/api/threads/{threadId}/unvote

Search (GET)

https://edstem.org/api/courses/{courseId}/threads/search?{params}
Where the parameters are query, sort, limit, offset, sort, limit and offset are already documented by smartspot2, and query is simply a string that you want to query from, it returns a standard thread response.

WebSocket API

wss://edstem.org/api/stream
Ed's WebSocket API isn't particularly complex, but it has a couple of things you need to get right.

Authentication

Similar to the HTTP endpoints, the WebSocket can be authenticated using a bearer token, however, the standard WebSocket API does not support headers.
Afaik there is no way to authenticate using an API token on the browser, as the X-token used to authenticate seems to be different from the API token.
Alternative WebSocket implementations such as NodeJS do support headers on the Websocket handshake and will work, e.g. React-Native WebSockets

API Request Formatting

Ed's WebSocket API works by sending a key-value pair { id: 1, type: "" },, where ID is a incremental identifier which resets on every WebSocket connection and type is the WebSocket API request type.

Unread Message Counts (WebSocket)

Note: This is the only WebSocket request type I reverse engineered, there are more, I just didn't need them.
{ id: {ID}, type: "thread.unreadCounts" },
It returns the unread counts in this format