Engineering for the future

derpinews:

Derpibooru’s been around for about 4 years now, and starting to mature as a site. We’re not implementing as many new features, especially when it comes to the core of the site, and we’re spending more engineering time on improving what we’ve got instead of adding new things.

Back when I first started writing the site, I wanted to avoid a lot of the issues that plagued other image board software. That required a fundamentally different data model, and at the time, MongoDB was the only option. I picked MongoDB as the core database for the site, and we’re still running on it today, albeit with ElasticSearch running alongside it to answer search queries. An average page load on Derpibooru will talk to ElasticSearch once, especially if there’s more than one image on the page, and MongoDB maybe 10-20 times, depending on the page (though we cache many of these).

However, with a million images in the database, over 100,000,000 user interactions (votes, faves, etc) and so on, we’ve been stung by MongoDB a few times. It’s not as fast as a traditional database for a lot of the queries we’re using it for. And it’s got issues with concurrency and consistency, which makes engineering certain aspects of the site difficult, and has lead to extended downtimes while we fix things. It’s becoming increasingly problematic at larger scales, causing us to move more parts of the site to ElasticSearch (forum post consistency is a result of this).

For the last year we’ve jokingly discussed moving away from MongoDB – it’s a huge amount of work – but last month byte[] (liamwhite1) actually started to sit down and port the software running the booru to use PostgreSQL, our selected ideal database. PostgreSQL has recently added support for the data model we’re using in MongoDB, namely arrays, so it’s an ideal fit for most of our work. It won’t replace ElasticSearch, but it’ll entirely replace MongoDB, improving performance, reliability and consistency in most of the site’s core.

This work is now almost complete, and we’ve spent some time planning how we migrate the site. This is by far the largest change to the site’s codebase since we introduced ElasticSearch around the time Ponibooru was closed. The work is far-reaching and has impacted practically every part of the site, so we’re anticipating there will be problems, and that we’ll need some time to fettle and fix them.

We’re going to need to turn everything off for a day or so while we move all the data across, and we’ll then need a bit of time to verify and validate. We’ve implemented a read-only mode so we can keep the site largely afloat during much of this time.

We’ll be starting this work on Monday, the 26th of October and anticipate that it will be entirely complete by Friday. With luck and care we’ll be done much sooner than that.

Love,
Clover (and team)

tl;dr We’re getting rid of MongoDB, how ‘bout that, but babies, don’t you panic, by the light of the night it’ll all seem alright, we’ll get you PostgreSQL instead. Site’ll be a bit rocky next week.