It was a restless night. I just fixed my personal website's software, Kittybox, so it wouldn't hang after the first few hours of working (I hope my bugfix finally worked! I've been chasing that bug for months). In an attempt to stimulate my bored brain I was reading some articles on the IndieWeb wiki and stumbled upon discussions that my posts on this very website sparked.

Except the links didn't work. Argh, I can't read my own posts, this won't do!

Of course, this was all my fault. But, thanks to past me having a lot of foresight, this should be rather easy to fix. First thing I decided to do was to patch articles containing links to my posts so the links would actually point to something - despite me wiping my website's main feed, all of the articles are still there because I know some people in the IndieWeb community have been linking to me. The second thing I did not do yet, and if I don't write it down, I might forget it all, so here's the post so I don't forget to do it.

Preserving compatibility

When I was designing the first versions of Kittybox (back then it didn't even had the name!) I was very inspired by 00dani.me's proposed approach of storing MF2-JSON directly in the database. First I stored it in flat files. Then, as I realized my file storage is incredibly slow, I migrated to Redis to keep my dataset in memory. Then several rewrites happened, and the Rust rewrite happened. The database's underlying format was still almost the same, so it was rather easy to port it over. Except I might've forgotten an important step.

In Kittybox, one post can contain several links. One of them is a UID - an authoritative link to the post, the one that Kittybox would use as a primary key in an SQL database if it used one. But since I use the filesystem as my database, the authoritative link gets transformed into an authoritative path which refers directly to the file with the MF2-JSON blob. All other links are supposed to be symlinks.

Except apparently when I was importing the posts in the new file storage backend, I somehow forgot to make all of those symlinks. And that's why the posts don't work.

Past Vika's foresight

Storing the posts in MF2-JSON as processed Micropub data was actually a very good idea. Pretty much all versions of Kittybox filled in alternative URLs (which would be u-url in MF2-HTML and .properties.url[] in MF2-JSON) which actually can be used to restore those permalinks!

Eventually permalink checking will need to be built into the software itself as a consistency check. I could probably write it like this:

use futures::stream::StreamExt;
let urls = json["properties"]["url"]
    .as_array()
    .unwrap_or_default()
    .into_iter()
    .filter_map(|s: serde_json::Value| s.as_str().ok())
    .map(|url| url_to_path(&self.root_dir, url);

let mut url_stream = tokio_stream::iter(urls);
url_stream.for_each_concurrent(2, |path| async move {
    if let Err(err) = tokio::fs::symlink_metadata(path).await {
        if err.kind() == std::io::ErrorKind::NotFound {
            let link = url_to_path(&self.root_dir, url);
            let basedir = link.parent();
            if basedir.is_err() {
                warn!("Database consistency check: couldn't calculate parent for {}", link);
                return;
            }
            let basedir = basedir.unwrap();
            let relative = path_relative_from(&canonical_path, basedir).unwrap();
            if let Err(Err) = tokio::fs::symlink(relative, link).await {
                warn!("Database consistency check: failed to restore symlink {} for {}: {:?}", canonical, path);
            }
        }
    }
});

I'm pretty sure this won't compile out of the box because of borrow checking and variables which need to be bound, but you get the idea. It can be performed on every read (for maximum correctness) or there could be scheduled tasks that sweep the database and perform those consistency checks - I think the second one is the better idea, since it allows me to check even posts that were completely forgotten by everyone. But I'll need to learn how to schedule tasks to run at certain intervals in Tokio - I'm sure there's a function for that though.

When will it be complete?

I hope it will be built into the software soon. For now, I will leave this as a to-do of sorts. A reminder to myself and an example for the others - of both my foresight and my mistakes.

And for now, I could potentially build a script that recursively walks my directory tree and restores symlinks via a cron job. It'll probably work just as well too. But I'll do it later. I wanna sleep...and coffeeeee....

good morning indieweb

i have returned somehow from having my site in semipermanent downtime

please don’t be gentle with it i need a little bit of sustained load so i can check a hypothesis

warp is amazing to cover your code with unit tests. Since everything is a filter, your logic can be tested in isolation (and components that are required, such as database connections, can simply be mocked).

Recording for future reference: I need easy access to creating new channels in Kittybox. The update for the top bar has made all of the channels accessible, so I want to separate my feeds to keep some of the in-the-moment things out of the main feed - for that I need to create a channel, and I don’t know of any Micropub clients that are capable of creating feeds the Kittybox way (specifically, posting an h-feed with a name property as a JSON object).

Inspired by the mess I’ve made through the years and petermolnar.net’s post I have decided to make all of my previous posts unlisted. The permalinks are not broken - I have preserved what I currently have in the database.

So, I guess I might as well introduce myself once again.

Hello. I’m Vika. I’m currently residing in Russia and I’m the developer of Kittybox, an IndieWeb-enabled all-in-one blogging solution. My favorite thing is homemade burgers and one of the most hated things in my life is all the mild inconveniences caused by capitalism crawling into our lives like a Ctulhu horror. They really add up.