The ongoing emergency at Twitter has made a lot of us take a serious look at social media. After a lot of debate here at Pi Towers, we’ve now spun up our own Mastodon instance. The best thing about it? It’s running on a Raspberry Pi 4 hosted at Mythic Beasts. Last time we talked about Mastodon, we told you why we were doing it. Today’s post talks about how you can join us.

You can join us on Mastodon by creating an account with one of the many public Mastodon service providers at the Mastodon website. Or you can read on, and spin up your own instance in the fediverse. Here’s Pete from Mythic Beasts to explain how. — Alasdair Allan

Into the fediverse

One email account can send and receive email from all email providers. Email is different from social media; with current social media you need an account with each different social media provider if you want to communicate with users of that social media service. You can’t switch social media provider without leaving your friends behind and starting anew on the next service. Or, more realistically, ending up having to check ten different accounts every day just to keep up.

The alternative is the fediverse. Here, there are thousands of providers of a social media service, but the services all interoperate and communicate. Raspberry Pi can post something on their server, the post can be boosted by Mythic Beasts on our server, before being replied to by a follower at fosstodon.org. Three different “providers” of social media services all seamlessly interoperate.

Different servers have different policies, and some specialise in specific areas or provide additional tools like maths or translation. At Raspberry Pi, the team took the decision that they should run their own server just for the company fediverse presence. You can tell it’s official, as it’s a Raspberry Pi-owned domain name; and Raspberry Pi can run their own publishing and moderation directly. The account can’t get banned or shadow banned; the worst that happens is nobody looks at the content because it’s boring. Which can be fixed by writing more interesting things. Raspberry Pi can’t pay for adverts and force content on you – the fediverse just doesn’t work like that.

Pi in the sky ☁️

Ideally, we want to run our Mastodon server on a Raspberry Pi because it’s a super-capable little computer. We also want it in the cloud at the core of the internet with reliable power, and lots and lots of network. Handily, Mythic Beasts run a Raspberry Pi cloud and offer managed services, so we can run the Raspberry Pi Mastodon service, keep it updated and (coming later) scale it up to a cluster of Raspberry Pis when the initial single server setup runs out of puff.

We’re going to use the base quad core 4GB Raspberry Pi 4 here, which is fine for a typical instance with moderate follower counts and small numbers of users. One thing to note is that a Raspberry Pi server in the cloud doesn’t get a public IPv4 address. There aren’t enough IPv4 addresses for the internet (4 billion in total) and we’ve run out. The waiting list for a block of 256 IPv4 addresses is now 9 months and 1100 companies long.

You can buy them off companies in blocks of 256 at around $50 each, significantly more than the base Raspberry Pi 4 at $35.

The future is IPv6, which has a nearly unlimited address space and this is what Mythic Beasts use for the Raspberry Pi cloud. However, IPv6 addresses can’t talk to IPv4 addresses so for every mastodon instance to follow every other we’re going to need translation. Mythic Beasts provide translation as standard. If you try and talk to something that’s IPv4 only, the DNS server makes up a fake IPv6 address that forwards your traffic onwards 64:ff9b::<your-ipv4-address-in-hex>.

For example,

$ ping -n mastodon.social
PING mastodon.social(64:ff9b::88f3:669c) 56 data bytes
64 bytes from 64:ff9b::88f3:669c: icmp_seq=1 ttl=43 time=19.2 ms

For inbound connections Mythic Beasts provide a proxy service on proxy.mythic-beasts.com which listens on IPv4 and forwards the connection to your Raspberry Pi over IPv6.

When you press order on the form, this turns on a Raspberry Pi server in the Mythic Beasts Raspberry Pi cloud running 32-bit Raspberry Pi OS. You need to add your ssh key through the control panel, and then you can log in as root.

Once you’re logged in, we’re going to try to install Mastodon. This guide is based on the official instructions with corrections and fixes for the Pi cloud and the 4.0.2 release.

Looking inside Mastodon

First we’ll start with a sketch of what Mastodon is, and how the pieces fit together.

What’s going on here? There are two data stores: Postgres (permanent) and Redis (temporary cache data). There’s a streaming server written in Node.js, along with the main Ruby on Rails application, which is called Puma. These two talk to Nginx which is a standard webserver forwarding the requests to the right places, which encrypts all the data to keep things secure. We have a scheduler called Sidekiq which runs tasks in the background.

Starting the install

First, we update our system to make sure we have all the latest packages and make sure we have the basic tools we need.

$ apt-get update; apt-get -y upgrade
$ apt install -y curl wget gnupg apt-transport-https lsb-release ca-certificates sudo

You should note that the official Mastodon documentation tells you to get Postgres direct from postgres. Right now, they don’t provide a binary for 32-bit Raspberry Pi OS, so we’re using the Raspberry Pi build of Postgres instead.

Then we install postgresql,redis, nginx and some tools for building other software we’ll need later on.

$ apt update
$ apt install -y \
  imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git-core \
  g++ libprotobuf-dev protobuf-compiler pkg-config gcc autoconf \
  bison build-essential libssl-dev libyaml-dev libreadline6-dev \
  zlib1g-dev libncurses5-dev libffi-dev libgdbm-dev \
  nginx redis-server redis-tools postgresql postgresql-contrib \
  certbot python3-certbot-nginx libidn11-dev libicu-dev libjemalloc-dev

This is where you take a break for your first cup of tea. Between them, these packages pull in about another 300 dependencies, which takes a while to install.

We install the Node.js repository so we can pull packages from there to build the streaming server later. For this, we download their setup script and run it, trusting NodeSource.

$ curl -sL https://deb.nodesource.com/setup_16.x | bash -
$ apt-get -y install nodejs

We need to set this to stable versions in the package manager so we can install the Node.js dependencies later:

$ corepack enable
$ yarn set version stable

…but this fails, saying, “Internal Error: Error when performing the request.”

The quick way to debug these things is to use strace, which follows all the calls the application makes. We rerun with strace and get a lot of output, but hidden in there is a clue.

$ strace yarn set version stable
…
connect(19, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("104.16.17.35")}, 16) = -1 EINPROGRESS (Operation now in progress)
epoll_ctl(13, EPOLL_CTL_ADD, 19, {EPOLLOUT, {u32=19, u64=19}}) = 0

You can see that it’s trying to connect to 104.16.27.35, which is an IPv4 address owned by Cloudflare, despite the fact Cloudflare offers IPv6 on all its services. Node.js prefers IPv4 because ‘it always works’.

There’s a piece of magic for badly behaved applications. It’s called tnat64. This library will intercept any calls to connect to IPv4 addresses, replacing them with a call to the IPv6 equivalent address provided by our NAT64 service.

$ tnat64 yarn set version stable

The application is written in Ruby on Rails. We deploy this by creating a user account just for Mastodon, building a Ruby environment dedicated to Mastodon with the supporting bundler tool for building Mastodon later. Compiling Ruby is a lengthy step, so you’ve definitely got time for another cup of tea and a chance to read the paper.

We create the user and switch to it:

$ adduser --disabled-login mastodon
$ su - mastodon

We then download and install Rbenv, which will build a Ruby environment for us based on 3.0.4 (the version required by Mastodon 4.0.0). The official docs say 3.0.3, because they still target Mastodon 3.5.3.

$ git clone https://github.com/rbenv/rbenv.git ~/.rbenv
$ cd ~/.rbenv && src/configure && make -C src
$ echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(rbenv init -)"' >> ~/.bashrc
$ exec bash
$ git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
$ RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.0.4
$ rbenv global 3.0.4
$ gem install bundler --no-document
$ exit

Now we can start configuring and installing Mastodon. We create a database inside Postgres for Mastodon to store its data.

$ sudo -u postgres psql
CREATE USER mastodon CREATEDB;
\q

Downloading and installing Mastodon

We become the Mastodon user again, and check out the latest stable version from git, setting it to a deployment setup. This is another lengthy step, so take the opportunity to recharge your teapot.

$ su - mastodon
$ git clone https://github.com/tootsuite/mastodon.git live && cd live
$ git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)
$ bundle config deployment 'true'
$ bundle config without 'development test'
$ bundle install -j$(getconf _NPROCESSORS_ONLN)
$ yarn install --pure-lockfile

Now we can tell the install script to compile and install it all, which includes all the startup/shutdown scripts. We can now to set up and configure Mastodon.

RAILS_ENV=production bundle exec rake mastodon:setup

We set ours up as a single-user server. We configure our domain name, with defaults for Postgres/Redis as they’re on localhost and we set to send email via localhost.

During the assets-precompile step, Node.js tries to download resources using IPv4 again. You have to re-run this with the tnat64 trick we used earlier.

$ RAILS_ENV=production tnat64 bundle exec rails assets:precompile
$ exit

You then need to set up the local mail server to relay mail through the Mythic Beasts mailservers, which authenticate based on your source IP.

$ dpkg-configure --reconfigure exim4-light

We set exim to smarthost mail out through mx.mythic-beasts.com. This may not be generally applicable depending on what email address you use and your SPF policies. (Fixing your email is beyond the scope of this blog post!)

DNS

We set up AAAA records pointing directly at the Pi, and A records pointing at the Mythic Beasts Proxy (46.235.225.189 and 93.93.129.174).

We tell the proxy through the Mythic Beasts control panel to forward our domain name to the IPv6 address on our Pi.

You don’t have to use our proxy; you can use any CDN that supports IPv6 origin.

Nginx

Finally we’re at the user-facing portion of the stack. We copy the default configuration file in:

$ cp /home/mastodon/live/dist/nginx.conf /etc/nginx/sites-available/mastodon
$ ln -s /etc/nginx/sites-available/mastodon /etc/nginx/sites-enabled/mastodon
$ /etc/init.d/nginx reload

…and edit this file to put our domain name in. We then set up ssl by running Certbot.

$ certbot --nginx -d <your-domain-here>
$ /etc/init.d/nginx reload

Start on boot

Systemd is the supervisor process for Linux. It needs to be told about Puma, Sidekiq and the streaming service, and to start them all up now.

$ cp /home/mastodon/live/dist/mastodon-*.service /etc/systemd/system/
$ systemctl daemon-reload
$ systemctl enable --now mastodon-web mastodon-sidekiq mastodon-streaming
$ systemctl start mastodon-web
$ systemctl start mastodon-sidekiq
$ systemctl start mastodon-streaming

The mastodon-web process takes about a minute to start up, after that, https://your-domain-here/ should offer your own instance of Mastodon. You can log in using the password that was auto-generated earlier.

We aren’t quite there yet. You can’t follow anyone on an IPv4 only server. Deep inside Mastodon there’s a list of private IP addresses, and it won’t go and fetch data from private IP addresses as that data might get republished out to the internet. This list includes any IPv6 address beginning 64:ff9b:: which is the magic IPv6 address space we use for IPv4 compatibility. We can allow this by adding it to the configuration file:

$ echo ALLOWED_PRIVATE_ADDRESSES=64:ff9b::/96 >>/home/mastodon/live/.env.production

then restart.

$ systemctl mastodon-web

Wait another minute for the restart, and you can now follow the whole internet.

Push to production

You now need to add all the other things you’d configure for a production web service:

• A firewall
• Nightly backups of the database and files
• Monitoring that will alert you if the service is offline
• Performance graphing

Mythic Beasts has this as a standard service for all managed customer applications. The service is auto-configured by script, so we’re not going to dive deep here.

The Raspberry Pi instance at https://raspberrypi.social/ is running the slightly faster 2GHz/8GB Raspberry Pi 4 and with over 15,000 followers is starting to look like it has some work to do.

Yet more network compatibility

You might notice that Raspberry Pi and Mythic Beasts are currently still running the previous version of Mastodon, (v3.5.3) as they were set up before 4.0.2 was released. If you’re really careful about reading the source code you’ll also notice that this version predates a vital IPv6 fix. When fetching remote content, Mastodon would look up all the IP addresses for a server, and pick two at random to try fetching content from. If a server had multiple IPv4 addresses, sometimes it would get just two IPv4 addresses back in the list, and fail to connect to both. The reverse can also happen: an IPv4 only server can be returned two IPv6 addresses and fail in the same way. The fix in 4.0.2 is to always guarantee two IPv6  and two IPv4 addresses, so that IPv6 will be preferred if it’s available, with IPv4 fallback if not.

For the Raspberry Pi Mastodon launch we worked around this with a horrible hack – clatd. Clatd gives your Raspberry Pi a default route for IPv4 that terminates on a virtual interface. This then forwards all your IPv4 connections to the 64:ffd:: IPv6 address that then gets turned back into IPv4 to go on the destination server. This works, but is horrible.

To explain – when raspberrypi.social wants to connect to social.mythic-beasts.com, a third of the time it goes direct if the IPv6 address is first in the list.

The other two thirds of the time, it opens an IPv4 connection to clatd, which opens an IPv6 connection to our NAT64 server, which opens an IPv4 connection to our proxy server, which then opens an IPv6 connection to social.mythic-beasts.com.

You can measure the speed difference: the IPv4/IPv6/IPv4/IPv6 path is routinely 50-100ms slower. With the upgrade to 4.0.2 we’ll be reverting this back to the standard NAT64/DNS64 setup in this example, and we’re profoundly grateful to ClearlyClaire for the IPv6 fixes.

Where now?

It’s likely that as usage of the instance grows (the Raspberry Pi account has grown from 800-odd followers when we initially spun the instance up to over 23,000 followers today), we’re going to have scale things into a cluster. When we get to that stage, we’ll have to talk again!

We spun up our new Raspberry Pi instance in single-user mode. However, if you’re thinking of hosting your own instance where you’ll be taking on sign-ups, you might want to think about registering with the copyright office and designating an agent to receive DMCA reports. Those operating in the UK or EU, or within the EEA, might also want to think about necessary steps around compliance with the GDPR and Digital Services Act. — Alasdair Allan