Host your own Mastodon instance on a Raspberry Pi

The ongoing emergency at Twitter has made a lot of us take a serious look at social media. After a lot of debate here at Pi Towers, we’ve now spun up our own Mastodon instance. The best thing about it? It’s running on a Raspberry Pi 4 hosted at Mythic Beasts. Last time we talked about Mastodon, we told you why we were doing it. Today’s post talks about how you can join us.

Our own Mastodon instance

You can join us on Mastodon by creating an account with one of the many public Mastodon service providers at the Mastodon website. Or you can read on, and spin up your own instance in the fediverse. Here’s Pete from Mythic Beasts to explain how.Alasdair Allan

Into the fediverse

One email account can send and receive email from all email providers. Email is different from social media; with current social media you need an account with each different social media provider if you want to communicate with users of that social media service. You can’t switch social media provider without leaving your friends behind and starting anew on the next service. Or, more realistically, ending up having to check ten different accounts every day just to keep up.

The alternative is the fediverse. Here, there are thousands of providers of a social media service, but the services all interoperate and communicate. Raspberry Pi can post something on their server, the post can be boosted by Mythic Beasts on our server, before being replied to by a follower at fosstodon.org. Three different “providers” of social media services all seamlessly interoperate.

Different servers have different policies, and some specialise in specific areas or provide additional tools like maths or translation. At Raspberry Pi, the team took the decision that they should run their own server just for the company fediverse presence. You can tell it’s official, as it’s a Raspberry Pi-owned domain name; and Raspberry Pi can run their own publishing and moderation directly. The account can’t get banned or shadow banned; the worst that happens is nobody looks at the content because it’s boring. Which can be fixed by writing more interesting things. Raspberry Pi can’t pay for adverts and force content on you – the fediverse just doesn’t work like that.

Pi in the sky ☁️

Ideally, we want to run our Mastodon server on a Raspberry Pi because it’s a super-capable little computer. We also want it in the cloud at the core of the internet with reliable power, and lots and lots of network. Handily, Mythic Beasts run a Raspberry Pi cloud and offer managed services, so we can run the Raspberry Pi Mastodon service, keep it updated and (coming later) scale it up to a cluster of Raspberry Pis when the initial single server setup runs out of puff.

A Pi in the sky ☁️

We’re going to use the base quad core 4GB Raspberry Pi 4 here, which is fine for a typical instance with moderate follower counts and small numbers of users. One thing to note is that a Raspberry Pi server in the cloud doesn’t get a public IPv4 address. There aren’t enough IPv4 addresses for the internet (4 billion in total) and we’ve run out. The waiting list for a block of 256 IPv4 addresses is now 9 months and 1100 companies long.

You can buy them off companies in blocks of 256 at around $50 each, significantly more than the base Raspberry Pi 4 at $35.

The future is IPv6, which has a nearly unlimited address space and this is what Mythic Beasts use for the Raspberry Pi cloud. However, IPv6 addresses can’t talk to IPv4 addresses so for every mastodon instance to follow every other we’re going to need translation. Mythic Beasts provide translation as standard. If you try and talk to something that’s IPv4 only, the DNS server makes up a fake IPv6 address that forwards your traffic onwards 64:ff9b::<your-ipv4-address-in-hex>.

For example,

$ ping -n mastodon.social
PING mastodon.social(64:ff9b::88f3:669c) 56 data bytes
64 bytes from 64:ff9b::88f3:669c: icmp_seq=1 ttl=43 time=19.2 ms

For inbound connections Mythic Beasts provide a proxy service on proxy.mythic-beasts.com which listens on IPv4 and forwards the connection to your Raspberry Pi over IPv6.

When you press order on the form, this turns on a Raspberry Pi server in the Mythic Beasts Raspberry Pi cloud running 32-bit Raspberry Pi OS. You need to add your ssh key through the control panel, and then you can log in as root.

This unit has 48 Raspberry Pi 4 servers in 3U allowing 1344 Raspberry Pi servers in a standard rack. It’s under construction; the Ethernet cables go in next, and the switch sandwiches on the top.

Once you’re logged in, we’re going to try to install Mastodon. This guide is based on the official instructions with corrections and fixes for the Pi cloud and the 4.0.2 release.

Looking inside Mastodon

First we’ll start with a sketch of what Mastodon is, and how the pieces fit together.

The architecture of a Mastodon instance and the fediverse.

What’s going on here? There are two data stores: Postgres (permanent) and Redis (temporary cache data). There’s a streaming server written in Node.js, along with the main Ruby on Rails application, which is called Puma. These two talk to Nginx which is a standard webserver forwarding the requests to the right places, which encrypts all the data to keep things secure. We have a scheduler called Sidekiq which runs tasks in the background.

Starting the install

First, we update our system to make sure we have all the latest packages and make sure we have the basic tools we need.

$ apt-get update; apt-get -y upgrade
$ apt install -y curl wget gnupg apt-transport-https lsb-release ca-certificates sudo

You should note that the official Mastodon documentation tells you to get Postgres direct from postgres. Right now, they don’t provide a binary for 32-bit Raspberry Pi OS, so we’re using the Raspberry Pi build of Postgres instead.

Then we install postgresql,redis, nginx and some tools for building other software we’ll need later on.

$ apt update
$ apt install -y \
  imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git-core \
  g++ libprotobuf-dev protobuf-compiler pkg-config gcc autoconf \
  bison build-essential libssl-dev libyaml-dev libreadline6-dev \
  zlib1g-dev libncurses5-dev libffi-dev libgdbm-dev \
  nginx redis-server redis-tools postgresql postgresql-contrib \
  certbot python3-certbot-nginx libidn11-dev libicu-dev libjemalloc-dev

This is where you take a break for your first cup of tea. Between them, these packages pull in about another 300 dependencies, which takes a while to install.

We install the Node.js repository so we can pull packages from there to build the streaming server later. For this, we download their setup script and run it, trusting NodeSource.

$ curl -sL https://deb.nodesource.com/setup_16.x | bash -
$ apt-get -y install nodejs

We need to set this to stable versions in the package manager so we can install the Node.js dependencies later:

$ corepack enable
$ yarn set version stable

…but this fails, saying, “Internal Error: Error when performing the request.”

The quick way to debug these things is to use strace, which follows all the calls the application makes. We rerun with strace and get a lot of output, but hidden in there is a clue.

$ strace yarn set version stable
…
connect(19, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("104.16.17.35")}, 16) = -1 EINPROGRESS (Operation now in progress)
epoll_ctl(13, EPOLL_CTL_ADD, 19, {EPOLLOUT, {u32=19, u64=19}}) = 0

You can see that it’s trying to connect to 104.16.27.35, which is an IPv4 address owned by Cloudflare, despite the fact Cloudflare offers IPv6 on all its services. Node.js prefers IPv4 because ‘it always works’.

There’s a piece of magic for badly behaved applications. It’s called tnat64. This library will intercept any calls to connect to IPv4 addresses, replacing them with a call to the IPv6 equivalent address provided by our NAT64 service.

$ tnat64 yarn set version stable

The application is written in Ruby on Rails. We deploy this by creating a user account just for Mastodon, building a Ruby environment dedicated to Mastodon with the supporting bundler tool for building Mastodon later. Compiling Ruby is a lengthy step, so you’ve definitely got time for another cup of tea and a chance to read the paper.

We create the user and switch to it:

$ adduser --disabled-login mastodon
$ su - mastodon

We then download and install Rbenv, which will build a Ruby environment for us based on 3.0.4 (the version required by Mastodon 4.0.0). The official docs say 3.0.3, because they still target Mastodon 3.5.3.

$ git clone https://github.com/rbenv/rbenv.git ~/.rbenv
$ cd ~/.rbenv && src/configure && make -C src
$ echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(rbenv init -)"' >> ~/.bashrc
$ exec bash
$ git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
$ RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.0.4
$ rbenv global 3.0.4
$ gem install bundler --no-document
$ exit

Now we can start configuring and installing Mastodon. We create a database inside Postgres for Mastodon to store its data.

$ sudo -u postgres psql
CREATE USER mastodon CREATEDB;
\q

Downloading and installing Mastodon

We become the Mastodon user again, and check out the latest stable version from git, setting it to a deployment setup. This is another lengthy step, so take the opportunity to recharge your teapot.

$ su - mastodon
$ git clone https://github.com/tootsuite/mastodon.git live && cd live
$ git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)
$ bundle config deployment 'true'
$ bundle config without 'development test'

Now we can tell the install script to compile and install it all, which includes all the startup/shutdown scripts. We can now to set up and configure Mastodon.

RAILS_ENV=production bundle exec rake mastodon:setup

We set ours up as a single-user server. We configure our domain name, with defaults for Postgres/Redis as they’re on localhost and we set to send email via localhost.

During the assets-precompile step, Node.js tries to download resources using IPv4 again. You have to re-run this with the tnat64 trick we used earlier.

$ RAILS_ENV=production tnat64 bundle exec rails assets:precompile
$ exit

You then need to set up the local mail server to relay mail through the Mythic Beasts mailservers, which authenticate based on your source IP.

$ dpkg-configure --reconfigure exim4-light

We set exim to smarthost mail out through mx.mythic-beasts.com. This may not be generally applicable depending on what email address you use and your SPF policies. (Fixing your email is beyond the scope of this blog post!)

DNS

We set up AAAA records pointing directly at the Pi, and A records pointing at the Mythic Beasts Proxy (46.235.225.189 and 93.93.129.174).

We tell the proxy through the Mythic Beasts control panel to forward our domain name to the IPv6 address on our Pi.

You don’t have to use our proxy; you can use any CDN that supports IPv6 origin.

Nginx

Finally we’re at the user-facing portion of the stack. We copy the default configuration file in:

$ cp /home/mastodon/live/dist/nginx.conf /etc/nginx/sites-available/mastodon
$ ln -s /etc/nginx/sites-available/mastodon /etc/nginx/sites-enabled/mastodon
$ /etc/init.d/nginx reload

…and edit this file to put our domain name in. We then set up ssl by running Certbot.

$ certbot --nginx -d <your-domain-here>
$ /etc/init.d/nginx reload

Start on boot

Systemd is the supervisor process for Linux. It needs to be told about Puma, Sidekiq and the streaming service, and to start them all up now.

$ cp /home/mastodon/live/dist/mastodon-*.service /etc/systemd/system/
$ systemctl daemon-reload
$ systemctl enable --now mastodon-web mastodon-sidekiq mastodon-streaming
$ systemctl start mastodon-web
$ systemctl start mastodon-sidekiq
$ systemctl start mastodon-streaming

The mastodon-web process takes about a minute to start up, after that, https://your-domain-here/ should offer your own instance of Mastodon. You can log in using the password that was auto-generated earlier.

We aren’t quite there yet. You can’t follow anyone on an IPv4 only server. Deep inside Mastodon there’s a list of private IP addresses, and it won’t go and fetch data from private IP addresses as that data might get republished out to the internet. This list includes any IPv6 address beginning 64:ff9b:: which is the magic IPv6 address space we use for IPv4 compatibility. We can allow this by adding it to the configuration file:

$ echo ALLOWED_PRIVATE_ADDRESSES=64:ff9b::/96 >>/home/mastodon/live/.env.production

then restart.

$ systemctl mastodon-web

Wait another minute for the restart, and you can now follow the whole internet.

Push to production

You now need to add all the other things you’d configure for a production web service:

  • A firewall
  • Nightly backups of the database and files
  • Monitoring that will alert you if the service is offline
  • Performance graphing

Mythic Beasts has this as a standard service for all managed customer applications. The service is auto-configured by script, so we’re not going to dive deep here.

The Raspberry Pi instance at https://raspberrypi.social/ is running the slightly faster 2GHz/8GB Raspberry Pi 4 and with over 15,000 followers is starting to look like it has some work to do.

CPU usage on raspberrypi.social

Yet more network compatibility

You might notice that Raspberry Pi and Mythic Beasts are currently still running the previous version of Mastodon, (v3.5.3) as they were set up before 4.0.2 was released. If you’re really careful about reading the source code you’ll also notice that this version predates a vital IPv6 fix. When fetching remote content, Mastodon would look up all the IP addresses for a server, and pick two at random to try fetching content from. If a server had multiple IPv4 addresses, sometimes it would get just two IPv4 addresses back in the list, and fail to connect to both. The reverse can also happen: an IPv4 only server can be returned two IPv6 addresses and fail in the same way. The fix in 4.0.2 is to always guarantee two IPv6  and two IPv4 addresses, so that IPv6 will be preferred if it’s available, with IPv4 fallback if not.

For the Raspberry Pi Mastodon launch we worked around this with a horrible hack – clatd. Clatd gives your Raspberry Pi a default route for IPv4 that terminates on a virtual interface. This then forwards all your IPv4 connections to the 64:ffd:: IPv6 address that then gets turned back into IPv4 to go on the destination server. This works, but is horrible.

To explain – when raspberrypi.social wants to connect to social.mythic-beasts.com, a third of the time it goes direct if the IPv6 address is first in the list.

The other two thirds of the time, it opens an IPv4 connection to clatd, which opens an IPv6 connection to our NAT64 server, which opens an IPv4 connection to our proxy server, which then opens an IPv6 connection to social.mythic-beasts.com.

You can measure the speed difference: the IPv4/IPv6/IPv4/IPv6 path is routinely 50-100ms slower. With the upgrade to 4.0.2 we’ll be reverting this back to the standard NAT64/DNS64 setup in this example, and we’re profoundly grateful to ClearlyClaire for the IPv6 fixes.

Where now?

It’s likely that as usage of the instance grows (the Raspberry Pi account has grown from 800-odd followers when we initially spun the instance up to over 23,000 followers today), we’re going to have scale things into a cluster. When we get to that stage, we’ll have to talk again!

We spun up our new Raspberry Pi instance in single-user mode. However, if you’re thinking of hosting your own instance where you’ll be taking on sign-ups, you might want to think about registering with the copyright office and designating an agent to receive DMCA reports. Those operating in the UK or EU, or within the EEA, might also want to think about necessary steps around compliance with the GDPR and Digital Services Act.Alasdair Allan

36 comments
Jump to the comment form

Avatar

Between writing the guide and publication we upgraded the Mythic Beasts and Raspberry Pi instances to 4.0.2 and reverted the clatd hack. Which makes the world a better and happier place.

Reply to Peter Stevens

Alasdair Allan
Avatar

Very nice, there is only one problem, where do we find all the Raspberry Pi showed in the picture?

Reply to Christian

Alasdair Allan

Well, you can get a hosted one at Mythic Beasts!

Reply to Alasdair Allan

Avatar

I think that was a comment on where to source our Raspberry Pi 4 from, rather than where to host it!!

13 months on from the production issues and I’ve only managed to buy ONE RasPi model 4. I’ve ditched plans to create a couple of projects using Zero 2W’s.

Reply to Fred Cee

Avatar

The foundation has lost their way and forgotten their original goal, the goal that the community helped them to reach. Now they sell most of their systems to businesses whilst young students have to ask friends to give them their old RPi3’s. If they have one. Then they post pictures from a company that uses them for server racks. Maybe it’s time they do the right thing and sell to education first and businesses last.

Reply to Richard collins

Avatar

Nail on the head. It’s very distressing if what you say about production being routed to industry is true. The danger is what I see hapening. Large organizations buying large quantities then bosting the price. $150US at Amazon while the primary sources like Adafruit show out of stock.

Avatar

I think you’re confused. Raspberry Pi and the Raspberry Pi foundation are two different entities now. You’re on the Raspberry Pi Trading Ltd website here. They are responsible of the business part not the mission.

Avatar

Thank you so much for posting this! Given the Twitter emergency, I was just recently wondering how hard it would be start my very own Mastodon instance. I had no idea it was as simple as this!

Reply to Tonald Drumpf

Avatar

I’m looking at doing this very soon, but running it via a docker container so I can quickly move it to other Pi’s if need be. I’ll share the docker-compose and dockerfile.yml once I’ve completed it

Reply to Kevin McAleer

Avatar

Fantastic, Kevin – the Docker route would be ideal!

Reply to Alan Robertson

Avatar

Huzzah! The last part about private addresses was the missing piece in the puzzle. My 4.0.2 is now full operational. Thanks.

Reply to PJ

Avatar

Glad to be of service!

Reply to Peter Stevens

Avatar

Hi Guys, I’m afraid I’m not as lucky as you. I was trying to install Mastodon on my P4/8GB using. Two freshly bought 128BG SD-Cards (one even labeled “high endurance”) broke during installation. Both went into r/o-mode. No chance to repair with e2fsck.

Reply to NH

Avatar

This is why the PiCloud uses netboot and all data is on a network file system. Once the box of Pis goes into the rack there’s no physical access and you can’t change an SD card without taking 48 offline at a time. So there’s no SD cards in the whole setup.

Reply to Peter Stevens

Avatar

Looks like. I’d like to buy 1 (one).

Reply to Gribbs

Avatar

Great, thanks. I will probably try this! One thing that might be important: once you add an active database system to a Raspberry Pi (like here, for a dynamic website with lots & lots of data traffic), running it from SD card is probably no longer a good idea. SD cards are not the best for writing many files continuously and may give out sooner than later. A large, quality SSD on the USB3 port would be much better for reliability, and faster too.

Reply to Ed

Alasdair Allan

Yup. The instance here boots and runs from a network mounted disk.

Reply to Alasdair Allan

Avatar

Why couldn’t I host it on the Pi on my home where I already have a domain name and SSL cert? The article doesn’t explain this or the criticality of IPv6 vs IPv4 if you’ve got your own domain (and why wouldn’t you if you’re intending to provide services to the internet).

Reply to Duane Moody

Alasdair Allan

You could absolutely host it on your home network! However, most people don’t have a static IP address, and don’t want to deal with dynamic DNS and the other overheads — like security concerns — that come with doing that.

Reply to Alasdair Allan

Avatar

What application is used to generate the CPU usage?

Reply to Garret

Avatar

That’s a Munin graph :)

Reply to Abigail

Avatar

Really really honest question: what is the “Twitter emergency”?

Reply to Daniel

Ashley Whittaker

I think we were being *slightly* facetious here. But for those who live their lives on Twitter, the prospect of it falling over would be an emergency. Like I would view Disney+ going down as an EMERGENCY. Or getting to the last open supermarket and realising they’re out of wine.

Reply to Ashley Whittaker

Avatar

Please go outside.

Reply to Cheef

Liz Upton

Unfriendly! (And inaccurate: I have seen Ashley out of doors at least twice.)

Reply to Liz Upton

Avatar

A couple things that I’m not sure if these are pi-specific differences, or what, but I wanted to mention them:
– In Debian 11.5, I don’t appear to have a dpkg-configure –reconfigure; I do have dpkg-reconfigure, which seemed to serve the same purpose.
– The example Mastodon server has an SSL server set up, with no certs available; this meant that it didn’t work to enable. I had to comment out the SSL section in order to get it to run, then run certbot, then uncomment the SSL bits once I had a cert in place. (cerbot also did some “helpful” redirection stuff in the nginx/default config I had to fix as well.)

Just two minor notes for those who might be trying to follow these instructions elsewhen; thanks for the clear guide, it was enough to get me off the ground on my self-hosted setup!

Reply to Christopher R Schmidt

Liz Upton

We’d always recommend you use Raspberry Pi OS – thanks for the note, though!

Reply to Liz Upton

Avatar

I am having a lot of trouble with the DNS setup, cannot get it to work at all. Can any provide advice to me on it?

Reply to Alvyn

Avatar

Now I read the recommendation of Raspberry pi OS I fear I will have to start again.using Ubuntu server hit the stage of
sudo -u postgress …CREATE USER
Unknown user postgres.
I’m guessing I broke something in earlier steps as I only remember creating the mastodon user…

Reply to DW

Avatar

I’m getting this failure:
curl -sL https://deb.nodesource.com/setup_16.x | bash –

## Installing the NodeSource Node.js 16.x repo…

## Populating apt-get cache…

+ apt-get update
Reading package lists… Done
E: Could not open lock file /var/lib/apt/lists/lock – open (13: Permission denied)
E: Unable to lock directory /var/lib/apt/lists/
W: Problem unlinking the file /var/cache/apt/pkgcache.bin – RemoveCaches (13: Permission denied)
W: Problem unlinking the file /var/cache/apt/srcpkgcache.bin – RemoveCaches (13: Permission denied)
Error executing command, exiting

Any suggestions?

Reply to Graham

Avatar

Perhaps you need a ‘sudo’ in front of the ‘bash’, to get it to run as the root user?

Reply to AndrewS

Avatar

I believe I tried that option, I’ll have another go. Thanks for your response.

Reply to Graham

Avatar

After several attempts, I think I am *almost* there – but stuck on DNS. Here is the latest certbot response …

– The following errors were reported by the server:
Domain:
Type: unauthorized
Detail: 2a00:1098:0:86:1000::33: Invalid response from
http:///.well-known/acme-challenge/OPq-FdxNxNloX-jWLgjIuKs5NjQvsdJYu9JknmMIOJk: 404

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.

Most probable explanation – I got a blind spot somewhere (perhaps literally) – but the A/AAAA IP addresses are Mythic Beasts 93.93.131.3 and 2a00:1098:0:86:1000:0:0:33

Right now I don’t see any way out. Perhaps some else here can suggest where it’s gone wrong?

:-)

Reply to Rick Crust

Avatar

I woke up in the middle of the night with the explanation. It was a literal blindspot. I have two RPis and one hosting account at Mythic Beasts – that’s multiple IP addresses, and they all look the same to my surviving half-eye. I had copied & pasted the wrong IPs. Not completed yet, but “Welcome to nginx!” is a step in the right direction.

So this was neither a Raspberry Pi nor a Mastodon issue – just a multi-tasking mistake.

Starting over again :-)

Reply to Rick Crust

Liz Upton

This is kind of brilliant and is EXACTLY how my middle-of-the-night epiphanies usually go. (The subconscious is a wonderful thing.)

Reply to Liz Upton

Leave a Comment