Orange cats are cute when they’re not causing trouble

This was a dumb issue but I think I might’ve solved it. Let me know if you keep noticing issues!

(I didn’t update my list of allowed ips from my cdn, so rate limiting was all wonky)

Hmm have you seen more of these errors by any chance? I’ve gotten a few alerts about 502s but didn’t see anything obvious at the same time

testing big pictures again

Thanks for the heads-up. pictrs should hopefully be better now. I upgraded it, gave it more ram, and migrated the database from the file-based sled repo to postgres.

I might still migrate the storage itself at some point, but I don’t think it’s a bottleneck at the moment.

Let me know if you keep noticing issues with it!

not 4k:

edit: pasting 4k image still fails

Haha well at least you helped figure out why pictrs crashes occasionally :)

I think I have a temporary fix (more memory!) for now, but I’ll work on it some more. A few months ago I mentioned migrating to new hardware - I haven’t actually done that yet, but do plan on doing that hopefully in the next month. It’ll give me more resources to dedicate to lemmy and co.

Looks like it was an issue with our pictrs instance. When I cleared my cache, I noticed avatars/etc weren’t loading either.

Should be fixed now!

Do you recall if they were all around the same time? I wonder if you opened it right after the bot posted a bunch of things all at once or something like that? I haven’t paid much attention to it, but I wouldn’t be surprised if it just runs on a cron or something.

Also, I would be curious on people’s opinions for de-federating from them. I’m not interested in seeing the reddit reposts, but I also don’t want to restrict what people can/can’t see too much.

We weren’t ever defederated from lemmit.online, but it’s possible someone recently subscribed to some communities there. When you’re not logged in, what is the home page sorting by for you?

I don’t actually see much from lemmit.online on the main feed right now even when not logged in, but it looks like the bot is marked as a bot account: https://eviltoast.org/u/bot@lemmit.online

If you haven’t already, you can also go to your account settings and disable “show bot accounts” to hide posts from bots as long as they mark themselves as a bot. It only works when you’re logged in though.

There’s actually a view votes link in lemmy-ui, but it’s only exposed to admins and mods:

It’d be nice if it was available to all since I think that’s how some of the other fediverse software does it. Lemvotes still looks pretty convenient though because it can get all of the vote info into one place.

If it’s for personal use, you could probably set up a small cloud vm at hetzner or wherever and just use it to reverse proxy back to the server in your house through a vpn.

Actually installing/setting it up isn’t too bad though. There’s a few matrix channels if you run into issues, but it seems like the ansible route is pretty popular. It at least has a few configs in the repo you can use as a reference when needed.

Essentially it is:

• postgres
• pictrs
• lemmy api
• lemmy-ui
• nginx / caddy / etc

I think you can get away with not having pictrs, but I don’t remember offhand. If you do set up pictrs, I’d recommend turning off the option in lemmy that proxies all images through your local pictrs instance. It’ll eat up disk space really fast if it’s on.

Very cool, thanks for sharing! We’re set up on https://lemmy-federate.com/ so hopefully that is part of what triggers it to pick up new communities faster. There’s basically a bot user that will subscribe to new communities automatically, as long as they’re from another community set up there.

I’ll look into Jerboa. I tried Voyager on my phone, and didn’t see a way to change my home instance from lem.ee to eviltoast. I should probably go read the directions.

In case you haven’t figured it out yet, Voyager supports multiple accounts so I think all you need to do is click the middle button on the bottom row (probably says lem.ee if that’s’ the current home instance. Mine says eviltoast.org). That should pop up the accounts dialog where you can click edit and add another. At least on Android, now sure about ios.

Thanks! I’ve settled on the new hardware, but it’ll be a few weeks before it gets set up and I can actually migrate to it.

I should write about how everything is hosted now, maybe I will once I get the new stuff set up. Other than hardware, I’m planning on migrating away from hashicorp nomad to kubernetes along with a more normal stack.

Expect me to dump more in the selfhosting and cats communities. I’ve got a hell of a homelab going, and I need to brag.

Combining some of the best hobbies (cats and homelabbing) :).

You’re right about duplicated efforts. One thing I’d like to see improved on lemmy is something like federated/decentralized communities. There are multiple selfhosted communities for example, it’d be nice if they could all be combined into one place so that you don’t have to look for all of them when subscribing.

I did find one consistent error related to postgres not having enough memory. That should be fixed now, but it looked like it was triggered by some internal queries from lemmy.

Hey @dbtng@eviltoast.org , thanks for the report.

There has been a lot more traffic than normally lately, mostly due to ai bot scraping/crawling. I think the “too many requests” errors you saw are probably from that. Hitting an overall rate limit in the webserver vs an individual limit.

Looking at the logs, there have also been more errors lately, but it’s hard to tell how many are real errors. For example, a “couldnt_find_post” error can show up as a 500 internal server error instead of a 404. I’ll dig into it some more

I normally use one of the apps (Jerboa or Voyager) and haven’t noticed any issues, but they might be retrying requests transparently.

It’s infra that was already hosting other things. Lemmy got added to it because there were spare resources in the cluster.

Oh man, I feel like eviltoast.org is on the expensive side of things now. There’s less than 20 mau, but I pay close to $400/mo. Not including object storage or hardware running at home.

Realistically this is hosted on hardware I was already renting for other reasons, so actual cost is zero for lemmy itself. I could probably migrate to a small vm for $20/mo and only lose redundancy.