Absolutely. This is exactly why It’s important that we get a donation strategy put into place sooner rather than later.

This is odd. I did not upgrade pictrs (the service responsible for image hosting). There may have been an error for some specific tables rows during the database migrates. Another possibility could have been an error during the schema update that I believe the new version had to go through.

Would you be able to DM me links to other posts that you come across with broken images? I will look to see if I can identify any common trends that could tell us more.

Thank you

We sure are!

Hi @Blaze@sopuli.xyz

It’s coming. This instance is deployed using lemmy-ansible but with some slight modifications. I need to review a few things to make sure the transition is smooth. As you also pointed out, it’s also summer time which isn’t helping with my free time situation.

I have a long term solution that will speed up upgrades but I only expect to get it in place by the fall. In the meanwhile, I’m planning on getting this instance upgraded to 19.5 in the coming weeks.

Hey,

I’m due to make a post about the instances finances soon. I’ll get one posted in the next bit.

I’m still covering costs out of pocket but my goal has been to have this instance fully funded by its members. I want this instance to stay true to its members, the fediverse and be put in a position where it can continue to thrive transparently without me. The admin team has been doing an outstanding job keeping this instance safe and moderated for everyone. I don’t thank them enough but they are the true heroes of this instance.

As some may know, I’ve been working with a local non profit for the past 6 months to have them leveraged in accepting accepting donations for this instance. They have been slow to get things done but its progressing. I may end up doing what I was trying to avoid and create my own non-profit instead if that’s what it takes. That direction does have its own challenges and would add additional responsibilities to my plate.

In the meanwhile, while some of the costs of the services have gone up slightly, I get joy knowing that I can continue providing this instance to all of you. I’m OK paying the costs to keeping the lights on until we can transition to donations.

deleted by creator

Yes, I’ll get this done

UPDATE: here you go https://oldsh.itjust.works/

There was a little hiccup this morning unrelated to the migration tonight. The Lemmy services like to be restarted every once in a while. Once we move to the new hardware I’ll be able to look into implementing a better logging system and hopefully be more proactive when situations like these happen.

The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn’t want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don’t plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn’t cutting it anymore I’ll probably have come across another opportunities on getting a new server with better NVME support.

I’ve started the preparation steps to minimize the maintenance window needed for this upgrade. These operations are running some background tasks and once they are done I’ll make an announcement to give everyone a few days heads up of the anticipated upgrade time and date. Should be soon.

Hey!

The best way is to @TheDude on our matrix channel. This action triggers notifications across all my devices, creating a barrage of alerts that almost always grab my attention, a.k.a a ddos to my sensory system.

@kersploosh@sh.itjust.works explained it well. It was related to a small bug on the pictrs service that I’ve already reported which they quickly fixed in a newer version. I will tackle this upgrade at the same time as I get this instance up to v19

Hi there

Sent you a message on the matrix chat. This should now be fixed.

Thank you

@Buddahriffic@lemmy.world Please send me a DM with the username and email address used to sign up and I’ll manually verify your account.

Could you provide some additional information on what is not functioning well? Screenshots with errors are also very valuable.

Hey all,

I have made some changes to how this instance sends email that should help with the email issues people have been experiencing. If anyone continues to have issues receiving emails please let me know

Thank you

Hey,

I did some research on this.

I will not share the list of accounts as I feel its everyone right to vote and to have their own opinions whether they align with yours or not.

I will however say, Its mostly the same accounts that are down voting you. The majority of the accounts are over 2 months old with content generated by them (they do not seem like bot account). There are a few that are just 10 days old. 99% of the down votes are from other instances with feddit.uk and feddit.nl being the primary source of them.

The big email email providers are not liking the amount of email coming from the server ip. The permanent fix is to use a mail sending service. I do plan on making that adjustment soon.

In the meanwhile send me a dm with your username and email address used and I’ll verify the account for you.

Thanks

Hey all, I’m always excited to upgrade to the latest versions that become available. Unless the upgrade is related to security I will tend to wait a few days to give other instances a chance to share their experiences. Before upgrading I usually go over the issues that have been submitted to help understand the risks and solutions to mitigate them.

That being said the new version does bring in a whole bunch of performance gains (mostly related to the database) that i’m looking forward to.

Lemmy.world is due to upgrade later today and I plan on upgrading shortly after them (either Monday or Tuesday).

I will put up a notification up prior to the upgrade.

Thanks

Hey all,

There has been a few issues over the last couple of weeks. Here’s a rundown of some of the issues we’ve had.

• Broken Images: A few weeks ago we had issues with broken images. This was due to me migrating our local image storage to object storage (like AWS S3). When I did the switch it broke all old images as they needed to be migrated. Pictrs at that time did not support concurrent uploads which means the migration would have taken days or weeks to complete during which time the image service would be offline. Instead I waiting for a newer version to be released that supported concurrent connections and did the migration in about 30-40 minutes one evening.

• 5xx server errors: Some of you may have experience a lemmy page with an error code on it. This was due to me trying to implement an additional proxy to shield and mitigate future risks. While rolling this out I hit a few blockers that caused downtime as we worked to rectify it. I’m glad to say that as of Today this has since been implemented.

In the addition to the above, the lemmyverse and especially this instance has been under bot attack almost daily. These bot attacks are eating resources and causing a query floods.

Lemmy is still very young and in its early phases. In time these issues will slowly go away.

P.S you don’t have to worry about me leaving you guys hanging.