Entrepreneur, developer, seasoned system and network engineer, aspiring hobbyist pilot, husband and father.

  • 10 Posts
  • 31 Comments
Joined 2 years ago
cake
Cake day: June 5th, 2023

help-circle

  • This is odd. I did not upgrade pictrs (the service responsible for image hosting). There may have been an error for some specific tables rows during the database migrates. Another possibility could have been an error during the schema update that I believe the new version had to go through.

    Would you be able to DM me links to other posts that you come across with broken images? I will look to see if I can identify any common trends that could tell us more.

    Thank you





  • Hey,

    I’m due to make a post about the instances finances soon. I’ll get one posted in the next bit.

    I’m still covering costs out of pocket but my goal has been to have this instance fully funded by its members. I want this instance to stay true to its members, the fediverse and be put in a position where it can continue to thrive transparently without me. The admin team has been doing an outstanding job keeping this instance safe and moderated for everyone. I don’t thank them enough but they are the true heroes of this instance.

    As some may know, I’ve been working with a local non profit for the past 6 months to have them leveraged in accepting accepting donations for this instance. They have been slow to get things done but its progressing. I may end up doing what I was trying to avoid and create my own non-profit instead if that’s what it takes. That direction does have its own challenges and would add additional responsibilities to my plate.

    In the meanwhile, while some of the costs of the services have gone up slightly, I get joy knowing that I can continue providing this instance to all of you. I’m OK paying the costs to keeping the lights on until we can transition to donations.





  • The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn’t want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don’t plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn’t cutting it anymore I’ll probably have come across another opportunities on getting a new server with better NVME support.









  • Hey,

    I did some research on this.

    I will not share the list of accounts as I feel its everyone right to vote and to have their own opinions whether they align with yours or not.

    I will however say, Its mostly the same accounts that are down voting you. The majority of the accounts are over 2 months old with content generated by them (they do not seem like bot account). There are a few that are just 10 days old. 99% of the down votes are from other instances with feddit.uk and feddit.nl being the primary source of them.




  • Hey all, I’m always excited to upgrade to the latest versions that become available. Unless the upgrade is related to security I will tend to wait a few days to give other instances a chance to share their experiences. Before upgrading I usually go over the issues that have been submitted to help understand the risks and solutions to mitigate them.

    That being said the new version does bring in a whole bunch of performance gains (mostly related to the database) that i’m looking forward to.

    Lemmy.world is due to upgrade later today and I plan on upgrading shortly after them (either Monday or Tuesday).

    I will put up a notification up prior to the upgrade.

    Thanks


  • Hey all,

    There has been a few issues over the last couple of weeks. Here’s a rundown of some of the issues we’ve had.

    • Broken Images: A few weeks ago we had issues with broken images. This was due to me migrating our local image storage to object storage (like AWS S3). When I did the switch it broke all old images as they needed to be migrated. Pictrs at that time did not support concurrent uploads which means the migration would have taken days or weeks to complete during which time the image service would be offline. Instead I waiting for a newer version to be released that supported concurrent connections and did the migration in about 30-40 minutes one evening.

    • 5xx server errors: Some of you may have experience a lemmy page with an error code on it. This was due to me trying to implement an additional proxy to shield and mitigate future risks. While rolling this out I hit a few blockers that caused downtime as we worked to rectify it. I’m glad to say that as of Today this has since been implemented.

    In the addition to the above, the lemmyverse and especially this instance has been under bot attack almost daily. These bot attacks are eating resources and causing a query floods.

    Lemmy is still very young and in its early phases. In time these issues will slowly go away.

    P.S you don’t have to worry about me leaving you guys hanging.


  • Honestly these type of scat spamming has been really discouraging on me. I spend time and money so that the community can have a reliable and nice instance to call home, only to have a few (not sure what to call them) people try to ruin it for everyone. If these people are reading this comment, just know that every time you make these kinds of unwanted posts you are taking my time away from my family and young kids to address the reports.

    My nights lately have been mostly used for coding some extra moderation tools that will moderate these in the future.

    I sincerely apologize for all of you who have been exposed to literal shit images over the last week. I’m working as fast as I can and will implement something soon.

    In the meanwhile i’ve added a few new admins to help with this moderation.