Yesterday, you probably saw this informal post by one of our head admins (Chris Remington). This post lamented some of the difficulties we’re running into with the site at this point, and what the future might hold for us. This is a more formal post about those difficulties and the way we currently see things.

Up front: we aren’t confident in the continued use of Lemmy. We are working through how best to make the website live up to the vision of our documents—and simply put, the vast majority of the limitations we’re running into are Lemmy’s at this point. An increasing amount of our time is spent trying to work around or against the software to achieve what we want rather than productively building this community. That leaves us with serious questions about our long-term ability to stay on this platform, especially with the lingering prospect of not having the people needed to navigate backend stuff.

Long-time users will no doubt be aware of our advocacy for moderator tools that we think the platform needs (and particularly that we need). Our belief in the importance and necessity of those tools has only hardened with time. Progress of those tools, however—and even organizing work on them—has been pretty much nonexistent outside of our efforts from what we can see.[1] In the three months since we started seriously pushing the ideas we’d like to see, we’re not aware of any of them being seriously considered—much less taken up or on the way to being incorporated into Lemmy.

In fact: even within the framework of Lemmy’s almost nonexistent roadmap and entirely nonexistent timetable on which to expect features it has been made clear to us that improving federation or moderation on the platform are not big priorities.[2] We have implicitly been told that if this part of the software is to improve we will need to organize that from scratch. And we have tried that to be clear. Our proposal is (and has been) paying people bounties for their labor toward implementing these features, in line with paying all labor done on our behalf—but we’ve received mixed messages from the top on whether this would be acceptable. (Unclear guidance and general lack of communication is symptomatic of a lot of our relation with the Lemmy devs in the past few months.)

Things aren’t much better on the non-moderator side of things. The problems with databases are almost too numerous to talk about and even Lemmy’s most ardent supporters recognize this as the biggest issue with the software currently. A complete rewrite is likely the only solution. Technical issues with the codebase are also extensive; we’ve made numerous changes on our side because of that. Many of the things we’re running into have been reported up the chain of command but continue to languish entirely unacknowledged. In some cases bugs, feature requests, and other requests to Lemmy devs have explicitly been blown off—and this is behavior that others have also run into with respect to the project. Only very recently have we seen any overtures at regular communication—and this communication has not hinted at any change in priorities.

All of what was just described has been difficult to get a handle on—and having fewer users, less activity, and more moderators has not done a whole lot to ease that. We honestly find that the more we dig and the more we work to straighten out issues that pop up, the more pop out and the more it feels like Lemmy is structurally unsound for our purposes. (One such example of what we’re working with is provided in the next section.)

In summary: we believe we can either continue to fight the software in basically every way possible, or we can prioritize building the community our documents preach. It is our shared belief that we cannot, in the long-term, do both; in any case, we’re not interested in constantly having to fight for basic priorities—ones we consider extremely beneficial to the health of the overall Lemmy network—or having to unilaterally organize and recruit for their addition to the software. We are hobbyists trying to make a cool space first and foremost, and it’s already a job enough to run the site. We cannot also be surrogates for fixing the software we use.

PenguinCoder: A brief sketch of the technical perspective

I’ve said a few words about this topic already, here and here. Other Beehaw admins have also brought some concerns to the Lemmy devs. Those issues still exist. To be clear: this is a volunteer operation and Lemmy is their software; they have a right to pick and choose what goes into it and what to put a priority on. But we have an obligation to keep users safe and secure, and their priorities increasingly stifle our own.

In the case of this happening for open source projects, the consensus is to make your own fork. But:

The problem with forking Lemmy is in starting from all the bad that is inherently there, and trying to make it better. That is way more work than starting fresh with more developers. IE, not using Rust for a web app and UI, better database queries from the start, better logging/functions from the start; not adding on bandaids. A fork of Lemmy will have all of Lemmy’s problems but now you’re responsible for them instead.

We don’t need a fork, we need a solution.

To give just one painful example of where an upstream solution is sorely needed: the federation, blocking, and/or removal of problem images.

  1. You post an image to Beehaw.
  2. Beehaw sends your content out to every other server it’s federated with
  3. Federated server accepts it (beehaw.org is on their allowlist), or rejects it (beehaw.org is on their denylist)
  4. If the server accepts it, a copy of your post or comment including the images are now on that receiving server as well as on the server you posted it to. Federation at work.
  5. Mod on beehaw.org sees your post doesn’t follow the rules. Removes it from beehaw.org. The other instances Beehaw pushed this content to, do not get that notice to remove it. The copy of your content on Beehaw was removed. The copy of your content on other servers was not removed.
  6. The receiving federated instance needs to manually remove/delete the content from their own server
  7. For a text post or comment that’s removed, this can be done via the admin/mod tools on that instance
  8. For a post or comment including a thumbnail, uploaded images, etc; that media content is not removed. It’s not tracked where in Lemmy that content was used at. Admin removal of media commences. This requires backend command line and database access, and takes about a dozen steps per image; sometimes more.

There are dozens of issues—some bigger, some smaller—like this that we have encountered and have either needed to patch ourselves or have reported up the chain without success.

Alternatives and the way forward

If possible the best solution here is to stay on Lemmy—but this is going to require the status quo changing, and we’re unsure of how realistic that is. If we stay on Lemmy, it is probable that we will have to do so by making use of a whitelist.

For the unfamiliar, we currently use a blacklist—by default, we federate with all current and newly-created nodes of the Fediverse unless we explicitly exclude them from interacting with our site. A switch to a whitelist would invert this dynamic: we would not federate with anybody unless we explicitly choose to do so. This has some benefits—maintaining federation in some form; staying on Lemmy; generally causing less entropy than other alternatives, etc. But the drawbacks are also obvious: nearly everything described in this post will continue, blacklist or whitelist, because a huge part of the problem is Lemmy.

Because of that we have discussed almost every conceivable alternative there is to Lemmy. We are interested in the thoughts of this community on platforms you have all used and what our eventual choice is going to be, but we are planning on having more surveys in the future to collect this feedback. We ask that you do not suggest anything to us at this time, and comments with suggestions in this thread will be removed.

As for alternatives we’re seriously considering right now: they’re basically all FOSS; would preserve most aspects of the current experience while giving us less to worry about on the backside of things (and/or lowering the bar for code participation); are pretty much all more mature and feature-rich than Lemmy; and generally seem to avoid the issues we’re talking about at length here. Downsides are varied but the main commonality is lack of federation; entropy in moving; questions of how sustainable they are with our current mod team; and more cosmetic things like customization and modification.

We’re currently investigating the most promising of them in greater depth—but we don’t want to list something and then have to strike it, hence the vagueness. If we make a jump, that will be an informed jump. In any case logistics mean that the timetable here is on the order of months. Don’t expect immediate changes. As things develop, we’ll engage the community on what the path forward is and how to make it as smooth as possible.


  1. Other administrators have probably vocally pushed for these things, but we’re not aware of any public examples we can point to of this taking place. Their advocacy has not produced results that we’re aware of in any case, which is what matters. ↩︎

  2. Perhaps best illustrated by the recent Lemmy dev AMA. We’ll also emphasize that Beehaw’s admin team is not alone in the belief that Lemmy devs do not take mod tools or federation issues particularly seriously. ↩︎

  • forestG@beehaw.org
    link
    fedilink
    arrow-up
    19
    ·
    1 year ago

    Well, I am sure many are tempted… I am waiting to see if a berson decides to carry the ring so I can offer my axe :p

    • ulkesh@beehaw.org
      link
      fedilink
      English
      arrow-up
      12
      ·
      1 year ago

      If I didn’t have a full-time job and a family, I’d probably already have begun. I would offer my bow! :D

      • PenguinCoder@beehaw.orgM
        link
        fedilink
        English
        arrow-up
        16
        ·
        1 year ago

        Tell me about it. But uhh… have my beer mug instead? There’s nothing I want to do more right now than make a better service (subjective of course) for Beehaw’s use. Time and obligations prevent that from being a quick enough solution though.

        • The Cuuuuube@beehaw.org
          link
          fedilink
          English
          arrow-up
          9
          ·
          1 year ago

          And for the early phases of a FOSS project like this you kind of need a benevolent dictator who already has the project started

            • The Cuuuuube@beehaw.org
              link
              fedilink
              English
              arrow-up
              7
              ·
              edit-2
              1 year ago

              Mmmhmmmmmmmmmmm. My inclusion of the word was not merely rote practice of the BDFL model lodging itself in my brain. I have some strong feelings about how code is a form of communication, and that the software you develop is intrinsically mapped to the ways you approach person to person interactions. Case in point, the very best software platform for Beehaw would be one whose developers felt the ideal platform would involve be(e)ing nice. This mentality would cause a developer to put heavy phasis on good moderation tools, ones that can give clear, meaningful, and thoughtful reasonings for moderation actions being taken as well as easy ways to parse these mod logs to learn the sorts of social norms the community does and doesn’t tolerate.

              Given all this the development of the software would be open to listening to diverse perspectives in feature requests. And I mean diverse perspectives in terms of “this group is vulnerable to online attacks”

      • Fushuan [he/him]@lemm.ee
        link
        fedilink
        English
        arrow-up
        9
        ·
        edit-2
        1 year ago

        I’m tempted to read the whole codebase to start a merge request to implement some modicum of mod tools. Things like deletion propagation, including uploaded media so stuff like GDPR deletion requests can be properly executed would be great, since this would enable or facilitate “admin purge” alongside it. I’m not that well versed in rust but I’ve got my fair share of experience coding. I’ll see how regular post propagation is done and see how deletion is treated (I feel like the solution might be something that instead of truly deleting the data it delets the content but leaves the records on the BD marked for deletion so it propagates correctly, and then after some time an automated task finally deletes them.

        If you want I can update you after forking and creating the merge request to work on. IF I begin this, it would be helpful to have some critique.

        I personally believe that scratching the whole this is too excessive, there are ways to modify the backend codebase with forks and merge requests if the core developers are open to pull requests. If they are not the project already died lmao.

        I’m checking their federation activities on the docs, it seems like user removals do federate but mod removals dont? I’ll check the code when I get out of work :). It might be interesting to look there. https://join-lemmy.org/docs/contributors/05-federation.html

        Second edit: It seems like image logging is already developed databasewise. There was quite a lot of discussion on this issue and it was merged a while ago. Frontend tools need to be developed also, but it’s a start: https://github.com/LemmyNet/lemmy/pull/3927

        • ulkesh@beehaw.org
          link
          fedilink
          English
          arrow-up
          5
          ·
          1 year ago

          You are a hero we need, but do not deserve. Thanks for any and all work you’re doing! If Lemmy can be fixed, I’d love to see it and maybe even help in some small way. If the maintainers are not willing to accept PRs, especially in a timely manner, then I agree, it’s not going to survive long.

          • Fushuan [he/him]@lemm.ee
            link
            fedilink
            English
            arrow-up
            5
            ·
            1 year ago

            As already stated in the edit, there are already contributions for user and image purges that are being merged, several accepted merge requests in the last week, so the community IS working and the devs are receptive. It seems like the code is developed as two separate servers, one for lemmy and the other for the image manager, and it interacts with any client as a REST server. Then, it connects into ActivityHub through a high level API that I have to check yet, but the functionalities people are asking don’t seem to be impossible to implement tbh. Admin / mod tools need to be implemented first as valid endpoint operations and then be exposed to the outer API so clients can then implement an interface for actual users to do stuff.

            The gist of it is that all these kind of stuff takes time, and the amount of people that is up for contributions is not that high, and when lemmy exploded this july, the priority was server stability.

      • TehPers@beehaw.org
        link
        fedilink
        English
        arrow-up
        7
        ·
        1 year ago

        I’ve played with some ideas, but turns out the ActivityPub protocol is extremely difficult to implement in strongly-typed languages, at least from my experience. Anyway, whoever picks this up, here’s my sword!