I pretty much agree there and some of our devs are huge fans of Rust, but the overall consensus was it just feels like it isnt there yet for web dev, or at least not without compromising something (your time or getting new devs or losing some to burnout). It's unfortunate that TS/JS is hugely co-opted by silicon valley types but the extreme focus on web dev and dev accessibility they have is so difficult to beat and with the FE already being in React/TS it helps unify the stack which is a pretty big deal for us with school/work/life often leaving us in awkward positions of having to ping someone to help out or drag a feature across the finish line.
ThePeoplesGuillotine [he/him]
- 0 Posts
- 36 Comments
Joined 4 years ago
Cake day: July 25th, 2020
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
Sure, I'll keep this a bit short though (well, i tried) as some of these are probably worth being a post topic in their own right.
Actix Web - The web server framework that Lemmy chose to use kind of sacrifices a lot for the sake of raw speed (or "cheats") and has undergone a change of hands after some drama awhile back. The design patterns of the framework also feel kind of janky for the purpose of a web api (they lean on the actor pattern iirc and a custom implementation for async)- But for us one of the problems was they allow customizing the thread pool but dont really state this obviously anywhere. One of our devs found this by looking in the source and it's something that is rather critical for handling a large number of concurrent requests. It also doesn't handle middleware very well, but I believe part of that is Lemmy is using an older version and the newer ones have significantly changed some of the apis. I don't think Actix is wholly lacking here but it's far more difficult to use well than anything in JS land and IMO I'm not a big fan of the mind set that goes into creating "the fastest framework ever" over just making something that works with clearly documented behavior.
Diesel - The ORM that Lemmy uses is the bigger offender here imo. As an ORM it does not support: self-joins, group bys, having clauses, window functions, custom sql aliasing and a few other things. Part of this is due to Diesel being young-ish, but the devs themselves state on their repo that some of these are "pointless". I primarily work in databases and all of this functionality are incredibly basic feature sets that any ORM should support. As a result- Lemmy was forced to create "views" as a workaround and some of the biggest architectural problems in the codebase are due to these Diesel workarounds. I can confidently say it would be easier to write raw sql for the entire project than use this, but Diesel also intentionally limits the "escape hatch" options to do so. To me, this lib feels more like some devs trying to "gamify" Rust's type checking system instead of making a real ORM and its by far the biggest slowdown to adding features. This is one of the bigger drivers for the rewrite, if we remove Diesel we're effectively restarting from zero, and there is so much basic functionality missing here that I would consider it a "non-starter".
Features - The mod/report system was exclusive until very recently, 2fa auth and server side token validation/revocation, a lot of smaller FE tweaks are as well..(user tags too? some other devs can flesh this out) Some of the bigger things would be we significantly rewrote/redid some of the database handling because the original implementation was consistently deadlocking itself and doing entire table scans at high frequency to obtain trivial data. For example- one of the views Lemmy uses performs a cross-join on data sets and turns something like 20k comments into a "9 million row" result set that is then filtered out. None of that is necessary at all and we removed it or fixed it, otherwise the debate/election threads wouldn't have been just slow, the site would've been down. Effectively today, the Chapo backend uses an entirely different set of SQL than Lemmy- so some of those and security changes are the big reasons, and these are things we relayed to upstream early on but were not prioritized over federation.. so here we are with what is functionally a fork in all but name.
Contribution - That's difficult for me to answer because I feel if this came down to duplicating efforts we wouldn't be where we are today and there is plenty of space in the Lemmy design for shared logic if they choose to architect the code to do so. For ex, the act of handling apub requests and translating the objects could be an upstream rust library that we use with typescript, but they would have to make that possible and start decoupling a lot of core logic. In the end, apub is an open spec and Lemmy is a specific implementation/usage of it. I feel what you could be asking instead is, would you rather contribute to one consumer client, or another? I kind of see it more like matrix/mastadon, where theres a "core spec" but plenty of different and good client implementations of it.
All my code projects go under a folder
open-sores
I'll give some tldr answers here but if you check my comment history on this post theres more involved answers-
1 - Some of us personally have/still and will continue to contribute upstream. The amount of code we share probably isnt as much as it seems, upstream sometimes requests frequent changes to what we do so it's never been as simple as "pull request it" and we have several proposals on their github still that are in various progress states. Ideally the rewrite makes it easier for this to happen- it will be quicker to implement an idea in Typescript, vet it against a large lemmy instance (ours) and propose it upstream or rewrite it in Rust, than it would be to write it in rust "our way" then have to rewrite it again anyways.
2 - Federation is still a core goal. There are some TS libs but none we've carefully vetted. For the initial "v1" state we might end up working on our own because Lemmy's websockets kind of tightly tie into how they use apub and we already have to port the ws code there
Hmm I think that's because of moving the repos around to split the FE out 😅
bonus points if youve personally declared a react function deprecated! (yea im kiddin)
No problem and feel free to ask anything anytime. It's part of why we made this comm and these are all perfectly valid questions to have. The secret is...we're all some doinkuses ourselves
we're stuck deciding if the vetting process is how many medium blogs on todo apps someone has or their hackernews post count
Long term I think the general overlap of people who know JS and ideologically agree with Lemmy will be greater and more sustainable than that of Rust- And with that said Rust itself is still very young in web dev space so even for a project as "simple" as this one we have hit annoyances, things that are unsupported or undocumented but are things that are trivial in other languages. So part of this long road is factoring in that a systems lang just "isnt there" yet and maybe wont be for years and will certainly put a damper on being productive or sustaining the dev pool.
The other thing to be aware of is code base incompatible is a relatively minor issue here. Federation relies on "activity pub" which is an open web standard for communication and so the only thing we really have to do is make sure we match the activity pub spec and that "hexbear objects" can translate to "lemmy objects", which they will because 'users', 'posts', 'comments', etc are pretty simple things to model and it's not technically difficult to add custom "extensions" to them that don't interfere (and we've already had to do this to prevent this instance from crashing). So I would think of it less about the codebase itself, but rather that we "communicate" the same way. That would only be a problem if upstream decides to stop using apub, which would be a breaking change to every Lemmy instance so it's rather unlikely.
tl;dr- we're rewriting the backend in a more group preferred lang, we still plan on federation support, we want to make big changes to overall architecture, we will upstream ideas and/or working implementations w/ evidence of "performing well", code is now public so y'all can take breathers from posting memes and contribute too
The specific language of choice here doesn't matter for scaling to 10k+ users- rather the decisions and implementation details of said language. Early Lemmy had some pretty bad performance issues at low user counts (100 concurrent?) due to some database queries and other choices. Some of the devs working on Chapo helped upstream fix those, but a lot of the early janky, sluggish and "bad" feeling performance of Chapo was due to decisions that upstream deprioritized or did not want to implement in favor of federation- which is fine! But we had to continue to prioritize those changes and the brick wall we hit is that finishing those out in rust is cumbersome and we've already deviated so far that merging changes between the two projects is a large effort. We plan to still contribute upstream on critical issues like these when possible, but by nature of our instance size we have to take a different prioritization path and likely will for some time.
The other side to this is that, yeah, anything is technically possible, but some choices make it a much more difficult road than others and at the end of the day we're merely lefties struggling under the boot of capital- so it makes sense to make the "technical road" as easy and effective as possible to combat engineer burn out or Life Happening.
The event loop of node makes it well suited for backends dealing with web requests and no heavy processing which I wouldn't see Lemmy deviating from too much and has a plethora of open source libs/tools tailor made for web dev space. The vast majority of our server load is database queries which are currently unfortunately tightly coupled to large websocket views so I think the exact choice here matters little over just engineering it well.
Some of us currently (or have) and will continue to contribute upstream, but there is also nothing fundamentally wrong or unhelpful by making a fork. At it's core the Lemmy ecosystem is about federating and having instances talking to each other- And all of this can be done in any particular language or feature set so long as the apub spec is maintained and the data sets remain compatible in some form. Personally I think it's healthy for the project to have choices as someone still needs to setup, configure and run a lemmy- some might like the lighter footprint the rust version provides, and some might like the easier setup (or ease of slight personal tweaks) a typescript might provide.
Nah, but you may refinance your existing debt with new terms and a slightly lower rate
Hell yeah. The NFL screwed game pass so much, it was fair at 99$ with all games and extra film angles but they had to go and make live games non-US only and put the price to 300 lmao
- ThePeoplesGuillotine [he/him]to
·4 years agoI top out at miracle whip spice OP
I bootleg stream sports- fuck cable deals and market blackouts. I schedule as much as possible during working hours, I love performing "time theft" of my wage thieves. I wear clothes until they are literally falling apart or my shoes have holes in them. In college I found book PDFs online or went through the class w/ no book To pair with that, my resume to get my first job contained many white lies and did not list my gpa
Americans would love to die cold, hungry and poor as long as someone else they didnt like did first
Historic phrase