We are aware of an issue with some clients not being able to sync (receiving 502 errors, etc…). It does not appear to affect all clients and the web view is still working. We will update this post as we know more and when we have resolved it.

Updates:

09-Sept 11:19 PM UTC We attempted to roll back from Lemmy v0.18.4-beta.8 to v0.18.4 (as well as even earlier beta versions) but the service failed to start. We’ve reverted back to v0.18.4-beta.8 for now and will continue investigating the cause of the sync issues.

10-Sept 3:30 AM UTC And… we are back, where we started. If you hadn’t noticed, we were down for almost 2.5 hours while we tried to rollback the database migrations introduced in the beta releases so we could go back to the stable v0.18.4. When that failed, we tried to restore from the backup we took just prior to the rollback attempt. When that failed, we used Digital Ocean’s restore to point in time and created a new DB cluster and switched to it. So… we are back on v0.18.4-beta.8 again, with the expectation that 3rd party clients are still broken. Unfortunately, we are not sure what we can do about this as we believe it is something in Lemmy that is breaking it. The mitigation we have put in place to avoid this going forward is that we no longer use the “latest” tag when loading a docker image and now hard lock to a specific version. We will update that version in a more intentional manner going forward.

10-Sept 4:15 AM UTC After some additional digging through the Lemmy diff for beta.5 to beta.8 (and then looking through the PRs that were merged to create this diff), the issue is indeed with the Lemmy code and is even called out in one of the PRs,. What this means is that the 3rd party clients that are not able to access the site will need to update their code to fix things on their side (or else they will be broken when v0.19 is released and all other instances update to it).