Alerts (old posts, page 7)

[Resolved] Planned bouncer ugrade on Thursday

Restart completed. Again, note that some upstream connections may take a while to be re-established due to rate limits.

(12:05 UTC — Apr 3)

Planned bouncer upgrade is scheduled for Thursday, April 3rd. On Thursday at 12:00 UTC we will be rolling out an upstream update for soju, the software that powers the chat.sr.ht bouncer. The bouncer will be restarted, causing everyone to be disconnected. Most clients will reconnect automatically. Note that some upstream connections may take a while to be re-established due to rate limits.

(09:55 UTC — Mar 31)

Maintenance on Database Systems

Mar 30, 21:00 EDT
Completed - The scheduled maintenance has been completed.

Mar 30, 16:30 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Mar 28, 10:05 EDT
Scheduled - Please be aware that we will be performing maintenance on our database systems.

The following services and functions may be unavailable:

- Purchased products maybe delayed during setup
- Updating customer data in the Control Panel and communication email for some Control Panel actions may not be sent out
- Some customers may be unable to view their invoice in the Control Panel

Additionally, our Customer Service Team may not be able to answer all questions related to your account.

All systems will be available to you as usual when the maintenance is completed, however temporary downtime is possible.

We apologize for any inconvenience this may cause.

[Resolved] LLM crawlers continue to DDoS SourceHut

We have deployed Anubis to git.sr.ht. After some internal discussions we have ultimately decided that the best course of action to protect git.sr.ht from LLM crawlers is to deploy Anubis. This software presents some users with a proof-of-work challenge which is solved by the user’s browser with JavaScript.

This challenge is automatically bypassed for logged-in users. If your browser does not support JavaScript (or you do not wish to enable it for any other reason), log in at meta.sr.ht to circumvent it.

Note that Anubis is only being used for the web frontend. API access and git operations are unaffected.

This solution is robust and reliable. We do not want to leave it enabled indefinitely, but considering that the user impact is minimal and it is sufficient to mitigate the LLM traffic, we consider the matter closed and are closing this notice. Thank you for your patience while we prepared our mitigations.

(09:00 UTC — Mar 24)

SourceHut continues to face disruptions due to aggressive LLM crawlers. We are continuously working to deploy mitigations. We have deployed a number of mitigations which are keeping the problem contained for now. However, some of our mitigations may impact end-users.

In particular, we have deployed Nepenthes to certain routes which are associated with large volumes of LLM-related traffic. You may encounter certain pages which are not usable as a result, especially if you are not logged in.

Mitigations only affect the web frontend of SourceHut: SSH access, git operations, API access, and so on, should behave normally.

We understand that some of our mitigations are user-impacting. We apologize for the inconvenience. These measures are temporary, but we do not have an estimate for when they will no longer be required. To be honest, we are running out of ideas for how to deal with these LLM bots. Your patience is appreciated.

If you are having problems using the SourceHut web UI:

First, log into your SourceHut account. Logged-in users bypass most of our mitigations. If that does not work, please contact support on IRC or via email.

If your cloud server is unable to reach SourceHut:

We have unilaterally blocked several cloud providers, including GCP and Azure, for the high volumes of bot traffic originating from their networks. If your cloud server is experiencing problems using SourceHut, and you have a legitimate reason to do so, you must email support to request an exception. Please explain your use-case and include a list of affected IPs and/or subnets.

We kindly ask the administrators of SourceHut integrations to program their software with responsible usage patterns. If possible, we request that you prefer webhooks over polling for updates. If your integration performs git operations, please prefer to use git fetch to update a persistent repository, or use a shallow git clone, rather than performing a fresh clone each time your automation runs. We also request that you set a User-Agent string for your traffic which identifies your software and includes an email address that we can contact with questions and feedback, as well as clearly identifying your traffic as non-malicious so we do not mistakenly apply mitigations to you.

If you are using git(1) for git operations, you can set a User-Agent by setting the GIT_HTTP_USER_AGENT environment varaible accordignly.

If you would like advice on making your integration more efficient, or setting up webhooks, please contact support for assistance.

(08:30 UTC — Mar 17)

SAP system maintenance

Mar 9, 07:00 EDT
Completed - The scheduled maintenance has been completed.

Mar 9, 00:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Mar 7, 10:39 EST
Scheduled - In order to assure that you can always safely use our services, we regularly perform system maintenance work.

During this time, you will not be able to access your billing information through your Control Panel. Furthermore, our Customer Service might also not be able to answer all questions related to your account. We apologize for any inconvenience this may cause you.

Of course, we will try to keep downtime to a minimum and once the upgrade is complete, all services will be restored.

Thank you for your understanding.

SAP system maintenance

Feb 16, 06:00 EST
Completed - The scheduled maintenance has been completed.

Feb 16, 00:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Feb 14, 05:33 EST
Scheduled - In order to assure that you can always safely use our services, we regularly perform system maintenance work.

During this time, you will not be able to access your billing information through your Control Panel. Furthermore, our Customer Service might also not be able to answer all questions related to your account. We apologize for any inconvenience this may cause you.

Of course, we will try to keep downtime to a minimum and once the upgrade is complete, all services will be restored.

Thank you for your understanding.

[Resolved] Planned maintenance on Tuesday

Maintenance completed. Access to all services is restored. (11:40 UTC — Feb 11)

Maintenance will take longer as expected. Due to unforeseen circumstances, the maintenance will take longer than anticipated. We are working to resolve the issue as quickly as possible. (11:00 UTC — Feb 11)

Maintenance is underway. The maintenance window is now open. (10:00 UTC — Feb 11)

Planned maintenance on February 11th will cause intermittent outages: Planned maintenance starting at 10AM UTC will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. Notably, chat.sr.ht will be rebooted, causing everyone to be disconnected from the bouncer. Most clients will reconnect automatically. The maintenance is expected to take no more than an hour.

Have a nice weekend! (10:50 UTC — Feb 6)

Disruption in MyWebsite with eCommerce integration

Feb 7, 11:57 EST
Resolved - The systems are stable and all services are functioning normally again. Our apologies for the inconvenience caused.

Feb 6, 11:10 EST
Monitoring - The solution has been successfully deployed. To take effect, a republish of the website is required.

Feb 6, 10:27 EST
Update - Our technicians are currently working on resolving this issue.

Feb 6, 08:26 EST
Identified - Some customers are experiencing issues while using their MyWebsite product with eCommerce integration.
The issue may affect the store management panel and the store pages in live sites.

Incident Report for MyWebsite Now

Feb 6, 12:32 EST
Resolved - The issue has been resolved.

Feb 6, 11:09 EST
Monitoring - The solution has been successfully deployed. To take effect, a republish of the website is required.

Feb 6, 10:45 EST
Identified - Some customers may be experiencing issues accessing their websites.

Degraded performance with our Support Tools

Feb 6, 08:32 EST
Resolved - The systems are stable and all services are functioning normally again.

Feb 6, 08:09 EST
Monitoring - We are monitoring the situation to ensure that the issue does not reoccur.

Feb 6, 07:46 EST
Identified - Due to a technical issue with our Support Tools, our Customer Service Team are not able to answer all questions related to your account.