Is Bubble down now?

I can get into the editor but my external website isn’t working --> https://getmarlow.com

Looks like https://zeroqode.com/ also isn’t available.

Thanks all – we have enough to go on to investigate, will let you know ASAP

3 Likes

I’m back up.

Thanks for the speedy response.

Up again. What was the issue?

Is anyone still down? I just restarted a system that looks like it got stuck in a bad state b/c of the previous database outage. Pretty sure I know how / why it got stuck and how to fix so that doesn’t happen again.

1 Like

Works for me

Hi Josh, thank you for your detailed explanation. Can you please share what cause the first outage as well?

First issue was caused by our primary application database crashing. I’m not sure yet what caused the crash – was digging through the forensics from the database when we saw the reports on this thread that apps weren’t fully recovered yet

5 Likes

Thanks Josh, mine is up now!

Working now, thanks for the quick response, Josh

Thank you for your reply. Can you please share here the details once you will find out? Just out of curiosity can you please advice if during the first outage dedicated cluster was also affected?

Resolved for me

Yes, though I don’t know what the ETA will be. I prioritized getting things back online as fast as possible vs preserving evidence for future debugging, so it may be hard / take a while to get to the bottom of this.

Re dedicated, I believe login may have been affected – dedicated instances use the main cluster bubble accounts to log into the editor, so login to the editor often goes down when the main cluster is down – but running apps themselves should have been completely unaffected, at least as far as I’m currently aware

Fun fact…system went down and I took a 30 minute break from Bubble.is. Forgot that news sites still existed. :upside_down_face:

1 Like

Thank you Josh, I’m fully agree that priorities to get everything is up and running and there is no doubts about it. On another hand it’s also important to get to the bottom of the problem which case a database crash. I know you guys mentioned that you are working on something to get more stable as this kind of outages are not acceptable for us a customers. Dedicated clusters are definitely the right direction in order to minimize this kind of issue, but for most of the customers with a small apps it’s overkill when it comes to the price. I really hope you will come out with something to sort this out. Thank you!

Keeping things in perspective - Google Calendar went down today. I don’t like downtime as much as anyone else - But being in the software industry, servers go down (for those of you who’ve ever had to buy shared/VPS hosting, you know the drill). I don’t care much as to why it went down today - It was taken care of quickly by the head honcho. That’s all we could ask for. :slight_smile: @josh Thanks.

6 Likes

So we found what I’m 80% sure was the issue (based on circumstantial evidence). In the database that went down, we were locking data access based on application name instead of both application name and application version. Normally that wouldn’t be a problem, but for our interactive tutorials, there’s one app name, “learn”, and a million different versions, since each user has their own private version of the app. We had a busy day yesterday with people taking lessons, and that caused a bottleneck causing queries to pile up, eventually choking the database and causing a crash. We’re working on deploying a fix to prevent it from happening again

8 Likes

I think it was the 20% chance, or maybe there was an issue with the fix – bubble went back down, we brought it back up again. Looks like same issue as before. Investigating further now

EDIT:
So the issue I mentioned above was real, but wasn’t the cause of the prolonged downtime yesterday or today. It looks like the issue was likely with the underlying system, so I’m in the process of replacing the server hosting the database, and upgrading the software to the latest version.

5 Likes