Request for a Bubble Software Release and Escalation Policy

Dear @emmanuel, @josh and @neerja

As you are aware, there was a lot of concern and anguish this week about a release of the Bubble environment last Friday which caused unexpected problems.

As I observed, and then became involved with problems in my own app, it became clear to me that there is no documented policy Bubble currently makes available to us users regarding how release updates are made to their platform, how they are documented for the users, or how we as users can escalate show stopping problems we have with our app in a way we know will get urgent attention.

This message is a request on two levels:

  • For Bubble to create such a policy for the users

  • For Bubble users to share here what they would like to see included in such a policy. There was discussion of this last week, but the details are lost in threads describing all the problems.

In the user community, I particularly call to those who documented their issues or concerns in the forum last week to make a contribution: @keith, @JonL, @jamesbond, @Jici, @seanhoots, @meyerhd2, @reger-alexander

I will document my own thoughts in a post here over the next couple of days.

Like many Bubble users, I am trusting my career plans for the next 5-10 years on Bubble, and I want to be sure we are all contributing to make sure we can keep that sometimes-hard-to-define thing called trust in the platform strong in our hearts.

Thanks in anticipation to everyone involved,

Antony.

10 Likes

and adding to the list of users (since only 10 can be documented in a post!)…

@J805, @JohnMark, @jon2, , @sudsy, @varshneyandson, @Bubbleboy, @bramat, @gaurav.

Thanks for taking the initiative and creating a topic for discussion @antony . I would like to see if it is within Bubble’s ability to let us update when we are ready to do so. It seems like they tried to do this but haven’t stuck with it, or maybe I misunderstood the intent of the new arrangement.

For a short time, Bubble would push an update and then would allow us to choose whether to update or not. This would also allow us to rollback if necessary if it broke our app as well.

It doesn’t seem like this is something really being followed and I would love to see this implemented more. I wonder if all updates can be sent that way, let’s say once a day or once a week, and allow us to update when we are ready to do so. That would allow us to rollback if there is a glitch.

Anyways, that would be my wish for Bubble if they have the capability to do so.

Rock on Bubble! :slight_smile:

3 Likes

Bubble does offer the ability to update Bubble or not on dedicated servers. So, if your app needs more stability then there is a way to get that today, although it’s quite a bit more expensive than their other plans. Just FYI.

I suspect there’s a material technical challenge to enabling people to do this on a non-dedicated server. That said, I can see a lot of people wanting this so it very well may be worth Bubble’s time to implement.

I can also imagine a different process whereby new changes are first pushed to apps on the free plan. If no odd usage patterns or bugs are reported, then it’s auto rolled out to the next tier and so on. As such, people can opt into higher tiers by paying more or they can pay less but then also be partially responsible for helping Bubble catch bugs before they push that code to a larger audience.

We opted for a dedicated server a long time ago because we needed the stability, so this no longer impacts me directly. That said, I do think it’s important for Bubble to be sure the people who need stability are able to get it. In many ways, it’s a core feature along with performance.

4 Likes

@antony thanks for starting this.
Interesting thing is that I was planning on creating a post to start a discussion on something very similar to this as it relates to plugins.

If I was to ask any bubble user which bubble version they’re currently on, I’m sure most people will not know. For whatever reason, bubble hasn’t been using the traditional software release versioning system. I’m very sure internally bubble has a versioning system but they don’t make this explicit to its users. Bubble plugins though have explicit versioning.

Almost every Wordpress developer knows the current version their system is on. This is very crucial especially because of plugins. Most wordpress plugins have some specific versions of Worpress that the plugin can work with. So I’ve seen some wordpress applications that are on some older versions just because some important plugin isn’t supported yet in the latest worpress versions.

I’m sure we’ve all experienced the implications of not having an explicit versioning system in Bubble.
A plugin that is working perfectly fine now stops working because bubble made some improvements that exposed some vulnerability in the plugin. It is my opinion that there shouldn’t ever be a case that something that was working should stop working without any changes.
If bubble was releasing updates in versions, it will mean that users can decide not to upgrade or downgrade if a release breaks existing functionality.

My second big issue about this release topic is plugin support. For most software there is always a support period, e.g. 1year support with upgrades. But currently it appears that when a developer creates a plugin he’s supposed to support it perpetually and this isn’t practical and its not going to happen. I can’t see myself or any other plugin developer sticking around for 10, 20, 30 yrs to be upgrading plugin whenever bubble makes an update that breaks the plugin.

We’ve already seen glimpses of this issue where we can’t hear from @mishav concerning probably the most used plugin, toolbox. If there was a proper versioning system and you know that the plugin works to say bubble version 4.2.1 but doesn’t work in bubble version 5.0.0 you can decide not to upgrade to avoid the plugin not working. As a plugin developer I want to be able to say, this plugin works for bubble version 2.1.0 upwards. If a bubble release breaks a plugin, I can inform users not to upgrade to the current version until a fix is provided. This way we will never encounter, “it was working and i haven’t changed anything but it has stopped working”.

Bottom line, bubble should start versioning every release and allow users the option to upgrade.

10 Likes

That would be ideal, but Wordpress(.org) is not SAAS/PASS as Bubble.

You would need to compare wordpress.com(their SAAS version) which I think is always in their lastest stable release.

But if Bubble manages to implement a release system as you very well described we have a winner indeed :slight_smile:

2 Likes

@seanhoots @JonL Your feedback is what we are hearing from our larger enterprise customers and we are considering options here.

10 Likes

Hi @JonL I think bubble kind of have a release system, its just that they don’t make it very explicit. Currently I think the latest bubble version is version 5.
So they’re kind of doing it already just that there are so many “forced” updates than are not versioned. By “forced” i mean, they just push the update and users don’t have any choice as to accept or not.
I really like the current agile approach where they can quickly add new features sometimes even withing a day of someone making a request.
But I wish there was more control from users to accept any update especially performance updates which sometimes break functionalities. Bug hotfixes excluded.

Look at what happened with the console errors that actually ended up making some bubble sites unusable. I visited @keith’s groupz site and my browser froze because of the thousand console errors bubble was throwing.
All these updates are things that should have been made explicit to us with the control to upgrade or even downgrade.

Currently I like how the plugin versioning works. Sometimes I update a plugin and introduce a bug. I’m quickly able to tell my users not to upgrade to the current version until i’ve fixed the bug.

I wish there was a dropdown for bubble versions like the way we have in plugins so that if a version is breaking some functionality, i can downgrade and get my site working while the issue is being addressed, avoiding any downtimes.

As I mentioned my second issue is with plugin supports. Without proper versioning of bubble it appears I have to stick around and support my plugins till the world ends or I pass away.

3 Likes

Welll… major releases with known breaking functionality do get released as versions and one can choose when to move (see Settings > Versions).

So, there’s that. Then we’ve got the level below that where Bubble introduces a new feature that warrants an update to the Releases & Roadmap file (indicated in the editor by the “gift” icon), sometimes accompanied by a longer explanation.

I want those things on a regular basis and certainly wouldn’t want such non-breaking enhancements to require me to “upgrade” my app via the editor. There’s just little point in that. Bubble is SaaS and we want the benefits of SaaS, right?

Then there’s regular ongoing releases that are underneath that level. Typically, we assume these are performance enhancements, backend improvements, and whatever that do not generally affect us except to improve the experience.

Of course, software being software, bugs / edge cases / unintended effects can happen. The recent apparent increase in warning messages (that odd feature had been around for a while, but it didn’t get obnoxious until last week) is an example here.

I’d assume that, internally, this was perceived as a feature that would not negatively impact users, but perhaps this is something that should have been tested more thoroughly before release. Because it was kind of immediately noticeable to certain types of users.

This just feels like one of those, “Yeah, this is something that should have had more impact testing” things.

The thing with Bubble: There’s no SLA on lower level plans (I’d assume there’s SLAs on dedicated, but I’ve never asked). Basically, you need to have a certain level of spend before you can get that sort of thing. I’m pretty sure you can get what it is that you’re asking for – more-or-less – but it’s only available on a higher-end plan.

2 Likes

Bubble needs us to feedback any changes. Auto update is still the best approach. In case of emergency, having in settings the option of ‘return to the last working state’ (and temporary) until another fix, so we don’t have to worry for a certain ‘crucial’ period and keeping our Bubble app updated anytime.

It’s a little confusing, honestly. The way it works is this:

  • For our shared hosting plans, all apps run on the same infrastructure (which is how we keep the prices affordable). We don’t currently have the ability to run multiple versions of the Bubble codebase at the same time on this infrastructure – we have to update all our main cluster applications in one go.
  • For dedicated hosting, because it’s separate infrastructure, we can update a dedicated installation at its own pace, and dedicated users have a button in the editor they can press that puts the latest Bubble code on their box

We generally release code multiple times per day. For 99% of the releases, we are either fixing bugs or adding new features. We keep the behavior of existing features (apart from bugs) exactly the same, to avoid breaking apps that depend on that behavior. However, occasionally we decide that an existing behavior is really bad, and it’s worth asking people to change their apps in order to get a better behavior. That’s what the Settings -> Versions tab of the editor is about: it’s a way for app owners to say “Hey, I’m ready to opt-in to the new behavior”. This has nothing to do with releasing new Bubble code to your app: it just changes the way the current code behaves.

Now, sometimes we mess up and release a bug that breaks existing behavior unintentionally and causes app downtime. We try hard not to do that, but software development is hard and sometimes we screw up. When that happens, we revert the change, and we (usually) write an automated test to prevent that particular failure or class of failure from happening again. This automated test suite runs every time we deploy new code, and as we add more and more tests, it makes it harder and harder for us to accidentally break things.

Dedicated users are mostly protected from those mistakes, because we don’t update the version that dedicated users can deploy until it’s been 4 hours since our last deploy (ie, until our deployment seems stable). So if we do something really bad, we generally fix it before dedicated users have an option to get that version of the code. (Sometimes subtler issues do slip through).

We’d like to find a way of offering something similar to our main cluster users, because we know how important uptime and reliability are for everyone. That said, there are hard technical challenges with doing that, so right now we don’t have immediate plans to build something, though it’s something we are considering.

14 Likes

Hi @josh

A partial solution will be us having access to Bubble logs implementation (or by email) so we can track faster any eventually bugs. Sometime it take days, sometime it take hours to found out a bug. Faster we know there’s a change, faster your response will be, and everybody stay happy :slight_smile: Just an idea.

3 Likes

I completely agree @JohnMark

@josh This is obviously tricky given the architecture of the platform. Can we at least talk about improving the current status quo of users taking to the forums when something breaks, fumbling around in the dark for several hours then @neerja explaining what happened?

The bubble team keeps repeating the mantra software development is hard. We get a lot of things right, but sometimes we make mistakes While I can empathize with that on human level, I cannot empathize as an entrepreneur trying to make a living on your platform.

This situation is not inevitable byproduct of software development. This is a manageable problem with solutions. :slight_smile:

Bubble is everyone’s SYSADMIN and without your help and communication we are forced to deal with whatever happens in the dark–tearing our hair out, trying to fix things that aren’t broken, embarrassing ourselves in front of clients, wasting time–you name it.

Some ‘cheap’ ideas that could help

  • Notification window in the editor where devs can send messages to all users. You can opt out
  • A button for users to report issues–either inside the editor or the debugger. Bubble could monitor submissions for an uptick in issues (or even other users could).
  • Recruit ‘super users’ who are intimate and knowledgable with bubble to provide feedback and potentially act as a liaison between bubble admins and the community.
  • Create a user area specifically for communicating with each other and admins about system status.
  • A posted update schedule (not holding my breath for this if you guys are constantly iterating in a looser development cycle, HOWEVER this is really the gold standard)
4 Likes

So @josh, do version-test and version-live get Bubble changes at the same time?

Is it not possible to release first to all version-test apps and then to version-live?

3 Likes

Hi there @josh

Thanks for getting involved with this debate so rapidly!

I don’t feel your reply addresses the real concerns people feel at the moment.

We aren’t so concerned about a technical solution in the long term.

We want to discuss new business practices and communication in the short term.

Are you willing to engage in a discussion on these topics?

Best wishes,
Antony.

They are the same thing, @JonL.

The question I have about what happened last week, is this:

Did this actually break anything except my production app, GRUPZ? It pissed me off at the time not because it broke that app (I was blissfully unaware of this until days later) but because I was working on List Shifter and it made debugging very hard.

This, of course, made me rage-quit Bubble for a few days (like one does). It was only later that one of my app users (who apparently gets a ton of value from my app and alerts me to anything… like literally anything that seems fishy) was like, “hey, the thing is broken.” And I’m like, “Well, FML, you are correct. I think I know what this is… Hold my beer.”

And then I filed a bug report and it was fixed. And I’m like, “Well, I told you this was a bad idea.” But it’s not clear to me this was anything that affected anyone other than me.

I super appreciate the awesome Bubble support, and concern from the community, but I’ve not seen any evidence this particular whackadoodle feature actually broke anything for anybody except me. (And, BTW, if the “impact” on other people was confusion – e.g., ‘wtf is this thing that looks like an error message?’ – I’d already posted that this was a thing that Bubble does and, well, welcome to my world, and also this thing is stupid and a bad idea and please knock ITF off.)

So, from my perspective, I see this as a case of, “Bubble should really listen to @keith, especially when he uses foul language”, but not some major issue that requires a petition and stuff.

1 Like

@keith - my app broke too.

1 Like

@keith If you have a glance at the initial post, it’s quite apparent that your experience was synonymous with others (mine included). Not to mention the several other bug reports.

1 Like

I break my app more often than Bubble does.

And if you aren’t breaking your app now and again … then you probably don’t have any users.

4 Likes

Thanks again to everyone participating in the thread – I appreciate that you care enough and rely on Bubble enough to want to push to improve the platform! To people’s points:

I’d be happy to share this with the community but I’m not sure how useful it would be. Here’s the most recent 10 entries from our internal deployment tracker:

05/09/19 04:31:40 pm EDT 2f728bb3cc7a231c
bug fix

05/09/19 02:31:40 pm EDT 00c88438cdae488c
CSM tweaks

05/09/19 12:22:30 pm EDT 656616fb1ae52d27
cleanup absolute positioning for datepickers in popups

05/09/19 12:10:33 pm EDT c63ce336323370d1
CSM tweaks

05/09/19 11:40:24 am EDT 5106f70d3c7297dd
Revert "http_request logs response/body regardless of error"

05/08/19 09:47:48 pm EDT 0c00785c16f04761
bug fix

05/08/19 08:45:20 pm EDT 8ef4efefbb2f3412
meta fix

05/08/19 01:57:37 pm EDT fb4cd7984b5672a8
CSM logic fix

05/08/19 01:24:51 pm bdd931d43d51bd43
http_request logs response/body regardless of error

05/08/19 09:41:55 am EDT 1e19e52bfabc37a4
bug fix

As you can see, we’re constantly making small changes, and it’s hard to correlate which ones corresponded to which user issues. Re:

I absolutely get why this is frustrating. Knowing that your app is broken, your customers are unhappy, your businesses is at risk, and not knowing if you did something wrong or if it’s a problem beyond your control is absolutely enraging and terrifying at the same time. I’ve been on the other side of this dynamic a lot: whenever something happens with Bubble’s upstream service providers such as AWS, the Bubble team feels exactly the same way. So I get it.

The reason it happens, though, isn’t because we’re missing features like a notification window or error reporting. We have https://status.bubble.is/ which lets us broadcast to all users, and we have the https://bubble.io/bug_report page for reporting bugs.

The issue is a signal-to-noise problem. We have a constant stream of bug reports. Many of them are things that have been broken for 6 months in an obscure corner of the product, and a user just happened to stumble across it and notice it. Others are a user building something really complicated, making a mistake somewhere, and reporting it as a bug to us because they don’t fully understand how Bubble is supposed to behave. So when we do release something that we need to roll back, it generally takes some time before we realize there’s a real problem. Most of the time window where you’re on the forum trying to figure out if something is broken is the window between us releasing the code and realizing there is a problem, not the window between us realizing there’s a problem and us fixing it (which tends to happen pretty quickly, often less than 5 minutes). I have seen numerous occasions where someone started a “is Bubble broken?” thread and it turned out to be a false alarm, or an issue affecting just that users’ app: it’s really not easy for us to tell.

I think the solutions here are: a) as the Bubble community gets bigger, it’ll be easier to detect statistical patterns, b) continual investment in our automated monitoring and testing, and c) us staffing more humans to read and respond to bug reports (note the open positions: Careers | Bubble) which will improve our response time.

As an entrepreneur trying to make a living on various platforms, I can empathize with that sentiment as well :slight_smile: But putting on my cold, logical business hat on for a second:

  • No platform is bug free or downtime free. Reliability costs money. Google’s Site Reliability Engineering team (one of the more respected organizations in the software industry) estimate that going from 99% uptime to 99.9% uptime, or 99.99% uptime to 99.999% uptime generally multiplies the costs of running a piece of software by an order of magnitude: in other words, 10 times the team size.
  • Businesses generally offer tiers of reliability and support, backed by SLAs, and customers pay more for higher tiers based on how critical their application is (it doesn’t make sense to pay for 99.9999% uptime if 99% uptime is enough to keep your users happy)

We’re a little primitive on this front right now: we don’t have formal SLAs, and only really two tiers of reliability: main cluster and dedicated. Formal SLAs is absolutely on our roadmap, but rolling it out requires growing our team, because right now we’re not adequately staffed to guarantee more than best-effort. This is something important to me, though, and we’re trying to hire to get to a place where we can offer a) formal SLAs, b) better reliability at lower prices. In the interim, though, our answer is dedicated: it’s not perfect, and we don’t provide formal SLAs on dedicated either, but we do pay higher technical costs to insulate dedicated users from many issues that affect main cluster issues.

Right now, it is not. I could imagine that as one potential option, but that raises other issues. For instance, if we release a new feature to test, and someone uses it in their app, then deploys their app to live before we deploy to version-live, their app would break. Managing that is potentially feasible but it seems like it could get very complicated.

Do my above comments do a better job at addressing your concerns? Your original ask was for a policy about:

I think I’ve explained how release updates get made to our platform. I know some people on this thread are advocating for a release schedule instead of continuous deployment, but I respectfully disagree: I think continuous deployment is the direction that pretty much all major web platforms are moving towards, because on average it leads to better reliability than big releases. (A good overview of the theoretical reasons why this is true is in the book The Lean Startup by Eric Ries).

In terms of documentation of releases, as I mentioned, it’s our policy never to make breaking changes to user apps without adding a documented version in the Settings → Versions tab. Most changes that break user apps in practice come from us trying to fix another bug, and inadvertently doing more harm than good – we don’t know we’ve done that until it’s too late!

In terms of escalating issues, see my comments above about SLAs and the challenges we have sorting out signal from noise. We do plan to offer premium support down the line, but we simply don’t have enough people on the team to offer it today (please send Careers | Bubble to all your New York-based friends!)

19 Likes