@jon2 We are 100% agreeing with you on the importance of reliability. The challenge as @StevenM mentioned is we have a global customer base so it’s peak time somewhere at any given point of time. Our focus is therefore first on preventing such situations.
2 Likes
For those of you who want to know how it ends up for me, here it goes:
As @gf_wolfer mentioned, many things can go wrong on such kind of meetings, internet connection specially. So, in order to avoid these circumstances, I often carry a PowerPoint with all the screenshots. But this day was a special one.
I have been ‘annoying’ this client for so long just to get a meeting, and out-of-nowhere he called me today at 11:30AM (ET), telling me that there was an important meeting at 3PM (ET) in the company, and he will give me some time to present to everyone involve in the decision-making process.
So, rather than preparing myself better, I was rescheduling my entire day, logistics to get there, etc. I got there couple of minutes before, I checked on my phone the app was working, so I was ready. Once I got into the conference room, I saw the CEO, a TV anchor, plus a top politician, and a bunch of other people. Then I told myself [laughing]: “You better don’t screw up this”.
I turned on the screen, plugged the HDMI cable, and once I hit enter to get into my site: “ERROR […] Missing Template”. I kept it calm, checked the WiFi again, went to Bubble Status and there I realized “I was fu**ed”. (Sorry for the F word, but no other could describe the situation better).
I unplugged the HDMI, asked for some seconds, while I was thinking what to do, since I have been in similar situations in the past, but not at that level. I hit to the forum, and I got to admit, the title I used might be a little bit over dramatized, but hey… probably was just a reflection of the impotence and frustration I was feeling. (@antony, I know you felt it too)
Anyway, I decided to explain that there was something going on with the server, probably due to maintenance. Immediately the engineers jumped up to ask why do I do maintenance updates during business hours, I elaborated a little bit more on the issue, but immediately cut the topic and started explain my app verbally. I could even beat a French mime for how many shapes I drew on the air.
While I was doing this, I was asking questions to certain people to get some interaction and extent my “presentation”, I even provoked -on purpose- a confrontation between an IT dude, and a marketing person (that was an easy one), just to keep gaining time.
After an hour of just ‘selling’, I was about to end up the session, and then the Bubble Cluster came back just on time for a final closure. By the way, @wegetdesign thanks for Google Cloud SLA tip, it helped me keep the old-school programmers at bay during the last Q&A part.
Thanks to everyone who got me some empathy, that was a hell of ride. @neerja , please check @jon2 suggestion. It might save some others in the future. Thanks again community!
CapiBalid
PS: I GOT THE [EXPLICIT] CONTRACT!!!
23 Likes
Additionally, might it be possible to give a heads-up prior to any major planned updates / deployments? (My hosting provider as well as Salesforce do this.)
Also (and pardon my ignorance here), is it possible to “roll out” updates at different times in different parts of the world?
And lastly, congrats, @CapiBalid, on clinching the deal! 
2 Likes
I feel your frustration man, I’ve been in the exact same spot. But awesome that you closed the contract, congrats! I’m sure there’s a lot of hard work, obstacles and disappointments until you reached that 
Well done @CapiBalid. Even with all the adversity, you still persisted. Thank you for the update
1 Like
Very glad you ended up with the contract @CapiBalid
Bubble Forums, come for the technical support, stay for the nocode success stories.
2 Likes
Hey @CapiBalid…
WELL DONE! Your explanation of how you handled the situation was so inspiring to me. In that moment where it all goes wrong, and the big prospect is sitting there waiting, it is so easy to be overwhelmed, and it sounds like you were totally cool and got the sale. Big congratulations.
I was also really inspired by your ideas @gf_wolfer… this age of cloud computing certainly calls for some more localised backup technology, and those are really sound ideas. I’m still sitting under a rock developing right now, but when I come out into the sunny world of client meetings, I will take those inspirations with me.
So something good has come from this outage!
All the best,
Antony.
4 Likes
Hi there @neerja, and also something I would love @josh and @emmanuel to read…
Firstly, I want to thank you personally for all the positive support you are giving us. It feels really great that you are always there with constructive things to say when there are issues, with rapid responses to our requests for information, and for us to know you are there at Bubble HQ to feed all our thoughts back into the team. Many of us are taking a huge leap in trusting Bubble with our dream businesses, and certainly for me you are making that feel a much safer decision.
Regarding the software upgrade and following outage issues… I would agree with all the ideas that have been put forward, and I would like to add another dimension to them.
I have been privileged enough to have worked for several amazing startup software companies during my career, and it is always interesting to see the common challenges each company faces as they grow. In this case, I am thinking of not just how software is released, but more importantly, the processes for testing.
Of course, a release is far more likely to be a success if the testing is thorough enough first. In my experience, building a high quality testing methodology becomes critical as the company grows from the handful of founders who know their product so well, to having a more diverse team involved. It takes real customer focused leadership to see this as a priority amongst all the other issues on the rapidly growing company’s to-do list.
From what I have seen it is usually the customer who shout about this first… and it is the companies who really take this on board and develop a top quality testing methodology who end up with a product everyone loves and can rely on as the new market becomes more mainstream.
So my hope is that the “miserable moment” experienced by @CapiBalid yesterday can bring focus to the importance of testing processes as Bubble grows.
Looking forward to all that you folks at Bubble will bring to us in the coming year,
Best wishes,
Antony.
4 Likes
As a software consultant, being able to “tap dance” when issues come up during a demo is a necessary skill. 
Also, having a backup presentation, or a pre-recorded video, that walks through what you were going to show live should always be part of major presentations that cannot be rescheduled. Heck, even the machine you’re using could crash, so having print outs of the slides or a spare computer can also be handy.
Unfortunately, I think most of us have to be burned a few times before we learn these life lessons.
If interested, this is a great read:
4 Likes
Congratz on landing the gig - you must be one hell of a tap-dancer. At least you no longer have to worry about testing the client’s character in a crisis. They sound like a good one.
Nothing sells whisky like an outage.
@CapiBalid, congrats for the contract, after all!
About two years ago I was very impressed by Bubble, however I soon realized it has some serious limitations. I had a situation, too - I used Bubble in one of my courses at the university to show students some basic design patterns and concepts. A part of the final exam should have been taken by using Bubble, and when the exam started - here comes the improvisation. Now I always plan for some degrees of freedom when dealing with Bubble.
I don’t have a clue about testing and maintenance processes of Bubble itself, but on other platforms there exist at least two instances, something like bubble.is and bubble.beta.is. The latter is open for the users, so they can test their apps before the changes go into production. Every two to three weeks the beta is cleared and all the apps from production are copied to beta. And two or even more instances worldwide, to put the maintenance under non-peak daily regimes. As I said, I don’t know what’s already in place.
Hey all, first of all, huge apologies to everyone affected by the outage yesterday. I’m really glad @CapiBalid was able to improvise and close the deal!! Very sorry to have created additional stress – entrepreneurship is hard enough without unexpected outages in the middle of demos.
This one was really a screw-up on my end. This wasn’t intended to be a code deploy: I went into one of our backend control systems to make a pretty small settings tweak to a bunch of our servers (they were scheduled by AWS to go down automatically for upgrades, and I was trying to disable that so we could upgrade them in a zero-downtime way at a future date). I made a typo in the command I ran, and ended up overwriting some settings that really shouldn’t have been overwritten, which caused our server management system to think that a bunch of resources had disappeared, causing mass chaos. I’m the only person on the Bubble team with regular direct access to that particular admin system, on the theory that I know it well enough / am careful enough not to do stuff like that – so I feel particularly stupid right now.
To shed a bit of light into our normal testing and procedures, we generally deploy new code multiple times a day, and run an extensive set of automated tests on each deploy. We feel like frequent, small deploys leads to fewer issues than occasional big deploys, and as mentioned above, with customers in every time zone there’s never a good time for us to go down. This overall philosophy works pretty well for the core product. That said, the last couple of times Bubble went down was due to issues with our server management system, which is a separate system we built to manage our infrastructure, and it’s exposed a couple weak points where an issue can have a big impact on our systems that slip by our normal testing procedures. We plan to go through the events yesterday in detail to build controls to prevent a similar outage from occurring in the future.
28 Likes
Thanks for the detailed transparency!
1 Like
Oh @josh… what it is to be human!
Thanks soooo much for coming into the forum to hold your hands up and explain what happened. In this world which seems to be managed by people just covering their own backs, it is wonderfully refreshing and reassuring to be working with a company who are just straightforward and honest!
4 Likes
On top of a fantastic product, statements like these are why I have become an ardent Bubbler and spent the last 12+ months building an app on Bubble. 
Keep up the great work @josh and the rest of the @bubble team.
8 Likes
Thank you for the honest response. I have worked in tech, for a large insurance company, for the past decade. And it seems the more common approach is to deny, deflect, and defend, whenever there’s an outage (at least when it’s with an “other” group and there’s no way to audit what happened). It is refreshing, and confidence building, when dealing with a company whose founder is willing to offer such transparency.
I do hope to see a status update, on the forums, when there’s been a material update to handling of the server management system. It’ll help me sleep better knowing to expect fewer incidents/outages like this one.
1 Like
+1 on this. I certainly appreciate the transparency as well. I gives me confidence hearing what happened in such a candid way.
@josh
HOWEVER
Can you shed any light on how this issue also managed to affect dedicated (Enterprise) plans?
-
From https://manual.bubble.is/dedicated-plans.html
This isolates you from the main Bubble user base, and will offer higher and more consistent performance.
Would you be able to share the metrics on the timezones of where the majority of your users are located?
Given that we are all human and make mistakes, get sick etc. Does bubble management have plans to add some policies and procedures to mitigate the risks of these kinds of predicatable mistakes?
- It is a little unnerving to hear that if you alone got sick, left bubble all of a sudden, then hundreds of thousands of bubble users —including enterprise customers—and their users could be put at risk.
- Cross-training non-founder-level staff. Maybe they could deploy a tweak like this during non-peak times when founders area away from work? Act as a sounding board and a 2nd set of eyes for critical tasks? Help proceduralize things a bit better? Frankly, I’m still questioning the reasoning behind the timing of this tweak.
A sincere thanks for the update on what happened. It was enlightening
1 Like
If you do frequent small deploys, do we get logged out/session ended when you deploy? Or do we not notice anything?
Why do we have to show a ‘new version released’ message to our customers but Bubble do not?
Because I’d like to do frequent small releases to Live as well but the messaging to the customer prevents this.
Is there any way we can also do frequent small releases but make this not noticeable? As I can’t say I notice when Bubble do a deploy…
This topic was automatically closed after 70 days. New replies are no longer allowed.