Possible Bubble limitations / dead-ends?

Questions
1

Hello,

I’m very excited about bubble and we’re seriously considering using it for our admin panel, essentially being a UI to interact with our API endpoints that CRUD data in our product’s SQL databases.

I’m just trying to think a little bit further down the line and see what could be some potential dead-ends or huge roadblocks with this, what would you say are the biggest limitations to Bubble ? and which ones are known to never be fixed ?

I’d highly appreciate any feedback :slight_smile:

Mehdi

1 Like
2

I just temporarily stopped a Bubble project similar to what you’re referring to. Data limits such as how much I can pull from an API in one call (before it times out), how that information gets stored (added entries into a database can sometimes take a while and may skip… leaving you with incomplete data) and how slow loops (called “Schedule API on List” in Bubble) are performed on data sets (example: I pull data from an API and I want each data to be it’s own entry in a database… Takes a long time) just killed me. The project has been put on hold because the time being spent to overcome these is eating into our budget. As an example, I had 188 items being brought over from an API that each needed to be stored as an entry in the database. It added only 77 and took 45 seconds. Sure, I could get it faster and make sure it got all 188, but this would take time to really deep dive, learn what I’m doing wrong, and correct it.

What you need can probably be done, but the time savings may not be what you are expecting. The initial start is very fast - But when you get into the details and try to optimize or deal with timeouts on API calls, you begin to realize limitations. The community is very creative with how they overcome limitations, however, it’s usually at the cost of having to deep dive and implement complex solutions to otherwise simple problems. And usually, it’s something that can be easily done in the programming world. Oh what I would do for “for” loops or “while” with a clear “I’m finished” return…

My recommendation: Before you make the switch, play around as much as you can. I do believe you can do anything in Bubble - But at the cost of having to work around unique limitations.

5 Likes
3

Thank you so much for your feedback w.fly.

I’ve read about loops quite a bit yes and I did notice that the API calls were strangely slow compared to when I execute them directly in postman or swagger.

About the performance and timeouts on the api is that something that could be solved by paying for more power units or is that an inherent limitation to the system ?

Mehdi

4

I love bubble I wouldn’t be able to accomplish what I have without it. I have no idea how to code.
I use bubble nightly (I’m a plumber by trade) for between 1 and 4 hours and would refer to it as my hobby.
Having not known anything else I would say it’s ace.
However a lot of the above rings true.
I have found myself making a lot of work arounds and have often hit dead ends so the above post is quite possibly true. If you know another language, have the knowledge then maybe bubble isn’t the right program however for someone like me bubble is perfect.

3 Likes
5

This is still an important limitation :

I can deal with the slowness running tasks on a list, but still not being able to lunch another Custom event (or another Schedule on a list of things) AFTER the last task has finished is painful
@Bubble

2 Likes
6

Anytime! And yes, more capacity/higher plans increase the speed from what I’ve read in the forums from folks with Professional/Dedicated Plans. However, it’s still a debate as to HOW MUCH speed you get vs. lower capacity plans. Bubble has put forth a great effort in giving information to help us determine this, including the ability to add temporary capacity so you can test first without going to a higher plan… But from what I’ve seen, there still is quite a bit of confusion as to how much impact extra capacity really has… To boot, some folks just have poorly optimized sites.

As far as inherent limitations, there are a few.

#1: Crons - (Known in Bubble as Schedule Recurring Event). Crons are the ability to run code at a specified time outside of your application. I used it mainly for doing automated cleanup on databases (if items in database are older than 30 days, remove). In the programming world, we use crons ALOT… However, in Bubble, how often you can run crons is dependent on your plan. Dedicated (I believe $500/month+?) gets you daily crons. I have a Personal plan for my current app, so I can only run 1 monthly. You can use outside Sources, like NodeRed (which is what I do), however, doing that just adds more development time.

#2: APIs - First, they are rate limited to 1,000 requests/minute. Second, I believe there is a timeout… In my Wrike implementation, I attempted to pull 800 tasks and it failed every single time after about 15-20 seconds. I seem to recall something in the documentation about 4 minutes being the official timeout…? In Postman, it took around 2 seconds to load all of them. I think it took much longer in Bubble because I took the data from the API and attempted to put it into a Repeating Group (Bubble’s version of a “looped” table) immediately.

#3 - Database/Lists: @NigelG is the expert on all things Database in Bubble. Database record limit. <-- Link to his post about the record limit. Another great post on this is Limitations running workflows on a list <-- Look at @josh post from Oct 17.

Hopefully that helps!

4 Likes
7

This was one of the big problems I had with my last app on Bubble. I tried the suggestion in that forum post, but the time it took to update the database then caused timing issues. So, when I would update ItemNumber, sometimes it wouldn’t actually update it, it would skip it. So the ItemNumber was much less than what was already in my database.

I believe this goes deeper into a recent forum post I found: Order of actions inside an event Another core issue here is the fact that your workflows don’t happen in order, rather they happen all at the same time. So if Step 1 of my workflow is to add an entry to the database from my API then Step 2 of my workflow is to add 1 to my ItemNumber to signify that I added another item, it may not actually happen in that order causing my ItemNumber to be wonky and not reflective of what is actually the real ItemNumber.

8

Lack of data consistency in the API Workflows is the biggest issue at the moment IMHO.

You can kick off a workflow and you have no idea what steps will execute and which ones won’t.

6 Likes
9

Thank you all so much for the feedback, it definitely gives us a lot to think about, bubble being a closed system on top of all this, it seems like the probability of this creating more problems long terms than it solves for us short term is pretty high :unamused:

Mehdi

10

Yep… I have the same problem… My app has a lot of API Workflow dependency… and I feel that I can’t have my app running until Bubble address that issue.

11

Could you share more details about the inconsistencies you’re seeing? We use the API a lot ourselves and I don’t believe we’re running into any inconsistencies.

2 Likes
12

Hey @sridharan.s how are you?

Sure… I can definitely share a few things that we are experience with our Bubble app. The first thing important to notice is that this is a well know issue within Bubble and the team itself already have that on their roadmap:

"Apps bug-free at scale
Workflow consistency when multiple workflows touch the same data at the exact same time"

You can see the reference on Releases | Bubble

You also can find a few comments from @emmanuel around the forum about this issue. Acording to him this is major on their roadmap.

So, in our case we deal with a lot of data that we retrieve from third parties APIs. When we setup a client’s account on Bubble we run a lots of API workflows and over 3.000 external API calls. And they pretty much depend on the result of the first “thing” so they can process the second, third thing and so forth. That’s when our problem starts. Also, we can’t have duplicated entries on our Data Base, but since Bubble is processing the same data (not the same action) multiple times it is common leave behind some data or duplicate the entries.

We are doing a bunch of different things to certify that missing and duplicated data are reduced but it always happen. We are now at a point where we have API Workflows going against our data base regularly looking for missing or duplicated data.

One thing we notice is that these problems start to happen when our application reaches 100% CPU. So, in other words, when you app reaches 100% CPU not only your app gets slower (which is fair I guess) but also depending the operations you are doing (API Workflows in our case) the results are not consistent and you will find problems with your data. I guess my biggest critic here is that API Workflows are asynchronous job, so I wouldn’t mind have them taking longer to be execute but I can’t afford having them being unstable with data processing.

This is an image from our usage capacity in the last 7 days… As you can see is pretty clear when we are adding a client’s account to our app.

34%20AM
34%20AM2488×1422 292 KB

Up to this point we have our app ready to go live but we are holding on to Bubble start guaranteeing Workflow consistency, otherwise it will be insane having 60, 100, 500, 3.000 clients running something that isn’t consistent.

You might ask why we don’t increase our Bubble plan to add more processing units… We already tested with over 12 units but our app always reach 100% when we are setting up an account After that we would need less then 5 units to have the app running… So financially it wouldn’t make sense having 14, 18, 20 units that we only need to use two or three times a week.

Also we are moving some of our heavy data (DB Tables with more then 5,000 entries) to an external Data Base where we can handle better that data and reduce those problems.

It would be REALLY nice if Bubble could allow us to add units by the hour automatically when our apps reach 100% capacity.

Not sure if I helped you at all… If you have more questions let me know!!

3 Likes
13

I also have serious issues with API workflows and performance issues around them. It has become a major roadblock that has kept me from moving forward with my app. You can see more here: Another post about performance issues

2 Likes
14

Thanks for the added details @scharan. A few related thoughts / insights for you (and the community) in case they’re helpful for anyone:

  • When your app hits 100% it does delay API requests. For many apps this is probably a smart prioritization of resources. For apps that depend on the API being timely, it’s not. Hopefully this is something Bubble will give us more control over down the road.
  • It’s odd to me that hitting 100% would result in duplicate or missing data for you (if you’re not seeing this when it doesn’t hit 100%). Two potential workarounds for you: 1) add the thing to your database before running the API calls that create/edit that thing. This way, the thing is always there and you won’t get different API calls creating it at the same time, 2) if possible, it may help to space out the API calls a bit (i.e., schedule them for 10 seconds or 1 minute apart or something like that) - this would reduce the likelihood 2 APIs collide with data and it’d also help space out the server requests so you’re less likely to hit 100% usage.
  • The issue with workflow inconsistency when multiple workflows touch the same data at the exact same time - I don’t believe this is an API issue per se, but instead an issue of any/all workflows (potentially) colliding when you’re editing the same data through numerous workflows at the same time. Bubble does offer guarantees if you’re running a single workflow (with multiple actions) so long as you’ve set it up appropriately. If you were to run your API calls as 1 workflow, instead of many, that’d likely eliminate this issue (but would mean that if 1 API call failed, the rest of the actions won’t fire - so you’d need a separate solution for that). I’m not 100% sure on this, but it’s my understanding and I believe this is a solution we’ve used in places (although my business partner owns that portion of our app so I could have misunderstood her).
  • I previously asked Bubble for the ability to ramp up/down the number (or size) of servers (e.g., on an hourly basis). This is critical for apps that have usage spikes like ours. My impression was that this isn’t on their roadmap, but I also believe there are enough apps that essentially require this to scale their business that Bubble will eventually need to build it to support power users. This is fairly simple to setup with custom apps in AWS so I can’t imagine it’d be that much work for Bubble. My guess is they’ll build it in the next 6-12 months, but I’m hoping they get to it sooner.
3 Likes
15

Tks for the reply @sridharan.s

So, about what you mentioned:

  1. About the delay on API calls, that ain’t the problem… We thought that might be the problem as well but we carefully looked at Bubble’s logs and also the external API Logs and we do see that data is returned and Bubble receives the data. We got a few "API Workflow errors"but most of them were problems with our setup. But at this point I can guarantee the problem isn’t with data being returned to Bubble.

  2. We are already doing booth things you mentioned : ). We are avoiding at all costs having data on server RAM so we constantly record data on “Temporary” tables so we can process latter. By the way @Kfawcett, reading your “Another post about performance issues” this is a great tip for you. We are also leaving plenty of “room” between processing things on a list… Some of our items have over 15 seconds difference.

  3. Yeah you’re right, it’s not exclusively about API Workflow but I do think they will address the issues we are currently having when they implement that functionality. About the 1 workflow guarantee, yes I guess you are right, they do offer that… but in my case it would be impossible doing everything I’m doing with only one workflow. Only 1 API workflow is for simple tasks I guess… We do a lot of scheduling API Workflow on a list.

  4. Yeah, My opinion is that they won’t do this any time sooner because they are making more money like it is right now. For example… we are running our app with 7 units… We might have to go up at least to 15units if we wanna take this to production… If that “scale” feature exist, our app would be running we around 3-4 units most of the time… I’m sure most of you feel that way.

16

A few added thoughts:

  • Have you looked at the Data API instead of running everything on a list? It’s Soooo much faster. Was a complete game changer for us bringing data back into our app. We used to have workflows that’d take several minutes to load 50-100 rows of data and now it’s loaded in a second or so.
  • Agree that allowing temporary boosts to server capacity could appear as if it’s not in their short-term interest, but it’s absolutely in their long-term interest because keeping customers on Bubble is essential for their business model. Plus, they make all of their money on large-scale customers, not small ones, (based on their current pricing model) so it’s absolutely in their financial interests to support power users and keep as many of them as possible using Bubble for as long as possible. I’d also like to add that Josh and Emmanuel are working on a vision much bigger and meaningful than simply creating a business or making some money for themselves - I think they’re working hard to create something that’s truly valuable for people. I can’t imagine them even thinking about making short-sighted decisions for selfish gain at the expense of their customers. Just doesn’t seem to be their approach.
2 Likes
17

Can you expand on this? I would really like to know how you did this and see if it is something I can integrate into my own app. Screenshots maybe? :slight_smile: Thanks.

18

@kramwe wrote up a thorough explanation here: How to write data to BubbleDB using Data API Bulk Data Loader

Enjoy!

4 Likes
19

Hey @sridharan.s

I’m familiar with the Data API. The problem is that we don’t have all the data ready to go. We have to get pieces fromAPI #1 and with that result go to API# 2 and get more data… We hit over 15 different APIs endpoint in order to get all our client’s info. So we do need the logic behind API Workflow, it isn’t just loading the data… At least my understanding of the Data API is that you can read and add data in Bubble’s DB really easy… but not process the data. Am I correct?

Tks!

20

What do you mean by “prcoess the data”?