Hi Leonidas,
Great question. As we are familiar with AWS we had thought keeping this in an RDS db separately, but we had two reasons for exploring it in bubble.
-
To see how it would handle the records and understand where the limitations of scale are. We are like many hoping to have an application that grows to a huge number of users. Our competitive has over 100m users, so we thought it prudent to look at this.
-
This table is so integral to our system, even though the data is supplied by a third party, we have a large portion of our functionality tied to these tables.
Further, we don’t actually have the original source data in our own SQL, we get it via CSV and then i imported it into mysql so I could better manage cutting it up and putting it into bite size chunks multiple scripts could handle and track the segmentation of the data. I can not open a 3.2M record csv in excel, so I know MySQL well enough to manage it all. So each quarter when the data is updated, I get CSV files.
My thought is to either keep it in another SQL DB local and build an application that feeds it to bubble or keep it in adjoining cluster at AWS/RDS.
To be honest, we are not sure how we will handle this in the long haul. We are hopeful that bubble can scale with a dedicated cluster, and with the ability to dial up the elasticache and db CPUs independently, that it would not be that much different than a traditional AWS cluster running laravel or django or some other middleware/application server, on a “hand coded” application server.
We see the huge benefits on development time and turns on iteration. We had originally thought, “let’s just build the MVP in bubble and then use that as reference to build the production app”, but we are now hopeful it can reside in bubble. Tools for big DP will be required I think for Bubble to become the large IT shops tool of productivity, which it could be(I believe) , freeing up IT staff to focus on their job and business people who own the business intelligence to build apps for their internal customers.
Anxious to see how everything works once we build the full app, but I am concerned at this point as if I go to the data tab and search for a unique number in the 3.2M database table, bubble gives me an error, “come back later” response. So I am not sure how to get around that without building page that admin this table separately. I assume an “any field” query like the canned one in data tab is not going to hack it.
I dialed up to 11 CPU units to get this to be able to achieve the 900-1000 Api/calls a minute on a shared/public instance of bubble.
More than you asked for, but would love your thoughts? We have only been 45 days into bubble.
Craig