I have a potential customer that exports a large dataset to excel/csv, >10,000 rows, each week. The data from this “dump” needs to be managed. Current methods don’t work for several reasons, plus it needs to be managed “in the field.”
Since Bubble has a low limit on the number of rows that can be imported at a time, and breaking up the data into smaller chunks would be inefficient, error-prone, and a deal-breaker, I’m wondering if there’s an effective work-around to this problem.
Does anyone know if there’s another service that the data could be imported into, then linked/uploaded to Bubble through API, etc? What about to a SQL database and use the SQL connector? If that would work I have no idea how to set up the SQL table, or whatever it’s called.
At the moment Bubble import limits are not ideal, very slow and limited to 4 minutes before ‘time out’. Currently, the process of using an API is also not ideal, because the field-time process is too long. For 10,000 records it might take hours. We look forward to an effective solution to this situation. I suggest to mount another server with at the limit a visual tool like ‘WinAutomation’ and support downloading and processing the file, do keep the essentials. Not an easy task unfortunately.
Thanks for your feedback. So it wouldn’t work to upload to SQL database and connect to it using the SQL connector? All records don’t need to be retrieved at once. Generally 10-50 with occasionally maybe twice that.
As far as I know you cannot import from sql into your bubble db directly. You could build a hybrid solution where most of the data resides in bubble db and refresh data resides in a sql db which is easily refreshed. The winautomation idea is probably the best plan if you need to keep everything in bubble db and can be surprisingly fast if built properly.
Thanks again for the feedback. I’m confused though. What then is the function of the SQL Connector plugin? I had plans to utilize that in the future for an app that I already have in service.
The sql connector allows you to talk to a sql db. Once connected it’s sql as usual provided you can write sql code.
Winautomation is a bot. You code it to do an task and churns away happily. Ie take the 10k list and upload it in batches of 200 without making any mistakes.
Thanks @JohnMark for that detailed example. I’m still fascinated by the level of support members of the Bubble community are willing to provide to others.
Unfortunately I’ve been swamped on my day job so I haven’t had the chance to play around with it yet. I downloaded it but that’s as far as I’ve gotten.
