Best way to bulk upload?

I need to upload records in bulk (CSV). Scale is ~500,000 records. I tried uploading parts of this before and it takes hours to upload ~60,000 records (while needing to keep the pc on, bubble open etc.). Is there a better way to upload this data ?

1 Like

New CSV uploader is better, but still pretty slow. I would imagine it is throttling itself.

It is taking long enough with 5000 rows, let alone half a million !!!

Alternatives …

Blockspring can read a google sheet. But I would worry about the performance of that too.

Zapier can run a workflow that could add the row (advantage is this is “background”) but … that is half a million workflows so unless you are on an unlimited plan … you have blown your limit.

Maybe talk to Emmanuel if this is one off load.

Sorry, not much help. But I have struggled with data upload thus far, and I am nowhere near your numbers.

1 Like

wont every single read of a row by blockspring count as one workflow run by bubble :frowning: I’m afraid I’ve a limit to workflows in my plan

Yes, that would be the same issues. As would using something like iMacros for Chrome to push a csv at a page.

Currently waiting for this …

Maybe a chat with the guys would be best.

@emmanuel is there a workaround for bulk uploads (ref. above) ? :sweat:

1 Like

The CSV uploader will take some time because we want to keep the server ‘airy’ but should do its job properly. Start with a 10k file and leave a tab open. It should be very reliable.
Then try 100k. Will of course take much longer but if that also works for your browser/computer/connection combo, 500k should work as well.

@georgeciobanu yeah but it takes like ages … can I atleast open bubble app on different computers and upload different files at the same time ? on a side note, I am 100% sure connection is not an issue at all as I’m on a university super high-speed network

And … It seems like it created two records for each row.

@gaurav you’re right we throttle these uploads it’s not your connection
@NigelG was that using the normal upload process? Is it possible you had two windows open, or you tried uploading the same file twice?

No, I would have noticed, it takes several hours to upload :slight_smile:

Data upload works nicely via CSV now, it is reliable if a little slow.

So I’m confused… Can I upload 2 files in two separate Windows simultaneously or not ?

@gaurav - yes just make sure the files are different.

Hello Bubble Family,

I need assistance here. I have tried to upload bulk data to my app but not successive. I got below screen showing 674 of 924 validated.

My app fields for type SPARE PARTS are as below:

This is the link to the similar public app but its behaving same way … https://bubble.io/page?name=index&id=responsive-tabs&tab=tabs-3

Unfortunately i cant upload the file here but file type is a CSV as shown below:

Please assist

Have you tried uploading is smaller chunks?

I hadn’t but i have just tried 20 items and its now loading.

So whats the optimum number of items that can be uploaded in bulk?

It should work for larger amounts, but there might be something specific with your data set or connection that makes it a little lower.

Ok fine, thanks for support

Any new updates on this topic? I am in the process of uploading a 37Kb CSV and it’s frankly taking a ridiculous amount of time considering the size of the data set (1000 items).

1 Like

I posted about this yesterday, but received no replies. I would like to know why the upload rate is so slow. It’s like breathing through a straw. It seems like an artificial throttling down, or is there a technical reason?

It was mentioned above that this is an intentional limitation, which I would understand, but not at this level and maybe not for an essential part of a paid feature.