Exporting large volume of data

[adrien]

Hey everyone,

I have two large datasets in Bubble (70k entries and growing) that I need to export (backup) on a regular basis. Sadly if I try to export the data as .csv from the admin panel, I only retrieve around 20k lines.

I tried creating buttons that trigger workflows that exports the data as .csv. Say, one button for the first 10k, then 10k to 20k for the second line, and so on, but the second button seems to fail for anything over 1k. And at this rate I would need 70 buttons, to download my full dataset which seems counterproductive.

What is the best way to retrieve large datasets?
Emmanuel recommended me to use a script that uses the Bubble GET API to get returns in JSON objects, which can be converted to text. But I must say I don’t know where to start.

Thanks in advance for your help.

Adrien

[adrien]

Hi everyone,

So I’ve been playing around with this today and made some progress.

I have been slicing my data by 10k entries, creating buttons that download the matching csv’s.

For example, this is what button #2 is fetching:

This is working well for button #1, and button #2. Each of them gives me 10,000 entries in my csv.

However anything above the 20000 mark (out of my 73k entries) that consistently fails. Even fetching 100 entries returns an empty .csv or one an error message in the .csv: “Error exporting data… please try again in a bit”.

Then I tried on a blank page to load a single entry’s creation date in a text element, specifically #25555, which came out blank.

Turns out #19999 works, so does #20000. But not #20001.

Would anyone have an idea around this? Maybe @emmanuel, is this the normal behavior?

Thanks in advance
Adrien

[alex5]

Hi Adrien,
Did you manage to find a solution to this problem? If so, can you share it?

Many thanks,

[adrien]

Hi Alex,
I upgraded to the dedicated plan and had support remove the limitation.