Forum Discussion

MalcolmMcDonald's avatar
MalcolmMcDonald
Qrew Cadet
15 days ago

Using Pipelines to update and load a CSV into tables

I am having a tough time figuring out the most efficient way to handle this use case. Currently, my pipeline has been running for 12 hours on about 45000 records — so my current method does not work!...
Pipelines
MarkShnier__You's avatar
MarkShnier__You
Icon for Qrew Legend rankQrew Legend
15 days ago

My suggestion is to bring in the data on a Connected Sync Table.  That will only take a few minutes to load, like 5 or 10 minutes at most.

Then can you use a relationship or something to bring in the values you need for making that Key field as a formula?   Then you can run a saved T2T copy to copy the Sync table to the target table.  A T2T copy run basically instantly, even for 45,000 records.  A Pipeline can run that save T2T copy every hour.