Forum Discussion

QuickbaseAdmin1's avatar
QuickbaseAdmin1
Qrew Trainee
2 months ago

Copy Records Step - Skipping All Records

Hello,

I am new to using the Copy Records step in Pipeline.

I have a pipeline running with no errors but also creating no data. It's activity log reports that it skips all of the source table's records, even when there is no query. The Merge Field is set to the Record ID#, which was the only option to choose.

The pipeline structure is simple:

  • Step A: Search a User table
  • Step B: For each User found, copy the records from a Master table to a Child table. 

Are there more steps to implement to use Copy records? 

  • I believe that "HN" trying to use the pipeline QuickBase step called Copy Records.  As far as I can tell when Quickbase provided that step to us they kind of missed the mark on that one. The reason is they don't give us any opportunity to attach those copied records to any kind of Parent. So I really don't see the point of that step. 

    Don is suggesting that you search the template records and then in a For Each loop create your child records.  If  there's not that many child records then that's all there is to it.

    If you had a situation where there was a 50 or 100+ records then it would be kinder to the app and your users to create a bulk upsert and then add the rows into the bulk upsert and then commit the upsert.   That way the the processing takes place in the pipeline and doesn't slow down your app for your human users.  

     

  • I am not sure why you would need anything around a Merge Key.    If I am understanding this correctly.

    Step A Search User Table

    Loop One

    Step B Search Master Table for the data

    Loop Two

    Step C Create Record with the data from Step A and Step B

    End Loop Two

    Loop One Continues

  • I believe that "HN" trying to use the pipeline QuickBase step called Copy Records.  As far as I can tell when Quickbase provided that step to us they kind of missed the mark on that one. The reason is they don't give us any opportunity to attach those copied records to any kind of Parent. So I really don't see the point of that step. 

    Don is suggesting that you search the template records and then in a For Each loop create your child records.  If  there's not that many child records then that's all there is to it.

    If you had a situation where there was a 50 or 100+ records then it would be kinder to the app and your users to create a bulk upsert and then add the rows into the bulk upsert and then commit the upsert.   That way the the processing takes place in the pipeline and doesn't slow down your app for your human users.  

     

  • Is Step B a Create Record or an Update Record?

    If you are trying to Update records with data from Step A, you need another Search step to find the record to update.   Then Step C will update it.

  • Hi Don, It's a Create Record step. I'd like to create new records each month based off template records in the source table.

    I was surprised that the Merge key is required because I am never wanting to update.

  • Thank you Mark and Don. Mark, you've validated my thoughts around the Copy Records step. I guess I don't understand the value of the Copy Records it seems to be a lesser bulk upsert step. I will stick with the tried and true bulk upsert steps.