Metered API Calls in a Batch Process

Thomas_937381

If I have several thousand rows in a data table, with one column containing JSON data, and I want to pass that to another system via a Web API: step in a metered way (X calls / minute), what is the best approach to doing so? Given that Tables: Start Pushbot for each row starts a subprocess for each row sequentially, I envision two potential solutions:

1. First use Tables: Apply formulas to columns to construct a timing delay column based off the rowIndex. This value could be used under the Delay Start option, but I envision two issues with this approach:

• With thousands of rows of data, and the smallest delay increment being minutes, it could be hard to optimize when the first API calls should start
• One might also want subsequent calls to only start based on a positive response following the preceding call

2. Scrap the idea of a batch process, and instead use Web API: Send POST request, with a webhook trigger to start a process row by row, using an iterative counter (+1 for each run). Two thoughts:

• This solves for the issue around the status returned from the API call in a prior call (start conditionally based on return status)
• With minutes being the smallest increment, I still run into an issue if I want to throttle these by X calls / minute

Is there a best approach for this problem? I see a similar one was also asked here.

Dylan_191182

@Thomas_937381 does the API you are calling send a 429 back when a rate limit is exceeded? If so, does it send when to retry the call back as a header (many APIs do this)? If so, you could add something to check for a 429 status code and, if found, reopen the API call with a delay based on when the API says you should retry. If you don't receive a header then you could just delay by a smaller period of time.

This way if the API ever increases its rate limit your process with adopt it automatically.

Thomas_937381

@Dylan_478989 Thanks for the guidance.