Metered API Calls in a Batch Process

Thomas_937381
Thomas_937381 Posts: 196
edited May 2020 in Questions

If I have several thousand rows in a data table, with one column containing JSON data, and I want to pass that to another system via a Web API: step in a metered way (X calls / minute), what is the best approach to doing so? Given that Tables: Start Pushbot for each row starts a subprocess for each row sequentially, I envision two potential solutions:

  1. First use Tables: Apply formulas to columns to construct a timing delay column based off the rowIndex. This value could be used under the Delay Start option, but I envision two issues with this approach:
  • With thousands of rows of data, and the smallest delay increment being minutes, it could be hard to optimize when the first API calls should start
  • One might also want subsequent calls to only start based on a positive response following the preceding call
  1. Scrap the idea of a batch process, and instead use Web API: Send POST request, with a webhook trigger to start a process row by row, using an iterative counter (+1 for each run). Two thoughts:
  • This solves for the issue around the status returned from the API call in a prior call (start conditionally based on return status)
  • With minutes being the smallest increment, I still run into an issue if I want to throttle these by X calls / minute

Is there a best approach for this problem? I see a similar one was also asked here.

Answers

  • Dylan_478989
    Dylan_478989 Posts: 92 admin

    @Thomas_937381 does the API you are calling send a 429 back when a rate limit is exceeded? If so, does it send when to retry the call back as a header (many APIs do this)? If so, you could add something to check for a 429 status code and, if found, reopen the API call with a delay based on when the API says you should retry. If you don't receive a header then you could just delay by a smaller period of time.

    This way if the API ever increases its rate limit your process with adopt it automatically.

  • Thomas_937381
    Thomas_937381 Posts: 196

    @Dylan_478989 Thanks for the guidance.