Workato: How to process an unlimited amount of records
If you’ve been following the feature development that Workato has been pushing out lately then you probably know about Callable recipes. Synchronous callable recipes, at the time of this writing, have the unique ability to be included in the “transaction” that started it off. You might call 5 different recipes from one single recipe and that would all take up just 1 transaction in Workato (this is from a billing standpoint). They have recently introduced the ability to do A-Synchronous callable recipes. This has multiple benefits but also some cons that we don’t get from the synchronous calls:
- You can send different sets of data to different recipes without hitting the 33 minute processing time limit.
- You can send the same set of data to the same callable recipe and stack your jobs up, without hitting the 33-minute limit
- You can now process an unlimited amount of records based on the method detailed below (as long as you don’t run out of transactions in Workato!)
- If you have an enterprise plan with Workato you can increase the number of jobs that a recipe processes at one time!
- Each asynchronous call results in an additional transaction, reducing the number you have for the rest of the month.
- You don’t get data back from the callable recipe you run.
Processing records in batch
Before they introduced asynchronous calls, you were really restricted to the 33-minute processing limit. This meant that you had to roughly calculate how long it would take to process each of your records and then limit the number of loops you went through. This then meant you’d need to find another way to “trigger” the original recipe to keep it running over and over. If you needed to process 100,000 records for the day it could potentially take hours upon hours to complete.
Now with asynchronous calls, we can stack up against the number of jobs on the callable recipes preventing our original recipe from hitting the 33-minute time limit. Let’s take a quick look at the basics of how this works.
In the above screenshot, we have a single loop. Let me break down the steps:
- This step creates our array of 500 items.
- In the Configure Repeat action we’re saying we’re going to loop through each item from step 1, so 500 loops.
- Here we’re going to “do something”, this can be creating your Journal Entries in Intacct or updating your Contacts, etc.
This particular setup would get you to 500 touched records. Now if you through in a search for records, you can bump that up to 10,000:
In this loop, I’ve added a search for records which is always limited to 200 results. For each of those, I loop through (step 4 in the screenshot above) and update them. You could use the bulk upsert command here and remove step 4 altogether.
Using Asynchronous callable recipes
Now with an asynchronous call to another recipe, I can move on to processing the next record (or set of records) without waiting for a response that would chew into our 33-minute time limit. In step 5, from above, we send the data from step 4 to the callable recipe and “stack” up the jobs on that callable recipe. This allows the callable recipe to process the jobs as fast as it can and we won’t error out with any time restrictions.
Nested loops with final call to start over
In the screenshot above, I have 3 loops, each running 500 times. This means we’re processing 200 records (the original search/update) times 500 (the first loop) times 500 (the second loop) times 500 (the third and final loop) to work through 25 billion records. Now I’m going to take a wild guess than say you won’t be able to have that many loops and not hit some limits but what you can do is process 100,000 records and then trigger a call to another recipe that calls this recipe again. These final steps (8 & 9 from the screenshot) are the key to looping through endless until all of your records have been processed. One caveat is that the callable recipe you’re working on, can not call itself. This means you’ll need an intermediate callable recipe to call this recipe again which is rather simple to set up.