4 - Scaling & Testing Your n8n Workflow
This tutorial covers scaling an n8n workflow for real-world use: pagination through the Reddit API, throttling to avoid rate limits, and documenting your tests.
An n8n workflow that works perfectly on ten test records can fall apart the moment it meets a real dataset with hundreds or thousands of entries, API call limits, and timeouts. This tutorial, the final one in a series on the Madison framework, walks through what it actually takes to make an n8n workflow production-ready: pagination, rate limit throttling, and the documentation and testing that turn a working prototype into something you can show off as a real deliverable.
The real-world scaling problem
Before touching any nodes, the video lays out the practical constraints a production workflow has to handle: processing hundreds or thousands of records, respecting API call limits that vary by service (some APIs, like Twitter's, restrict calls to as little as once every 15 minutes), and avoiding timeouts or interruptions on workflows that run long. Demonstrating that a workflow accounts for these constraints, rather than just working once in a demo, is part of what separates a student project from professional-level work.
Implementing pagination with the Reddit API
The pagination demo uses a real market-research workflow built to scrape popular Reddit discussions about OpenAI, targeting the OpenAI subreddit for brand-related conversation. The workflow connects Reddit credentials and sets a limit parameter of 25, meaning Reddit sends back 25 posts per response. To get more than that single batch, the pagination mode is set to "update a parameter in each request," using Reddit's own "after" tag: n8n takes the ID Reddit returns and uses it to request the next batch of posts. A safety check stops the loop once Reddit stops returning an "after" ID, signaling the end of the results, and the maximum page count is capped, in this case at two pages of 25 items each, for a total of 50 posts across five pages of ten responses. The same pagination approach scales to 500 or more responses if a workflow needs it.
Throttling and rate limiting
To avoid tripping API rate limits, the workflow adds a Wait node between requests, typically a 5 to 10 second delay, positioned between HTTP request nodes making frequent calls. For workflows that need more sophisticated handling than a fixed delay, a function node can implement exponential backoff instead, increasing the wait time progressively if requests start failing. Both techniques are aimed at the same goal: keeping a workflow within an API's rate limits without needing to babysit it manually.
Professional documentation and testing
The last stretch of the video moves from the technical build to what makes a workflow look and function like a professional deliverable. That means writing a comprehensive README covering setup instructions and required credentials, including an architecture diagram that explains the workflow's components, and documenting known limitations and success rates. On the testing side, it means creating actual test cases that cover both normal and edge cases, documenting the results, and showing concrete optimization metrics, including improvements measured against an earlier version of the same project.
Key takeaways
- A production-ready n8n workflow needs to account for large record counts, API rate limits, and the risk of timeouts on long-running workflows.
- Pagination in n8n can use the "update a parameter in each request" mode with an API's own paging tag, such as Reddit's "after" ID, plus a safety check to stop looping once results run out.
- A Wait node with a 5 to 10 second delay handles basic throttling between requests; a function node with exponential backoff handles more advanced rate limit avoidance.
- Professional documentation includes a full README, an architecture diagram, and a written account of known limitations and success rates.
- Systematic testing means writing actual test cases covering normal and edge cases and documenting the measured results and improvements.
Who this is for
This is the final video in an n8n tutorial series built around the Madison framework, and it's aimed at anyone building an n8n-based project who needs to move past a working prototype into something that holds up under real data volume and can be evaluated on its documentation and testing, not just whether it ran once. It's a natural fit for contributors to the Madison open-source marketing project under Humanitarians AI, and the next video in the series moves on to building a simple Streamlit or Gradio interface for the same kind of workflow.
Full transcript(auto-generated, with timestamps)
[0:01]Hello everyone. Welcome to our final n8n tutorial in the medicine framework series. I'm Kriti and today we are focusing on taking your workflow to the next level by making it scalable and production ready. By the end of this video, you'll know how to handle larger data sets and demonstrate professional level implementation. Now if you need to consider how your workflow would handle real-world scale and these are some of the factors that you need to address. Your workflow should process hundreds or thousands of records. For the APIs that you're going to be using in your projects, sometimes there might be restriction on how many API calls you can make per minute.
[0:41]For example, for these Twitter APIs, there's a restriction that you can only make one API call for every 15 minutes. And if your workflow is extremely lengthy, then the workflow might time out or face interruptions. So, you need to demonstrate these scaling consideration and show that you're thinking for the real-world scale. Most APIs limit how many results you can get in a single request. Let's implement pagination to handle larger data sets and let me just give a demonstration of this with the help of Reddit APIs in n8n. Here I have a market research project for OpenAI where I have built a workflow that automatically scrapes most popular
[1:24]Discussions about OpenAI from Reddit using their official API. So, this URL, it targets OpenAI subreddit to find brand-related discussions. And I have connected my Reddit credentials. In my previous videos, I explained how to do this with the help of Twitter API. In a very similar manner, I have used Reddit APIs here. I have connected my Reddit account. Next, these are the parameters that I'm using basically to help me narrow my focus and this is the important parameter here, limit parameter. This is set to 25, which tells Reddit to send only 25 posts in one single response package. Next, scroll down and you will find the
[2:08]Pagination option. And in the pagination mode drop down, I'm choosing update a parameter in each request option. Reddit basically requires us to flip the page by updating a specific tag, which is after in my case. While some APIs use the response contains next URL, and that requires or that provides a full clickable link to the next page. But for my specific task, I'm going to be using this update a parameter in each request. And in the parameters, I want to add this after. Basically, this is telling N8N to take this ID and use it for the next batch of posts. I have also included a safety check that
[2:56]Tells the workflow to stop looping once Reddit stops sending an after ID, meaning we have reached the end of the search results. And I am currently setting the maximum pages to two, and since I selected the 25 limit or 25 items per page here, I'm fetching around 50 posts in total. Finally, I'm adding also adding a small delay between two requests. That's basically it for my HTTP request node. And here I'm fetching only the required fields like post title, number of upvotes, the body text, etc. And in my last node, I am sorting my responses with the help of the field upvote. So, as a final
[3:44]Response, I have top posts discussion about the brand OpenAI. And as you can see, I I 50 responses. Five pages, 10 responses each in five pages. You can do the same. You can have up to 50 responses or even 500 responses. And when you have that that many responses, make sure to use the pagination technique like just like this. Next, I'm going to talk about rate limiting and throttling. To avoid API rate limits, we need to add throttling to our workflow. And we can achieve that by adding a wait node in between API calls. So, imagine you have a bigger workflow or you have multiple
[4:22]HTTP requests. Let me use the same Reddit API. So, yeah, in this case is when you're making frequent API calls, it's always good to have a wait node. Just add a 5 or 10 seconds wait. This will help you avoid all the API rate limits that these APIs have. And if you want more advanced control, you can use a function node to implement exponential backoff. Moving on to the documentation and testing. For this project, you need to demonstrate professional documentation and testing. And here's how you can do it. Create a comprehensive read me that explains all the setup instructions with all the required credentials. Show your architecture diagram and
[5:10]Explain your workflow components. Also, write about known limitations and success rates. Also, include the systematic testing part. Create some test cases, cover the normal and edge cases, and document your test results. That is really important and show your actual metrics. You can also show your improvements from your previous project or the previous assignment. So, that's mostly it for this video. You now have the skills to scale your Anypoint workflow and prepare it for production use. These improvements will help you excel in your project and create a Madison framework contribution that stands out. In our next video, we will shift the focus on creating a simple user
[5:53]Interface for your medicine agent using either Streamlit or Gradio. This will transform your technical back-end into something non-technical users can actually interact with. Up next, creating a simple UI with Streamlit. Thank you so much for watching this video. I hope this was helpful.
