2 - Working with APIs
This tutorial connects an n8n pipeline to real-world APIs, covering how to choose data sources, set up OAuth credentials safely, and build a workflow that pulls, branches, and merges API data for a Madison agent.
An AI agent is only as good as the data feeding it, and pulling real data into a workflow means dealing with API keys, authentication, and rate limits instead of a clean CSV. This second video in the Madison framework tutorial series, presented by Kiti, walks through connecting an n8n pipeline to real-world APIs so a Madison agent has actual data to work with, not placeholder examples.
Picking APIs that match your problem
Before any technical setup, the video makes the case that API selection has to match your specific problem statement, and that a single source usually isn't enough: aim for at least two to three relevant data sources. For a brand analysis project, that means looking at the YouTube Data API, the Twitter API, or web scraping tools. For market research, it means product review APIs, Google Trends, or price comparison sites. For a customer service project, support ticket databases or FAQ scraping fit better. The through-line is that the source has to provide data directly relevant to what you're actually trying to measure, not just whatever API happens to be easy to set up.
Setting up credentials without leaking them
The tutorial demonstrates credential setup using the Twitter/X developer portal at developer.x.com, which comes with a clear security warning attached: never share your API keys or commit them to public repositories. After signing up and landing in a default project, the process runs through the settings panel to user authentication setup, where app permissions need to be set to read and write and the app type set to "web app." The callback URI has to point back to your local n8n instance, and once that's saved, X generates a client ID and secret key. Back in n8n, adding the credential means clicking the plus icon, searching for "X," and selecting OAuth 2 API, then pasting in the client ID and secret and clicking "Connect my account" to authorize the app. Once authorized, the credential appears in n8n ready to use.
Building the API collection workflow
With credentials in place, the tutorial builds a workflow named "API collection" starting with a manual trigger, followed by an HTTP Request node configured to send a GET request. The example endpoint pulls public profile data for the OpenAI Twitter account, returning structured JSON with fields like user ID, name, username, and profile description. For authentication, the node is set to "Predefined Credential Type" using the X OAuth2 credential configured earlier. Running the node confirms the connection works, with output visible in both JSON and table format. For larger datasets, like full sets of tweets or reviews, the tutorial notes you'd typically add a JSON parse node to structure the data and a write binary file node to save it, though that step isn't needed for a simple profile lookup.
Chaining and merging multiple sources
Since a single API rarely covers a full problem statement, the tutorial covers two ways to combine sources in one workflow. The first is running parallel branches directly off the trigger node, adding a second HTTP Request node alongside the first. The second is a sequential setup, where the output of one API call feeds the next, for example pulling a list of products from one API and then looping through those products to pull detailed reviews from a second API. A Merge node then combines the resulting datasets into one before anything gets saved.
Key takeaways
- Choose at least two to three APIs that map directly to your specific problem statement rather than defaulting to whatever's convenient.
- Never share API keys or commit them to public repositories; store credentials inside n8n's credential manager instead.
- A basic API collection workflow starts with a manual trigger and an HTTP Request node set to GET, authenticated with a predefined OAuth2 credential.
- Larger datasets typically need a JSON parse node and a write binary file node to structure and save the output.
- Multiple APIs can be combined with parallel branches off the trigger, a sequential chain where one API's output feeds the next, or a Merge node to combine datasets.
- Aim for a data harvest of at least 50 to 100 records, and document your data sources as you go.
Try it yourself
If you're building toward your own Madison agent, this is a workflow you can rebuild directly: pick two or three APIs relevant to your specific problem, set up OAuth credentials in n8n without ever hardcoding a key, and chain an HTTP Request node into a Merge node to combine your sources. This is one part of the INFO7375 Branding & AI series from Humanitarians AI, with data cleaning and error handling covered in the next video in the series.
Full transcript(auto-generated, with timestamps)
[0:00]Hello everyone. Welcome back to Madison framework tutorial series. I'm Kruti and today we are going to connect N8N to real world data sources through APIs. By the end of this video, you will know how to authenticate with APIs and pull in the data you need for your Madison agent. Before we dive into the technical setup, let's talk about choosing the right APIs for your more Madison project. You need to use at least two to three data sources relevant to your specific problem. For brand analysis project, consider YouTube data API, Twitter API or web scraping tools. For market research, look at product review APIs, Google Trends or price comparison sites.
[0:34]For customer service projects, support ticket databases or FAQ scraping would be appropriate. The key is selecting the sources that provide data directly relevant to your problem statement. Now I will demonstrate how to generate an API key from the developer portal. Most APIs require authentication. Let's go through the process of setting up credentials in N8N. First, you will need to create an account with your chosen API provider and generate API keys from their developer portal. For this example, I will demonstrate with the Twitter API, but the process is similar for most of the services. And the security tip here is never share your API keys or commit them to public
[1:13]Repositories. You can just Google Twitter developer account and you'll land on this. There is extensive documentation available in this link. And the developer portal link is developer.x.com. Sign up create your account. Once you are here, you see a default project that is already created. So go to the settings, click on the settings button and scroll down. You see this user authentication setup. So I just did the authentication, so it's already set up, but initially you will see a setup button here. Click on it and change these permissions. This should be app permissions should be read and write and type of app should be web app.
[1:59]And for app info, the URI callback URI is the callback from your localhost N8N. So, this is the credential. You will see that in the N8N in the next section that I'm going to show it. For now, you can copy paste this. The same thing for OAuth 2 and OAuth 1. And just click on the save button, and then you will see a client ID and secret key generated. Come to this keys and token section, scroll down, and yes, you're going to see the same thing here again. Next step is adding these credentials in your local N8N. Let's do that and open your localhost N8N and click on this
[2:38]Plus icon. For now, we are creating a new credential, so click on credential and search for X. So, this is a tried and tested method, so select OAuth 2 API and click on continue. Copy paste the same client and secret key here. Yes, I have copy pasted my client ID and secret key, and I'll click on this connect my account. It's asking to authorize app, and yes, connection successful. That's it. And you can see a new credentials a new credential is being added here. Now, let's build a workflow that connects to an API and pulls data. Create a new workflow, and let's name it API collection.
[3:32]And start uh creating a new workflow by adding uh the first step, that is manual trigger. And after this, let's add an HTTP request node. Search for HTTP request and configure it to your API. Set the request method to get and enter the endpoint URL. Here we are making a simple get request to the Twitter API to fetch the public profile information about a specific Twitter account, in this case OpenAI. So this endpoint returns structured JSON data about user including their user ID, name, username, profile description, etc. For the authentication, select predefined. The credential type should be XO Auth 2 because that is the one that we have
[4:26]Already configured. And uh yeah, this is already set up now. So yeah, that's it. And let's go back to the canvas. I will test this connection by running this node. Yep, it is executed successfully and you can see the output here in the JSON format and also in the table. So as you are receiving the JSON data back, congrats, you have successfully connected to an API. Now, if you are working with larger data sets like tweet results or reviews, then we would structure this uh the JSON using JSON parse node and saving it using a write binary file. But for this particular small user profile
[5:14]Lookup, I'm not going to do going to be doing that. And as a next step, you need at least two to three data sources. So let's quickly see how to combine multiple APIs in one single workflow. What you can do is you can actually add parallel branches from your trigger node. So let's select another HTTP. For now, I'm just going to leave it as it is. And add another node. You can do this or alternatively, you can use the result from one API call to inform your next API call creating a sequential workflow. Yes, something like this. For example, you might first get a list
[5:56]Of products from one API and then loop through those products to get the detailed reviews from another API. And you can also know use a a merge node. And this is useful for combining data from multiple sources into one data set before saving it. You now know how to connect n8n to APIs and collect real-world data for your medicine project. In the next video, we'll look at cleaning and structuring that data and handling errors that might occur during the collection. Remember, we're focusing on getting relevant data and meaningful analysis, so aim for at least 50 to 100 records, but more is better. And please document your data sources as
[6:38]Required. Up next is data cleaning and error handling in n8n. Thank you for watching this video.
