Building AI News Monitoring Agent with n8n & FastAPI | Mycroft | Nerd Stuff with Humanitarians AI
The Mycroft News Monitoring Agent processes nearly 2,000 news articles a day through a hybrid n8n and FastAPI architecture, and this walkthrough covers how the pipeline, models, and vector database actually fit together.
Monitoring, analyzing, and extracting insight from nearly 2,000 news articles a day is not something you do by hand, and building an automated pipeline for it surfaces a real architectural question: where should the workflow logic live, and where should the actual machine learning processing happen? The Mycroft News Monitoring Agent, part of the open-source Mycroft framework for AI-powered investment analysis, answers that with a hybrid approach combining n8n for orchestration and FastAPI microservices for the heavy Python-based ML work.
A few terms worth knowing first
An RSS feed, short for really simple syndication, is a web feed format that lets automated systems check for website updates, effectively a machine-readable subscription to new content. FinBERT is a natural language processing model fine-tuned specifically on financial text for sentiment analysis, built on the BERT architecture but trained to understand financial context, so it recognizes what terms like bearish or volatility actually signal for markets. Embeddings are numerical vector representations of text that capture semantic meaning, turning words into something a computer can compare mathematically. RAG, or retrieval augmented generation, combines database retrieval with AI generation so the system fetches real data before generating a response instead of relying on the model to recall facts from memory. A vector database stores and searches those embeddings by similarity rather than exact keyword matches, closer to finding related concepts than finding literal text.
Why split the work between n8n and FastAPI
The processing pipeline follows a consistent shape: ingest news from multiple sources, deduplicate to remove redundant articles, analyze sentiment and generate embeddings, then store everything and make it queryable through a RAG interface. n8n handles the orchestration layer because it gives visual, easily debuggable workflows, while FastAPI handles the parts that genuinely need Python, since it can run the ML models and heavier processing that a visual workflow tool is not built for. n8n's schedule trigger runs the whole ingestion cycle every 20 minutes, which keeps updates close to real time without overloading either the news sources or the databases.
Fetching and parsing RSS at scale
The actual RSS fetching is handled by the FastAPI service rather than n8n directly, called through HTTP request nodes. That choice comes down to Python's feedparser package, which makes parsing the wildly inconsistent formats of real-world RSS feeds trivial. Different XML structures, namespaces, and encoding quirks all get handled automatically. The fetch endpoint aggregates feeds from sources like TechCrunch and VentureBeat and others, and a separate parse endpoint then extracts and normalizes the article content, since raw feeds arrive with inconsistent fields, some carrying full article text and others just short snippets. The parse endpoint returns standardized fields regardless of the source: title, content, source, publish date, and URL.
Sentiment analysis and dual embeddings
Once articles are normalized, sentiment analysis runs through a FastAPI endpoint that processes the article content using the self-hosted FinBERT model. For embeddings, two models run side by side as a deliberate experiment rather than pure redundancy: Google's Gemini API using the text-embedding-004 model, and HuggingFace's FinLang model. The comparison is meant to answer a specific question, which model gives better retrieval results for financial and tech news and which one captures domain-specific terminology more accurately, with those performance comparisons planned for a future video. Both embedding calls happen directly from n8n, since they are already optimized API services that do not need FastAPI's heavier processing layer.
Why Qdrant over PGVector
All processed data, including the embeddings, gets stored in Qdrant, a vector database chosen deliberately over PGVector. Qdrant stores not just the embedding vectors but the full article metadata alongside them, the article text, sentiment scores, timestamps, and source information. The advantage that matters most in production is that n8n's RAG implementation works seamlessly with Qdrant's metadata filtering, so querying articles from a specific date range or with particular sentiment scores becomes straightforward inside the n8n workflow itself.
Tracking real production metrics
Every stage of processing gets measured, and three metrics matter most in practice: ingestion rate, the number of articles pulled into the pipeline; signal percentage, the share of unique articles that actually make it through processing; and loss percentage, the share of articles that error out due to paywalls, anti-scraping measures, or similar obstacles. The system currently faces an 81% deduplication challenge, meaning a large share of incoming articles are redundant and get filtered out before reaching sentiment analysis and embedding.
Self-hosting models without killing performance
Self-hosting a model like FinBERT efficiently comes down to lifecycle management rather than anything exotic. FastAPI's lifespan decorator defines an async context manager that loads the model and tokenizer into memory once, at application startup, as global variables that stay resident for every subsequent request. When the application shuts down, those objects are properly deleted to free memory. The payoff is that incoming requests never trigger a cold start, since the model is already loaded and ready, which avoids both the severe slowdown of reloading a model on every request and the memory leaks that come from improper cleanup. This same lifecycle pattern works for any self-hosted HuggingFace model, not just FinBERT.
Key takeaways
- The Mycroft News Monitoring Agent uses a hybrid architecture: n8n for visual, debuggable workflow orchestration, and FastAPI for the Python-heavy ML processing.
- Python's feedparser package handles the inconsistent XML structures, namespaces, and encoding of real-world RSS feeds automatically.
- FinBERT is BERT fine-tuned specifically on financial text, letting it correctly interpret financial-context terms like bearish or volatility.
- Two embedding models, Gemini's text-embedding-004 and HuggingFace's FinLang, run in parallel as a deliberate comparison rather than for redundancy alone.
- Qdrant was chosen over PGVector specifically because its metadata filtering integrates seamlessly with n8n's RAG implementation.
- FastAPI's lifespan decorator loads a model into memory once at startup, eliminating cold starts and avoiding memory leaks from repeated loading.
- Deduplication currently removes 81% of incoming articles, tracked alongside ingestion rate and loss percentage as the system's core production metrics.
Who this is for
This is for developers building production-grade ML pipelines who need Python's ML ecosystem alongside a visually debuggable orchestration layer, and for anyone curious how a real system handles self-hosted models, dual embeddings, and vector search at scale. The full code and setup documentation are open source as part of the Mycroft framework from Humanitarians AI.
Full transcript(auto-generated, with timestamps)
[0:00]What if you could automatically monitor, analyze, and extract insights from thousands of news articles every day? Today, I'm showing you exactly how I built a production system that processes nearly 2,000 articles daily. I'm Ashish, and this is the Microoft News Monitoring Agent, a sophisticated AI powered news aggregation and analysis system I've been developing. It's part of the broader Microsoft framework, which is an open-source educational experiment in AI powered investment analysis. This system focuses specifically on monitoring AI and technology news to support investment decision-m through intelligent news monitoring and analysis. Everything I'm showing you today is open source, and I'll be sharing what actually works in
[0:50]Production versus what just sounds good in theory. Before we dive in, let me quickly explain some key terms you'll hear throughout this video. RSS feed. This stands for really simple syndication. It's a web feed format that allows automated access to website updates. Think of it as a subscription to a website's new content that machines can read. Finnbbert. This is a pre-trained natural language processing model specifically fine-tuned on financial text for sentiment analysis. It's built on the BERT architecture but understands financial context. So when it sees bearish or volatility, it knows exactly what that means for markets. Embeddings. These are numerical representations of text that capture
[1:43]Semantic meaning. Basically, we're turning words into vectors that computers can understand and compare. Rag or retrieval augmented generation. This is a technique that combines database retrieval with AI generation for more accurate contextual responses. Instead of hallucinating, the AI fetches real data first. Vector database, a specialized database that stores and searches embeddings based on similarity rather than exact matches. It's like finding needles in haystacks, but the needles are concepts, not keywords. Now, let's look at the overall architecture. The system uses a hybrid approach combining N8N for workflow orchestration with fast API microservices for specialized Python processing. Why this combination? N8N gives us visual workflows that are
[2:43]Easy to debug and modify while fast API handles the heavy ML processing that needs Python libraries. It's the best of both worlds. The processing pipeline works like this. First, we ingest news from multiple sources. Then, we dduplicate to remove redundant articles. Next, we analyze sentiment and generate embeddings. Finally, we store everything in our databases and make it available through a rag interface. Let's dive into the N8N workflows. This is where the magic happens. First, we have the schedule trigger. You can see here it's set to run every 20 minutes. This frequency gives us near realtime updates without overwhelming our sources or databases. For new sources integration, N8N orchestrates
[3:37]The workflow, but the actual fetching is handled by our fast API service. Why fast API for fetching? The Python feed parser package makes parsing different RSS formats trivial. RSS feeds are notoriously inconsistent. Different XML structures, namespaces, and encoding issues. Feed parser handles all these edge cases automatically. We use HTTP request nodes to call the fetch endpoint. The fetch endpoint aggregates RSS feeds from TechCrunch, Venturebe, and several other tech publications. The data transformation happens in two stages. First, the fast API parse endpoint, extracts and normalizes article content. Raw feeds come in different formats. Some have full content, others just snippets. Some use different field names for the same data.
[4:35]The parse endpoint handles all this complexity, returning standardized fields, title, content, source, publish date, and URL. Next, sentiment analysis. We call the fin endpoint passing the article content as a parameter. This fast API endpoint processes sentiment analysis using our self-hosted fin model. The beauty of this design is that all the heavy ML processing happens in the fast API service while N8N handles the orchestration and flow control. For embeddings, I'm actually running an experiment. I use two different models. Google's Gemini API with the text embedding 0004 model and hugging faces Finn Lang model. This isn't just for redundancy. I'm evaluating their performance in this specific use case.
[5:31]Which one gives better retrieval results for financial and tech news. Which one has better semantic understanding of domain specific terms? I'll be sharing these performance comparisons in a future video. These are called directly from N8N since they're already optimized API services. For database operations, everything goes into QDR, our vector database. This was a deliberate choice over PG vector. While QDR stores the embeddings as vectors, it also stores all the article metadata, the article text, sentiment scores, timestamps, source information, everything. The key advantage here is that N8N's rag implementation works seamlessly with QDN's metadata filtering. When I need to query articles from a specific date range or with certain sentiment scores,
[6:24]QDR node makes this trivial in N8N. For metrics collection, we track every stage of processing. How many articles came in? How many were duplicates? How many are successfully processed? These metrics are crucial for keeping track of the key performance indicators. One, ingestion rate, number of articles ingested into the pipeline. Two, signal percentage, percentage of unique articles processed by the system. Three, loss percentage, percentage of articles errored during process due to payw wall, anti-scraping, etc. The design decision to use N8N for orchestration with fast API handling the actual processing was deliberate. N8N gives us visual debugging. When processing hundreds of articles, I can see exactly where the
[7:20]Pipeline failed and what data caused the issue. But keeping the fetching, parsing, and ML models in fast API microservices means better performance, proper error handling, and the ability to update or scale these components independently. It's a perfect separation of concerns. An A10 for workflow logic, fast API for computational work. Now, let's look at the code side. I'll quickly show you the fast API endpoints, then dive deep into something really important. How to self-host models efficiently. Here are our main endpoints. Each endpoint is purposefully simple, single responsibility, easy to test and maintain. But here's the secret sauce. How to self-host models like Finnird without killing your memory or having
[8:16]Cold starts. This technique works for any hugging face model. We use fast APIs life cycle management. The key is loading the model once at startup, not on every request. Here's how it works. We define an async context manager using the lifespan decorator. When the fast API app starts up, it loads the model and tokenizer into memory as global variables. These stay resident in memory for all requests. When the app shuts down, we properly delete them to free memory. The model is loaded once during startup. Now when a request comes in, the model is already loaded. No cold starts, no repeated loading, consistent memory usage. This pattern is crucial
[9:06]For performance. Without it, you'd either reload the model on every request, which is incredibly slow, or risk memory leaks from improper cleanup. Everything I've shown you today is open- source. The complete code, documentation, and setup instructions are available on GitHub, link in the description. Check out the detailed documentation if you want to run this yourself. I've included environment setup, API key requirements, and troubleshooting guides. I'd love to hear your thoughts. What would you optimize? Have you solved similar dduplication challenges? Drop a comment below and let's discuss. If you're interested in the broader Microoft framework for AI powered investment analysis, subscribe for updates. I'll be covering the other
[9:54]Components in future videos. Thanks for watching and happy
More from INFO7375 Branding & AI
4:513 - Datacleaning n8n
5:011 Getting Started n8n with HAILogo
14:52Automate Everything with n8n! No Code & Low Code Workflow Automation | Humanitarians AI
15:15Skunks & Fellows: Automate Everything with n8n! | No-Code Workflow Magic
4:543 Datacleaning n8n with HAILogo
9:46