Bhavana's Week 08: Don't Wait to be Found
Time-sensitive content like job listings cannot wait for Google's normal crawl schedule, so this week's build knocks on Google's door directly instead.
A job listing that takes days to show up in Google search results might as well not exist by the time it's indexed. Job postings, event pages, and price changes all share the same problem: they have a shelf life, and the normal way search engines discover content, a crawler working through a site on its own schedule, is built for pages that stay useful indefinitely, not ones that need to be seen today.
Knocking instead of waiting
Bhavana Bhariday, a Humanitarians AI Fellow working on a job search platform, frames the difference clearly. A crawler is like a mail carrier on a regular route: it reaches your street eventually, which works fine for a page meant to stay relevant. But a job listing that gets filled shouldn't linger in search results, and a brand new one shouldn't sit for days waiting to be discovered. The fix is the Google Indexing API, which knocks on Google's door directly instead of waiting for the crawler to wander by. This week's build turned that idea from a design on paper into three working pieces: an identity, a script, and a memory.
Identity: proving the script is allowed to speak for the site
An unattended backend script can't log into Google the way a person would, so it needs a different kind of credential. The system uses a service account, essentially a robot identity, to prove to Google that it's authorized to act on behalf of the site. The script loads a service account key, requests the indexing scope, and gets an authorized session with no human involved in any individual run. That's what makes it possible to submit URLs automatically after every scrape, rather than requiring someone to manually flag new listings.
The indexing script: telling Google what changed
With identity in place, the second piece is the script that does the actual telling. After each scrape, it queries the database for jobs created in the last six hours that are still active, converts each one into its public URL, and batches those URLs to Google's indexing API. The practical effect is that a newly scraped listing can be known to Google within the hour, instead of waiting on whatever schedule the crawler happens to be running.
Memory: why a tracking table isn't optional
The third piece is the one that makes the whole system sustainable rather than a one-time trick. A tracking table records every URL already sent to the indexing API, along with what happened and when: the job ID, the URL, whether it was an update or a removal, the API response, and the timestamp. This table lives on the backend only, locked to the service role, and it's the difference between a system that submits once and one that keeps submitting correctly forever.
The reason this matters comes down to quota. Google caps the Indexing API at 200 URLs a day by default. Without a memory of what's already been sent, every run would resubmit every job the system has ever seen, burning through the entire daily quota on repeats before it ever reached the genuinely new listings the system exists to surface. The tracking table is what makes the quota survivable, because it lets the script tell the difference between a job that needs submitting and one it already handled.
Putting it together
The recap ties the three pieces into a single flow: the Indexing API turns waiting into knocking, the service account gives an unattended script the identity to speak for the site, the script pushes new and removed jobs to Google within hours instead of days, and the tracking table remembers what's already been sent so the 200-a-day quota goes toward new listings rather than duplicates. The system is built and wired to run automatically after every scrape, with the live service account credential as the last step before it goes fully live.
Key takeaways
- Time-sensitive content like job listings needs direct indexing, not the standard crawl schedule.
- A service account gives an unattended backend script the identity to submit URLs to Google without a human in the loop.
- The indexing script queries for recently created, still-active listings and batches their URLs to Google's Indexing API.
- A tracking table recording every submitted URL is what prevents the system from burning its daily quota on duplicate submissions.
- Google's Indexing API defaults to a 200-URL daily cap, making that tracking memory a requirement, not an optimization.
Who this is for
This build log is aimed at developers working on SEO or backend systems for time-sensitive content, from job boards to event listings, and it comes out of the Humanitarians AI Fellows program, where Fellows document their weekly technical progress in public.
Chapters
Full transcript(auto-generated, with timestamps)
Knocking on Google's Door Directly
[0:00]Hi, I'm Bavana, a Humanitarian's AI Fellow. Week eight of my job seeker report, the job search app where I handle SEO, making Google find and rank our listings. Last week the fast indexing system was a design on paper. This week it's real, three pieces that tell Google about new jobs directly instead of waiting. Executive summary in one image. For time sensitive jobs, you don't wait for
Identity: Setting up a Service Account
[0:25]Google to wander by, you knock on the door. Normally the crawler is a mail carrier on a usual route. It reaches your street eventually, which is fine for a page that stays useful. But a filled job shouldn't linger, and a new one shouldn't wait days. The indexing API is knocking directly, and this week I built the three pieces that make it work. An identity, a script, and a memory. First piece, identity. An unattended back-end script
The Indexing Script: Submitting URLs
[0:50]Can't log in like a person. So it uses a service account, essentially a robot identity, to prove to Google it's allowed to speak for the site. It loads a service account key, requests the indexing scope, and gets an authorized session. No human in the loop every run. Second piece, the script that does the telling. After each scrape, it asks the database for jobs created in the last six hours
Memory: Why a Tracking Table is Mandatory
[1:16]That are still active, turns each into its public URL, and batches them to Google's indexing API. A new listing scraped, Google knows within the hour, not whenever the crawler wanders back. Third piece, memory. A tracking table, CO index status, records every URL already sent with what happened and when. The job ID, the URL, whether it was an update or a removal, the response, the time stamp. Back-end only, locked to the service role. This table is the
Recap: Managing the Quota
[1:47]Difference between a system that submits once and one that submits forever. And here's why that table isn't optional. Google caps the indexing API at 200 URLs a day by default. Without a memory of what's already been sent, every single run would resubmit every job it's ever seen, and burn the entire daily quota on repeats, before it ever reached the genuinely new listings the system exists to surface. The memory is what makes the quota survivable. The recap. The Indexing API turns waiting into knocking. A service account gives an unattended script the identity to speak for the site. One script pushes new and removed jobs to Google in hours. And
A tracking table remembers what's been sent, so a 200 a day quota goes to new listings, not repeats. It's built and wired to run after every scrape. The service account credential is the last step before it goes live. The rule for time-sensitive content. Don't wait to be found. Your turn. If you've got content with a shelf life, listings, events, prices, ask Claude how you tell a search engine the moment something changes, instead of waiting to be crawled. And ask the question that actually makes it work. What do you record so you never send the same update twice? Run this and design the memory first. Don't wait to be found.
