Data pipelines in Claude Code
Ingestion, pagination, and backfill, written against the API's real behaviour.
A pipeline is where the details decide everything: how the cursor behaves at the end of a feed, what a batch call does with an id that no longer resolves, how to backfill without re-reading what you have. These are answers you get by calling the endpoint, not by reading about it. Claude Code can do both in one place, and it can read the files it just wrote and run them.
- 1A KonbiniAPI accountThe free plan includes 100 credits, which is enough to run this a few times.
- 2Claude Code connectedOAuth.
- 3A real targetA public account, post, or keyword to run it against — the placeholders below are not live values.
Build with it.
Ask Claude Code for what you need. It calls the tool, sees the real response, and writes code against the shape that came back rather than one it guessed — each call costs the same credits as the matching REST call.
Good for
- Stand up a scheduled ingest for a set of accounts or subreddits
- Use the batch endpoints to refresh up to 100 stored ids per call
- Write backfill that resumes instead of starting over
- Handle the failure cases — deleted posts, private accounts, empty pages
Build a job that ingests new posts from these subreddits daily and stores them in Postgres. Call the endpoint first so you can see the cursor behaviour.
We store Reddit post ids. Write a refresh job that uses the batch endpoint to re-read them 100 at a time.
Write a backfill script that pages through @[creator]'s videos and can resume from where it stopped.
Our ingest breaks when an account goes private. Call the endpoint for one and handle whatever it returns.
Add rate-limit-aware retry to this ingestion job, based on how the API actually responds.
Swap the bracketed parts for your own project — the platform, the endpoint, and where the result should land.
Other Claude Code workflows
Run this in Claude Code.
One API key works for MCP and the REST API. The free plan includes 100 credits and no credit card.