Data pipelines in VS Code
Ingestion, pagination, and backfill, written against the API's real behaviour.
A pipeline is where the details decide everything: how the cursor behaves at the end of a feed, what a batch call does with an id that no longer resolves, how to backfill without re-reading what you have. These are answers you get by calling the endpoint, not by reading about it. VS Code can do both in one place, and the workspace config travels with the repo.
- 1A KonbiniAPI accountThe free plan includes 100 credits, which is enough to run this a few times.
- 2VS Code connectedOAuth. GitHub Copilot reads the same configuration.
- 3A real targetA public account, post, or keyword to run it against — the placeholders below are not live values.
Build with it.
Ask VS Code for what you need. It calls the tool, sees the real response, and writes code against the shape that came back rather than one it guessed — each call costs the same credits as the matching REST call.
Good for
- Stand up a scheduled ingest for a set of accounts or subreddits
- Use the batch endpoints to refresh up to 100 stored ids per call
- Write backfill that resumes instead of starting over
- Handle the failure cases — deleted posts, private accounts, empty pages
Build a job that ingests new posts from these subreddits daily and stores them in Postgres. Call the endpoint first so you can see the cursor behaviour.
We store Reddit post ids. Write a refresh job that uses the batch endpoint to re-read them 100 at a time.
Write a backfill script that pages through @[creator]'s videos and can resume from where it stopped.
Our ingest breaks when an account goes private. Call the endpoint for one and handle whatever it returns.
Add rate-limit-aware retry to this ingestion job, based on how the API actually responds.
Swap the bracketed parts for your own project — the platform, the endpoint, and where the result should land.
Run this in VS Code.
One API key works for MCP and the REST API. The free plan includes 100 credits and no credit card.