- Use cases
- Social media data for research
Social media data for research
Collect posts and comments for a study, with cursors you can resume.
The job
A study needs repeatable collection. Page through a feed with its cursor, keep the raw response, and record when each page was fetched.
Endpoints
Failed requests and cached repeats cost 0 credits. Send max_age_days to accept a recent cached copy. See credits and caching.
| Endpoint | Returns | Cost |
|---|---|---|
| YouTube search | 1 credit | |
| YouTube video comments | 1 credit | |
| TikTok videos by hashtag | 1 credit | |
| Bluesky profile posts | 1 credit | |
| Telegram channel posts | 1 credit |
Recipe
- Define the query and the date window.
- Page with the cursor until has_more is false.
- Store the raw response and the fetch time.
1import os2import requests34API = "https://api.socialscrape.dev"5HEADERS = {"x-api-key": os.environ["SOCIALSCRAPE_API_KEY"]}678def get(path, **params):9 res = requests.get(API + path, headers=HEADERS, params=params, timeout=60)10 res.raise_for_status()11 return res.json()["data"]1213# YouTube search14page = get("/v1/youtube/search", query="climate policy", type="video", upload_date="month")15# YouTube video comments16comments = get("/v1/youtube/video/comments", url=page["items"][0]["url"], sort="top")1718print(page["has_more"], page["cursor"])19print(len(comments["items"]))1const API = "https://api.socialscrape.dev";2const headers = { "x-api-key": process.env.SOCIALSCRAPE_API_KEY! };34async function get(path: string, params: Record<string, string | number | boolean>) {5 const query = new URLSearchParams(Object.entries(params).map(([k, v]) => [k, String(v)]));6 const res = await fetch(`${API}${path}?${query}`, { headers });7 if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);8 return (await res.json()).data;9}1011// YouTube search12const page = await get("/v1/youtube/search", { query: "climate policy", type: "video", upload_date: "month" });13// YouTube video comments14const comments = await get("/v1/youtube/video/comments", { url: page.items[0].url, sort: "top" });1516console.log(page.has_more, page.cursor);17console.log(comments.items.length);Calls use api.socialscrape.dev and return the shapes in the reference.
Good to know
A cached response carries cached_at, so you can tell fresh collection from a repeat. Check your ethics board and each platform's terms for your use.