CrewAI
Use SocialScrape from CrewAI agents with the crewai-socialscrape package, eight tools for profiles, posts, comments, transcripts and search.
crewai-socialscrape is the official CrewAI integration. It gives an agent tools for the jobs agents do most (a profile, its posts, one post, the comments, a video transcript, a search) and a tool that runs any other endpoint by id. It builds on the Python SDK, so retries, timeouts and long transcription jobs are handled for you.
Install
pip install -U crewai-socialscrape
export SOCIALSCRAPE_API_KEY="YOUR_KEY"
Python 3.11 or newer. Create a key in the dashboard; a new account starts with 100 free credits. The source is on GitHub and the package is on PyPI.
Give an agent the tools
from crewai import Agent, Crew, Task
from crewai_socialscrape import get_socialscrape_tools
analyst = Agent(
role="Social media analyst",
goal="Answer questions about creators and what their audiences say",
backstory="You look at the data before you answer and you say where each number came from.",
tools=get_socialscrape_tools(), # reads SOCIALSCRAPE_API_KEY
)
task = Task(
description=(
"Compare NASA's TikTok and YouTube channels: followers, the latest posts, "
"and what people say in the comments of the newest YouTube video."
),
expected_output="A short comparison with the numbers and three points from the comments.",
agent=analyst,
)
result = Crew(agents=[analyst], tasks=[task]).kickoff()
print(result.raw)
get_socialscrape_tools() returns eight tools that share one client:
| Tool | What it does |
|---|---|
socialscrape_get_profile | A profile, channel or artist: name, bio, links, verification and follower counts. |
socialscrape_list_posts | The latest posts or videos of a profile, channel or page, one page per call. |
socialscrape_get_post | One post, video, tweet, pin or clip from its URL. |
socialscrape_list_comments | The comments on a post or video, one page per call. |
socialscrape_get_transcript | The transcript of a video from its URL. See AI transcription for what happens when a video has no captions. |
socialscrape_search | Keyword search on one platform, one page per call. |
socialscrape_list_endpoints | Every endpoint id with its parameters and price. It makes no request, so it is free. |
socialscrape_call_endpoint | Runs any endpoint in the API reference by its id, with its parameters by name. |
The platform is an argument, for example platform="tiktok", handle="nasa". For a post, its comments or a transcript the platform is read from the link. Tools that list things return a cursor; pass it back for the next page, as described in pagination.
The tool classes are SocialScrapeProfileTool, SocialScrapePostsTool, SocialScrapePostTool, SocialScrapeCommentsTool, SocialScrapeTranscriptTool, SocialScrapeSearchTool, SocialScrapeEndpointsTool and SocialScrapeCallEndpointTool. Give an agent only the ones it needs:
from crewai_socialscrape import SocialScrapeCommentsTool, SocialScrapeTranscriptTool
researcher = Agent(
role="Video researcher",
goal="Summarise what a video says and how viewers react",
backstory="You read the transcript and the comments before you write.",
tools=[SocialScrapeTranscriptTool(), SocialScrapeCommentsTool()],
)
Use one tool directly
from crewai_socialscrape import SocialScrapeProfileTool, SocialScrapeTranscriptTool SocialScrapeProfileTool().run(platform="tiktok", handle="nasa") SocialScrapeTranscriptTool().run(url="https://www.youtube.com/watch?v=dQw4w9WgXcQ")
Every tool answers with a JSON string that holds data (the object, or a list page with items, cursor and has_more), credits_charged and credits_remaining. The objects are the same models the REST API returns.
To reach an endpoint that has no tool of its own, look it up and call it by id:
from crewai_socialscrape import SocialScrapeCallEndpointTool, SocialScrapeEndpointsTool SocialScrapeEndpointsTool().run(search="ad library") SocialScrapeCallEndpointTool().run( endpoint="facebook.ad_library.search", params={"query": "running shoes"}, )
Parameters are checked against the endpoint before the request is made, so a misspelt name is answered at once and costs nothing.
Errors
A request the API refuses is free, as described in errors. A tool returns the error as its answer, with the error code first (Error: not_found: ...), so an agent can read it and try something else. A missing API key raises at the first call.
Options
Each tool, and get_socialscrape_tools(), takes these arguments:
| Argument | What it does |
|---|---|
api_key | The API key. Left out, it is read from SOCIALSCRAPE_API_KEY. |
max_age_days | Accept a cached result up to this many days old. A cached result costs 0 credits; see caching. |
trim | On by default: asks for the smaller response, without null fields and the extra object, which saves context. |
client, async_client | A SocialScrape or AsyncSocialScrape client from the SDK that you configured yourself. |
base_url | API base URL override. |
The tools work with run and arun.
Credits
A tool call is one API request and is charged like one: see credits. socialscrape_list_endpoints makes no request and is free. Since an agent decides how many calls to make, CrewAI's max_usage_count on a tool is a simple way to cap a run.
Other agent frameworks
For LangChain and LangGraph there is a toolkit and a document loader. For Claude, ChatGPT, Cursor and other MCP clients, use the MCP server instead.