Asynchronous web crawler managing browser sessions, crawling strategies, and crawl result aggregation.

Defines extraction strategies: LLM, cosine similarity, topic modeling, CSS selectors for crawling content.

Interfaces with VirusTotal API to fetch threat intelligence and reputation data for files, domains, IPs, and URLs.

Wraps Playwright's async element locator to perform browser interactions like clicks and text input with navigation tracking and outcome validation for web scraping.

Wraps Playwright page functionality for browser automation and web scraping with methods for page interaction, data extraction, and element handling.

Models URL data for crawl results, supporting markdown generation via Pydantic.

Generates a JavaScript injection script to spoof Web Worker fingerprinting surfaces like navigator and WebGL, ensuring consistent evasion in worker contexts.

Finds elements using CSS or XPath selectors with automatic type detection, optional timeout, and configurable error handling.

Scans local filesystem for phone numbers via google-libphonenumber, extracting and formatting numbers for OSINT workflows.

-*- coding: utf-8 -*- ------------------------------------------------------------------------------- Name: sfp_ripe Purpose: Queries Internet registryes like RIPE (incl. ARIN) to

Shodan API integration for network recon, exposing host, port, and vulnerability intel.

Processes scraping operations in NestJS, orchestrating web data extraction and crawl job handling.

Web capability — `web.fetch` + `web.search`, stdlib-only, allowlist-gated. The first capability that lets an agent read the open web, so egress is default-deny twice over: 1. **Con

Executes arbitrary JavaScript in the browser page context, optionally on a specific element, with extensive control over evaluation parameters. It enables dynamic page manipulation

-*- coding: utf-8 -*- ------------------------------------------------------------------------------- Name: sfp_spider Purpose: SpiderFoot plug-in for spidering sites and returning

Transfers URL and prompt data for checking selector cache status in element selection operations.

Locates web elements by combining common HTML attributes (id, class, name, tag, text) into a single flexible method, supporting timeouts and multiple matches.

Parses a User-Agent string into structured browser metadata, including platform, vendor, and version, to enable consistent browser emulation.

Generates IPv4/IPv6 address ranges from CIDR blocks via netaddr, returning expanded IP lists for network targeting.

Applies a browser fingerprint profile to a tab and its workers via CDP overrides and JS injection, ensuring consistent identity across pages, iframes, and workers.

Generates realistic, randomized HTTP headers (User-Agent, Accept-Language, sec-ch-ua, etc.) based on browser/OS/device constraints. Supports HTTP/1 and HTTP/2 header sets.

browserPool module manages Puppeteer browser pool for scraping, handling lifecycle and reuse.

Sherlock Module This module contains the main logic to search for usernames at social networks.

Assembles a complete JavaScript injection script that overrides browser fingerprint surfaces CDP cannot control, using a shared native-toString hook and per-section builders.

108. Browser as required base abstraction to ensure multiple implementations.

Command-line interface entry point for username search across social platforms.

Exception raised when a required field cannot be extracted and no default value is provided, ensuring missing data is explicitly handled.

Manages asynchronous HTTP requests and responses for web scanning tasks.

Cleans HTML by removing comments, scripts, styles, and other non-content tags while trimming attribute values to 100 characters for web scraping processing.

Exception raised when a DOM element cannot be located, signaling missing or stale elements during automation.

Generates a unique CSS selector for a given DOM element, prioritizing semantic tags, IDs, meaningful classes, and falling back to nth-child paths.

Sherlock Sites Information Module This module supports storing information about websites. This is the raw data that will be used to search for usernames.

Core fingerprint generation engine producing browser fingerprints from various input signals.

Splits a CSS selector at iframe boundaries to enable cross-iframe element traversal, returning segment tuples for stepwise lookup.

Comprehensive fault injection patterns for OS command injection, directory traversal, XSS, SQL injection, and more. Categorized by attack type and platform.

Creates a CDP command to override the browser's User-Agent string, ensuring consistent HTTP headers and JavaScript properties, with optional Client Hints support.

Parses Set-Cookie headers from HTTP responses into structured HAR cookie objects, extracting name, value, and attributes like HttpOnly, Secure, Path, and Domain for network recordi

Validates active Chrome tab matches expected URL/pattern with extension verification.

Chrome automation getter retrieving website info (history, cookies, profile) for evaluators.

Generates realistic HTTP headers based on browser profiles and statistical models.

Abstract base class providing common download logic for all platform extractors.

Fetches repo URL from GitHub API; injects button via DOM manipulation.

-*- coding: utf-8 -*- ------------------------------------------------------------------------------- Name: sfp_networksdb Purpose: SpiderFoot plug-in to search NetworksDB.io API f

-*- coding: utf-8 -*- ------------------------------------------------------------------------------- Name: sfp_onyphe Purpose: SpiderFoot plug-in to check if the IP is included on

Extracts indicators of compromise (URLs, IPs, domains, hashes, emails, file paths, suspicious tools) from free-text investigation context using regex, with deduplication and false-

skills/threat_analyst/reputation_intel.py Fetches reputation intelligence for IPs and domains from public APIs: - AbuseIPDB: IP abuse history - Talos: IP intelligence - AlienVault

Pure functions to extract emails, phones and social links from raw HTML/text. No browser/IO dependencies, fully unit-testable against HTML fixtures.

HTTP fetch server retrieving web content with markdown conversion.

Waits for a selector to appear in the DOM with timeout handling.

-*- coding: utf-8 -*- ------------------------------------------------------------------------------- Name: sfp_accounts Purpose: Identify the existence of a given acount on variou