• Content Harvesting & Extraction
  • Concept Extraction - Extracting topics that are present as tags from extracted content
  • Summarization - Providing a summarization of content that is extracted from a URL.
  • Entity Extraction - Locate and classify named entities, such as the names of persons, organizations, locations from extracted content.
  • Taxonomy & Classification - Identifying potential categories, and classifications present in extracted content.
  • Relation Extraction - Detection and classification of semantic relationship identified within extracted text from URL.
  • 越墙看国外网加速软件 - Extraction of news and blog article from the content retrieved from a URL.
  • Discussion Extraction - Extraction of structured and threaded conversations such as forums, and comments from extracted content.
  • Date Extraction - Pulling of the dates from an article, blog post, and other content published via a URL.
  • Author Extraction - Pulling of author names from an article, blog post, and content published by a specific individual.
  • Product Extraction - The extraction of specific product related information like title, description, pricing, and images from content extracted from web pages.
  • Related Phrases - Identifying related phrases with content extracted from website URL.
  • Pagination Extraction - Identifying the breadcrumbs, and pagination for URLs where content is being extracted and providing necessary details to navigate and harvest as needed.
  • 越墙看国外网加速软件 - Provide the ability to create, access, and manage dictionaries from extracted content, and for applying to content that is harvested.
• Media Acquisition
  • Image Extraction - The extraction of images from content pulled from URLs.
  • Video Extraction - The extraction of videos from content pulled from URLs.
  • Image Tagging - Tagging of images with specific topics when extracted from URLs.
  • Image Color Extraction - Identifying of colors present in images extracted from URLs.
  • Face Detection - Identifying faces within images and photos extracted from URLs.
  • Barcode Recognition - Identifying barcodes that are present in images extracted from URLs.
  • 越墙看国外网加速软件 - The identification of vehicle license plates identified in photos and images extracted from URLs.
• Document Processing
  • Feed Detection - Identifying the types of different documents and feeds available via URLs.
  • PDF Extraction - The extraction of content from PDF documents harvested from URLs.
  • Word Documents - Extraction of content from Word documents that are harvest via URLs.
• 越墙看国外网加速软件
  • 越墙看国外网加速软件 - Pulling of HTML tables from content extracted via URLs.
  • 越墙看国外网加速软件 - Extraction of structured data from spreadsheets harvested from URLs.
  • CSV Files - Identification and processing of CSV files extracted from URLs.
  • JSON Files - Identification and processing of JSON files extracted from URLs.
  • 越墙看国外网加速软件 - The parsing of common microformats from content extracted from URLs.
  • XML Extraction - The extraction of structured data from XML files harvested from URLs.
• Crawling
  • Seed URLs - Allowing users to provide URLs that seed the web crawling process.
  • 越墙看国外网加速软件 - Allow for the ability to provide URL patterns that will applied during the crawling process
  • Scripting - Allow for scripting as part of the web crawling cycle.
  • Conditional Expressions - Allow for the defining of conditional expressions that can be applied during the web crawling process.
  • 越墙看国外网加速软件 - Allow the application of XPath when processing XML files during the web crawling process.
  • 越墙看国外网加速软件 - Opening up the ability to apply regular expression as part of evaluation during the crawling process.
  • Injection - Allow for the injection of data and content into the crawling of web pages.
  • 越墙看国外网加速软件 - Giving users control over defining timeout limitations to be applied during the crawling cycle.
• Machine Learning
  • Semantic Text Analysis - Relating syntactic structures, from text to the level of the writing as a whole, to their language-independent meanings.
  • Semantic Similarity - Look for the similarities that exist after relating syntactic structures in content to the level of the writing as a whole, to their language-independent meanings.
  • Sentiment Analysis - Identify and categorize opinions expressed in content to interpret it's attitude towards a particular topic, product, etc., is positive, negative, or neutral.
  • Emotion Analysis - Evaluating content for signs of human emotion presence such as sad or happy, complimenting and augmenting sentiment analysis.
• Content Access
  • Content Latest Index - Providing access to the latest content crawled and harvest against a domain or list of domains.
  • Historical Index - Allowing access to historical content that has been retrieved crawling and scraping URLs.
  • Storage - Offering storage for content, data, and media that is scraped from URLs.
  • Search - Providing search tools for access content that is scraped from URLs.
• DNS
  • Domain Lists - Providing lists of domains, and subdomains to seed scraping and harvesting activities.
  • Domain Metadata - Providing details about domains that can be used in the scraping and crawling process.
• Automation & Orchestration
  • API - Provide an API for programmatically managing all aspects of the scraping and crawling process.
  • Webhooks - Provide a webhook architecture for triggering or being notified of specific events when scraping and crawling.
• Analytics
  • Reporting - Providing reporting on crawling and scraping activities.
  • URL Metrics - Provide metrics on specific URLs, offering detail on its activity.
  • Spam Score - Providing a spam score for domains or URLs included as part of crawling and spider process.
  • Rankings - Providng rankings that can be applied to domains, URLs, content, data, media, and other building blocks of the crawling and scraping activities.
• International
  • 越墙看国外网加速软件 - Providing language detection resources for applying against content and data that is extracted from URLs.
  • Geo IP Address - Providing the ability to harvest and scrape content and data from specific IP addresses located in specific geographic regions and countries.
• Utilities
  • 越墙看国外网加速软件 - Allowing for the deployment, configuration, and operation of proxies that get applied as part of scraping and crawling.
  • Cookies - Provide tooling for managing and configuration cookies that are used as part of scraping and crawling.
  • Headers - Giving control over what headers are sent as part of the scraping and crawling process.
  • User Agents - Allow for the customization of the user agent that identifies scraping and crawling tooling.
  • IP Address - Provide the ability to select from different IP address pools, change, and optimize IP addresses used in the scraping and crawling process.
  • 越墙看国外网加速软件 - Providing logging for scraping and crawling with the ability to search, and configure as needed.
  • Batch Calls - Provide the ability to make more than one call at once, tackling mulitple objects at once.
  • Scheduler - Enable the ability control the schedule for the execution of scraping and crawling jobs.
  • 越墙看国外网加速软件 - Allow for low latency scraping and crawling, limiting the impact on targets.
• Integrations
  • Dropbox - Integration of crawling and scraping activities into Dropbox.
  • Amazon S3 - Integration of crawling and scraping activities into Amazon S3.
  • Google Sheets - Integration of crawling and scraping activities into Google Sheets.
  • Plot.ly - Integration of crawling and scraping activities into Plot.ly.
  • Silk - Integration of crawling and scraping activities into Silk.
  • Tableau - Integration of crawling and scraping activities into Tableau.

I am always working to add more building blocks, and regularly step back to see how they all fit together, trying to understand the world of Scraping works. Tracking on the common patterns being used across different areas of the API sector.

This is a research site dedicated to API design. Where we actively publish anything from the blog on API design, as well as curated links, companies, organizations, tools, and a handful of the common building blocks we've identified across the space.

We've been researching, spidering, and gathering information on APIs since 2010 and breaking it down into meaningful stops along the life cycle. Providing a buffet of information you can consider as you are working to develop or consume APIs.

If you have any questions, feel free to submit a GitHub Issue, Tweet at us, or email at [email protected]