Scraping Starters
Slot provides powerful CLI commands for scraping starter templates from various sources. These tools help you build and maintain your own collection of templates.
GitHub Scraping
The GitHub scraper finds repositories that match specific criteria and downloads their contents for analysis.
Basic Usage
slot scrape github <token> [options]
Options
-s, --stars <number>
Minimum stars for GitHub repositories
-t, --topics <items>
GitHub topics to search for (comma-separated)
-L, --languages <items>
GitHub languages to search for (comma-separated)
-k, --keywords <items>
NPM keywords to search for (comma-separated)
-b, --batch <size>
Batch size for parallel processing
-d, --delay <ms>
Delay between requests in milliseconds
-r, --rph <limit>
Requests per hour limit
-l, --limit <number>
Maximum number of results to fetch
Examples
# Scrape React starters from GitHub
slot scrape github $GITHUB_TOKEN -s 100 -t react-starter,react-boilerplate -L typescript
# Scrape Next.js templates with rate limiting
slot scrape github $GITHUB_TOKEN -t nextjs,starter -r 100 -d 1000
# Scrape NPM packages with keywords
slot scrape npm -k react-starter,react-boilerplate -l 50
# Scrape NPM packages with rate limiting
slot scrape npm -k nextjs-starter -b 10 -r 100
NPM Scraping
The NPM scraper finds starter template packages based on keywords and download counts.
Basic Usage
slot scrape npm [options]
Options
-k, --keywords <items>
NPM keywords to search for (comma-separated)
-b, --batch <size>
Batch size for parallel processing
-d, --delay <ms>
Delay between requests
-r, --rph <limit>
Requests per hour limit
-l, --limit <number>
Maximum results to fetch
Examples
# Search for React starter kits
slot scrape npm -k react-starter,react-boilerplate
# Find Next.js templates with limits
slot scrape npm -k nextjs-starter -l 50 -r 100
# Search for Vue.js starters in batches
slot scrape npm -k vue-starter,vue-template -b 10
Rate Limiting
Be mindful of API rate limits when scraping. Use the --rph
and --delay
options to avoid hitting rate limits. For GitHub, authenticate with a token that has appropriate permissions.
Next Steps
After scraping templates, you can:
- Analyze the templates to extract features and quality metrics
- Search through the templates to find ones that match your needs