Skip to content

A Node.js webapp to scrape the stars of specified GitHub users

Notifications You must be signed in to change notification settings

umaar/export-github-stars

Repository files navigation

GitHub Star Export

Actions Status

Export all your GitHub stars.

preview.png

This webapp queries the GitHub API to query for the stars of a GitHub user, and stores them in an SQLite database. 10,000 stars will occupy ~6mb worth of data.

This repo uses persistent queues, and supports resumable downloading of stars. You can fetch stars of many users.

preview-jobs.png

To run

  1. Get a GitHub token from https://github.com/settings/tokens

  2. Create a .env file:

# Contents of .env

GITHUB_TOKEN=your token here

# Any name for your new database
DATABASE_NAME=output
  1. Run these commands:
npm i
make migrate-db-dev
make start

TODO:

  • Send emails using an email service about database size changes
  • Automatically update user stars every ~ few hours
  • Insert stars/repos using transactions in bulk
  • Implement ajax for pagination
  • Cache users and their star counts
  • Use http://nvd3.org/ to visualise a users stars over time
  • For many stars, should I use something like this https://github.com/tbranyen/hyperlist
  • Configure pooling correctly: knex/knex#2820 (comment) e.g.
"pool": {
  "min": 2,
  "max": 6,
  "createTimeoutMillis": 3000,
  "acquireTimeoutMillis": 30000,
  "idleTimeoutMillis": 30000,
  "reapIntervalMillis": 1000,
  "createRetryIntervalMillis": 100,
  "propagateCreateError": false // <- default is true, set to false
},

About

A Node.js webapp to scrape the stars of specified GitHub users

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published