SC Data is a Python package and related tools making use of
sparecores-crawler to pull and
standardize data on cloud compute resources. This repository actually
runs the crawler every 5 minutes to update spot prices, and every hour
to update all cloud resources in an internal SCD table and public
SQLite snapshot as well.
Stable version from PyPI:
pip install sparecores-data
Most recent version from GitHub:
pip install "sparecores-data @ git+https://git@github.com/SpareCores/sc-data.git"
For easy access to the most recent version of the SQLite database
file, import the db object of the sc_data Python package, which
runs an updater thread in the background to keep the SQLite file
up-to-date:
from sc_data import db
print(db.path)By default, the SQLite file will be updated every 600 seconds, which
can be overwritten by the sc_data_db_refresh_seconds builtins
attribute or the SC_DATA_DB_REFRESH_SECONDS environment variable.
Similarly, you van set the following environment variables:
SC_DATA_NO_UPDATESC_DATA_DB_PATHSC_DATA_DB_URLSC_DATA_DB_REFRESH_SECONDSSC_DATA_HTTP_TIMEOUT