A Two-Year Web Scraping Pipeline

Sun, 15 Dec 2024 00:00:00 +0000

Summary

This project implements an end-to-end pipeline to collect, store, and visualize publicly available video-ranking data from a major online video platform. The aim was to explore geographic differences in popularity, track trends over different time windows (daily, weekly, monthly, yearly, all-time), and build a robust system that runs on a schedule with monitoring and backups.

Note: the project only collected information that is publicly visible on the site, and used rate limiting and scheduling safeguards to avoid putting load on it. See the Ethics, legal & safety section below for the precautions taken. The pipeline was retired in August 2025, after changes to the site cut the amount of usable data — and the insight that could be drawn from it — significantly.

Git on Niklas Lange

A Two-Year Web Scraping Pipeline

Summary