Machine Learning System to Predict Phishing Sites Using HTML, URL & Metadata

Abstract

Since the start of the internet, phishing has been a persistent problem, with attempts becoming more sophisticated and common in recent years. Prevention models are therefore needed to classify sites correctly as either phishing or legitimate at scale. This paper builds and compares three ML models — Random Forest, Logistic Regression and K-Nearest Neighbours — and a CNN model, using URL, metadata and HTML features collected from phishing and legitimate websites. The models were evaluated on both historical and modern datasets, with a key finding being that all models experienced significant accuracy degradation on modern data, with ML models dropping by ~30% and the CNN by ~48%, despite performing best on historical data.

The question I actually cared about

Nearly every phishing-detection paper reports accuracy on a historical benchmark, and the numbers are excellent — 95% and up. What almost none of them report is how the same model does against phishing sites that went live after the training data was collected. Attackers are not a fixed distribution; they adapt. So a model that is 99% accurate on a 2023 corpus may be worth very little in 2026.

That gap is what this project set out to measure. Four research questions:

1. RQ1 — How effectively can machine learning models detect phishing websites using a combination of URL, HTML, and metadata features?
2. RQ2 — Which feature categories contribute most to detection performance?
3. RQ3 — How well do models trained on historical datasets generalise to modern phishing data?
4. RQ4 — To what extent does detection performance degrade when evaluated on newly collected real-world phishing websites?

What I built

A six-stage Python pipeline: data collection, WHOIS enrichment, preprocessing, an ML/DL dataset split, model training, and evaluation.

Two datasets, deliberately separated in time.

• Historical — 45,373 labelled pages (50/50 phishing and legitimate) from an existing raw-URL-and-HTML phishing corpus published on Kaggle.
• Modern — 15,571 pages I collected myself. Legitimate pages were streamed from the CC-MAIN-2025-47 Common Crawl snapshot with warcio (8,000 pages, WARC paths shuffled for a representative sample, anything under 500 characters discarded). Phishing pages came from the live OpenPhish feed, fetched with requests and parsed with BeautifulSoup, looping until I had enough. Final split: 51.4% legitimate, 48.6% phishing.

WHOIS metadata for every domain. tldextract to pull the root domain, python-whois to query it, 0.5 s between requests to stay under rate limits. Creation date, expiry, last update, registrar, name servers, organisation, country, status and contact emails. This was by far the slowest part of the pipeline, and the status field was missing for 27% of entries.

After merging and dropping rows where WHOIS returned nothing, the working dataset was 60,574 rows — 74.3% historical, 25.7% modern.

Then the data forked two ways. The classical models got 38 engineered numerical features (16 URL, 14 HTML, 8 metadata). The CNN got raw text through three separate branches: character-level tokenisation for the URL, word-level for HTML and for the concatenated metadata string.

Every model was trained on an 80/20 split of the historical data only, then evaluated twice: once on the held-out historical test set, and once on the entire modern dataset.

On historical data, everything works

The CNN reached 98.96% accuracy with an AUC of 0.999 and a false positive rate of 0.29%. Random Forest led the classical models at 96.84%, with the lowest false negative rate of the three at 2.87% — the metric I weighted most heavily, since a missed phishing site does more damage than a false alarm. Five-fold cross-validation put every standard deviation below 0.004, and McNemar tests confirmed each pairwise difference was significant (p < 0.0001).

Read only this table and the project looks like a success.

On modern data, everything breaks

The classical models landed between 65% and 68% — bad, but still better than a coin flip. The CNN fell to 51.01%, which is a coin flip. Its AUC dropped to 0.536.

The failure mode is worth spelling out. The CNN’s false negative rate on modern data was an excellent-looking 2.09%, and its recall was 0.9791. Both numbers are meaningless, because its false positive rate was 93.37%: the model had learned to answer “phishing” to almost everything. A detection system that blocks 93% of legitimate traffic cannot be deployed at any threshold.

That the CNN degraded nearly 20 percentage points worse than the worst classical model is, I think, the most useful thing in the dissertation. Character- and word-level tokenisation lets the CNN learn patterns that are specific to the historical corpus — the particular domains, templates and toolkits in use when it was collected. The engineered features are cruder but describe structure rather than surface text: URL path length, HTML link ratios, script and form counts. Those hold their meaning as the attackers rotate their infrastructure.

Which features carry the signal

HTML was the strongest single category, by a wide margin over metadata. Random Forest ablation:

Averaged across all three classical models, HTML features scored 0.973, URL 0.870 and metadata 0.685. html_length was the single most important feature in the Random Forest at 0.175, and 8 of the top 15 came from HTML.

The interesting non-result: adding metadata to HTML did nothing. HTML + Metadata (94.55%) came in marginally below HTML alone (94.61%). All 38 features together still beat every subset, so the three categories do carry complementary signal — but WHOIS metadata, the part of the pipeline that cost the most engineering effort and wall-clock time, contributed the least.

What it means

If you deploy a phishing classifier and do not retrain it, it decays — and the more powerful the architecture, the faster it decays. The model that would win a benchmark comparison here is precisely the model that fails hardest in production a couple of years later. Frequent retraining on freshly collected data is not an optimisation; it is a condition of the system working at all.

Limitations

Worth being straight about these:

• Volume. More data on both sides would have helped. Modern collection was slower than planned because tooling for live phishing capture is limited.
• Label noise. The “legitimate” pages are assumed legitimate — they were sampled randomly from Common Crawl and verifying all ~8,000 by hand was not feasible, even though a non-trivial share of the open web is suspicious.
• Hardware. Everything ran on consumer CPU. That capped the CNN at 10 epochs, which is on the low side.

Future work

Mitigating degradation directly is the obvious next step: periodic retraining, and mixing historical with modern data during training to see whether it blunts the distribution shift. Transformer-based models are also worth testing, since they may capture structure in raw HTML and URL text that a CNN of this shape cannot. And the older-versus-newer HTML itself deserves a closer look — understanding what changed between the two corpora would explain the degradation rather than just measuring it.

Stack

Python 3.9 throughout. scikit-learn for the classical models, PyTorch for the CNN, pandas and NumPy for the data work. Collection ran on requests, BeautifulSoup, warcio, python-whois and tldextract; statsmodels for significance testing; matplotlib and seaborn for the figures.