Haider Toha › Projects

global sentiment engine

by Haider Toha · 10 June 2025

a real-time nlp pipeline that ingests multilingual news and social content from 150+ countries, performs gpu-accelerated transformer-based sentiment analysis, aggregates results by country and time and visualises global emotional state on an interactive webgl 3d globe.

github • live site

---

system architecture

---

data collection

the pipeline ingests from 300+ rss feeds and multiple social platforms. all collectors implement a common interface ensuring consistent data normalization with source type, source name, title, url (used for deduplication), content, country code and publication timestamp.

rss feed coverage:

• wire services (4): reuters, ap, afp, euronews
• north america (17): npr, pbs, nbc, cbs, abc, usa today, washington post, cbc, ctv, toronto star, national post, milenio, el universal mx...
• europe western (45): bbc, guardian, sky, telegraph, france 24, le monde, deutsche welle, der spiegel, ansa, la repubblica, el pais, nos, rtbf, swi orf, rte...
• europe nordic (18): svt, aftonbladet, nrk, vg, dr, tv2 dk, yle, helsinki times, iceland monitor...
• europe eastern (30): tvn24, gazeta wyborcza, hungary today, romania insider, ekathimerini, kyiv independent, moscow times, n1 serbia/croatia/bosnia...
• asia (65): nhk, japan times, yonhap, cgtn, scmp, cna, straits times, jakarta post, bangkok post, times of india, ndtv, dawn...
• middle east & africa (55): al jazeera, arab news, haaretz, jerusalem post, gulf news, daily sabah, news24 sa, punch nigeria...
• americas (40): folha, clarin, el mercurio, el tiempo, jamaica observer...
• oceania (15): abc australia, nz herald, fiji times, rnz pacific...

country detection pipeline:

---

sentiment analysis pipeline

text preprocessing:

• remove urls (https?://\S+)
• strip html tags
• remove hashtags/mentions
• normalise whitespace
• preserve unicode (multilingual support)
• truncate to 512 tokens (model max context)

supported models:

the inference pipeline runs through huggingface with cuda acceleration when available. model outputs are normalised to a -1.0 to +1.0 scale regardless of the underlying label format. "LABEL_0" with 0.85 confidence becomes score: -0.85, label: "negative". "5 stars" with 0.91 confidence becomes score: +0.91, label: "positive".

batch processing:

single inference is inefficient. the system batches articles for gpu parallelization. articles are grouped into batches of 32, filtered for minimum length (10 chars), padded to max length in batch and processed in a single forward pass. results are mapped back to original articles.

---

aggregation and weighting

not all sources are equal. credibility weights:

weighted sentiment calculation:

$\text{Weighted Average} = \frac{\sum (\text{sentiment\_score} \times \text{source\_weight})}{\sum \text{source\_weight}}$

hourly aggregates track country code, hour, simple mean, weighted mean, article count and foreign keys to the top positive/negative articles for that period.

---

globe visualization

the frontend uses react three fiber for declarative threejs. the globe component includes an earth sphere with texture, country markers positioned by sentiment and orbitcontrols for user interaction.

lat/long to 3d position:

$\phi = (90 - \text{lat}) \times (\pi/180) \theta = (\text{lon} + 180) \times (\pi/180) x = -r \sin(\phi) \cos(\theta), \quad y = r \cos(\phi), \quad z = r \sin(\phi) \sin(\theta)$

sentiment to color:

each country marker uses meshstandardmaterial with emissive properties that intensify on hover (0.3 → 0.6). data fetching uses swr with 60-second refresh intervals, deduplication and automatic revalidation on focus.

---

performance characteristics

database growth averages ~2kb per article, 1000-3000 articles per hour, 50-150mb daily storage with 30-day retention.

---

error handling and fallbacks

collection failures:

• per-source isolation: one source failure doesn't stop the job
• automatic retry: 3 attempts with exponential backoff
• timeout: 30 seconds per source

inference failures:

• text too short (<10 chars): return none, skip article
• model exception: log error, return none for article
• batch failure: fall back to individual processing
• oom: reduce batch size dynamically

graceful degradation:

• postgresql credentials missing → falls back to sqlite
• reddit/mastodon credentials missing → skip those collectors
• gpu unavailable → falls back to cpu inference
• model load fails → api returns degraded status

---

limitations

• model accuracy: ~85% on standard benchmarks; sarcasm and cultural context remain challenging
• language coverage: while xlm-roberta supports 100+ languages, accuracy varies; best for high-resource languages
• real-time latency: 1-hour collection interval; breaking news delayed up to 60 minutes
• geographic granularity: country-level only; no city or region subdivision
• source availability: rss feeds may block, rate-limit or discontinue without notice
• attribution ambiguity: articles covering multiple countries assigned to most-mentioned
• bias propagation: model inherits biases from training data

---

stack: python 3.10+, fastapi, uvicorn, sqlalchemy 2.0, apscheduler, pytorch 2.0, transformers 4.37+, httpx | next.js 14, react three fiber, three.js, tailwindcss 3.4, swr, typescript 5