Multi-Platform Healthcare Sentiment Analysis Using Kappa-Validated Lexicon Labelling and SMOTE-Enhanced Naive Bayes

Abstract

This study develops an automated sentiment analysis system for classifying reviews of the RSI Sunan Kudus mobile application, collected from Google Play Store, Google Maps, and YouTube (N = 1,428). Sentiment labels were automatically assigned using a domain-adapted Indonesian lexicon (87 positive / 100 negative terms), with label reliability validated against a human-annotated gold-standard subset (n = 100, two annotators; κ = 0.900, almost perfect agreement), yielding 85.71% classifier accuracy against adjudicated labels. After excluding neutral reviews, a binary Multinomial Naive Bayes classifier was evaluated via 5-fold stratified cross-validation, achieving 91.01 ± 1.13% accuracy and 89.20 ± 1.48% Macro F1-score. Class imbalance (ratio = 2.39:1) was addressed using SMOTE within each training fold in feature-vector space. An ablation study across six model-vectorizer combinations identified LinearSVC with Bag-of-Words as best-performing (Macro F1 = 96.03 ± 1.34%); a preprocessing ablation showed the six-stage normalisation pipeline did not meaningfully improve Macro F1 over simpler variants. Cross-platform transfer experiments revealed substantial generalisation gaps (e.g., Macro F1 dropping from 81.1% to 26.8% for a Google-Maps-trained model applied to YouTube), underscoring the value of multi-platform data collection. Findings indicate three primary service pain points: OTP/login failures, long waiting times, and application connectivity issues.