Representative Learning for Distributed Support Vector Machines
Joseph A. Casey, Boxiang Wang, Keren Li
Abstract
Distributed support vector machine (SVM) learning is challenging under heterogeneous local data and limited communication. We develop a Representative Learning framework based on Lifted Empirical Risk Minimization, replacing each covariate by a local Gaussian distribution. For linear SVMs, the resulting expected hinge loss has an exact closed form, is smooth for nondegenerate lifting covariances, and converges to the ordinary hinge loss as the covariance vanishes. The lifted formulation enables clouded representatives that encode both location and covariance. We study a model-independent Mean Representative (MR) and a Score-Matching Representative (SMR) that matches the class-wise lifted gradient at its construction parameter. We show that SMR covariance scaling is stable with respect to within-group heterogeneity and that the gradient discrepancy between full data and frozen representatives is controlled by the maximum within-group diameter. Under strong convexity and smoothness, this yields an inexact-gradient convergence guarantee. Experiments under heterogeneous class allocation show that lifted representatives remain competitive with centralized solutions, while SMR more closely recovers the centralized lifted-SVM optimizer. We further extend the framework to nonlinear SVMs through a shared kernel dictionary, with experiments on Ionosphere and Australian Credit Approval showing analogous behavior. These results provide a distributional representative framework for communication-efficient distributed SVM learning under heterogeneity.