Trust-Region Based Stochastic Variational Inference for Distributed and Asynchronous Networks

doi:10.1007/s11424-022-2085-5

Journal of Systems Science and Complexity ›› 2022, Vol. 35 ›› Issue (6): 2062-2076.DOI: 10.1007/s11424-022-2085-5

Previous Articles Next Articles

Trust-Region Based Stochastic Variational Inference for Distributed and Asynchronous Networks

FU Weiming¹, QIN Jiahu^1,2, LING Qing³, KANG Yu^1,4, YE Baijia¹

1. Department of Automation, University of Science and Technology of China, Hefei 230027, China;
2. Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei 230088, China;
3. School of Computer Science and Engineering, and Guangdong Province Key Laboratory of Computational Science, Sun Yat-Sen University, Guangzhou 510006, China;
4. Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China

Received:2022-02-07 Revised:2022-05-07 Online:2022-11-25 Published:2022-12-23
Contact: QIN Jiahu,Email:jhqin@ustc.edu.cn
Supported by:
This research was supported in part by the National Natural Science Foundation of China under Grant Nos. 61922076, 61873252, 61725304, and 61973324; in part by Guangdong Basic and Applied Basic Research Foundation under Grant No. 2021B1515020094; and in part by the Guangdong Provincial Key Laboratory of Computational Science under Grant No. 2020B1212060032.

PDF ( 4 ) Cited

FU Weiming, QIN Jiahu, LING Qing, KANG Yu, YE Baijia. Trust-Region Based Stochastic Variational Inference for Distributed and Asynchronous Networks[J]. Journal of Systems Science and Complexity, 2022, 35(6): 2062-2076.

Stochastic variational inference is an efficient Bayesian inference technology for massive datasets, which approximates posteriors by using noisy gradient estimates. Traditional stochastic variational inference can only be performed in a centralized manner, which limits its applications in a wide range of situations where data is possessed by multiple nodes. Therefore, this paper develops a novel trust-region based stochastic variational inference algorithm for a general class of conjugate-exponential models over distributed and asynchronous networks, where the global parameters are diffused over the network by using the Metropolis rule and the local parameters are updated by using the trust-region method. Besides, a simple rule is introduced to balance the transmission frequencies between neighboring nodes such that the proposed distributed algorithm can be performed in an asynchronous manner. The utility of the proposed algorithm is tested by fitting the Bernoulli model and the Gaussian model to different datasets on a synthetic network, and experimental results demonstrate its effectiveness and advantages over existing works.

Key words: Asynchronous networks, Bayesian inference, distributed algorithm, stochastic variational inference, trust-region method

[1] Blei D M, Ng A Y, and Jordan M I, Latent dirichlet allocation, Journal of Machine Learning Research, 2003, 3: 993–1022.
[2] Corduneanu A and Bishop C M, Variational bayesian model selection for mixture distributions, Artificial Intelligence and Statistics (Eds. by Jaakkola T and Richardson T), Morgan Kaufmann, Waltham, 2001.
[3] Letham B, Letham L M, and Rudin C, Bayesian inference of arrival rate and substitution behavior from sales transaction data with stockouts, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, USA, 2016.
[4] Hoffman M D, Blei D M, Wang C, et al., Stochastic variational inference, Journal of Machine Learning Research, 2013, 14(1): 1303–1347.
[5] Hu Y, Niu D, Yang J, et al., FDML: A collaborative machine learning framework for distributed features, Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, USA, 2019.
[6] Liu S, Pan S J, and Ho Q, Distributed multi-task relationship learning, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, Canada, 2017.
[7] Fang H, Liu S, and Wei Y, Distributed clustering algorithm for energy efficiency and load-balance in large-scale multi-agent systems, Journal of Systems Science and Complexity, 2018, 31(1): 234– 243.
[8] Hua J and Li C, Distributed variational Bayesian algorithms over sensor networks, IEEE Transactions on Signal Processing, 2015, 64(3): 783–798.
[9] Anwar H and Zhu Q, ADMM-based networked stochastic variational inference, arXiv preprint, arXiv: 1802.10168, 2018.
[10] Fu W, Qin J, and Zhu Y, Distributed stochastic variational inference based on diffusion method, Acta Automatica Sinica, 2021, 47(1): 92–99.
[11] Mohamad S, Bouchachia A, and Sayed-Mouchaweh M, Asynchronous stochastic variational inference, Proceedings of INNS Big Data and Deep Learning Conference, Sestri Levante, Italy, 2019.
[12] Ye B, Qin J, Fu W, et al., Distributed bayesian inference over sensor networks, IEEE Transactions on Cybernetics, DOI: 10.1109/TCYB.2021.3106660, 2021.
[13] Zhang J, Raman P, Ji S, et al., Extreme stochastic variational inference: Distributed inference for large scale mixture models, International Conference on Artificial Intelligence and Statistics, Naha, Japan, 2019.
[14] Cattivelli F S and Sayed A H, Diffusion LMS strategies for distributed estimation, IEEE Transactions on Signal Processing, 2010, 58(3): 1035–1048.
[15] Boyd S, Parikh N, and Chu E, Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers, Now Publishers Inc, Hanover, USA, 2011.
[16] García-Fernández Á F and Grajal J, Asynchronous particle filter for tracking using non-synchronous sensor networks, Signal Processing, 2011, 91(10): 2304–2313.
[17] Niu F, Recht B, Re C, et al., HOGWILD!: A lock-free approach to parallelizing stochastic gradient descent, Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain, 2011.
[18] Hughes M C and Sudderth E, Memoized online variational inference for Dirichlet process mixture models, Advances in Neural Information Processing Systems, 2013, 26: 1133–1141.
[19] Hoffman M D and Blei D M, Structured stochastic variational inference, Proceedings of the 18th International Conference on Artificial Intelligence and Statistic, San Diego, USA, 2015.
[20] Nedic A, Olshevsky A, and Shi W, Achieving geometric convergence for distributed optimization over time-varying graphs, SIAM Journal on Optimization, 2017, 27(4): 2597–2633.
[21] Nocedal J and Wright S, Numerical Optimization, Springer Science & Business Media, New York, USA, 2006.
[22] Theis L and Hoffman M D, A trust-region method for stochastic variational inference with applications to streaming data, Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015.
[23] Hoffman M and Blei D, Stochastic structured variational inference, Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, San Diego, USA, 2015.
[24] Blei D M, Kucukelbir A, and McAuliffe J D, Variational inference: A review for statisticians, Journal of the American Statistical Association, 2017, 112(518): 859–877.
[25] Betancourt M, The convergence of markov chain monte carlo methods: From the Metropolis method to Hamiltonian Monte Carlo, Annalen der Physik, 2019, 531(3): 1700214.
[26] Durrett R, Probability: Theory and Examples, Cambridge University Press, New York, USA, 2019.
[27] Kingston D B and Beard R W, Discrete-time average-consensus under switching network topologies, Proceedings of the 2006 American Control Conference, Minneapolis, USA, 2006.

[1]	LI Dequan,WANG Xiaofan,YIN Zhixiang. ROBUST CONSENSUS FOR MULTI-AGENT SYSTEMS OVER UNBALANCED DIRECTED NETWORKS [J]. Journal of Systems Science and Complexity, 2014, 27(6): 1121-1137.
[2]	Yiguang HONG;Xiaoli WANG. MULTI-AGENT TRACKING OF A HIGH-DIMENSIONAL ACTIVE LEADERWITH SWITCHING TOPOLOGY [J]. Journal of Systems Science and Complexity, 2009, 22(4): 722-731.

Viewed

Full text

Abstract