eprintid: 3545 rev_number: 5 eprint_status: archive userid: 69 dir: disk0/00/00/35/45 datestamp: 2016-10-04 08:56:19 lastmod: 2016-10-04 08:56:19 status_changed: 2016-10-04 08:56:19 type: article metadata_visibility: show creators_name: Breschi, Valentina creators_name: Piga, Dario creators_name: Bemporad, Alberto creators_id: creators_id: dario.piga@imtlucca.it creators_id: alberto.bemporad@imtlucca.it title: Piecewise affine regression via recursive multiple least squares and multicategory discrimination ispublished: pub subjects: QA75 divisions: CSA full_text_status: none keywords: PWA regression; System identification; Clustering; Recursive multiple least squares; Multicategory discrimination note: SCOPUS ID: 2-s2.0-84986593950 abstract: In nonlinear regression choosing an adequate model structure is often a challenging problem. While simple models (such as linear functions) may not be able to capture the underlying relationship among the variables, over-parametrized models described by a large set of nonlinear basis functions tend to overfit the training data, leading to poor generalization on unseen data. Piecewise-affine (PWA) models can describe nonlinear and possible discontinuous relationships while maintaining simple local affine regressor-to-output mappings, with extreme flexibility when the polyhedral partitioning of the regressor space is learned from data rather than fixed a priori. In this paper, we propose a novel and numerically very efficient two-stage approach for {PWA} regression based on a combined use of (i) recursive multi-model least-squares techniques for clustering and fitting linear functions to data, and (ii) linear multi-category discrimination, either offline (batch) via a Newton-like algorithm for computing a solution of unconstrained optimization problems with objective functions having a piecewise smooth gradient, or online (recursive) via averaged stochastic gradient descent. date: 2016 date_type: published publication: Automatica volume: 73 publisher: Elsevier pagerange: 155 - 162 id_number: 10.1016/j.automatica.2016.07.016 refereed: TRUE issn: 0005-1098 official_url: http://www.sciencedirect.com/science/article/pii/S0005109816302849 referencetext: S.T. Alexander, A.L. Ghirnikar. A method for recursive least squares filtering based upon an inverse QR decomposition. IEEE Transactions on Signal Processing, 41 (1) (1993), pp. 20–30 L. Bako, K. Boukharouba, E. Duviella, S. Lecoeuche. A recursive identification algorithm for switched linear/affine models Nonlinear Analysis. Hybrid Systems, 5 (2) (2011), pp. 242–253 Bemporad, A., Bernardini, D., & Patrinos, P. (2015). A convex feasibility approach to anytime model predictive control. Technical report, IMT Lucca, February. http://arxiv.org/abs/1502.07974. Bemporad, A., Breschi, V., & Piga, D. (2016). Piecewise Affine Regression via Recursive Multiple Least Squares and Multicategory Discrimination. Technical report, TR-IMT-DYSCO-2016-01. Available online at: http://www.dariopiga.com/TR/TR-IMT-DYSCO-2016-01.pdf. A. Bemporad, A. Garulli, S. Paoletti, A. Vicino. A bounded-error approach to piecewise affine system identification. IEEE Transactions on Automatic Control, 50 (10) (2005), pp. 1567–1580 K.P. Bennett, O.L. Mangasarian. Multicategory discrimination via linear programming. Optimization Methods & Software, 3 (1994), pp. 27–39 L. Bottou. Stochastic gradient descent tricks. Neural Networks: Tricks of the Trade, Springer (2012), pp. 421–436 Breschi, V., Bemporad, A., & Piga, D. (2016). Identification of hybrid and linear parameter varying models via recursive piecewise affine regression and discrimination. In European control conference (pp. 2632–2637), Aalborg, Denmark. K. Crammer, Y. Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research (JMLR), 2 (2002), pp. 265–292 Ferrari-Trecate, G. (2005). Hybrid Identification Toolbox (HIT). G. Ferrari-Trecate, M. Muselli, D. Liberati, M. Morari. A clustering technique for the identification of piecewise affine systems. Automatica, 39 (2) (2003), pp. 205–217 G.M. Fung, O.L. Mangasarian. Multicategory proximal support vector machine classifiers. Machine Learning, 59 (2005), pp. 77–97 Garulli, A., Paoletti, S., & Vicino, A. (2012). A survey on switched and piecewise affine system identification. In 16th IFAC symposium on system identification (pp. 344–355), Brussels, Belgium. A.L. Juloski, S. Weiland, W.~P.M.~H. Heemels. A bayesian approach to identification of hybrid systems. IEEE Transactions on Automatic Control, 50 (10) (2005), pp. 1520–1533 F. Lauer. On the complexity of piecewise affine system identification. Automatica, 62 (2015), pp. 148–153 F. Lauer, Y. Guermeur. Msvmpack: a multi-class support vector machine package. Journal of Machine Learning Research (JMLR), 12 (2011), pp. 2269–2272 http://www.loria.fr/~lauer/MSVMpack Y. Lee, Y. Lin, G. Wahba. Multicategory support vector machines, theory, and application to the classification of microarray data and satellite radiance data. Journal of American Statistical Association, 99 (2004), pp. 67–81 H. Nakada, K. Takaba, T. Katayama. Identification of piecewise affine systems based on statistical clustering technique. Automatica, 41 (5) (2005), pp. 905–913 H. Ohlsson, L. Ljung. Identification of switched linear regression models using sum-of-norms regularization. Automatica, 49 (4) (2013), pp. 1045–1050 S. Paoletti, A.L. Juloski, G. Ferrari-Trecate, R. Vidal. Identification of hybrid systems a tutorial. European journal of control, 13 (2) (2007), pp. 242–260 J. Roll, A. Bemporad, L. Ljung. Identification of piecewise affine systems via mixed-integer programming. Automatica, 40 (1) (2004), pp. 37–50 Ruppert, D. (1988). Efficient estimations from a slowly convergent Robbins–Monro process. Technical report, Cornell University Operations Research and Industrial Engineering. V. Vapnik. Statistical learning theory. Wiley, New York (1998) Xu, W. (2011). Towards optimal one pass large scale learning with averaged stochastic gradient descent. arXiv preprint arXiv:1107.2490. citation: Breschi, Valentina and Piga, Dario and Bemporad, Alberto Piecewise affine regression via recursive multiple least squares and multicategory discrimination. Automatica, 73. 155 - 162. ISSN 0005-1098 (2016)