Randomness in neural networks:an overview

Simone Scardapane 1 and Dianhui Wang2*

Abstract Neural networks, as powerful tools for data mining and knowledge engineering,can learn from data to build feature-based classifiers and nonlinear predictivemodels. Training neural networks involves the optimization of nonconvex objectivefunctions, and usually, the learning process is costly and infeasible for applicationsassociated with data streams. A possible, albeit counterintuitive,alternative is to randomly assign a subset of the networks’ weights so that theresulting optimization task can be formulated as a linear least-squares problem.This methodology can be applied to both feedforward and recurrent networks,and similar techniques can be used to approximate kernel functions. Manyexperimental results indicate that such randomized models can reach sound performancecompared to fully adaptable ones, with a number of favorable benefits,including (1) simplicity of implementation, (2) faster learning with less interventionfrom human beings, and (3) possibility of leveraging overall linear regressionand classification algorithms (e.g., ?1 norm minimization for obtainingsparse formulations). This class of neural networks attractive and valuable to thedata mining community, particularly for handling large scale data mining inreal-time. However, the literature in the field is extremely vast and fragmented,with many results being reintroduced multiple times under different names. Thisoverview aims to provide a self-contained, uniform introduction to the differentways in which randomization can be applied to the design of neural networksand kernel functions. A clear exposition of the basic framework underlying allthese approaches helps to clarify innovative lines of research, open problems,and most importantly, foster the exchanges of well-known results throughoutdifferent communities.

Stochastic configuration networks ensemble with heterogeneous features for large-scale data analytics

Dianhui Wang ?, Caihao Cui

Abstract This paper presents a fast decorrelated neuro-ensemble with heterogeneous features for large-scale data analytics, where stochastic configuration networks (SCNs) are employed as base learner models and the well-known negative correlation learning (NCL) strategy is adopted to evaluate the output weights. By feeding a large number of samples into the SCN base models, we obtain a huge sized linear equation system which is difficult to be solved by means of computing a pseudo-inverse used in the least squares method. Based on the group of heterogeneous features, the block Jacobi and Gauss–Seidel methods are employed to iteratively evaluate the output weights, and a convergence analysis is given with a demonstration on the uniqueness of these iterative solutions. Experiments with comparisons on two large-scale datasets are carried out, and the system robustness with respect to the regularizing factor used in NCL is given. Results indicate that the proposed ensemble learning techniques have good potential for resolving large-scale data modelling problems.