Random neural networks defined as single-hidden-layer neural networks whose weights and biases inside the activation function

Random neural networks (RaNNs), specifically those defined as single-hidden-layer neural networks with weights and biases inside the activation function, are an intriguing subset of neural network models that differ from traditional approaches. Here’s an overview of their structure, properties, and applications:

1. Definition of Random Neural Networks

Single-Hidden-Layer Structure: These neural networks consist of an input layer, a hidden layer with a certain number of nodes (neurons), and an output layer.
Random Weights and Biases: The weights and biases used inside the activation function of the hidden layer are initialized randomly and kept fixed after initialization. This contrasts with traditional neural networks where weights are adjusted during training.
Training: Only the weights between the hidden layer and the output layer are learned during the training process.

2. Mathematical Formulation

A single-hidden-layer random neural network with input $\mathbf{x} \in \mathbb{R}^d$ and $N$ hidden neurons can be described as:

$f(\mathbf{x}) = \sum_{j=1}^{N} \beta_j \sigma(\mathbf{w}_j \cdot \mathbf{x} + b_j),$

where:

$\sigma(\cdot)$ is the activation function.
$\mathbf{w}_j \in \mathbb{R}^d$ and $b_j \in \mathbb{R}$ are the weights and biases for the $j$ th neuron, drawn randomly from some distribution (e.g., normal or uniform distribution).
$\beta_j$ are the trainable weights connecting the hidden layer to the output layer.

3. Key Characteristics

Fixed Hidden Layer Weights: Once the weights $\mathbf{w}_j$ and biases $b_j$ are initialized, they remain unchanged during training. This reduces the computational complexity of training compared to fully trainable networks.
Universal Approximation: Despite having fixed weights in the hidden layer, random neural networks can still serve as universal function approximators under certain conditions if the number of hidden neurons $N$ is sufficiently large and the activation function $\sigma(\cdot)$ is chosen appropriately.
Activation Functions: Common choices include ReLU, sigmoid, and tanh. However, the network behavior depends significantly on the chosen activation due to the randomness in weights and biases.

4. Advantages of Random Neural Networks

Reduced Training Complexity: Since only the output weights $\beta_j$ are updated during training, the optimization problem is simpler and faster.
Avoids Overfitting: The randomness and fixed nature of the hidden layer can help in regularizing the network and preventing overfitting.
Flexibility in Initialization: The performance of RaNNs can often be enhanced by choosing suitable distributions for the initialization of $\mathbf{w}_j$ and $b_j$ based on the problem at hand.