Neural Networks
​What are neural networks
-
A set of algorithms or layers that recognize patterns and help to output similar outputs to logistic or linear regression
-
Neural networks help to find the relationship between the input and output
-
They can happen in both supervised learning and unsupervised learning. Some examples of neural networks can include facial recognition, voice identifiers, and spam emails
-
The unique part of neural networks is that it goes through numerous activation layers to get to the final layer
Deep Learning
-
A subset of machine learning that is typically utilized in unsupervised learning
-
More than one hidden layer
-
Each layer of nodes train on different features than the previous layer and the more the complex features the node recognizes and each layer takes the distinct features from the previous layer (known as feature hierarchy)
-
Deep learning can take numerous images and help put them together based on similar features. It can do the same thing with spam emails or spam calls.
Illustration of a neural network (Adapted from James Write from ResearchGate)
Note that each type of neural network will have a different structure and that this is just an example
1. Each dot is a node and each node will have a structure such as the one shown below (Adapted from pathmind). Remember this same structure from logistic and kind of linear regression.
Below are vital concepts of Neural Networks
The Softmax Layer
-
Softmax is an activation function, part of the numerous layers, and is known as the outer layer.
-
Softmax layers limit the output to 0 to 1, allowing this output to be a probability. Notice that this is exactly what happens with a logistic regression algorithm. Therefore, using a softmax activation function means that you are using logistic regression. It is typically the final layer.
ReLu
-
ReLU, also known as rectified linear unit, is the most popular type of activation function and is used especially in Convolutional Neural Networks. Mathematically, it is known as:
-
max(0,x)
-
Soft Plus
-
Activation = ln(ex+1)
-
This is the integral of the softmax layer and is useful in backpropogation (described below)
​​
BackPropagation
-
Supervised learning using gradient descent. Back-propagation method calculates the best weights to use based on error in the previous epoch (an epoch is an iteration in a neural network)
-
Uses the derivative of the activation function
-
Called Back Propagation because the weights are updated backwards, from output to input
-
This is used mostly for feed forward neural networks (described later) in supervised learning
Hidden layer
-
Layer between input and output layers where they take a set of weighted inputs and produce an output through the activation function.
Optimization​
1. Optimization is essentially what type of function is computing the optimized weights. It typically uses back-propagation.
2. It needs a loss function to start with and tries to minimize it
3. An example optimization function that we will use in our application section frequently is rms prop which uses a continuously updated average of the squares of the previous gradients to compute a learning rate vector for each input instead of one for all inputs.
Regularization
1. This concept helps to avoid overfitting through stopping the model from becoming too complicated.
2. It further reduces the variance in the model without increasing the bias substantially.
HyperParameters
1. Variables used to control the learning process. They are called HyperParameters because they have to be manually set and tuned before the program is run.
2. They govern the entire network structure.
3. Some examples include the number of nodes in hidden layers or the optimization function.
What type of neural Networks are there and how do they look like:
-
This chart is popularly used to teach neural networks and their shapes. This sheet was taken from Fjodor Van Veen from the Asimov Institute:
​Now there will be examples of typical neural network structures as well as actual programming and separate pages for the two most popular ones.
Perceptron
-
Perceptron has no hidden layer, it just takes an input, dot products this with the weight, and goes to the output layer
-
The most simple neuron (The images of these networks have been adapted from Andrew Tch in Towards Data Science)
Feed Forward Neural Network
- All nodes are connected to eachother
-
There is only one hidden layer
-
There is no cycle and the network simply feeds forward with its inputs and weights
Deep Feed Forward Neural Network
- ​Like a feedforward neural network but multiple hidden layers
-
This is the most popular type of neural network.
-
The foundation of Deep Learning networks
Recurrent Neural Network
- Used in language processing and speech recognition
-
Typically utilizes sigmoid and ReLU as activation functions
-
Measures the distance between a fixed point and an input and any point within that is in the Radial Basis, and is sliced and classified based on that
Convolutional Neural Network
Training and Validation Accuracy
-
Training Accuracy - when you apply the model and backpropagation on training data
-
Validation Accuracy - applying the trained set of weights and test inputs on the test data
-
When the model starts to stagnate it overfits, the model is now memorizing the data and closely fit to a limited set of data points
-
A sample graph is shown below:
