1. Principles of a perceptron
A perceptron takes several binary inputs and produces a single binary output. How does it compute the output? Each input is associated with a certain weight wi and the total value of wi xi will be compared against a certain threshold to decide the output. Look at the figure below.
- Is the weather good (yes/no)? Influence scale = 10.
- Is your girlfriend going with you (yes/no)? Influence scale = 3.
- Is the place near a metro station (yes/no)? Influence scale = 4.
Note that the influence scales here represent the weights you personally assign to these factors. For example, you really hate being caught in the rain so much; or you don't really mind if you go alone without your girlfriend and make new friends. You should also infer a threshold to make your decision. That's it, a perceptron has just been used to model how you make a decision. What happens when we combine multiple perceptrons? This structure is called a neural network, as shown in the figure below. In practice, such network is used to take into account more factors and make a better decision.
Let us describe the perceptron more formally. First, the summation of weights and inputs can be rewritten as a dot product of w.x = Σ wi xi . Second, the threshold is in fact a bias, b = −threshold. This bias can be thought as a measure of how easy it is to get the perceptron to give an output '1'. Or to put it in more biological terms, it is a measure of how easy it is to get the perceptron to fire. The formula to get the output thus becomes, y = w . x + b. Thirdly, the binary output of '0' or '1' beyond a certain threshold mimics a type of function called a step function.
2. Simple neural network
A neural network is basically a multi-layer perceptron or MLP as shown in the figure above. The leftmost layer is called the input layer and the neurons inside it are called input neurons. The rightmost layer is called an output layer with just a single output neuron. In between, there exists 2 layers of what is called hidden layers, which take inputs from the input layer and project outputs to the output layer. A neural network in such structure is also popularly known as a feedforward neural network since the outputs from one layer are projected to the the next layer to the right and so on, with no feedback allowed. There are a few other neural network architecture, which will be said at a later time.
The power of a neural network lies in the capability to be trained. Neural networks can be 'trained' to behave in a certain way, or to produce a certain output given some patterns of inputs. Engineers can call this an iterative fine-tuning process. We use neural networks to help us decide something. For example, you want to decide whether a blurred image is a picture of a handwritten digit (number "0" to "9"). Training the neural network here means we let the network to readjust the weights and biases within the network according to the given inputs, in this case, different images.
How can we readjust the weights and biases? We let a small change in a weight or bias to cause only a small change in output Δy (like fine-tuning). In this way, we would get our network to behave more in the manner we want. For example, suppose the network was mistakenly classifying an image as an "8" when it should be a "9". We could figure out how to make a small change in the weights and biases so the network gets a little closer to classifying the image as a "9". This process is then repeated over and over again to make the output more and more accurate to decide "9". The network is said to be learning.
But there is one problem. With the current setup, a small change in an input xi will yield quite a big jump in the binary output, it can completely be flipped (yes/no, '1' or '0'). This is primarily due to the all-or-none nature of the perceptron output. Scientists were not satisfied with such characteristic, so they defined a new type of perceptron where the output follows a sigmoid rather than a step function characteristic. Thus, a neuron with a sigmoid function don't just produce output '1' or '0'. It turns out that with this characteristic, Δy is a linear function of the changes Δw and Δb. This linearity makes it easy to choose small changes in the weights and biases to achieve any desired small change in the output. Formally, output characteristic of an artificial neuron or perceptron can be defined by the activation function, where a sigmoid is one of the examples.
Suppose we have our multi-layer perceptron and this is so-called our model. Mathematically, the method to "train" the model of a neural network is called backpropagation algorithm. This term is coned after Rumelhart et al who proposed an efficient numerical solution of the training problem. Training or learning in terms of backpropagation here means to minimize discrepancy or error (or something bad) by adjusting weights and biases.
To portray the discrepancy or error, we need a cost function, which defines how 'good' our model is at the moment as learning progresses. Now we wish to obtain a set of parameters (weights and biases) such that the discrepancy between the predicted and actual output of the model (neural network) as defined by the cost function is minimized. To minimize this so-called cost function, we can use an optimization algorithm called gradient descent, by iteratively moving in the direction of steepest descent as defined by the negative of the gradient (or slope).
Suppose we have a cost function F that depends on imaginary parameter x (x1 and x2). Let us imagine a valley defined in the dimension of x1 and x2, such as the one shown below, and we want to roll a ball down this valley. Making the ball roll down at different dimension of x1 and x2 is akin to saying that the change in F is negative, ΔF < 0. We can build a relationship such that: ΔF ≈ ∇F ⋅ Δx ; where ∇F is a the gradient vector of F, which carries partial differentiation operators. At the moment, it is enough to think a gradient vector simply as something that relates changes in x to changes in F, just as we would expect something called a gradient to do. Suppose we make the change in x to be Δx = −η∇F, where η > 0. Then mathematically, ΔF ≈ −η ∇F⋅∇F = −η∥∇F∥2 , which means that ΔF is guaranteed to be negative and F will forever decrease not increase. The ball is for sure rolling down the valley!
The whole idea of iteration is as follows. From an arbitrary ball position of in dimension x, we first compute the change in Δx. so that to find a new position of x → x' = x − η∇F. Note that the arrow denotes an update rule, the variable takes up a new value. This update rule can be thought as defining the gradient descent algorithm. It gives us a way of repeatedly changing the ball position in order to find a minimum value of the function. If we keep doing this over and over again, we will keep decreasing F, until theoretically we reach a global minimum. Once this global minimum has been found, it is said that the training algorithm has converged. Note that η is called learning rate where it is usually kept small, to control learning and to prevent the update behavior to be chaotic.
To summarize, the way the gradient descent algorithm works is to repeatedly compute the gradient vector ∇F, and then to move in the opposite direction step by step, so as to "fall down" the valley. Updates of weight and bias parameters occur in an efficient way until the error is minimized.
Suppose we have a cost function F that depends on imaginary parameter x (x1 and x2). Let us imagine a valley defined in the dimension of x1 and x2, such as the one shown below, and we want to roll a ball down this valley. Making the ball roll down at different dimension of x1 and x2 is akin to saying that the change in F is negative, ΔF < 0. We can build a relationship such that: ΔF ≈ ∇F ⋅ Δx ; where ∇F is a the gradient vector of F, which carries partial differentiation operators. At the moment, it is enough to think a gradient vector simply as something that relates changes in x to changes in F, just as we would expect something called a gradient to do. Suppose we make the change in x to be Δx = −η∇F, where η > 0. Then mathematically, ΔF ≈ −η ∇F⋅∇F = −η∥∇F∥2 , which means that ΔF is guaranteed to be negative and F will forever decrease not increase. The ball is for sure rolling down the valley!
The whole idea of iteration is as follows. From an arbitrary ball position of in dimension x, we first compute the change in Δx. so that to find a new position of x → x' = x − η∇F. Note that the arrow denotes an update rule, the variable takes up a new value. This update rule can be thought as defining the gradient descent algorithm. It gives us a way of repeatedly changing the ball position in order to find a minimum value of the function. If we keep doing this over and over again, we will keep decreasing F, until theoretically we reach a global minimum. Once this global minimum has been found, it is said that the training algorithm has converged. Note that η is called learning rate where it is usually kept small, to control learning and to prevent the update behavior to be chaotic.
To summarize, the way the gradient descent algorithm works is to repeatedly compute the gradient vector ∇F, and then to move in the opposite direction step by step, so as to "fall down" the valley. Updates of weight and bias parameters occur in an efficient way until the error is minimized.
Some notes:
1) In more complex networks for a multiclass prediction, softmax is used as the activation function instead of a sigmoid.
2) The following website is informative to understand backpropagation.
No comments:
Post a Comment