What Are Neural Network Parameters?

Key takeaways

Neural network parameters are internal variables that a machine learning model configures as it moves through the training process.

• Weights and biases, two types of parameters, influence how nodes within neural networks interact with data.

• Neural networks can automatically optimize their parameters during the training process based on feedback from the model’s errors.

• You can use techniques such as regularization and hyperparameter tuning to fine-tune your model's outputs.

Discover the impact neural network parameters have when training a machine learning model. Then, if you’re interested in learning more about machine learning, the Deep Learning Specialization from DeepLearning.AI can help you learn to build and train neural networks and identify key parameters, while giving you the opportunity to train test sets and gain experience using TensorFlow.

What is a neural network?

A neural network, sometimes called an artificial neural network (ANN), falls under the umbrella of artificial intelligence (AI). It is a type of machine learning that mimics how humans think. These models have layers of nodes hidden between the input and output layers. Each node in each layer connects to nodes in the previous and subsequent layers, allowing the network to pass information through as a web of nodes that interacts with the data at each point. As a neural network grows to consist of more than three layers, it becomes known as a deep learning network. Adding these additional layers allows the AI model to manipulate data in increasingly complex ways. 

Neural network parameters definition

Parameters are the internal variables that a machine learning model configures as it moves through the training process. The configuration of your parameters has an impact on your output. For example, one type of neural network parameter is weight, which is the numerical value that measures the strength of the connections between nodes. The weight influences how the model manipulates or changes the data as it passes through the node. A slight weight change can make an exponential difference in your output, depending on how many nodes and layers are within your neural network. 

While parameters explain the internal configuration of an AI model, hyperparameters are the external variables. These can include the number of layers and nodes within a neural network and can help determine the value of features like learning rate and model architecture. 

Neural networks parameters example

An example of a neural network parameter and its impact on the model is the number of hidden layers, where you can increase the training speed of your model by limiting the number of hidden layers. Alternatively, you can increase the number of hidden layers for a greater learning capacity, but this will also raise the amount of computational power you need to train your model.

Parameters in neural networks: Weights and biases

Neural network parameters are defined by the concepts of weight and bias, two values that determine how the model will interact with data at each node. If a neural network mimics how a human brain works, and the nodes in each hidden layer are analogous to neurons, then weights are comparable to the synapses between neurons that fire to send data through the biological network. Biases are similar to biases in the human brain in that they provide a constant that affects how all nodes interact with data. The bias will ultimately shape every conclusion the AI model makes. 

For example, if you want to decide where to eat, you will likely consider several variables, some very important and others less so, before deciding. You might consider what time of day it is, what kind of meal you want, what type of food you’re craving, how hungry you are, and how much money you have. You might have a food allergy or a dietary preference that determines which restaurants you consider in the first place. If you place high importance on sticking to your budget, that will play a more prominent role in deciding where you eat than a factor you prioritize as less important, such as the type of food you are actually hungry for, for instance.

In a neural network, weights and biases are the mechanisms for similarly weighing decisions. A strong connection between nodes will weigh that data as more important, while weaker connections signal that data is less critical to the output. As you train your neural network, the model will fine-tune these parameters to help you get a more accurate output based on feedback on the model's errors in each attempt. 

Hyperparameters in neural networks

The model automatically sets neural network parameters during its training process by adjusting weights and biases. In contrast, hyperparameters are the variables that you, as a data scientist, configure when creating the model. Optimizing your hyperparameters can help you achieve a more accurate output from your AI model. Finding the optimum settings can take time and may require experimenting while using a process called hyperparameter tuning. 

Hyperparameters include: 

• Number of hidden layers and nodes within layers: The number of hidden layers in a neural network and the number of nodes in each layer influence what kind of problems your network can solve and how complicated the network’s analysis of your data can be. 

• Learning rate: The rate at which a neural network learns refers to how much the model changes its weights during each training iteration. Adjusting its weights for accuracy is a balance that requires finding the optimal learning rate. 

• Convergence rate: When a model reaches optimum weights and can predict outputs accurately, the model is “converged.” The time it takes for the neural network to reach convergence is the convergence rate. 

• Epochs: Once the entire training data set has passed through the neural network, one epoch has been completed. You will determine how many epochs your model will complete (how many times it will work through the training materials) to create the most accurate result. The correct number of epochs helps you avoid common problems such as underfitting and overfitting, when the model becomes too general or too specific to accurately predict outputs on unfamiliar training data.

• Activation function: A neural network’s activation function is a mechanism that tells the model which neurons to activate based on their relevance to the problem it’s solving. The activation function is important because it allows you to build models that work with linear and nonlinear data or data containing dynamic relationships between variables. 

Optimizing and tuning neural network parameters

Optimizing or tuning your neural network patterns to find the correct balance can allow you to create a neural network that accurately predicts the correct output for training data and new information it has never seen before. Two techniques you may use are regularization and hyperparameter tuning. 

• Regularization: Regularization is a technique that helps you create a model that can generalize to new information. You train a model with a data set, but your goal is to create a model that can accurately generalize what it learned in training and apply those principles to new data. You can use regularization techniques like Lasso regression, Ridge regression, and Elastic Net regularization to help you avoid problems like underfitting and overfitting. 

• Hyperparameter tuning: Hyperparameter tuning is the process of determining the optimal hyperparameters under which your neural network returns accurate information. For example, you can influence how your neural network updates its weights with factors such as weight initialization (how you select the values for your starting weights) and learning algorithms, such as backpropagation, which explain how the model will get feedback to adjust its weights. You can tune your hyperparameters manually or use techniques like Bayesian optimization, grid search, and random search to automate hyperparameter tuning. 

Read more: Neural Network Regularization Techniques

Explore our free machine learning resources

Subscribe to our weekly LinkedIn newsletter, Career Chat, for updates on popular tools, skills, and certifications. Then, check out some of our other free resources to keep learning about machine learning:

• Watch on YouTube: Decision Tree Machine Learning Explained Simply 

• Take the quiz: Which Machine Learning Course Should You Take? Find Out in 1 Minute

• Read our Career Chat issue: Machine Learning: What It Is + How It Works

Whether you want to develop a new skill, get comfortable with an in-demand technology, or advance your abilities, keep growing with a Coursera Plus subscription. You’ll get access to over 10,000 flexible courses. 

Build job-ready skills with Coursera Plus

Start 7-day free trial

• 
• 
• 
• 

Start 7-day free trial