What Is Probabilistic Programming?

Key takeaways

Probabilistic programming utilizes software to make inferences and determine the likelihood of different outcomes within a given set of variables. Here are some important things to know:

• Probabilistic programming applies programming language concepts to make statistical modeling easier and more powerful.

• It uses a wide selection of probabilistic programming languages and frameworks, including Stan, TensorFlow, and Pyro.

• You can use probabilistic programming to enhance the effectiveness of machine learning and AI.

Learn more about what probabilistic programming is and what it can do for you. Afterward, start advancing your career in machine learning and AI by enrolling in the Deep Learning Specialization from DeepLearning.AI. You’ll have the opportunity to build and train deep neural networks, identify key architecture parameters, implement vectorized neural networks, and apply deep learning to applications in as little as three months.

What is probabilistic programming?

It helps to think of probabilistic modeling as a tool you can use for statistical modeling, a method for creating probabilistic models. These models incorporate randomness, which you can then use to make probabilistic inferences and predictions about the future. Essentially, its purpose is to determine the probability distribution, or the likelihood of different outcomes, based on how frequently parameters take on specific values.

Experts already build statistical models manually, but the process is complex and hard to automate. However, using probabilistic programming, you can help save considerable time and effort within this procedure. Alternatively, you can think of a probabilistic programming language as a regular programming language with randomness, which you can achieve using methods such as the Metropolis-Hastings algorithm, which helps with sampling and probabilistic inference within probabilistic programming frameworks. 

How probabilistic programming works

Probabilistic programming works by letting developers express complex distributions through code, which you can complete using probabilistic programming languages like webPPL or Hakuru. For example, you might use poisson(5) in Hakaru to represent a Poisson distribution, which estimates the number of times something might happen during a specific time, with a rate of five. 

The key idea in probabilistic programming is that it helps make statistical modeling feel like programming. It incorporates probability distributions, which also allows complex development of algorithms like AI.

Why probabilistic programming matters

Managing uncertain information can pose a serious challenge, particularly in applications like machine learning. By automating distribution, query, and inference algorithm design, probabilistic programming simplifies machine learning and makes building machine learning programs more accessible, while supporting more efficient and accurate models that require less data. As such, it helps expand statistical modeling to create powerful new tools. Probabilistic programming is also a critical mechanism that allows AI models to reason in the presence of uncertainty. 

Is probabilistic programming useful​?

Probabilistic programming is useful in a variety of applications, including machine learning, AI model development, and more:

- Machine learning: Probabilistic programming streamlines machine learning using automated distribution, inference algorithm design, and query to enable deeper exploration.

- AI: Probabilistic programming is a core AI concept that enables complex modeling and boosts AI capabilities.

- Statistical modeling: At its core, probabilistic programming is a tool for statistical modeling, meaning it can help automate tasks that previously required experts to construct by hand.

Foundations of probabilistic programming and core concepts

Probabilistic programming’s roots come from Bayesian programming, which dynamically updates a statistical hypothesis using Bayes' Theorem. One of the core concepts of probabilistic programming is the use of latent random variables, as using latent, or unobserved, variables makes reasoning about a problem easier. Another key concept of probabilistic programming is statistical inference, where you can infer latent variables from observations and apply statistical inference to a variety of applications, including machine learning.

Read more: Latent Variable Modeling Explained: Methods, Examples, and Applications

Probabilistic programming languages and applications

You can use numerous probabilistic programming languages and frameworks. Each has its own strengths. For instance, PyMC may enable you to explore a greater number of models without implementing inference methods on your own. Stan, on the other hand, facilitates truncation, which reports data measurements that fall between lower and upper bounds, which may help improve sample quality. 

Explore a selection of probabilistic programming languages in more detail:

• Stan: A flexible probabilistic programming language that defines different parts of a Bayesian model in blocks like parameters, model, transformed parameters, and generated quantities, each matching a step in the modeling workflow.

• Slicstan: Offers a compositional interface to Stan, removing block syntax and enabling reusable functions, resulting in shorter programs. 

• TensorFlow modeling language: Wraps pure functional TensorFlow computations as generative functions, enabling scalable use of deep neural networks in models and proposal distributions.

• Pyro: A Python-based, PyTorch-supported language that enables flexible deep probabilistic modeling by combining modern deep learning with Bayesian methods. 

• PyMC: Another Python-based tool that offers easy use and features algorithms like advanced Markov chain Monte Carlo (MCMC) for flexibility that makes it useful for many Bayesian modeling and machine learning problems.

What is an example of a probabilistic model? 

For an example of a probabilistic model, imagine you want to measure the average time it takes for a bus to arrive at a stop using the Poisson distribution and previous arrival times.

 

The function for this example would look like this: 

x∼Poisson(λ) 

Where x is the arrival time, and λ is the quantity to make the prediction. In probabilistic modeling, the predicted quantity is a parameter, and the data is a noisy function of it called the likelihood. You might use predicted quantity for things like predictive maintenance in manufacturing to foresee and prevent possible failures or malfunctions.  You can write the likelihood as a density function showing how likely x is given a λ:

f(λ) = xe-xx!

How to implement probabilistic programming​

To start, you will need to choose a probabilistic programming language. The language you should use depends on your specific purpose for probabilistic programming. Also, consider whether you’ll need to change the base language during the course of your project. For example, if you choose PyMC, you will work exclusively in Python, whereas with Stan, you can switch between the command-line interface and other languages, including Python and R.

You might also choose tools like TensorFlow Probability to streamline the process. TensorFlow Probability offers easy access to optimizers, variational inference (VI) and MCMC, tools like a joint distribution abstraction models, and access to various probability distributions. 

In this instance, the basic process would look something like this: 

1. Install TensorFlow Probability using pip install upgrade tensorflow-probability.

2. Begin building the model, first using distributions and bijectors, which can help you calculate summary statistics, probability, quantities, and transform distributions.

3. Add joint distributions and probabilistic layers.

4. Develop the third layer, probabilistic inference using commands like tfp.mcmc for Markov chain Monte Carlo, tfp.vi for variational inference, and tfp.optimizer for Stochastic optimization methods, which enable you to fine-tune the model.

Discover free resources for machine learning engineers

Discover our free online resources to help further your career in machine learning, such as exploring your potential career path in Machine Learning Career Paths: Explore Roles & Specializations. You can also check out more free resources below:

• Watch on YouTube: Career Spotlight: Machine Learning Engineer

• Prepare for an interview: 10 Machine Learning Interview Questions (+ Tips to Answer Them)

• Learn how much machine learning engineers are making: Machine Learning Engineer Salary: How Much Can You Make?

Accelerate your career growth with a Coursera Plus subscription. When you enroll in either the monthly or annual option, you’ll get access to over 10,000 courses—simply check the course page to make sure your selection is available.

Build job-ready skills with Coursera Plus

Start 7-day free trial

• 
• 
• 
• 

Start 7-day free trial