Explore
For Enterprise
Join for Free
Log In

Arts and Humanities
Business
Computer Science
Data Science
Information Technology
Health
Math and Logic
Personal Development
Physical Science and Engineering
Social Sciences
Language Learning
Degrees
Certificates

Explore all of Coursera

Browse
Search
For Enterprise
Log In
Join for Free

Notebook for lesson 3

Loading...

Natural Language Processing in TensorFlow

deeplearning.ai

4.6 (540 ratings) | 9.7K Students Enrolled

Course 3 of 4 in the TensorFlow in Practice Specialization

If you are a software developer who wants to build scalable AI-powered algorithms, you need to understand how to use the tools to build them. This Specialization will teach you best practices for using TensorFlow, a popular open-source framework for machine learning. In Course 3 of the deeplearning.ai TensorFlow Specialization, you will build natural language processing systems using TensorFlow. You will learn to process text, including tokenizing and representing sentences as vectors, so that they can be input to a neural network. You’ll also learn to apply RNNs, GRUs, and LSTMs in TensorFlow. Finally, you’ll get to train an LSTM on existing text to create original poetry! The Machine Learning course and Deep Learning Specialization from Andrew Ng teach the most important and foundational principles of Machine Learning and Deep Learning. This new deeplearning.ai TensorFlow Specialization teaches you how to use TensorFlow to implement those principles so that you can start building and applying scalable models to real-world problems. To develop a deeper understanding of how neural networks work, we recommend that you take the Deep Learning Specialization.

Skills You'll Learn

Natural Language Processing, Tokenization, Machine Learning, Tensorflow, RNNs

Reviews

4.6 (540 ratings)

5 stars
385 ratings
4 stars
113 ratings
3 stars
28 ratings
2 stars
9 ratings
1 star
5 ratings

GI

Jun 22, 2019

Amazing course by Laurence Moroney. But only after finishing Sequence Models by Andrew NG, I was able to understand the concepts taught here.

SM

Aug 15, 2019

Course explains some of key concept very well, like sequence model to predict next word using existing corpus.

From the lesson

Word Embeddings

Last week you saw how to use the Tokenizer to prepare your text to be used by a neural network by converting words into numeric tokens, and sequencing sentences from these tokens. This week you'll learn about Embeddings, where these tokens are mapped as vectors in a high dimension space. With Embeddings and labelled examples, these vectors can then be tuned so that words with similar meaning will have a similar direction in the vector space. This will begin the process of training a neural network to udnerstand sentiment in text -- and you'll begin by looking at movie reviews, training a neural network on texts that are labelled 'positive' or 'negative' and determining which words in a sentence drive those meanings.

A conversation with Andrew Ng2:08

Introduction2:14

The IMBD dataset1:03

Looking into the details4:01

How can we use vectors?2:30

More into the details2:47

Notebook for lesson 110:09

Remember the sarcasm dataset?1:07

Building a classifier for the sarcasm dataset1:55

Let’s talk about the loss function1:26

Pre-tokenized datasets0:43

Diving into the code (part 1)1:25

Diving into the code (part 2)2:30

Notebook for lesson 35:10

Taught By

Laurence Moroney
AI Advocate

Transcript

So here's the notebook for lesson three. So a few things to check just before you begin is go to the runtime, click change runtime type, make sure you're running Python three and GPU. Then run this cell. If you run this cell, if it does not say 2.0.0- alpha, then come into this cell, uncomment this line, and then execute it. But as I have 2.0.0-alpha already I'm not going to do that. Once you've done it, you'll see a button to reset, click the page would reset, and then you can run this cell and just ensure that you have 2.0.0-alpha before continuing, otherwise, try to re-install or reset the runtime with reset all runtimes. The next thing to do now is to start loading your dataset. So we using TFDS for that and this is the code to install TFDS if it fails. If I run it now, we'll see that it worked, I didn't get any problems. Again, if you see issues with TFDS, then you can just pip install it with this code on common train if we go. I can now create my training data and test data sets, I can now create my tokenizer, I can print out my tokenizer sub words, and we can see there's lots of them including punctuation stuff. For take a sample string like TensorFlow, from basics to mastery and tokenize that, I'll say my tokens are this. If I take a look at how those tokens map 6307 is ten, 2327 is sor, etc. So I now have a tokenized string based on subword tokenization and I built that using the subwords8k vocabulary. So now let's create our model. So our model I'm just going to do another embedding with 64 dimensions. I'm going to pass the tokenizer vocabulary size into the embedding so we can learn from that. If I train I'm going to train for 10 epochs, epochs are pretty slow. I'm going to compile the model with binary-crossentropy and add an optimizer, and I'm going collect accuracy metrics, and then my model that fits pretty simple. You see I haven't really done a whole lot of pre-processing. It's very nice that the data's already pre-process for me. So I'm just going to pass it the training data, I'm going to pass it the number of epochs, and then I'm going to say the test data is my validation data test. So if I start training, we'll see that it is pretty slow. So this number here will be direct 25,000 records, so it's going to build that up as it doesn't know the record size on the first epoch. So that's going to take a few moments to build up. We've seen it's now approaching 25,000, we know there are 25,000 in the epoch will end. So this epoch took about four minutes. So now we can see there are more epochs. Epoch two has just kicked in. So on epoch one, we had about four and a half minutes, the accuracy was only 52 percent, and we'll see over time that this isn't going to improve. I'm going to pause the recording now and then come back when epoch 10 hits. Now, we can see epoch 10 coming to a close, and as it finishes, we can see our accuracy numbers went from 52.3 percent to 54.2 percent. Pretty flat throughout the training. Validation accuracy is similar. It started on measurable and then went 53.18-53.49. So it's pretty clear nothing much is really happening despite all the time we're spending on this. When I plot this out, we can see my accuracy might look really good, but when you look at the scale, it's very little, and the validation accuracy is the same. Now, the reason why this is happening of course is just because we're working on subwords, because we're training on things that it's very hard to pull semantics and meaning out of them and the results that we're getting are little better than 50 percent. But if you think about it in a binary classifier, a random guess would be 50 percent. So this leads us to a problem where we've taken a little bit of a step back, but that's okay. Sometimes you take one step back to take two steps forward, and that's what we'll be learning with RNNs next week, and that's as you start learning things when they're put in sequence. So now sor, after ten, ten so actually has some meaning, but ten could be misunderstood as the number and sor doesn't really mean anything. So when we start putting this together with RNNs and learning things where sequences are important, then we'll see how this step back will hopefully help us make a step forward.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2019 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play

Coursera
About
Leadership
Careers
Catalog
Certificates
MasterTrack™ Certificates
Degrees
For Enterprise
For Government

Community
Learners
Partners
Developers
Beta Testers
Translators

Connect
Blog
Facebook
LinkedIn
Twitter
Instagram
YouTube
Tech Blog

More
Terms
Privacy
Help
Accessibility
Press
Contact
Directory
Affiliates