Before you can work with data you have to get some. This course will cover the basic ways that data can be obtained. The course will cover obtaining data from the web, from APIs, from databases and from colleagues in various formats. It will also cover the basics of data cleaning and how to make data “tidy”. Tidy data dramatically speed downstream data analysis tasks. The course will also cover the components of a complete data set including raw data, processing instructions, codebooks, and processed data. The course will cover the basics needed for collecting, cleaning, and sharing data.
Reading From APIs
Loading...
From the course by Johns Hopkins University
Getting and Cleaning Data
5027 ratings
From the lesson
Week 2
Welcome to Week 2 of Getting and Cleaning Data! The primary goal is to introduce you to the most common data storage systems and the appropriate tools to extract data from web or from databases like MySQL.
Meet the Instructors
Jeff Leek, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Roger D. Peng, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Brian Caffo, PhDProfessor, Biostatistics
Bloomberg School of Public Health
This cover, lecture covers getting data from the APIs.
APIs are application programming interfaces.
So for example, most internet companies like Twitter or Facebook will have
an application inter, programming interface, or
an API, where you can download data.
For example, you can get data about, you know,
which users are tweeting, or what they're tweeting about.
Where you can get information about what people are posting on Facebook.
You can usually get these with GET
requests with specific URLs as the arguments.
And so what we're going to be able to do is to use
the HTTR package to be able to get data from these web sites.
Usually the first thing that you'll need to do is create an account.
And, in this case, I don't mean an account like a user account You actually have
to create an account with the api or
with the development team out of each particular organization.
So, for example, here we're going to go
to dev.twitter.com/apps and we're going to create an application.
And in this case, we're going to make an
application using the Twitter handle from my blog, simplystats.
And so, what we've done now is, we've logged on using our Twitter details.
And so, you can see I'm logged on as simply statistics.
And, what I can do is, I can click on this button to create a new application.
And what that'll do is, that will give me some numbers that I'm
going to later use to be able to authenticate the application through R.
And access data.
So you can see here, I've actually already
created an application, which I call Simply Statistics blog.
And so if I click on that application, then what
I'm taken to is a page that looks something like this.
So you can see, this is the application, and
this is the URL of the application up there.
And there's some information that I haven't filled
in about my organization and my organization's website.
Down here under the Oduff settings, I've actually see that I have read
and write access to the API and then there are a bunch of numbers.
I black them out here cause these numbers are
private to you in the application that you build.
But you'll have a number a number for your consumer key Consumer secret.
A request hoping URL, and a authorization URL.
You are going to want to copy that down all
of those numbers, once you've created an application,
because we're going to use them later in the R code to be able to access the A-P-I.
[SOUND] So, here's what we're going to do.
We're going to use the HTTR package.
So,we're going to load that package and then we're going to use this command here.
Oauth app so what this is going to
do is start the authorization process for your application.
And so what you need to do is to you name the application, this isn't
actually going to get sent to the a p i, this is just for your convenience.
So I've named it twitter here, because it's going to be the twitter APP.
From me and then I'm going to input here in quotes my consumer key that I got from
the application website and I'm going to put my consumer
secrete here that I got from the application website.
Then what I'm going to do is take that application and
I'm going to sign in and I'm going to sign in using the
token Here that I col, collected from the website as
well as my TokenSecret that I collected from the website here.
So once I've done this, I've actually sort
of created the credentials that will allow me
to access data that is privately held by
Twitter that's only available to people with an application.
So now what I do is I'm using the GET command,
this is very similar to what I did when I was
reading data off the Internet, now I'm going to use a very
specific URL, this is the URL that corresponds to the twitter API.
So that's this part of the URL right here.
This is the version of the API that I'm using,
1.1, which is the current version of the twitter API.
Then I'm going to look, then I pass it some more
components which correspond to which data I'd like to get out.
So in this case, I'd like to get the
statuses on my home timeline, and I'm going to
get it out as a JSON file, which is
currently the only kind of data supported by Twitter.
And then I'm going to pass it, instead of authenticating with the username or a
password, I'm going to pass it the authentication
I used with this O Auth sign in.
And so what I'll get back is actually the page that corresponds
to this url which actually is going to be just some JSON data.
So what I can do is actually use the content function to extract
the JSON data, so this content function
will actually recognize that it's JSON data.
And create it's going to use the from JSOBN function from the R JSON
IO package, so that will return sort of a structured R object.
Which is a little bit hard to read I font so what
I'm going to do is I'm going to use the JSON lite package to reformat
it as a I data frame, so what I do is I take
the JSON structure that I got out from the original command to content.
And I convert that struc, structured R objects back into JSON.
And then I use the JSON lite version of
the from JSON argument to create a data frame.
And so, what this data frame is now is a data
frame where each row corresponds to a tweet in my home timeline.
And so here you can see I'm going to look at the first row of this data frame.
Which is at the time that I extracted the data.
The most recent tweet in my home timeline.
I'm going to look at the first four columns.
The first four columns tell me the time that the tweet was created, the ID number
for the tweet, the ID string here and then it's actually the text of the tweet.
So this is the tweet that was the most
recent tweet when I collected the data off the internet.
So you can do this to actually extract information from Twitter.
This is actually just the request for how do you
actually collect information about what's happening on my home timeline.
But, there's a whole bunch of other requests that you can do.
There's a bunch of different data that you can get access to through Twitter.
And so, how did I know what URL to use?
You actually go into the do, the documentation for the Twitter API, so this
is the same website as before only I'm going to look at the Twitter documentation
and what I'll get to see is what URL, see it says resource URL
down here, and that's the URL I'm going to be passing to my Get command.
It also has a bunch of parameters down here.
And so, if you look up how to format which parameters to
send to an API You can actually send a bunch of different parameters
say you can it how many tweets to collect from what time
to collect it and so forth depending on what URL you are accessing.
If you want to get other information, not just the information about your
home timeline, you can actually go the main Twitter documentation so this is
the documentation for the latest version of the API version 1.1 and
it'll give you information about Getting a bunch of different kinds of information.
So, you can get all sorts of information about, just, mentions.
You can get information about the user timeline.
You can get information about the home timeline.
You can get information about retweets.
You can download all that information, and you
can also search through, Twitter, the Twitter feeds,
and get, specific, tags that you're interested in
or hashtags that you might be interested in following.
So in general, you have to look at
the documentation, HTTR allows get, post, put, delete
requests if you are authorized as you can see, I was authorized to read and write.
Particular to the time line of simply stats,
because that's who I created the application through.
You can also authenticate with a user name or a password, in some cases.
Most modern API's though use something like
oauth, like we saw just now with twitter.
HTTR works well with Facebook, Google, Twitter, Gethb and so forth.
If you go to the HTTR demo component on Gethub you
can actually see a bunch of examples of how you access
the different APIs for the different websites and that's actually where
I modeled this lecture on was the demo for the Twitter API.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
