Who am I?

Resul Umit

• post-doctoral researcher at the University of Oslo

• interested in representation, elections, and parliaments

  • a recent publication: Parliamentary communication allowances do not increase electoral turnout or incumbents’ vote share

• working with Twitter data

  • a publication based on Twitter data: The voices of Eurosceptic members of parliament (MPs) echo disproportionately louder on Twitter

    • 400,000+ tweets from 1,000+ MPs
    • automated classification, using a bag-of-words approach
    • manual validation

  • an app based on Twitter data: LikeWise — a Shiny app that facilitates searching the tweets a user liked

• more information available at resulumit.com

The Workshop — Overview

• Two days, on how to collect, process, and analyse data from Twitter

  • ~200 slides, 75+ exercises
  • support for applications to Twitter, developing real projects

• Designed for researchers with basic knowledge of R programming language

  • does not cover programming with R
    • e.g., writing functions
      
      
  • existing ability to work with data in R will be very helpful
    • especially while processing and analysing data
    • but not absolutely necessary — this ability can be developed during and after the workshop as well

The Workshop — Motivation

• Twitter provides attractive opportunities for academic research

  • a simple search for "twitter data" returns 74,000+ results on Google Scholar
    • at the beginning of March 2022

• Research based on Twitter data requires a set of skills

  • typically, these skills are not part of academic training

The Workshop — Motivation — Opportunities

• Popularity of the network

  • about 220 million users, more than three quarters are daily active
  • your subjects are likely Twitter users
    • e.g., for me, members of parliament

The Workshop — Motivation — Opportunities

• Popularity of the network

  • about 220 million users, more than three quarters are daily active^*
  • your subjects are likely Twitter users
    • e.g., for me, members of parliament

• Richness of the data

  • about 500 million tweets per day^*
  • with up to 90 variables for each tweet

The Workshop — Motivation — Opportunities

• Popularity of the network

  • about 220 million users, more than three quarters are daily active
  • your subjects are likely Twitter users
    • e.g., for me, members of parliament

• Richness of the data

  • about 500 million tweets per day
  • with up to 90 variables for each tweet

• Accessibility of the data

  • most Twitter data are public
    • i.e., relatively few private profiles
      
      
  • APIs (application programming interfaces) enable programmatic access to Twitter
    • e.g., downloading tweets with R, as opposed to viewing tweets as visualised via browsers

The Workshop — Motivation — Challenges

• Research based on Twitter data requires certain skills

  • e.g., availability of API is meaningless for researchers who cannot code yet
  • processing and analysing data are no less challenging without programming skills

• The required skills are often uncovered in the academic training of social scientists

  • e.g., methodology courses
  • but help is available elsewhere, including
    
    
    • collaboration with researchers with the skills
      • at the cost of sacrificing some control over your research and research agenda
        
        
    • acquiring the skills through individual effort
      • e.g., workshops such as this one

The Workshop — Motivation — Aims

• To provide you with an understanding of what is possible

  • we will cover a large breath of issues, not all of it is for long-term memory
    • hence the slides are designed for self study as well
      
      
  • awareness of what is possible, Google, and perseverance are all you need

• To start you with acquiring and practicing the skills needed

  • practice with ready-written code
  • start working on a real project

The Workshop — Contents

The Workshop — Organisation

• I will go through a number of slides...

  • introducing things
  • demonstrating how-to do things

• ... and then pause, for you to use/do those things

  • e.g., prepare your computer for the workshop, and/or
  • complete a number of exercises

• We are here to help

  • ask me, other participants
  • consult Google, slides, answer scripts
    • type, rather than copy and paste, the code you will find on the slides or the script

The Workshop — Organisation — Slides

• Codes and texts that go in R console or scripts appear as such — in a different font, on gray background

  • long codes and texts will have their own line(s)

# read in  the tweets dataset
df <- read_rds("tweets.rds") %>%
# split the variable text, create a new variable called da_tweets   
  unnest_tokens(output = da_tweets, input = text, token = "tweets") %>%
# remove rows that match any of the stop words as stored in the stop_words dataset 
  anti_join(stop_words, by = c("da_tweets" = "word"))

The Workshop — Organisation — Slides

• Codes and texts that go in R console or scripts appear as such — in a different font, on gray background

  • long codes and texts will have their own line(s)

• Results that come out as output appear as such — in the same font, on green background

  • except very obvious results, such as graphs

• Specific sections are highlighted yellow as such for emphasis

  • these could be for anything — codes and texts in input, results in output, and/or texts on slides

• The slides are designed for self-study as much as for the workshop

  • accessible, in substance and form, to go through on your own

Considerations — Research Questions & Hypotheses

• Ideally, we have one or more research questions, hypotheses

  • developed prior to data collection, analysis
    • based on, e.g., theory, claims, observations
      
      
  • perhaps, even pre-registered
    • e.g., at OSF Registries

• Not all questions can be answered with Twitter data

  • see relevant literature for what works, what does not
    • e.g., for political science, the review by (Jungherr, 2016)
    • for public health, the review by (Sinnenberg, Buttenheim, Padrez, Mancheno, Ungar, and Merchant, 2017)

Considerations — Potential Biases

There are at least two potential sources of bias in Twitter data

• sampling

  • Twitter users are not representative of the people out there
    • see, for example, (Mellon and Prosser, 2017)
      
      
  • Tweeting behaviour has a strategic component
    • see, for example, (Umit, 2017)

• mediation

  • the behaviour on Twitter is mediated through written and unwritten rules
    • e.g., there is a button to like, but no dislike
      • might systematically bias the replies towards negative
        
        
    • e.g., the common use of the like function as a bookmark
      • what would a study of Twitter likes be measuring?

Considerations — Constraints over Data Access

• Twitter has restrictions on data access

  • how much data is available to download
  • how quickly, how frequently, how far dating back etc.

• These restrictions vary across API types

  • e.g., Standard v1.1 is the most restrictive APIs
    • other first generation APIs are the Premium v1.1 and Enterprise: Gnip 2.0 APIs — both with paid subscriptions
    • there are also the second generation APIs, including the newly announced Academic Research access

• These restrictions also vary within APIs types, across different operations

  • e.g., collecting tweets in real time vs. collecting historical tweets
    • but also, collecting historical tweets from a specific user vs. tweets from any user

Considerations — Constraints over Data Redistribution

• Twitter restricts content redistribution

  • e.g., only the tweet and/or user IDs can be made publicly available in datasets over 50,000 observations
    • e.g., not the tweets themselves
    • and no more than 1.5M IDs
      • with some exceptions for academic research
        
        
  • see Twitter Developer terms for further details
    
    

• Reproducibility of research based on Twitter data is limited in practice

  • i.e., reproducibility after publication, by others
  • technically, they can retrieve the same tweets with IDs
    • demanding for reproducers
    • may even be impossible
      • e.g., some tweets, or whole accounts, might be deleted before replication attempts

Considerations — Changes in the Twitter APIs

• Twitter is currently switching to a new generation of APIs

  • replacing APIs v1 with v2
    • each with various types of APIs
      
      
  • the switch is not complete, outcome is not clear
    • see the early access options

• Twitter might change the rules of the APIs game at any anytime, again

  • making the existing restrictions more or less strict
    • e.g., while you are in the middle of data collection
      
      
  • breaking your plans, code

Considerations — Changes in the Twitter APIs — Notes

• Existing codes to collect tweets may or may not be affected, depending on

  • how the APIs v2 will look in the end
    • it is still a work in progress
      
      
  • how the rtweet package^* will adopt
    • it is currently going through a major revision

Considerations — Changes in the Twitter APIs — Notes

• Existing codes to collect tweets may or may not be affected, depending on

  • how the APIs V2 will look in the end
    • it is still a work in progress
      
      
  • whether and how the rtweet package will adopt
    • it is currently going through a major revision

• Not all changes are bad

  • among others, APIs v2 introduces the Academic Research access
    
    
    • 'to serve the unique needs and challenges of academic researchers'
      • ranging from master's students to professors
        
        
    • access to all public tweets
      • by up to 10M a month at a time

Considerations — Law and Ethics

• It is often impossible to get users' consent

  • i.e., for collecting and analysing their data on Twitter
    
    
  • Twitter itself has no problem with it, but others might disagree
    • e.g., your law makers, (funding and/or research) institution, subjects, conscience

• Check the rules that apply to your case

  • rules and regulations in your country, at your institution

• Reflect on whether using Twitter data for research is ethical

  • even where it is legal and allowed, it may not be moral

Considerations — Data Storage

Twitter data frequently requires

• large amounts of digital storage space

  • Twitter data is typically big data
    • many tweets, up to 90 variables
      
      
  • e.g., a dataset of 1M tweets requires about 300MB
    • when stored in R data formats

• private, safe storage spaces

  • due to Twitter Developer terms
  • but also local rules, institutional requirements

Considerations — Language and Context

• Some tools of text analysis are developed for a specific language and/or context

  • e.g., dictionaries for sentiment analysis

    • might be in English, for political texts, only

  • these may not be useful, valid for different languages, and/or contexts

• Some tools of text analysis are developed for general use

  • e.g., a dictionary for sentiments in everyday language

  • these may not be useful, valid for a specific context

    • e.g., political texts

Course Materials — Overview

Materials have the following structure

twtr_workshop-materials
   |
   |- data
   |  |
   |  |- mps.csv
   |  |- status_ids.rds
   |  |- tweets.rds
   |
   |- analysis
   |  |
   |  |- solutions.R   
   |  |- tweets.Rmd
   |  |- tweets_answers.Rmd
   |  |- users.Rmd
   |  |- users_answers.Rmd

Course Materials — Contents

• data/mps.csv

  • a dataset on the members of parliament (MPs) in the British House of Commons, at the end of January 2021
  • it includes variables on electoral results as well as Twitter usernames

• data/status_ids.rds

  • a dataset with a single variable: status_id
  • lists the status IDs of all tweets posted by the MPs listed in mps.csv, during January 2021

• data/tweets.rds

  • similar to data/status_ids, except that
    • the time period is now limited to 15 to 31 January, reducing the number of observations below 50,000, allowing for all variables to be posted online

Course Materials — Contents

• exercises/solutions.R

  • an R script with solutions for exercises until the end of Part 4

• exercises/tweets.Rmd

  • an R Markdown file with exercises for Part 6
  • the solution to these exercises are in tweets_answers.Rmd

• exercises/users.Rmd

  • an R Markdown file with exercises for Part 5
  • the solution to these exercises are in users_answers.Rmd

Twitter — Authorisation

Authorization to use Twitter APIs requires at least three steps^*

1) open a user account on Twitter

• a personal or an institutional (perhaps, for a research project) one
• done once, takes minutes

2) with that user account, apply for a developer account

• so that you are recognised as a developer, have access to the developer portal
• done once per account, takes days to get approved manually

3) with that developer account, register a Twitter app

• so that you have the keys and tokens for authorisation
• repeated for every project, takes minutes

Twitter — Authorisation — Notes

• It is possible to interact with Twitter APIs without steps 2 and 3

  • rtweet has a its own Twitter app — rstats2twitter — that anyone can use
    • anyone with a Twitter account, who authorises rstats2twitter via a pop-up browser

• I recommend

  • following only the step 1 (open an account) now, which
    • you might already have done
    • is otherwise automatic
    • allows us to use rstats2twitter and follow the workshop
      
      
  • leaving the remaining steps until after the workshop
    • to allow you to think and write your applications carefully
    • to get my feedback if you prefer to do so

Twitter — Keys and Tokens — Notes

• Keys and tokens are personal
  
  
  • should not be shared with others
    • e.g., through replication files, when asking a question on Stack Overflow

• Keys and tokens can be re-generated anytime
  
  

  • on your applications page, under the Keys and tokens tab
  • this requires updating your R script with the new tokens as well

• Twitter allows for further, optional settings involving keys and tokens

  • relevant, mostly, for apps to do more than just collecting tweets
  • e.g., settings for 3-legged OAuth
    • like rstats2twitter, to allow for other users to authenticate through a browser pop up
  • not covered in this workshop

Twitter — Keys and Tokens — Notes — Definitions

• Consumer key and Consumer secret

  • identifiers for an application
  • provide project authorization
    • identify the application
    • check whether the application has been granted access
  • like your application's username and password

• Access token and Access token secret

  • identifier for users of an application
    • this may or may not include anyone other than the owner
    • e.g., for rstats2twitter, there are many users
  • provide user authentication
    • identify the user
    • check whether the user should have access to a request

Data Collection — Authentication

There are two different methods of authentication

• through rtweet's rstats2twitter app

  • the app makes requests on your behalf
  • you simply approve this, via a browser that pops up
  • nice and easy, but comes with some limitations

• through your own app

  • you make requests on your behalf
  • through your own app that you register as a developer
  • takes a little effort, but comes with additional stability and functionality
    • removing the dependency on rstats2twitter, over which you have no control
    • getting rid of the pop up windows
    • not only for collecting tweets, but also posting your own tweets
      • and, reading and writing your own direct messages

Data Collection — Define Your Token — Alternatives

 tw_token <- create_token(
    app = "",
    consumer_key = "",
    consumer_secret = "",
    access_token = "",
    access_secret = ""
   )

 library(rtweet)
 source("keys_tokens.R")

Data Collection — Define Your Token — Alternatives

TWITTER_APP=name_of_my_app
TWITTER_CONSUMER_KEY=akN...
TWITTER_CONSUMER_SECRET=HJK...
TWITTER_ACCESS_TOKEN=345...
TWITTER_ACCESS_SECRET=SDF...

 library(rtweet)
tw_token <- create_token(
    app = Sys.getenv("TWITTER_APP"),
    consumer_key = Sys.getenv("TWITTER_CONSUMER_KEY"),
    consumer_secret = Sys.getenv("TWITTER_CONSUMER_SECRET"),
    access_token = Sys.getenv("TWITTER_ACCESS_TOKEN"),
    access_secret = Sys.getenv("TWITTER_ACCESS_SECRET")
 )

Other Resources^*

• R for Data Science (Wickham and Grolemund, 2021)

  • open access at https://r4ds.had.co.nz

• Text Mining with R: A Tidy Approach (Silge and Robinson, 2017)

  • open access at tidytextmining.com
  • comes with a course website where you can practice

• A Tutorial for Using Twitter Data in the Social Sciences: Data Collection, Preparation, and Analysis (Jürgens and Jungherr, 2016)

  • open access at http://dx.doi.org/10.2139/ssrn.2710146

Data Collection — Overview — APIs

• We will collect data through APIs

  • i.e., Twitter's Standard v1.1 APIs
  • provides more variables than available through browsers
  • comes with rules and restrictions
    • enforced through authentication

• Collecting data through web scraping is also possible

  • e.g., with GetOldTweets3 — a python library
    • scrapes, scrolls down, and scrapes again to collect all matching data
      
      
  • does not require, is not limited by, Twitter APIs
  • limited with what is available on browsers
  • may or may not be ethical and/or legal

Data Collection — Overview — APIs — Types

• In general, there are two main types of APIs

  • REST and Streaming
  • applies to APIS elsewhere, not just at Twitter
  • functions, arguments, behaviour differ slightly

• REST APIs are for single, one-off requests

  • e.g., search for tweets posted in the last 6 to 9 days
  • but also, post or delete tweets

• Streaming APIs are for continuous requests

  • e.g., collect tweets as they are being posted

Data Collection — Overview — APIs — Types

• At Twitter, there is a further differentiation among the APIs

  • e.i., API v2, Enterprise: Gnip 2.0, Premium v1.1, Standard v1.1
  • with each, you can make single or continuous requests

• Rules and restrictions differ from one type to another

  • as does the cost
  • some remove the restrictions on how much data we can access
  • restrictions on how quickly we can access data exist in all types
    • these restrictions are called rate limits

• Rules and restrictions can also differ within one type

  • for different operations
    • e.g., for collecting historical vs. live data

Data Collection — Overview — Standard v1.1 APIs

• We will collect data through Twitter's Standard v1.1 APIs

  • free of charge
  • thanks to rweet's rstats2twitter app, can be used immediately
  • comes with the strictest of restrictions
    • e.g., searches tweets posted in the last 6 to 9 days

• You can surpass these restrictions later on

  • academic researchers can apply for Academic Research access
  • others can purchase an alternative
  • the principles of data collection are likely to remain the same
    • rweet has the search_30day and search_fullarchive functions for the Premium V1.1 APIs
    • the package may require an update, in line with the Academic Research access

Data Collection — Overview — APIs — Limitations

Our attempts to collect data will be limited for various reasons, including

• the intended restrictions by Twitter
  • by the limitations of Standard v1.1
    • e.g., by rate limits
      • maximum number of requests
      • per app, type of request, time interval

• any lack of tweets or users matching our search criteria
  • stricter the criteria, more likely to occur

• connections timing out
  • depends on type of requests, and your internet connection
  • more likely for continuous searches

• reasons unknown to humankind
  • sometimes things just happen

Data Collection — rtweet — Overview

• A powerful R package for collecting Twitter data

  • created by Michael W. Kearney (University of Missouri)
  • used widely, replacing previous packages for this task
    • e.g., twitteR
  • last updated on CRAN two years ago
    • the package is currently being updated on GitHub

• A lot has already been written on this package. See, for example,

  • the package repository on GitHub
  • the package documentation
  • this journal article by its creator
  • this book by Bob Rudis — a user of the package
  • numerous tutorials, such as this, this, and this

• Comes with its own app, rstats2twitter
  • allows for collecting tweets without a developer account
  • offers the option of using your own keys and tokens, if/once you have them

Data Collection — rtweet — Basics

There are four main groups of functions to collect historical data, starting with

• search_
  • such as search_tweets or search_users
  • 4 functions, for general use
    • 2 for standard APIs, 2 for premium APIs

• lookup_
  • such as lookup_tweets or lookup_users
  • 5 functions, for expanding an existing variable into a dataset
    • e.g., starting with a list of IDs for tweets or users

• get_
  • such as get_followers or get_friends
  • 11 functions, for specific tasks

• lists_
  • such as lists_members or lists_statuses
  • 6 functions, for tasks related to Twitter lists specifically

Data Collection — rtweet — Basics

There is also one function to collect tweets in real time

• stream_tweets

  • queries the streaming API
  • returns a small random sample of all tweets as they are posted
  • can be filtered by keywords, users, and/or locations

• For other functions, see the package documentation

  • e.g., the functions starting with post_
    • allowing for posting your tweets, direct messages, from within R

Data Collection — search_tweets

search_tweets(q,
              n = 100,
              type = "recent",
              include_rts = TRUE,
              geocode = NULL,
              max_id = NULL,
              parse = TRUE,
              token = NULL,
              retryonratelimit = FALSE,
              verbose = TRUE,
              ...
)

Data Collection — search_tweets

search_tweets(q = "#publish")

Data Collection — Notes

• Twitter usernames, or handles, are stored under variable screen_name

  • can be misleading, as users also have display names

• Twitter allows user to change their usernames and display names

  • user IDs, however, do not change
    • user_id is a better variable for reproducible research

• The date and time data are matched to Greenwhich Mean Time

  • stored under the variable created_at
  • no matter where users actually are at the time of tweeting

• You may wish to exclude retweets

  • depending on the research question and design
  • by setting include_rts = FALSE

Data Collection — search_tweets

search_tweets(q = "publish",
              n = 200,
              type = "popular")

Data Collection — search_tweets

search_tweets(q = "publish perish",
              n = 200,
              type = "popular")

Data Collection — search_tweets

search_tweets(q = "publish OR perish",
              n = 200,
              type = "popular")

Data Collection — search_tweets

search_tweets(q = "\"publish or perish\"",
              n = 200,
              type = "popular")

Data Collection — search_tweets

search_tweets(q = "publish -perish",
              n = 200,
              type = "popular")

Data Collection — search_tweets

search_tweets(q = "publish lang:de",
              n = 200,
              type = "popular")

• Note that

  • query parameters, such as lang, are followed by a colon :
    • other parameters include filter, from, to, since, until, min_retweets etc.
      
      
  • there is an official guide for a comprehensive but not complete list of search operators
    • alternatively, fill in this advanced search form, and observe the resulting syntax

Data Collection — search_tweets

search_tweets(q = "publish -lang:de",
              n = 200,
              type = "popular")

Data Collection — search_tweets — Notes

search_tweets(q = "publish lang:en filter:replies",
              n = 200,
              type = "mixed")

search_tweets(q = "publish",
              n = 200,
              type = "mixed",
              lang = "en", 
              filter = "replies")

Data Collection — search_tweets — Notes

• This function returns a data frame

  • as do many other functions in the package
  • because, by default, parse = TRUE

• Under the hood, Twitter APIs return nested lists

  • nested lists can be tidied into rectangular shape, but not tidy data as they are returned
  • rtweet does most of the data preparation for us

Data Collection — search_users

search_users(q, 
             n = 100, 
             parse = TRUE, 
             token = NULL, 
             verbose = TRUE)

• Note that

  • there is no retryonratelimit argument
  • you can still use the complete list of search operators and this advanced search form for help with queries

Data Collection — Notes — rate_limit

rate_limit(token = NULL, 
           query = NULL, 
           parse = TRUE)

• Note that

  • these limits are for the number of requests, per user and/or per app
  • if no token is specified, the function uses the rstats2twitter app
  • rate limits decrease as you use them, increase again after a time threshold
  • Twitter Developer terms do not allow for multiple tokens to be used for the same project

Data Collection — Notes — rate_limit

Check you remaining rate limits, for all operations

rate_limit()

# A tibble: 261 x 7
   query                   limit remaining reset      reset_at            timestamp           app           
   <chr>                   <int>     <int> <drtn>     <dttm>              <dttm>              <chr>         
 1 lists/list                 15        15 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
 2 lists/:id/tweets&GET      900       900 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
 3 lists/:id/followers&GET   180       180 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
 4 lists/memberships          75        75 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
 5 lists/:id&DELETE          300       300 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
 6 lists/subscriptions        15        15 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
 7 lists/members             900       900 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
 8 lists/:id&GET              75        75 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
 9 lists/subscribers/show     15        15 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
10 lists/:id&PUT             300       300 14.78 mins 2022-03-06 10:35:45 2022-03-06 10:20:59 rstats2twitter
# ... with 251 more rows

Data Collection — Notes — rate_limit

Check your remaining rate limits for specifically the search_tweets function

rate_limit(query = "search/tweets")

# A tibble: 1 x 7
  query         limit remaining reset         reset_at            timestamp           app           
  <chr>         <int>     <int> <drtn>        <dttm>              <dttm>              <chr>         
1 search/tweets   180       171 14.77129 mins 2022-03-06 10:42:12 2022-03-06 10:27:26 rstats2twitter

Data Collection — Notes — rate_limit

Single out the number of remaining rate limit for specifically the search_tweets function

rate_limit(query = "search/tweets")$remaining

## [1] 171

• Note that

  • this is not the number of tweets you can search for
  • but rather the number of times that you can search for tweets

Data Collection — lookup_tweets

lookup_tweets(statuses, 
              parse = TRUE, 
              token = NULL)

• Note that

  • this function would be useful for replicating studies
  • status IDs are visible on browsers
    • if you click on a specific tweet

Data Collection — lookup_tweets

Collect data on one or more status IDs

lookup_tweets(statuses = c("567053242429734913", "266031293945503744", "440322224407314432"))

Collect data on status IDs in a data frame

lookup_tweets(statuses = df$status_id)

Data Collection — lookup_users

lookup_users(users, 
             parse = TRUE, 
             token = NULL)

Data Collection — lookup_users

Collect data on one or more status IDs

lookup_users(users = c("drob", "hadleywickham", "JennyBryan"))

Collect data on status IDs in a data frame

lookup_users(users  = df$screen_name)

Data Collection — lookup_friendships

lookup_friendships(source, 
                   target, 
                   parse = TRUE, 
                   token = NULL)

Data Collection — get_timeline

get_timeline(user,
             n = 100,
             max_id = NULL,
             home = FALSE,
             parse = TRUE,
             check = TRUE,
             token = NULL,
             ...
)

Data Collection — get_timeline

get_timeline(user = "drob",
              n = 200)

Data Collection — get_timeline

get_timeline(user = c("drob",
                     "hadleywickham"),
             n = 200)

Data Collection — get_timeline — Home Timeline

get_timeline(user = "hadleywickham",
             n = 200, 
             home = TRUE)

Data Collection — Notes — retryonratelimit

• the retryonratelimit argument is not available for all functions in the package

  • e.g., search_users
  • does not mean you will not run into limits

• You can create your own safety net

  • e.g., with loops, where the system sleeps between iterations
    • until a specific rate limit increases

Data Collection — Notes — retryonratelimit — Iteration

datalist <- list() # create an empty list, to be filled later
for(i in 1:length(df_users$screen_name)) { # for one user, in the data frame df_users, at a time
  if (rate_limit(query = "application/rate_limit_status", token = tw_token)$remaining > 2 &
      rate_limit(query = "get_timeline", token = tw_token)$remaining > 20) { # if your are still under rate limit for this task
  dat <- get_timeline(df$screen_name[i], n = 3200, # collect the tweets
                      token = tw_token)
  datalist[[i]] <- dat # fill the list with data, for one user at a time
  }else{ # if there is no limit, wait a little
  wait <- rate_limit(query = "get_timeline")$reset + 0.1
  Sys.sleep(wait * 60)
  }
}
df_tweets <- as.data.frame(do.call(rbind, datalist)) # put all data in one data frame

Data Collection — get_followers

get_followers(user,
              n = 5000,
              page = "-1",
              retryonratelimit = FALSE,
              parse = TRUE,
              verbose = TRUE,
              token = NULL
)

Data Collection — get_followers

Collect a list of Hadley Wickham's followers on Twitter

get_followers(user = "hadleywickham",
              n = 10000,
              retryonratelimit = TRUE)

Data Collection — get_friends

get_friends(users,
            n = 5000,
            retryonratelimit = FALSE,
            page = "-1",
            parse = TRUE,
            verbose = TRUE,
            token = NULL
)

Data Collection — get_friends

Collect a list of users followed by Jenny Bryan and Hadley Wickham on Twitter

• with 20 friends from each

get_friends(users = c("hadleywickham", "JennyBryan"),
            n = 20)

Data Collection — get_favorites

get_favorites(user,
              n = 200,
              since_id = NULL,
              max_id = NULL,
              parse = TRUE,
              token = NULL
)

Data Collection — get_favorites

Collect a list of tweets liked by Jenny Bryan

get_favorites(user = "JennyBryan")

Data Collection — get_retweets

get_retweets(status_id, 
             n = 100, 
             parse = TRUE, 
             token = NULL, 
             ...
)

Data Collection — get_retweets

• Collect the most recent 50 retweets

  • of the post announcing the Academic Research Access

get_retweets(status_id = "1354143047324299264")

Data Collection — get_trends

get_trends(woeid = 1,
           lat = NULL,
           lng = NULL,
           exclude_hashtags = FALSE,
           token = NULL,
           parse = TRUE
)

Data Collection — get_trends

Collect the trends data for London

• using the woeid argument

get_trends(woeid = 44418)

Collect the same trends data for London

• using the lat and long arguments instead

get_trends(lat = "51.50", lng = "0.12")

Data Collection — lists_memberships

lists_memberships(user = NULL,
                  n = 200,
                  cursor = "-1",
                  filter_to_owned_lists = FALSE,
                  token = NULL,
                  parse = TRUE,
                  previous_cursor = NULL
)

Data Collection — lists_memberships

Collect data on lists where Jenny Bryan is listed

lists_memberships(user = "JennyBryan")

Collect data on lists where Jenny Bryan or Hadley Wickham is listed

lists_memberships(user = c("JennyBryan", "hadleywickham"))

Data Collection — lists_members

lists_members(list_id = NULL,
              slug = NULL,
              owner_user = NULL,
              n = 5000,
              cursor = "-1",
              token = NULL,
              parse = TRUE,
              ...
)

Data Collection — lists_members

Collect data on the list of MPs in the House of Commons

• using the list_id argument

lists_members(list_id = "217199644")

Collect the same data, with different arguments

• using the owner_user and slug arguments

lists_members(owner_user = "TwitterGov", slug = "UK-MPs")

Data Collection — lists_statuses

lists_statuses(list_id = NULL,
               slug = NULL,
               owner_user = NULL,
               since_id = NULL,
               max_id = NULL,
               n = 200,
               include_rts = TRUE,
               parse = TRUE,
               token = NULL
)

Data Collection — lists_statuses

Collect tweets posted by the members of the UK MPs list

• using the list_id argument

lists_statuses(list_id = "217199644")

Collect the same data, with different arguments

• using the owner_user and slug arguments

lists_statuses(owner_user = "TwitterGov", slug = "UK-MPs")

Data Collection — lists_subscribers

lists_subscribers(list_id = NULL,
                  slug = NULL,
                  owner_user = NULL,
                  n = 20,
                  cursor = "-1",
                  parse = TRUE,
                  token = NULL
)

Data Collection — lists_subscribers

Collect data on users subscribed to the UKMPs list

• using the list_id argument

lists_subscribers(list_id = "1405362")

Collect the same data, with different arguments

• using the owner_user and slug arguments

lists_subscribers(owner_user = "TwitterGov", slug = "UK-MPs")

Data Collection — lists_subscriptions

lists_subscriptions(user, 
                    n = 20, 
                    cursor = "-1", 
                    parse = TRUE, 
                    token = NULL)

Data Collection — lists_subscriptions

Collect data on the lists that TwitterGov is subscribed to

lists_subscriptions(user = "TwitterGov")

Data Collection — stream_tweets

stream_tweets(q = "",
              timeout = 30,
              parse = TRUE,
              token = NULL,
              file_name = NULL,
              verbose = TRUE,
              ...
)

Data Collection — stream_tweets

stream_tweets(q = "",
              timeout = Inf)

Data Collection — stream_tweets

stream_tweets(q = "",
              timeout = 30)

Data Collection — stream_tweets

stream_tweets(q = "switzerland, schweiz,
                   suisse, svizzera",
              timeout = 30)

Data Collection — stream_tweets

stream_tweets(q = c("UniLuzern", "hslu",
                   "phluzern"),
              timeout = 30)

Data Collection — stream_tweets

stream_tweets(q = c(6.02, 45.77, 
                  10.44, 47.83),
              timeout = 30)

Data Preperation — Overview

• The rtweet package does a very good job with data preperation to start with

  • returns data frames, with mostly tidy data
  • although Twitter APIs return nested lists
  • some variables are still lists
    • e.g., hastags

• Further data preparation depends on your research project

  • most importantly, on whether you will work with texts or not
  • we will cover some common preparation steps

Data Preperation — Overview — Strings

• Most researchers would be interested in textual Twitter data

  • tweets as a whole, but also specifically hashtags etc.

• There are many components of tweets as texts

  • e.g., mentions, hashtags, emojis, links etc.
  • but also punctuation, white spaces, upper case letters etc.
  • some of these may need to be taken out before analysis

• I use the stringr package (Wickham, 2019) for string operations

  • part of the tidyverse family
  • you might have another favourite already
    • no need to change as long as it does the job

Data Preperation — Overview — Numbers

• There is more to Twitter data than just tweets themselves

  • e.g., the number of followers, likes etc.
    • see Silva and Proksch (Silva and Proksch, 2021) for a great study on numbers

• I use the dplyr package (Wickham, François, Henry, and Müller, 2022) for most data operations on numbers

  • part of the tidyverse family
  • you might have another favourite already
    • no need to change as long as it does the job

Data Preperation — Remove Mentions

tweet <- "These from @handle1 are #socool. 👏 A #mustsee, @handle2! 
          👉 https://t.co/aq7MJJ2"

str_remove_all(string = tweet, pattern = "[@][\\w_-]+")

Note that

• this removes all occurrences
• there is also the str_remove fucntion, which removes the first occurrence only

Data Preperation — Remove Hashtags

tweet <- "These from @handle1 are #socool. 👏 A #mustsee, @handle2! 
          👉 https://t.co/aq7MJJ2"

str_remove_all(string = tweet, pattern = "[#][\\w_-]+")

Data Preperation — Exercises — Notes

• The exercises in this part are best followed by

  • using tweets.rds or a similar existing dataset
  • saving a new variable at every step of preparation
  • observing the newly created variables
    • by selecting only the old and new variable
    • to confirm whether the code works as intended

• The mutate and select functions, from the dplyr package, can be helpful, as follows

df_tweets <- read_rds("data/tweets.rds")
df_tweets %>%
  mutate(no_mentions = str_remove_all(string = text, pattern = "[@][\\w_-]+")) %>% 
  select(text, no_mentions) %>% 
  View()

Data Preperation — Remove Links

tweet <- "These from @handle1 are #socool. 👏 A #mustsee, @handle2! 
          👉 https://t.co/aq7MJJ2"

str_remove_all(string = tweet, pattern = "http\\S+\\s*")

Data Preperation — Remove Emojis

tweet <- "These from @handle1 are #socool. 👏 A #mustsee, @handle2! 
          👉 https://t.co/aq7MJJ2"

iconv(x = tweet, from = "latin1", to = "ASCII", sub = "")

Data Preperation — Remove Punctuations

tweet <- "These from @handle1 are #socool. 👏 A #mustsee, @handle2! 
          👉 https://t.co/aq7MJJ2"

str_remove_all(string = tweet, pattern = "[[:punct:]]")

Note that

• this removed all punctuation, including those in links

Data Preperation — Remove Punctuations

tweet <- "This is a sentence.There is no space before this sentence."

str_remove_all(string = tweet, pattern = "[[:punct:]]")

Note that

• if tweets are typed with no spaces after punctuation, this might lead to merged pieces of text
  • alternatively, try the str_replace_all function to replace punctuation with space

Data Preperation — Replace Punctuations

tweet <- "This is a sentence.There is no space before this sentence."

str_replace_all(string = tweet, pattern = "[[:punct:]]", replacement = " ")

Data Preperation — Remove Repeated Whitespace

tweet <- "There are too many spaces after this sentence.   This is a new sentence."

str_squish(string = tweet)

Note that

• white spaces can be introduced not only by users on Twitter, but also by us, while cleaning the data
  • e.g., removing and/or replacing operations above
  • hence, this function might be useful after other operations

Data Preperation — Change Case

tweet <- "lower case. Sentence case. Title Case. UPPER CASE."

str_to_lower(string = tweet)

Note that

• there are other functions in this family, including
  • str_to_sentence, str_to_title, str_to_upper

Data Preperation — Change Unit of Observation

Research designs might require changing the unit of observation

• aggregation
  • e.g., at the level of users, locations, hashtags etc.
  • summarise with dplyr

• dis-aggregation
  • e.g., to the level of words
  • tokenise with tidytext

Data Preperation — Change Unit of Observation — Aggregation

Aggregate at the level of users

• the number of tweets per user

# load the tweets dataset
df <- read_rds("tweets.rds") %>%
# group by users for aggregation  
  group_by(user_id) %>%
# create summary statistics for variables of interest
  summarise(sum_tweets = n())

Data Preperation — Change Unit of Observation — Aggregation

What is aggregated at which level depends on your research design, such as

• aggregate the tweets into a single text
• at the level of users by source

# load the tweets dataset
df <- read_rds("tweets.rds") %>%
# group by users for aggregation  
  group_by(user_id, source) %>%
# create summary statistics for variables of interest
  summarise(merged_tweets = paste0(text, collapse = ". "))

Data Preperation — Change Unit of Observation — Dis-aggregation

Disaggregate the tweets, by splitting them into smaller units

• also called tokenisation

Note that

• by default sep = "[^[:alnum:].]+", which works well with separating tweets into words
  • change this argument with a regular expression of your choice
• this creates a tidy dataset, where each observation is a word
  • all other tweet-level variables are repeated for each observation

# load the tweets dataset
df <- read_rds("tweets.rds") %>%
# split the variable text  
    separate_rows(text)

Data Preperation — Change Unit of Observation — Dis-aggregation

The tidytext has a function that works better with tokenising tweets

• with token = "tweets", it dis-aggregates text into words
  • except that it respects usernames, hashtags, and URLS

# load the tweets dataset
df <- read_rds("tweets.rds") %>%
# split the variable text, create a new variable called da_tweets    
    unnest_tokens(output = da_tweets, input = text, token = "tweets")

Data Preperation — Change Unit of Observation — Dis-aggregation

Tokenise variables to levels other than words

• e.g., characters, words (the default), sentences, lines

# load the tweets dataset
df <- read_rds("tweets.rds") %>%
# split the variable text into sentences, create a new variable called da_tweets    
    unnest_tokens(output = da_tweets, input = text, token = "sentences")

Data Preperation — Change Unit of Observation — Dis-aggregation

Tokenise variables other than tweets

• recall that rtweet stores multiple hastags, mentions etc. as lists

# load the tweets dataset
df <- read_rds("tweets.rds") %>%
# unlist the lists of hashtags to create strings  
  group_by(status_id) %>%
  mutate(tidy_hashtags = str_c(unlist(hashtags), collapse = " ")) %>%
# split the string, create a new variable called da_tweets    
  unnest_tokens(output = da_hashtags, input = tidy_hashtags, token = "words")

Data Preperation — Remove Stop Words

Remove the common, uninformative words

• e.g., the, a, i

Note that

• this operation requires a tokenised-to-word variable
• stop words for English are stored in the stop_words dataset in the tidytext variable
• list of words for other languages are available elsewhere, including
  • the stopwordslangs function from the rtweet package
  • the stopwords function from the tm package
    • e.g., use tm::stopwords("german") for German

# load the tweets dataset
df <- read_rds("tweets.rds") %>%
# split the variable text, create a new variable called da_tweets    
  unnest_tokens(output = da_tweets, input = text, token = "tweets") %>%
# remove rows that match any of the stop words as stored in the stop_words dataset 
  anti_join(stop_words, by = c("da_tweets" = "word"))

Analysing Users — Overview

• Twitter analysis might focus on users

  • e.g., members of parliaments
  • as opposed to their tweets on Twitter
    • not always mutually exclusive
      
      
  • might be supplemented with non-Twitter data
    • e.g., data/mps.csv

• There are at least two types of user-based analysis

  • count things, and describe or correlate
    • e.g., who has the most followers
      • are female MPs more or less likely to have large number of followers?
        
        
  • network analysis
    • e.g., who retweets whom, how often?

Analysing Users — Correlates of being on Twitter

Analysing Users — Who has the most followers?

Analysing Users — Correlates of having more followers

Analysing Users — Who tweets the most often?

Analysing Users — Correlates of tweeting more often

Analysing Users — Who do they talk to?

Analysing Users — Network Analysis — Overview

• Twitter data is suitable for network analysis

  • a social network
  • studying the relations between users

• There are at least five networks

  • networks of followers
    • directed, not reoccurring
    • e.g., who is following whom
      
      
  • networks of retweeters, quoters, repliers, and likers
    • directed, reoccurring
    • e.g., who is replying to whom, how often

Analysing Users — Network Analysis — Basics

• Networks are composed of nodes and edges

  • e.g., who (a node) replies (an edge) to whom (another node), how often (the strength of the edge)

• The nodes and edges are often kept separate for analysis

  • e.g., in separate datasets, frames
  • nodes are given an ID number

• We will use two pacakges for network analysis

  • tidygraph for data manipulation
  • ggraph for visualisation

Analysing Users — Network Analysis — tidygraph

read_rds("data/tweets.rds") %>%
  filter(is_retweet == TRUE) %>% 
  group_by(screen_name, retweet_screen_name) %>%
  summarise(rts = n()) %>% 
  head()

# A tibble: 6 x 3    
# Groups:   screen_name [1]     
  screen_name  retweet_screen_name   rts
                         
1 _OliviaBlake AlexDaviesJones         1
2 _OliviaBlake CommonsPAC              2
3 _OliviaBlake DanJarvisMP             3
4 _OliviaBlake DrRosena                1
5 _OliviaBlake EmmaHardyMP             1
6 _OliviaBlake FriendsLoxley           2

Analysing Users — Network Analysis — tidygraph

read_rds("data/tweets.rds") %>%
  filter(is_retweet == TRUE) %>% 
  group_by(screen_name, retweet_screen_name) %>%
  summarise(rts = n()) %>% 
  as_tbl_graph()

# A tbl_graph: 7377 nodes and 16131 edges
#
# A directed multigraph with 5 components
#
# Node Data: 7,377 x 1 (active)
  name         
          
1 _OliviaBlake 
2 _RobbieMoore 
3 AaronBell4NUL
4 ab4scambs    
5 abenaopp     
6 ABridgen     
# ... with 7,371 more rows
#
# Edge Data: 16,131 x 3
   from    to   rts
    
1     1    19     1
2     1   550     2
3     1   119     3
# ... with 16,128 more rows

Analysing Users — Network Analysis — tidygraph

read_rds("data/tweets.rds") %>%
  filter(is_retweet == TRUE) %>% 
  group_by(screen_name, retweet_screen_name) %>%
  summarise(rts = n()) %>% 
  as_tbl_graph() %>% 
  activate(edges) %>%
  mutate(multi_rts = if_else(rts > 1, 1, 0))

# A tbl_graph: 7377 nodes and 16131 edges
#
# A directed multigraph with 5 components
#
# Edge Data: 16,131 x 4 (active)
   from    to   rts multi_rts
        
1     1    19     1        0
2     1   550     2        1
3     1   119     3        1
4     1   154     1        0
5     1   167     1        0
6     1   551     2        1
# ... with 16,125 more rows
#
# Node Data: 7,377 x 1
  name         
          
1 _OliviaBlake 
2 _RobbieMoore 
3 AaronBell4NUL
# ... with 7,374 more rows

Analysing Users — Network Analysis — ggraph

Once the nodes and edges are ready, use the ggraph package to visualise

• an extension of ggplot2
• many verbs are intuitively similar

ggraph(rt_network) +
  geom_edge_link() +
  geom_node_point(aes(color = party)) +
  theme_graph()

Analysing Users — Reply networks

Analysing Users — Retweet networks

Analysing Users — Who are more central in the retweet networks?

Analysing Tweets — Overview

• Twitter analysis often focuses on tweets

  • e.g., text in the tweets, but also hashtags
  • as opposed to users who tweet them
    • not always mutually exclusive
  • hard to supplement ordinary users with non-Twitter data

• There are at least two types of tweet-based analysis

  • count things, and describe or correlate
    • e.g., how often a hashtag is tweeted
      • are shorter hashtags more likely to be tweeted than longer ones?
  • categorising tweets
    • e.g., with dictionaries
      • but also with other techniques such as machiene learning

Analysing Tweets — When were the tweets posted?

Analysing Tweets — What day of the week?

Analysing Tweets — What time of the day?

Analysing Tweets — Which hastags were the most frequent?

Analysing Tweets — Which words were the most frequent?

Analysing Users — Dictionary Methods — Overview

• Dictionary methods are based on pre-categorisation of words

  • e.g., the word happy might be categorised as positive

    • sad would be negative

  • e.g., the word happy might be categorised as 0.2 sophisticated

    • contented might be 0.4 sophisticated

• These catagories are than matched with the text we have

  • to calculate scores for, e.g., each tweet

Analysing Users — Dictionary Methods — Notes

• There are many ways to calculate scores

  • depending on your research design

• Positive score could be

  • sum(positive)
  • sum(positive) - sum(negative)
  • (sum(positive) - sum(negative)) / (sum(positive) + sum(negative))

• We will use

  • the sentiment dictionary stored at tidytext::get_sentiments("nrc")
  • the concreteness dictionary stored at doc2concrete::mturk_list

Analysing Tweets — Sentiments by Hours of the Day

Analysing Tweets — Sentiments across the time frame

Analysing Tweets — Sentiments in different types of tweets

Analysing Tweets — Concreteness in different types of tweets

Analysing Tweets — Concreteness by Hours of the Day

References

Cheng, J. and W. Chang (2022). httpuv: HTTP and WebSocket Server Library. R package version 1.6.5. https://github.com/rstudio/httpuv.

Jungherr, A. (2016). "Twitter use in election campaigns: A systematic literature review". In: Journal of Information Technology & Politics 13.1, pp. 72-91.

Jürgens, P. and A. Jungherr (2016). "A tutorial for using Twitter data in the social sciences: Data collection, preparation, and analysis". In: Available at http://dx.doi.org/10.2139/ssrn.2710146.

Kearney, M. W. (2020). rtweet: Collecting Twitter Data. R package version 0.7.0. https://CRAN.R-project.org/package=rtweet.

Mellon, J. and C. Prosser (2017). "Twitter and Facebook are not representative of the general population: Political attitudes and demographics of British social media users". In: Research & Politics 4.3, pp. 1-9.

Robinson, D. and J. Silge (2021). tidytext: Text Mining using dplyr, ggplot2, and Other Tidy Tools. R package version 0.3.2. https://github.com/juliasilge/tidytext.

Silge, J. and D. Robinson (2017). Text mining with R: A tidy approach. O'Reilly.

Silva, B. C. and S. Proksch (2021). "Fake It 'Til You Make It: A Natural Experiment to Identify European Politicians' Benefit from Twitter Bots". In: American Political Science Review 115.1, pp. 316-322.

References

Sinnenberg, L., A. M. Buttenheim, K. Padrez, et al. (2017). "Twitter as a tool for health research: a systematic review". In: American Journal of Public Health 107.1, pp. 1-8.

Umit, R. (2017). "Strategic communication of EU affairs: an analysis of legislative behaviour on Twitter". In: The Journal of Legislative Studies 23.1, pp. 93-124.

Wickham, H. (2019). stringr: Simple, Consistent Wrappers for Common String Operations. R package version 1.4.0. https://CRAN.R-project.org/package=stringr.

Wickham, H. (2021). tidyverse: Easily Install and Load the Tidyverse. R package version 1.3.1. https://CRAN.R-project.org/package=tidyverse.

Wickham, H., R. François, L. Henry, et al. (2022). dplyr: A Grammar of Data Manipulation. R package version 1.0.8. https://CRAN.R-project.org/package=dplyr.

Wickham, H. and G. Grolemund (2021). R for data science. O'Reilly.

Xie, Y. (2022). xaringan: Presentation Ninja. R package version 0.23. https://github.com/yihui/xaringan.