Who am I?

Resul Umit

• post-doctoral researcher at the University of Oslo

• interested in representation, elections, and parliaments

  • a recent publication: Parliamentary communication allowances do not increase electoral turnout or incumbents’ vote share

• teaching workshops also on

  • writing reproducible research papers
  • working with Twitter data
  • creating professional websites
  • automated web scraping

• more information available at resulumit.com

The Workshop — Overview

• Half a day, on how to

  • version our academic work
  • store the different versions securely — in multiple places
  • collaborate with co-authors — in writing as well as versioning and storing

• Using Git and GitHub, through

  • command line
  • GitHub Desktop, RStudio

• Most useful for those working with pain text

  • e.g., Markdown, rather than Microsoft Word
  • but not exclusively so
    • projects based on binary files would also benefit
    • you may switch to plain text in the future

The Workshop — Motivation

• Research projects have many versions, as they are typically completed

  • over a long period of time
  • in numerous sittings
  • by multiple authors
  • on multiple computers

• With many versions created over time, in different sittings ..., there emerges various challenges

  • keeping track of changes and versions
  • reverting to a previous version when necessary
  • storing versions safely in multiple locations
  • communicating the versions to other authors and/or computers
  • working on the same project with co-authors at the same time

The Workshop — Worklow

The Git-and-GitHub workflow has two steps

• Git creates, saves, and tracks versions

  • on our own machine
  • but also can also save them elsewhere, such as on GitHub

• Github keeps a copy online

  • allowing for safe storage in the cloud
  • but also for collaboration, through a common version

The Workshop — Worklow — Comparison

• The Git-and-GitHub workflow might seem daunting to adopt at first

  • originally designed for software development
    • here, adopted to academic work
  • but offers many benefits over alternatives
    • efficient, free

• Existing workflows might seem easier to keep

  • we all version control and collaborate, one way or another
    • e.g., renaming, saving, and emailing to co-authors
    • using Dropbox, Google Docs, Overleaf
  • but less efficient in the long run

• Give the Git-and-GitHub workflow a chance

  • if you are curious enough to be here on this slide, this is probably the right workflow for you

The Workshop — Contents

The Workshop — Organisation

• Sit in groups of two

  • participants learn as much from their partner as from instructors
  • play the role of co-authors

• Type, rather than copy and paste, the code that you will find on these slides

  • typing is a part of the learning process
  • keyboard shortcuts for copy/paste does not work in the shell

• When you have a question

  • ask your partner
  • google together
  • if it cannot wait, turn your microphone on and ask
  • if it can wait, type it in the chat

The Workshop — Organisation — Slides

• Codes and texts that go in appear as such — in a different font, on gray background

• Results that come out in appear as such — in the same font, on green background

• Specific sections are highlighted yellow as such for emphasis

• The slides are designed for self-study as much as for the workshop
  • accessible, in substance and form, to go through on your own

The Workshop — Aims

• To make you aware what is possible with Git and GitHub

  • we will cover a large breath of issues, not all of it is for long-term memory
    • one reason why the slides are designed for self study as well
      
      
  • awareness of what is possible, Google, and perseverance are all we need

• To encourage you to convert into the Git and GitHub workflow

  • practice with a mock project, co-author
  • start converting a real one right after the workshop
    • happy to help via emails afterwards

Workshop Materials — Overview

git_workshop-materials
   |
   |- data
   |  |
   |  |- journals.csv
   |
   |- manuscript
   |  |
   |  |- journals.docx
   |  |- journals.pdf
   |  |- journals.Rmd
   |
   |- README.md

Workshop Materials — Contents — Data

data\journals.csv

• a dataset created with the fabricatr package (Blair et al., 2019), imagined to explore the Google Scholar rankings of fictitious journals

• includes the following variables

  • name: journals (1090 random titles)
  • origin: geographic origins (five continents)
  • branch: major discipline of journals (four branches)
  • since: time of first publication (years)
  • h5_index: H5 Index (integers)
  • h5_median: H5 Median (integers)
  • english: English (1) vs. other-language (0) journals
  • subfield: subfield (1) vs. generalist (0) journals
  • issues: number of issues published per year (integers)

Workshop Materials — Contents — Manuscript

• manuscript\journals.docx

  • the Microsoft Word document, holding the text for the manuscript of the project

• There are two other versions of paper.docx in the same folder

  • saved in PDF and R Markdwon formats respectively

    • manuscript\journals.pdf
    • manuscript\journals.Rmd

Workshop Materials — Contents — README

README.md

• a Markdown document, holding the basic information about the project

GitHub — Create a new GitHub repository — Notes

There are two types of repositories

• public repositories

  • free for all to create
  • free for everyone else to see, copy, propose changes

• private repositories

  • in general, a paid service
  • with GitHub Education, it free for academics, students
    • requires a separate application

GitHub — Create a personal access token

• Accessing GitHub through the shell requires using a password
  • not the same password to login to GitHub website
  • personal access token (PAT) is the new generation of passwords

• Generate a PAT at https://github.com/settings/tokens
  • note it down for now
  • on how to caching PATs safely, see https://happygitwithr.com/credential-caching.html

Other Resources^*

• Git Reference Manual

  • available at https://git-scm.com/docs

• Pro Git (Chacon and Straub, 2021)

  • open access at https://git-scm.com/book/en/v2

• Git Cheat Sheet

  • available at https://training.github.com/downloads/github-git-cheat-sheet.pdf

• Git Visual Cheat Sheet

  • available at https://ndpsoftware.com/git-cheatsheet.html

Version Control — Shell

Version Control — Git Command Structure

$ git

Version Control — Git Command Structure

$ git push

Version Control — Git Command Structure

$ git --version

Version Control — Git Command Structure

$ git push origin master

Version Control — Git Command Structure

$ git commit -m "I revised the abstract"

Version Control — cd

$ cd

Version Control — cd

$ cd ../pc/Desktop/resul_git_workshop

Note that

• cd is not a git command
  • it does not start with git
    
    
• you can drag and drop the directory onto the shell
  • instead of typing the full path

Version Control — ls

$ ls

Version Control — ls

$ ls data/

Version Control — git init

Use the git init command to turn a directory into a git repository directory

$ git init

Version Control — git init — Notes

.git
  ├── refs
  │   ├── tags
  │   └── heads
  ├── objects
  │   ├── pack
  │   └── info
  ├── info
  │   └── exclude
  ├── hooks
  │   (12 files)
  ├── HEAD
  ├── config
  └── description

Version Control — git add

$ git add README.md

Version Control — git add

$ git add README.md  data/journals.csv

Version Control — git add

$ git add .

Version Control — git add — Notes

• The git add command alone does not create a version
  • it must be followed by the git commit command, without which
    • there is no version saved yet
    • the tracked files are said to be .yellow-h[staged], or in the .yellow-h[staging area]

• Use the git add command (and then, the git commit command)
  • not only to add a new file to be versioned
  • but also, once that file changes, to snapshot a new version of that file every time

• A file in the staging area can be taken back, before its commit
  • by one of the following, depending on whether they have been versioned before

$ git rm --cached README.md

$ git rm --staged README.md

Version Control — git commit

Use the git commit command to take a snaphot of, or to version, a repository

$ git commit -m "adding the README file"

Version Control — git commit

Use the git commit command to take a snaphot of, or to version, a repository

• with the changes in the staging area

$ git commit -m "adding the README file"

Note that

• committing requires registering a message
  • to yourself and to co-authors to indicate what has changed

Version Control — git commit

Use the git commit command to take a snaphot of, or to version, a repository

• with the changes in the staging area

$ git commit -m "adding the README file"

Note that

• committing requires registering a message
  • to yourself and to co-authors to indicate what has changed
  • typed after the –m flag

Recall that

• committing requires one or more files in the staging area in the first place
  • staged with the git add command

Version Control — git status

Use the git status command to check the status of your repository

• e.g., are there any untracked, modified, deleted, staged etc. files?

$ git status

Version Control — git log

Use the git log command to see the changes in reverse order

$ git log

Version Control — git log

Use the git log command to see the changes in reverse order

• the oneline flag returns a summary

$ git log --oneline

Note that

• commit hash is abbreviated to the first 7 digits, enough to uniquely identify each commit
  • i.e., 241b728 from 241b7285a00d47d69655d4a4afcd50989d5b50a8
• HEAD refers to the snapshot of your last commit
• master is the default branch name
  • we have only one branch at the moment, but can have multiple
  • think of a branch as a version of versions, or a copy of the whole repository

Version Control — git branch

Use the git branch command to create a copy of your repository, on your local machine

$ git branch coding_branch

              add &          add &
             commit         commit
master: c1 ----------> c2 ----------> c3
                                      |                            
                                      |                            
                                     b|                              
                                     r|                            
                                     a|                              
                                     n|                            
                                     c|                            
                                     h|                            
                                      |  
                                      v    
                      coding_branch:  c3

Version Control — git branch

Use the git branch command to create a copy of your repository, on your local machine

$ git branch coding_branch

Note that

• this line takes an argument for branch name
  • the default repository is called master

Version Control — git branch

Use the git branch command to create a copy of your repository, on your local machine

$ git branch coding_branch

Note that

• this line takes an argument for branch name
  • the default repository is called master
    
    
• we are still in the master branch
  • check which branch are you in with the following command

$ git branch -v

Version Control — git branch — Notes

• Branches can be developped
  • one at a time

              add &          add &
             commit         commit
master: c1 ----------> c2 ----------> c3
                                      |                             
                                      |                             
                                     b|                               
                                     r|                             
                                     a|                               
                                     n|                             
                                     c|                             
                                     h|                             
                                      |    add &          add &     
                                      v   commit         commit     
                     coding_branch:   c3 ----------> c4 ----------> c5

Version Control — git branch — Notes

• Branches can be developped
  • one at a time
  • at the same time

              add &          add &          add &
             commit         commit         commit
master: c1 ----------> c2 ----------> c3 ----------> c5
                                      |                             
                                      |                             
                                     b|                               
                                     r|                             
                                     a|                               
                                     n|                             
                                     c|                             
                                     h|                             
                                      |    add &          add &     
                                      v   commit         commit     
                     coding_branch:   c3 ----------> c4 ----------> c5

Version Control — git diff

Use the git diff command to see what has changed, for changes

• not staged or committed yet

$ git diff

Version Control — git diff

$ git diff --staged

Version Control — git diff

$ git diff fdcb0ad

$ git diff HEAD~2

Version Control — git diff

$ git diff 241b728 1dfb2a5

$ git diff HEAD~1 HEAD~2

Version Control — git diff

$ git diff 241b728 1dfb2a5 README.md

$ git diff HEAD~1 HEAD~2 README.md

Version Control — git diff

$ git diff master..coding_branch

Version Control — git checkout

$ git checkout 241b728

$ git diff HEAD~1

Version Control — git checkout

$ git checkout 241b728 README.md

$ git checkout HEAD~1 README.md

Version Control — git checkout

$ git checkout coding_branch

Version Control — git merge

Use the git merge command to integrate different versions of your repository into a single version

$ git merge coding_branch

              add &          add &
             commit         commit
master: c1 ----------> c2 ----------> c3 ------------------------> c6
                                      |                             ^
                                      |                             |
                                     b|                             |m  
                                     r|                             |e
                                     a|                             |r  
                                     n|                             |g
                                     c|                             |e
                                     h|                             |
                                      |    add &          add &     |
                                      v   commit         commit     |
                     coding_branch:   c3 ----------> c4 ----------> c5

Version Control — git merge

Use the git merge command to integrate different versions of your repository into a single version

$ git merge coding_branch

Note that

• this line includes one argument
  • from where to merge

Version Control — git merge

Use the git merge command to integrate different versions of your repository into a single version

$ git merge coding_branch

Note that

• this line includes one argument
  • from where to merge
    
    
• this merges into the current branch
  • move into the desired branch first
    • using the git checkout command

Version Control — git merge — Conflicts

• Merge is done automatically if there are no conflicts
  • conflicts emerge when versions to be merged include edits on the same line of the same file
  • edits on different lines are not a problem as changes are tracked line by line

• Merge conflicts require human intervention
  • by editing the conflicting lines
    • after the merge attempt
    • marked in the document with the conflict
  • and commiting the corrected file

Version Control — git remote add

Use the git remote add command to define a different place, to save an backup of your repositories

• e.g., a repository on GitHub
  • could also be a directory on an external hard drive

$ git remote add origin https://github.com/resulumit/git_workshop.git

Version Control — git remote add

Use the git remote add command to define a different place, to save an external copy of your repositories

• e.g., a repository on GitHub
  • could also be a directory on an external hard drive

$ git remote add origin https://github.com/resulumit/git_workshop.git

Note that this includes

• a valid address for the external copy
  • e.g., there is a repository on my GitHub account called git_workshop

Version Control — git remote add

Use the git remote add command to define a different place, to save an external copy of your repositories

• e.g., a repository on GitHub
  • could also be a directory on an external hard drive

$ git remote add origin https://github.com/resulumit/git_workshop.git

Note that this line includes

• a valid address for the external copy to be saved
  • e.g., there is a repository on my GitHub account called git_workshop
• an arbitrary name for this external repository
  • traditionally called origin, could also be anything else

Version Control — git push

Use the git push command to save a copy of your local repository to its remote repository

$ git push origin master

Version Control — git push

Use the git push command to save a copy of your local repository to its remote repository

$ git push origin master

Note that

• this line includes two arguments
  • where to push
    • already defined with the git remote add command

Version Control — git push

Use the git push command to save a copy of your local repository to its remote repository

$ git push origin master

Note that

• this line includes two arguments
  • where and what to push
    • could be any branch

Version Control — git push

Use the git push command to save a copy of your local repository to its remote repository

$ git push origin master

Note that

• this line includes two arguments
  • where and what to push
    
    
• pushing will be rejected if the remote repository has edits that the local repository does not
  • necessitating a pull first

Version Control — git push

Use the git push command to save a copy of your local repository to its remote repository

$ git push origin master

Note that

• this line includes two arguments
  • where and what to push
    
    
• pushing will be rejected if the remote repository has edits that the local repository does not
  • necessitating a pull first
    
    
• if this is your first time using GitHub, you will be prompted to authenticate
  • follow the instructions on your screen and in your email
    • using the PAT that you created in Part 2

Version Control — git pull

Use the git pull command to get a copy of the remote repository, and merge it into the local repository

$ git pull origin master

Version Control — git pull

Use the git pull command to get a copy of the remote repository, and merge it into the local repository

$ git pull origin master

Note that

• this line includes two arguments
  • where to pull from
    • already defined with the git remote add command

Version Control — git pull

Use the git pull command to get a copy of the remote repository, and merge it into the local repository

$ git pull origin master

Note that

• this line includes two arguments
  • where and what to pull from
    • could be any branch

Version Control — git pull

Use the git pull command to get a copy of the remote repository, and merge it into the local repository

$ git pull origin master

Note that

• this line includes two arguments
  • where and what to pull from
    
    
• pulling involves fetching and merging
  • might lead to conflicts, and necessitate solving them manually

What to Version — Overview

There might be good reasons to exclude some others from versioning

• for local-only repositories

  • files that we (re-)create automatically as outputs
    • e.g., paper.pdf, as opposed to paper.Rmd

• for repositories that are also on GitHub

  • files that contain information that we do not want others to see

    • e.g., personal API keys

  • files that we do not have the right to share with others

    • e.g., secondary data with user agreements otherwise

  • files that are too large

    • individual files cannot be larger than 100MB

What to Version — .gitignore

• .gitignore specifies which file(s) and/or folder(s) should be excluded from version control

  • is a file itself, to be saved in the root directory
  • allowing for us to use the git add . shortcut

• .gitignore lists one item per line

  • each line has a pattern, which determines whether one or more files or folders are to be ignored

• See these

  • documentation at https://git-scm.com/docs/gitignore
  • templates at https://github.com/github/gitignore

What to Version — .gitignore — Examples

/manuscript/

What to Version — .gitignore

/manuscript/
/manuscript/paper.pdf

What to Version — .gitignore

/manuscript/
/manuscript/paper.pdf
paper.pdf

What to Version — .gitignore

.pull-left[

/manuscript/
/manuscript/journals.pdf
journals.pdf               
*.pdf

What to Version — .gitignore — Notes

• There are many other pattern formats

  • see the documentation at https://git-scm.com/docs/gitignore

• Starting to ignore a file or folder that is already being tracked requires clearing the cache

  • after changing and saving .gitignore, enter the following line in the shell
  • with your speficic /path/to/file

git rm --cached /path/to/file

• The following command clears all cache

  • might be useful after changes to .gitignore that involves several files or folders
  • but should be used with care, on an otherwise up-to-date repository

git rm -r --cached .

Collaboration — Project Setup

• The setup depends on the users' role, on whether they are
  • the owner who creates the GitHub repository, or
  • the collaborator who is then added to that repository

• Once the project is setup
  • it continues to be associated with the owner's GitHub profile
  • at the same time, it is listed under the collaborator's profile as well
  • both the owner and the collaborator have the same rights, unless otherwise restricted

Collaboration — Project Setup — Owner

• The setup for the owner is largely the same as in any single-author, single-computer scenario
  • introduce version control to a local project with Git,
  • create a remote repository for that project on GitHub, and
  • associate the local project with the remote repository

• As an additional step, the owner needs to invite their collaborator(s) to the project
• following, from the relevant GitHub repository,

Settings -> Manage access -> Invite a collaborator

Collaboration — Project Setup — Collaborator

• Notice that the remote part of the setup is done by the owner for the collaborator
  • subject to acceptance of the invitation
    • invitations are available directly at https://github.com/notifications, but also sent via email
    • on acceptance, projects appear among the repositories of the collaborator

• The local part of the setup still needs to be done
  • by using the git clone command with the repository address

$ git clone https://github.com/USER_NAME/REPOSITORY_NAME.git

Colloboration — Workflow

1) Pull

• on the Git tab in RStudio, click Pull to move the up-to-date records from GitHub to your computer
  • if your collaborator has not pushed anything since your last pull, you will be noticed that Already up-to-date.
  • collaborative projects require pulling as well as pushing because your collaborator(s) might have pushed their commits to GitHub
  • pulling frequently minimises the risk of merge conflicts

2) Edit and save; commit and push

• the same procedure as in any single-author, single-computer scenario
  • as described on this slide forward
• pushing frequently minimises the risk of merge conflicts

Colloboration — Workflow — Alternative

• The workflow above is rather simple, but has some disadvantages, including
  • not easy, albeit still possible, to see the edits of the collaborators
  • not clear who is in charge of the overall progress
  • not possible to discuss edits
  • not possible to compromise on conflicting edits

• An alternative workflow exits
  • work on different branches of the same project
  • version control to your own branch
  • create pull requests with comments
  • merge the branch into master

Colloboration — Workflow — Alternative

1) Branch

• using the git branch command to create a branch

2) Edit and save; commit and push

• the same procedure as in any single-author, single-computer scenario
• except that now pushing into a branch
• notice, on GitHub, that your commit is in the new branch, while master remains unchanged

3) Pull request

• On GitHub, click

  Pull requests -> New pull request

• choose what is to be pulled, and write a note to your collaborator who can accept or reject the merge

  • if there are merge conflicts, the collaborator solves them on GitHub before merging

Applications — RStudio

Integrating RStudio into the workflow requires

1. initial setup, done once^*

   • unless for a new computer or, if ever, a new GitHub account
   • a bit technical, but worth the hassle

2. project setup, repeated for every project

   • shorter, less complicated

Applications — RStudio — Notes

• The principles covered in Part 3 to Part 5 remain the same

  • e.g., add, commit, and push

• There are now two alternative ways to proceed

  • typing git commands in the Terminal

  • as opposed to in a stand alone shell

  • clicking the buttons on the Git tab

  • e.g., Diff, Commit

References

Blair, G., Cooper, J., Coppock, A., Humphreys, M., Rudkin, A. and Fultz, N. (2019). fabricatr: Imagine your data before you collect it. R package, version 0.10.0.