5. When do git and bash give me an advantage?#
so far we have used git and bash to accomplish familiar goals, and git and bash feel like just extra work for familiar goals.
Today, we will start to see why git and bash are essential skills: they give you efficiency gains and time traveling super powers (within your work, only, sorry)
5.1. Setting the stage#
Discuss your prepare work with your neighbors. Share what project you looked at and discuss the following. If you all looked at the same repo, split up and look at different ones now.
What things were common across different repos?
What was different?
5.2. Important references#
Use these for checking facts and resources.
5.3. Setup#
First, we’ll go back to our github inclass folder
cd gh-inclass-sp24-brownsarahm/
Find your PR that I opened for you today that has the title, “2/6 in class activity”
I am going to use the gh
CLI for this. It might not work on Windows unless you got gh and GitBash configured correctly.
gh pr list
Showing 1 of 1 open pull request in compsys-progtools/gh-inclass-sp24-brownsarahm
#5 2/6 in class activity organizing_ac about 1 hour ago
It shows me the number of total PRs and then for each one:
the PR number (remember there is one sequence of numbers for issues and PRs)
PR title
branch name
when it was created
Mine was 5, so I can view it with this command:
gh pr view 5
2/6 in class activity #5
Draft • brownsarahm wants to merge 1 commit into main from organizing_ac • about 1 hour ago
+17 -0 • No checks
this draft PR https://github.blog/2019-02-14-introducing-draft-pull-
requests/ adds some example files that we will use during class
on 2/6. \n\n wait until class time (or when you make up class
by following the notes) to do anything with this PR
View this pull request on GitHub: https://github.com/compsys-progtools/gh-inclass-sp24-brownsarahm/pull/5
by default it shows the contents of the PR message and then the link to view in browser
Alternatively, we can use the --web
option to open it in browser.
gh pr view 5 --web
Opening github.com/compsys-progtools/gh-inclass-sp24-brownsarahm/pull/5 in your browser.
Then we merged in the PR after converting it from a draft PR to a regular PR using the “Ready for Review” button.
Then back locally, we check where we are
git status
On branch fun_fact
Your branch is up to date with 'origin/fun_fact'.
nothing to commit, working tree clean
5.4. Organizing a project (workign with files)#
A common question is about how to organize projects. While our main focus
in this class session is the bash
commands to do it, the task that we are
going to do is to organize a hypothetical python project
Put another way, we are using organizing a project as the context to motivate practicing with bash commands for moving files.
A different the instructor might go through a slide deck that lists commands and describes what each one does and then have examples at the end. Instead, we are going to focus on organizing files, and I will introduce the commands we need along the ways.
We’ll go back to main first
git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
Then we will get the update from main.
git pull
remote: Enumerating objects: 20, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 19 (delta 1), reused 17 (delta 0), pack-reused 0
Unpacking objects: 100% (19/19), 2.46 KiB | 114.00 KiB/s, done.
From https://github.com/compsys-progtools/gh-inclass-sp24-brownsarahm
1e2a45f..9f39946 main -> origin/main
* [new branch] organizing_ac -> origin/organizing_ac
Updating 1e2a45f..9f39946
Fast-forward
API.md | 1 +
CONTRIBUTING.md | 1 +
LICENSE.md | 1 +
_config.yml | 1 +
_toc.yml | 1 +
abstract_base_class.py | 1 +
alternative_classes.py | 1 +
example.md | 1 +
helper_functions.py | 1 +
important_classes.py | 1 +
philosophy.md | 1 +
scratch.ipynb | 1 +
setup.py | 1 +
test_abc.py | 1 +
tests_alt.py | 1 +
tests_helpers.py | 1 +
tests_imp.py | 1 +
17 files changed, 17 insertions(+)
create mode 100644 API.md
create mode 100644 CONTRIBUTING.md
create mode 100644 LICENSE.md
create mode 100644 _config.yml
create mode 100644 _toc.yml
create mode 100644 abstract_base_class.py
create mode 100644 alternative_classes.py
create mode 100644 example.md
create mode 100644 helper_functions.py
create mode 100644 important_classes.py
create mode 100644 philosophy.md
create mode 100644 scratch.ipynb
create mode 100644 setup.py
create mode 100644 test_abc.py
create mode 100644 tests_alt.py
create mode 100644 tests_helpers.py
create mode 100644 tests_imp.py
next we are going to pretend we worked on the project and made a bunch of files
I gave a bunch of files, each with a short phrase in them.
none of these are functional files
the phrases mean you can inspect them on the terminal
We can view the contents of a file with cat
.
cat
concatenates the contents of a file to stdout, which is a special file that our terminal reads
cat setup.py
file with function with instructions for pip
Important
file extensions are for people; they do not specify how the file is actually written . these are all actually plain text files, despite different file exensions.
Before we start organizing, we will make a new branch.
git checkout -b organization
Switched to a new branch 'organization'
git status
On branch organization
nothing to commit, working tree clean
5.5. Files, Redirects, git restore#
Let’s first look at the README file
cat README.md
# GitHub Practice
Name: Sarah Brown
Echo allows us to send a message to stdout.
echo "hello"
hello
Now we are going to put these two things together, but first let’s connect what we are about to learn to something you have probably seen before.
Think about a time you opened a file within a program that you wrote. For example
fopen
in Cor
open
in Python
in both cases one parameter is the file to open, what other parameters have you used?
Typically we also have to tell it what mode to open the file with, and some options are:
read
write
append
We can also redirect the contents of a command from stdout to a file in bash
. Like file operations while programming there is a similar concept to this mode.
There are two types of redirects, like there are two ways to write to a file, more generally:
write (
>
)append (
>>
)
We will use append first.
We can add contents to files with echo
and >>
echo "hello" >> README.md
This command does not return anything, because instead of writing to stdout, it wrote to the file.
We can see the impact by looking at the README file.
cat README.md
# GitHub Practice
Name: Sarah Brown
hello
We can redirect other commands too:
git status >> curgit
we see this created a new file
ls
API.md example.md
CONTRIBUTING.md helper_functions.py
LICENSE.md important_classes.py
README.md philosophy.md
_config.yml scratch.ipynb
_toc.yml setup.py
about.md test_abc.py
abstract_base_class.py tests_alt.py
alternative_classes.py tests_helpers.py
curgit tests_imp.py
and we can look at its contents too
cat curgit
On branch organization
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: README.md
Untracked files:
(use "git add <file>..." to include in what will be committed)
curgit
no changes added to commit (use "git add" and/or "git commit -a")
This also highlights a bit more about how the redirect works.
The curgit
file was created first, and then the command was run, with its output goign to that file instead of to stdout.
Important
Note this also explains that <
is for redirecting inputs (stdin is also a file like stdout and has input, like your keyboard)
this is not a file we actually want, which gives us a chance to learn another new bash command:
rm
for remove
rm curgit
Important
Note that this is a true, full, and complete DELETE, this does not put the file in your recycling bin or the apple trash can that you can recover the file from, it is gone for real.
We will see soon a way around this, because git can help.
use rm
with great care
git status
On branch organization
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: README.md
no changes added to commit (use "git add" and/or "git commit -a")
Now we have made some changes we want, so let’s commit our changes.
When we want to commit all files that are changed or deleted we can use the -a
option. This does not do anything to untracked files. Thi option is short for all, and can also be used like --all
, so you cannot pick a subset of files out with this option.
git commit -a -m 'add to readme'
[organization 29ffc88] add to readme
1 file changed, 1 insertion(+)
and we confirm that it is what we want
git status
On branch organization
nothing to commit, working tree clean
clean working area, ready to keep going
Now, let’s go back to thinking about redirects. We saw that with two >>
we appended to the file. With just one what happens?
echo "hello" > README.md
Now we look at the file:
cat README.md
hello
It wrote over. This would be bad, we lost content, but this is what git is for!
It is very very easy to undo work since our last commit.
This is good for times when you have something you have an idea and you do not know if it is going to work, so you make a commit before you try it. Then you can try it out. If it doesn’t work you can undo and go back to the place where you made the commit.
We can also use git status
for a hint.
git status
On branch organization
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: README.md
no changes added to commit (use "git add" and/or "git commit -a")
Notice that it tells us what to do (use "git restore <file>..." to discard changes in working directory)
. The version of README.md that we broke is in the working directory but not commited to git, so git refers to them as “changes” in the working directory.
Let’s follow the hint
git restore README.md
First we will check with git
git status
On branch organization
nothing to commit, working tree clean
back to a clean working tree is a good sign!
and it looks like it did before the >
line. and we can check the file too
cat README.md
# GitHub Practice
Name: Sarah Brown
hello
back how we wanted it!
5.6. What if I forget the -m
#
Now we will add some text to the README so we have a single point of reference about what these files are “about”
echo "|file | contents |
> > | ------| ------- |
> > | abstract_base_class.py | core abstract classes for the project |
> > | helper_functions.py | utitly funtions that are called by many classes |
> > | important_classes.py | classes that inherit from the abc |
> > | alternative_classes.py | classes that inherit from the abc |
> > | LICENSE.md | the info on how the code can be reused|
> > | CONTRIBUTING.md | instructions for how people can contribute to the project|
> > | setup.py | file with function with instructions for pip |
> > | test_abc.py | tests for constructors and methods in abstract_base_class.py|
> > | tests_helpers.py | tests for constructors and methods in helper_functions.py|
> > | tests_imp.py | tests for constructors and methods in important_classes.py|
> > | tests_alt.py | tests for constructors and methods in alternative_classes.py|
> > | API.md | jupyterbook file to generate api documentation |
> > | _config.yml | jupyterbook config for documentation |
> > | _toc.yml | jupyter book toc file for documentation |
> > | philosophy.md | overview of how the code is organized for docs |
> > | example.md | myst notebook example of using the code |
> > | scratch.ipynb | jupyter notebook from dev |" >> README.md
this explains each file a little bit more than the name of it does. We see there are sort of 5 groups of files:
about the project/repository
code that defines a python module
test code
documentation
extra files that “we know” we can delete.
We also learn something about bash:
using the open quote "
then you stay inside that until you close it. when you press enter the command does not run until after you close the quotes
git status
On branch organization
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: README.md
no changes added to commit (use "git add" and/or "git commit -a")
This time we will add separately
git add .
and commit without using -m
git commit
This brings us into vim. vim is a ubiquitous, but not novice friendly terminal text editor.
What is is doing is making a temporary file for you, opening it in the vim editor and waiting for you to write a commit message.
Important
What you need to know about vim:
it has 2 modes, command and edit
in command mode pressing keys does not enter text, but does commands
in edit mode, it says
~insert~
at the bottomto get into edit mode press a
to get into command mode press escape
to exit type
:wq
and press enter or return
Once you edit your temp file so that there is a line without a #
that has your commit message and exit, your commit gets created.
[organization 260c9c3] describe the files
1 file changed, 19 insertions(+)
5.7. Moving files and patterns#
First, we’ll make a directory with mkdir
mkdir docs
We can see that it worked with ls
ls
API.md example.md
CONTRIBUTING.md helper_functions.py
LICENSE.md important_classes.py
README.md philosophy.md
_config.yml scratch.ipynb
_toc.yml setup.py
about.md test_abc.py
abstract_base_class.py tests_alt.py
alternative_classes.py tests_helpers.py
docs tests_imp.py
next we will move a file there with mv
mv philosophy.md docs/
what this does is change the path of the file from .../gh-inclass-sp24-brownsarahm/philosophy.md
to .../gh-inclass-sp24-brownsarahm/docs/philosophy.md
Important
moving a file means changing its path; it does not rewrite the file.
While it is useful to think of a path like an address, a path is not tied to a specific memory location on the disk. The operating system keeps an index of the human readable paths that relates them to atual memory locations on the storage device.
mv
updates that index, not the actual storage device.
This doesn’t return anything, but we can see the effect with ls
ls
API.md example.md
CONTRIBUTING.md helper_functions.py
LICENSE.md important_classes.py
README.md scratch.ipynb
_config.yml setup.py
_toc.yml test_abc.py
about.md tests_alt.py
abstract_base_class.py tests_helpers.py
alternative_classes.py tests_imp.py
docs
the philosophy.md
file is no longer visible in the current working directory.
We can also use ls
with a relative or absolute path of a directory to list tht location instead of our current working directory.
ls docs/
philosophy.md
5.7.1. Move can also rename files#
We have one file with test_
and a bunch of others with tests_
. We want them all to follow the same pattern, so let’s rename.
We can use mv
to change the name as well. This is because “moving” a file and
is really about changing its path, not actually copying it from one location to
another and the file name is a part of the path.
mv test_abc.py tests_abc.py
This changes the path from .../test_abc.py
to .../tests_abc.py
to. It is doing the same thing as when we use it to move a file from one folder to another folder, but changing a different part of the path.
ls
API.md example.md
CONTRIBUTING.md helper_functions.py
LICENSE.md important_classes.py
README.md scratch.ipynb
_config.yml setup.py
_toc.yml tests_abc.py
about.md tests_alt.py
abstract_base_class.py tests_helpers.py
alternative_classes.py tests_imp.py
docs
5.7.2. Moving multiple files with patterns#
let’s look at the list of files again.
ls
API.md example.md
CONTRIBUTING.md helper_functions.py
LICENSE.md important_classes.py
README.md scratch.ipynb
_config.yml setup.py
_toc.yml tests_abc.py
about.md tests_alt.py
abstract_base_class.py tests_helpers.py
alternative_classes.py tests_imp.py
docs
We have a lot of tests_
files, we will put them all in one folder. First we create the folder:
mkdir tests
We can use the *
wildcard operator to move all files that match the pattern.
mv tests_* tests/
Again, we confirm it worked by seeing that they are no longer in the working directory.
ls
API.md alternative_classes.py
CONTRIBUTING.md docs
LICENSE.md example.md
README.md helper_functions.py
_config.yml important_classes.py
_toc.yml scratch.ipynb
about.md setup.py
abstract_base_class.py tests
and that they are in tests
ls tests/
tests_abc.py tests_helpers.py
tests_alt.py tests_imp.py
success!
We will do the same thing for the .yml
files to move them to the docs folder
mv *.yml docs/
Next we are going to move the Python files that are for the “package” that we have made. We will make a folder for this too.
mkdir pkg
Here we want to move all of the .py
files that have a _
in the name, because the only one we do not want to move is setup.py
mv *_*.py
usage: mv [-f | -i | -n] [-v] source target
mv [-f | -i | -n] [-v] source ... directory
I made an error here so it gave me usage for the command.
mv
requires both a source and target, and I did not provide one.
Once I fix it
mv *_*.py pkg/
we see that it works.
Warning
Some students got a different error message, instead of being told they
had used the command wrong, their error message said that important_classes.py
is not a directory. That tells me that what happened was the *_*.py
was expanded to a list of files and then mv
tried to work on the list of files using all but the last one as sources and tried to use the last one as the destination.
To debug, I tried comparing bash versions, but it appears that we had the same version, so why we saw different behavior is TBD.
this is worth a community badge if you can explain why
bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)
Copyright (C) 2007 Free Software Foundation, Inc.
5.9. Summary#
stdout is a file, that is displayed on the terminal
“moving” a file does not re-write the data to a different part of the disk, it updates its adress only
the wildcard operator
*
allows us to use patterns with bash commandstouch
can accept a list of filesbash lists are space delimited without any brackets
the
.gitignore
file prevents files from being in your repo
5.9.1. New commands#
command |
explanation |
---|---|
|
repeat ‘message’ to stdout |
|
write redirect |
|
append redirect |
|
remove (delete) |
5.9.2. New git commands#
command |
explanation |
---|---|
|
the |
5.10. Prepare for next class#
Bring git questions or scenarios you want to be able to solve to class on Thursday (in your mind or comment here if that helps you remember)
Practice using the new bash commands we have learned to read and understand the workflow files in your KWL repo. Look for bash commands that we have seen before. Try to modify files on a prepare branch so that your name is already filled in when your experience badge (inclass) action runs.
5.11. Badges#
badge steps marked lab are steps that you will be encouraged to use lab time to work on. In this case, in lab, we will check that you know what to do, but if we want you to do revisions those will be done through the badge.
Update your KWL chart with the new items and any learned items.
Clone the course website. Append the commands used and the contents of your
fall2023/.git/config
to a terminal_review.md (hint: history outputs recent commands and redirects can work with any command, not only echo). Edit the README.md, commit, and try to push the changes. What happens and what GitHub concept that we have not used yet might fix it? see yourvocab-
repo for a list of key github concepts. (answer in theterminal_review.md
)lab Organize the provided messy folder in a Codepsace (details will be provided in lab time). Commit and push the changes. Answer the questions below in your kwl repo in a file called
terminal_organization.md
clone your
messy_repo
locally and append thehistory.md
file to yourterminal_organization.md
badge steps marked lab are steps that you will be encouraged to use lab time to work on. For this one in particular, I am going to give you the messy repo in lab.
Update your KWL chart with any learned items.
Get set up so that you can pull from the course website repo and push to your own fork of the class website by cloning the main repo, then forking it and adding your fork as an additional remote. Append the commands used and the contents of your
spring2024/.git/config
to a terminalpractice.md (hint:history
outputs recent commands and redirects can work with any command, not only echo). Based on what you know so far about forks and branches, what advantage does this setup provide? (answer in theterminal_practice.md
)lab Organize the provided messy folder (details will be provided in lab time). Commit and push the changes. Clone that repo locally.
Organize a folder on your computer ( good candidate may be desktop or downloads folder), using only a terminal to make new directories, move files, check what’s inside them, etc. Answer reflection questions in a new file, terminal_organization_adv.md in your kwl repo. Tip: Start with a file explorer open, but then try to close it, and use only command line tools to explore and make your choices. If you get stuck, look up additional commands to do acomplish your goals.
# Terminal File moving reflection
1. How was this activity overall Did this get easier toward the end?
2. How was it different working on your own computer compared to the Codespace form?
3. Did you have to look up how to do anything we had not done in class?
4. When do you think that using the terminal will be better than using your GUI file explorer?
5. What questions/challenges/ reflections do you have after this?
6. Append all of the commands you used in lab below. (not from your local computer's history, from the codespace history)
5.12. Experience Report Evidence#
redirect the history to a file
history >> makeup_2023-09-21.md
then move the file created to your KWL repo on your experience report branch.
5.13. Questions After Today’s Class#
5.13.1. Why exactly does the mv command also work as the rename function?#
There is also a rename
utility, but mv
can do both because “moving” a file is just changing its path.
5.13.2. Can you access the . folders not using terminal?#
Yes, but I do not know the specifics for each operating system. Usually I think you can right click or control a setting somewhere.
5.13.3. Why do some open source projects have a lot of open issues, maybe even more issues than files?#
Some open source projects get innundated with bug reports or questions. Some projects even use issues for other things. For example, the Carpentries trainers repo uses issues to keep track of proposals for running a leadership committee of a group of people. The journal of open source software uses the joss-reviews repo to do peer reviews, it has 12 files and 291 open issues + 6017 closed issues.
5.13.4. How can the “*” be used consistently to move a bunch of file? because we used it with the text infront of the similarity and behind it. i’m just wondering how it’s used for multiple files, because it was used both before the similarity and after the similarity.#
It fills in for any number of characters.
5.13.5. I have some badges from 2 weeks ago that are awaiting a second review after changes had been fixed.#
Important
Do not merge an unapproved badge
Re-request a review
5.13.6. can you close pull requests on the terminal?#
Not with git because pull requests are not a git feature, but it is a feature of github. The gh
CLI can do this.
5.13.7. If you do mv * will it move all files?#
In the current working directory.
5.13.8. Are there any other ways of using mv that haven’t been covered yet?#
No those are basically the two purposes.
5.13.9. how often should we be practicing with the terminal for git?#
Ideally, you work on badges on at least several of the days we do not have class so that you are working with it close to every day.
You could also start trying to use them for your other classes.
5.13.10. Would it be beneficial to organize files with github rather than bash?#
GitHub cannot organize files and doing do in broswer would be slow and difficult. We will see that GitHub code spaces give us a virtual machine that we can work with.