How can I automate things with bash?

13. How can I automate things with bash?#

So far we have used bash commands to navigate our file system as a way to learn about the file system itself. To do this we used commands like:

mv
cd
pwd
ls
find
rm

do you remember what each of those does?, no need to respond, just think through it for yourself

Bash is a unix shell for the GNU operating system and it has been adopted in other contexts as well. It is the default shell on Ubuntu linux as well for example (and many others). This is why we teach it.

A Unix shell is both a command interpreter and a programming language. As a command interpreter, the shell provides the user interface to the rich set of GNU utilities. The programming language features allow these utilities to be combined.

Read the official definition of bash and a shell in the bash manual

flowchart TD subgraph shell pl[programming language] i[command interpreter] end

13.1. Inspecting an example script#

Today we will start by inspecting the github action file, first we’ll navigate to the folder.

Community badge

read in and annotate using myst/jupyter tools the file of the action file with the excerpts that show the different parts

cd fall24-brownsarahm/

gh repo view --web

Opening github.com/compsys-progtools/fall24-brownsarahm in your browser.

13.2. Variables in Bash#

From the action files what do you think the syntax for a variable in bash is? Give an example by creating a variable called MYVAR with the value my_val

We can create variables

MY_NAME='sarah'

notice that there are no spaces around the =. spaces in bash separate commands and options, so they cannot be in variable declarations.

and use them with a $ before the variable name.

echo $MY_NAME 

sarah

This variable is local, in memory, to the current terminal window, so if we open a separate window and try echo $NAME it will not work. We can also see that it does not create any file changes.

A common mistake is to put a space around the = sign, this is actually considered good style in many other languages.

MY_VAR = 'sarah'

-bash: MY_VAR: command not found

In bash, however, this creates an error. When there is a space after MY_VAR, bash tried to interpet MY_VAR as a bash command, but then it does not find it, so it gives an error.

echo $MY_NAME 

sarah

13.3. Environment Variables#

Environment variables are global.

A common one is the PATH

echo $PATH

/Library/Frameworks/Python.framework/Versions/3.12/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/TeX/texbin

The $ is essential syntax for recalling variables in bash. If we forget it, it treats it as a literal

echo PATH

PATH

so we get the variable name out instead of the variable value

the variables do still work if we chnge our path:

cd ../gh-inclass-brownsarahm/

echo $MY_NAME 

sarah

You can store environment variables to be set each time you start a terminal in your profile.

On MacOS this file is: ~/.bash_profile
on linux it is ~/.baschrc

cat ~/.bash_profile

drsmb="/Users/brownsarahm/Documents/web/drsmb-co.github.io"

alias pip=pip3
alias python=python3

export PS1="\u@\W $ "


# Setting PATH for Python 3.12
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.12/bin:${PATH}"
export PATH

A common one you might want to set is:

PS1 the primary prompt line. Content you can use is documented in the gnu docs
alias lets you set another name for a program, for exmaple I use pip to call pip3

13.4. GH Actions as an example of scripts#

Check out the github action marketplace to see other actions that are available and try to get a casual level of understanding of the types of things that people use actions for.

13.5. Bash Loops#

We can also make loops like

for loopvar in list
do
# loop body
echo $loopvar
done

So, for example:

for name in Sarah Aiden Jad
> do
> echo $name
> done

Notes:

The > is not typed, it is what happens since the interpreter knows that after we write the first line, the command is not complete.
a list in bash is items with spaces, no brackets here Sarah Aiden Jad

Sarah
Aiden
Jad

and it outputs each name as expected

When we get the command back with the up arrow key, it puts it all on one line, because it was one command. The ; (semicolon) separates the “lines”

we can change the loop variable to be anything, but good practice is something semantic, unlike the following:

for HI in Sarah Aiden Jad; do echo $HI; done

Sarah
Aiden
Jad

and we can add more to the list

for HI in Sarah Aiden Jad Jordan Johsua; do echo $HI; done

Sarah
Aiden
Jad
Jordan
Johsua

13.6. Nesting commands#

Note: bash provides a date command that returns the date

date

Tue Oct 22 13:14:32 EDT 2024

ls

CONTRIBUTING.md	README.md	scratch.ipynb	src
LICENSE.md	docs		setup.py	tests

We can run a command to generate the list by putting it inside $() to run that command first. Think kind of like PEMDAS and the $ in bash is for variables.

for file in  $(ls)
> do
> echo $file
> done

the $() tells bash to run that command first and then hold its output as a variable for use elsewhere

CONTRIBUTING.md
LICENSE.md
README.md
docs
scratch.ipynb
setup.py
src
tests

13.7. Conditionals in bash#

We can also do conditional statements

if test -f docs
> then
> echo "file"
> fi

the key parts of this:

test checks if a file or directory exists
the -f option makes it check if the item is a file
what to do if the condition is met goes after a then keyword
the fi (backwardsif) closes the if statment

this outputs nothing, because docs is a folder, but if we change the file it checks:

if test -f README.md; then echo "file"; fi

file

We can put the if inside of the loop, but this is getting long so typing it all without errors is hard.

13.8. Script files#

We can put our script into a file

nano filecheck.sh

cat filecheck.sh 

for file in $(ls)
do
  if test -f $file
  then
    echo $file "is a file"
  fi
done

and run it with bash <filename>

bash filecheck.sh 

CONTRIBUTING.md is a file
LICENSE.md is a file
README.md is a file
filecheck.sh is a file
scratch.ipynb is a file
setup.py is a file

the extension can be anything

mv filecheck.sh filecheck

bash filecheck

CONTRIBUTING.md is a file
LICENSE.md is a file
README.md is a file
filecheck is a file
scratch.ipynb is a file
setup.py is a file

and it still works

but the extension is good practice, so we will put it back

mv filecheck filecheck.sh

13.9. `gh` CLI operations#

When you are working sometimes it is helpful to be able to manipulate (or create) issues, pull requests or even releases from the command line.

This is how I post announcements. I work on the notes (a markdown file) in vs code and then I use the vscode terminal to commit, push, create a tag, and create a release. I can post the notes and notify you all that they are posted without leaving VScode at all; this makes it much simpler/faster than it would be using Brightspace

We can also search and filter them by piping the output to grep which searches the contents of a file (including stdin). We previously searched the file names with find. So find searches the paths that exist and grep actually reads the contents of the files, it does so faster than many other languages would be.

gh issue list -s all

Showing 3 of 3 issues in compsys-progtools/gh-inclass-brownsarahm that match your search

ID  TITLE                     LABELS  UPDATED          
#3  Create an about file              about 1 month ago
#2  Create a Add a classmate          about 1 month ago
#1  Create a README                   about 1 month ago

let’s create one more issue

gh issue create

Creating issue in compsys-progtools/gh-inclass-brownsarahm

? Title question
? Body <Received>
? What's next? Submit
https://github.com/compsys-progtools/gh-inclass-brownsarahm/issues/6

no body, title “question”

gh issue list -s all

Showing 4 of 4 issues in compsys-progtools/gh-inclass-brownsarahm that match your search

ID  TITLE                     LABELS  UPDATED               
#6  question                          less than a minute ago
#3  Create an about file              about 1 month ago
#2  Create a Add a classmate          about 1 month ago
#1  Create a README                   about 1 month ago

grep can be used with pattern matching as well

Learning more about grep is a good explore badge topic

gh issue list -s all | grep "Create"

CLOSED	Create an about file		2024-09-12T17:21:42Z
OPEN	Create a Add a classmate		2024-09-12T17:18:02Z
CLOSED	Create a README		2024-09-17T16:50:52Z

We can use awk to separate out only the number from the output

gh issue list -s all | awk '{print $1}'

we can also use multiple pipes

gh issue list -s all | grep "Create" | awk '{print $1}'

3
2
1

and we can put htis in a script

for issue in $(gh issue list -s all | grep "Create" | awk '{print $1}')
do
gh issue view $issue >> combined_issues.md
done

then run that script:

bash combine.sh

and loook at the output

cat combined_issues.md

title:	Create an about file
state:	CLOSED
author:	github-actions
labels:	
comments:	0
assignees:	
projects:	
milestone:	
number:	3
--
Add a file `about.md` that has your name and expected graduation in it 
follow along to close this issue with  commit.
title:	Create a Add a classmate
state:	OPEN
author:	github-actions
labels:	
comments:	0
assignees:	
projects:	
milestone:	
number:	2
--
owner:
- [ ] give a class mate access to the repo
- [ ] assign this issue to them

classmate:
- [ ] add `classmate.md` with your name and expected graduation on  a bracn `classmate`
- [ ] open a PR that will close this issue
```
title:	Create a README
state:	CLOSED
author:	github-actions
labels:	
comments:	0
assignees:	brownsarahm
projects:	
milestone:	
number:	1
--
A README on GitHub, is ideally a markdown file, github will render that nicely

### Steps
- [x] Add self as assignee to this issue
- [x] check off another item
- [x] use edit to see how they look in plain text mode, the check this off there

create the file and close this issue with your commit message

contents:
```
# GitHub Practice

Name: <your name here>
```

13.10. Prepare for Next Class#

think about what you know about networking
make sure you have putty if using windows
get the big ideas of hpc, by reading this IBM intro page and some hypothetical people who would attend an HPC carpentry workshop. Make a list of key terms as an issue comment

13.11. Badges#

Review

Update your KWL Chart learned column with what you’ve learned
write a bash script to make it so that cating the the files in your gh_inclass repo does not make the prompt move

Practice

Update your KWL Chart learned column with what you’ve learned
prevent badges.json and badges.yml from being tracked by git in your fall24 repo
write a bash script that prevents jupyter-book from giving warnings about files not having a heading by using the file name as a temporary title. (see the badge from 10/3 where you should have converted your repo, or do that one first, this counts as an extension on that if you have not done it)

Hint: Use sed’s insert option and head as needed.

Explore idea

modify your script to use a small llm from ollama to automatically insert a sensible title by summarizing the file. using a commercial chat interface does not qualify, but using an llm locally does

13.12. Experience Report Evidence#

13.13. Questions After Today’s Class#

13.13.1. Explore topics from questions#

zsh vs bash

13.13.2. Where can I find the documentation for all bash syntax?#

the bash manual is the official reference

13.13.3. How do we know when to use $ in bash when using varaibles?#

Whenever you want to use the value of a variable, not declare it.

13.13.4. How can I look at the bash profile on windows? As the commands for Linux and MacOS didn’t work when we looked at /.bash_profile.#

I think by default it does not rely on one, but you can create one, you should create the file .bash_profile

13.13.6. What is the use for `#!/bin/bash` Why is it not needed all the time?#

#!/bin/bash tells the interpretter which shell to use. Since we ran our file like bash <filename> we did not need it.

We would need it for running like ./<filename>

the #! is calked the shebang. more in the bash docs