How can I automate things with bash?

14. How can I automate things with bash?#

So far we have used bash commands to navigate our file system as a way to learn about the file system itself. To do this we used commands like:

  • mv

  • cd

  • pwd

  • ls

  • find

  • rm

Bash is a unix shell for the GNU operating system and it has been adopted in other contexts as well. It is the default shell on Ubuntu linux as well for example (and many others). This is why we teach it.

flowchart TD shell --> |is accessed through| terminal bash --> |is instance of| shell zsh --> |is instance of| shell git --> |is a program accessed though | shell

A Unix shell is both a command interpreter and a programming language. As a command interpreter, the shell provides the user interface to the rich set of GNU utilities. The programming language features allow these utilities to be combined.

Read the official definition of bash and a shell in the bash manual

flowchart TD subgraph shell pl[programming language] i[command interpreter] end

14.1. Inspecting an example script#

Today we will start by inspecting the github action file, first we’ll navigate to the folder.

cd kwl-sp24-brownsarahm/

Next we’ll look at which files we have

ls .github/workflows/

create_about.yml	experiencereport.yml	getassignment.yml
experienceinclass.yml	forgottenexperience.yml

Then we will look at the getassignmet one.

cat .github/workflows/getassignment.yml 

name: Create badge issues (Do not run manually)
on:
  workflow_dispatch

# once you edit, change the name

jobs:
  check-contents:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    
    # Install dependencies
    - name: Set up Python 3.9
      uses: actions/setup-python@v4
      with:
        python-version: 3.9
    
    - name: Install Utils
      run: |
        pip install git+https://github.com/introcompsys/courseutils@main
    - name: Get badge requirements
      run: |
        # prepare badge lines
        pretitle="prepare-"$(sysgetbadgedate --prepare)
        sysgetassignment --type prepare | gh issue create --title $pretitle --label prepare --body-file -
        # review badge lines
        rtitle="review-"$(sysgetbadgedate --review)
        sysgetassignment --type review | gh issue create --title $rtitle --label review --body-file -
        # practice badge lines
        pratitle="practice-"$(sysgetbadgedate --practice)
        sysgetassignment --type practice| gh issue create --title $pratitle --label practice --body-file -
      env:
        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
#  edit the run step above for the level(s) you want. 
#  You should keep the prepare, because they are required for experience badges 
#     You may choose to get only the review or only the practice (and change this any time)

14.2. Variables in Bash#

We can create variables

MYVAR="my_val"

notice that there are no spaces around the =. spaces in bash separate commands and options, so they cannot be in variable declarations.

and use them with a $ before the variable name.

echo $MYVAR

This variable is local, in memory, to the current terminal window, so if we open a separate window and try echo $NAME it will not work. We can also see that it does not create any file changes.

my_val

A common mistake is to put a space around the = sign, this is actually considered good style in many other languages.

NAME = "Sarah"

-bash: NAME: command not found

In bash, however, this creates an error. When there is a space after NAME, bash tried to interpet NAME as a bash command, but then it does not find it, so it gives an error.

Removing the space works again:

NAME="Sarah"
echo $NAME

Sarah

echo $NAME

Sarah

The $ is essential syntax for recalling variables in bash. If we forget it, it treats it as a literal

echo NAME

echo NAME

NAME

so we get the variable name out instead of the variable value

14.3. Environment Variables#

Environment variables are global.

A common one is the PATH

echo $PATH

/Users/brownsarahm/anaconda3/bin:/Users/brownsarahm/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin

mine has my anaconda install, yours may be different

You can store environment variables to be set each time you start a terminal in your profile.

  • On MacOS this file is: ~/.bash_profile

  • on linux it is ~/.baschrc

cat ~/.bash_profile

alias pip=pip3
alias python=python3

export PS1="\u@\W $ "

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/brownsarahm/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/Users/brownsarahm/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/Users/brownsarahm/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/Users/brownsarahm/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

A common one you might want to set is:

  • PS1 the primary prompt line. Content you can use is documented in the gnu docs

  • alias lets you set another name for a program, for exmaple I use pip to call pip3

14.4. Bash Loops#

We can also make loops like

for loopvar in list
do
# loop body
echo $loopvar
done

So, for example:

for name in Sarah Cam Kai
> do 
> echo $name
> done

Notes:

  • The > is not typed, it is what happens since the interpreter knows that after we write the first line, the command is not complete.

  • a list in bash is items with spaces, no brackets here ( Sarah Cam Kai)

Sarah
Cam
Kai

and this does as expected

When we get the command back with the up arrow key, it puts it all on one line, because it was one command. The ; (semicolon) separates the “lines”

for name in Sarah Cam Kai; do  echo $name; done

Sarah
Cam
Kai

14.5. Nesting commands#

Let’s look at the action file again

name: Create badge issues (Do not run manually)
on:
  workflow_dispatch

# once you edit, change the name

jobs:
  check-contents:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    
    # Install dependencies
    - name: Set up Python 3.9
      uses: actions/setup-python@v4
      with:
        python-version: 3.9
    
    - name: Install Utils
      run: |
        pip install git+https://github.com/introcompsys/courseutils@main
    - name: Get badge requirements
      run: |
        # prepare badge lines
        pretitle="prepare-"$(sysgetbadgedate --prepare)
        sysgetassignment --type prepare | gh issue create --title $pretitle --label prepare --body-file -
        # review badge lines
        rtitle="review-"$(sysgetbadgedate --review)
        sysgetassignment --type review | gh issue create --title $rtitle --label review --body-file -
        # practice badge lines
        pratitle="practice-"$(sysgetbadgedate --practice)
        sysgetassignment --type practice| gh issue create --title $pratitle --label practice --body-file -
      env:
        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
#  edit the run step above for the level(s) you want. 
#  You should keep the prepare, because they are required for experience badges 
#     You may choose to get only the review or only the practice (and change this any time)

We can run a command to generate the list by putting it inside $() to run that command first. Think kind of like PEMDAS and the $ in bash is for variables.

This is useful in combination with the the loop

for file in $(ls)
> do 
> echo $file 
> done

README.md
community_contributions.md
experiences

the $() tells bash to run that command first and then hold its output as a variable for use elsewhere

If we modify it

for file in $(ls -a); do  echo $file ; done 

.
..
.git
.github
.templates
README.md
community_contributions.md
experiences

test hello

14.6. Conditionals in bash#

We can also do conditional statements

if test -f .git
> then
> echo "file"
> fi

the key parts of this:

  • test checks if a file or directory exists

  • the -f option makes it check if the item is a file

  • what to do if the condition is met goes after a then keyword

  • the fi (backwardsif) closes the if statment

This returns nothing because it is not a file

If we switch it, we get output:

if test -f README.md; then echo "file"; fi

file

We can put the if inside of the loop.

if test  .git; then echo "exists"; fi

exists

echo $(test hello)



for file in $(ls -a); do if test -f $file ;then  echo $file; fi ; done

README.md
community_contributions.md

14.7. Script files#

We can put our script into a file

nano filecheck.sh

So that the file

cat filecheck.sh 

for file in $(ls)
do 
  if test -f $file
  then
     echo $file
  fi
done

and run it with bash <filename>

bash filecheck.sh 

README.md
community_contributions.md
filecheck.sh

cd ../



ls

ex				spring2024
gh-inclass-sp24-brownsarahm	test
kwl-sp24-brownsarahm		tiny-book

cd spring2024/

14.8. Prepare for Next Class#

Review your idethoughts.md from a few weeks ago and add some summary notes.

14.9. Badges#

  1. Update your KWL Chart learned column with what you’ve learned

  2. Write a bash script that updates your badges.json file and then generates your progress report using courseutils we will use this in lab on 3/25

  1. Update your KWL Chart learned column with what you’ve learned

  2. Write a bash script that updates your badges.json file and then generates your progress report using courseutils we will use this in lab on 3/25

  3. Update your action file for getting assignments to use the new courseutils (at compsys-progtools instead of introcompsys)

14.10. Experience Report Evidence#

14.11. Questions After Today’s Class#

14.11.1. What parts of the terminal are saved local per window and what parts aren’t?#

local variables local, but there are also global variables

most programs should be per window.

14.11.2. Are there functions in bash scripting?#

Yes see the docs

14.11.3. what is YMAL syntax similar to in terms of other programming langauges#

It is kind of like a dicionary, but yaml is just a file type, like json or xml, not a a programming language. It is a way to store data that is parse-able.

14.11.4. What kind of action could be useful?#

Anything you might do a alot and would prefer to have automated. Fore xample in calss we have the actions for the badges, the market place has examples.

there are also the ideas in the discussions.

14.11.5. How to stop a loop (break statement)#

break exists in bash

14.11.6. How to check values (greater than/less than)#

those are operators defines on the conditional page

14.11.7. What differentiates a terminal and a shell?#

a shell is the language/interpreter terminal is how you access it.

Sometimes they collapse like powershell is a terminal and shell.

14.11.8. How can I change the experience workflow to automatically fill in the date?#

This is nontrival because you would need to edit the file instead of just copying it. It is a good build badge.

Contents