Command the Command Line

Part III - Improving Your Workflow

Customization

In this chapter, we'll look at various ways to customize the environment provided by a Unix-like system. These techniques can help maintain awareness of the system state, increase productivity, or simply make the terminal look nicer.

Re-cap:

vm$ sh ./greet.sh
Hello.
vm$ ./greet.sh
./greet.sh: Permission denied
vm$ chmod +x ./greet.sh
vm$ ./greet.sh
Hello.
vm$

In Chapter 13 - Scripting, we saw how a shell script could be invoked from the command line. This pattern is much more convenient than using the script as an input to the shell utility, but it hides an important detail: in both cases, the script is executed as a standalone process.

The hidden process

vm$ pwd
/home/sally
vm$ cat change-env.sh
#!/bin/bash
FOO=8
cd /
vm$ ./change-env.sh
vm$ echo [ $FOO ]
[ ]
vm$ pwd
/home/sally
vm$

If the script's purpose is to modify the system state (e.g. by modifying files, starting processes, etc.), then this distinction is not very important. However, if we want to use the script to modify the environment, then the process boundary is a problem.

A script executed in this way might set environment variables or change directories, but this will not effect the "calling" context. The goal of this chapter is to automatically modify our shell's environment, so we'll need to learn a new way of executing scripts before we can go any further.

"Sourcing" files with .

vm$ help .
.: . filename [arguments]
    Execute commands from a file in the current shell.

    Read and execute commands from FILENAME in the current shell.
    The entries in $PATH are used to find the directory
    containing FILENAME. If any ARGUMENTS are supplied, they
    become the positional parameters when FILENAME is executed.

    Exit Status:
    Returns the status of the last command executed in FILENAME;
    fails if FILENAME cannot be read.
vm$

. (a.k.a. "dot") is a standard (though oddly-named) shell utility that does exactly this.

vm$ pwd
/home/sally
vm$ cat change-env.sh
#!/bin/bash
FOO=8
vm$ . change-env.sh
vm$ echo [ $FOO ]
[ 8 ]
cd /
vm$ pwd
/
vm$

Using the "dot" utility is essentially saying, "interpret the commands in this file as though I entered them directly into this terminal window myself." This is sometimes referred to as "sourcing a file."

vm$ cat change-prompt.sh
# Set the command prompt to a Microsoft Windows-style
# value. It's just a bunch of characters, after all!
PS1='C:\> '
vm$ . change-prompt.sh
C:\> ​

We can use this right away to start writing scripts that customize our environment. (Recall the $PS1 variable discussed in Chapter 6 - Command Invocation.) Our current knowledge of customizations is still limited, but even now, we can appreciate a problem with this approach. Running a configuration script like this every time we logged in the system would become tiresome very quickly.

Thankfully, we can instruct the system to automatically run our configuration scripts on our behalf. Accomplishing this is somewhat more complicated than it might seem at first, so we'll take some time to discuss the details before returning to more customization options.

Startup scripts

To facilitate environment customation, every shell has a different set of hidden files that it will execute as it initializes. The file we should modify depends not only on the shell we are using, but also the "invocation mode" of the shell.

Shell invocation modes

vm$ cat shell-classifications.txt
                |     Login     |    Non-login    |
----------------+---------------+-----------------+
Interactive     |               |                 |
                |               |                 |
----------------+---------------+-----------------+
Non-interactive |               |                 |
                |               |                 |
----------------+---------------+-----------------+
vm$

Whether bash, sh, zsh, or some other application, all shells recognize two orthogonal "invocation modes": interactive versus non-interactive, and login versus non-login. These modes effect how the shell behaves as it starts.

If you open a shell or terminal (or switch to one), and it asks you to log in (Username? Password?) before it gives you a prompt, it's a login shell.

https://askubuntu.com/questions/155865/what-are-login-and-non-login-shells

This is a frequently-discussed topic on the web, but even among those supplying answers, there is some confusion about what this means.

Shell invocation mode considerations

The best way to understand these distinctions is via three different considerations.

"Login" shells

"Interactive" shells

We separate "conventional meaning" from "requirements" because given the correct options, a shell can be run in any "mode" regardless of the current context. While it may be a little technical, it's not magic!

Shell invocation modes

Conventional examples

vm$ cat shell-classifications-examples.txt
                |     Login     |    Non-login    |
----------------+---------------+-----------------+
Interactive     | connecting    | opening a       |
                | via SSH       | terminal window |
----------------+---------------+-----------------+
Non-interactive | very rare     | running a       |
                | circumstances | shell script    |
----------------+---------------+-----------------+
vm$

All this means that the different invocation modes simply determine which files a shell "sources" as it starts up. The distinctions exist to allow for fine-grained control over how the system prepares the environment in different contexts.

However, the distinction between "login" and "non-login" contexts is largely a vestige from the past, when terminals were slow and computing time was expensive. For our purposes, whether or not a shell is a "login shell" will not be particularly relevant.

Shell invocation modes

Files sourced

mode bash sh zsh
login /etc/profile ~/.bash_profile ~/.bash_login ~/.profile /etc/profile ~/.profile $ENV /etc/zprofile $ZDOTDIR/.zprofile /etc/zshrc $ZDOTDIR/.zshrc /etc/zlogin $ZDOTDIR/.zlogin
non-login ~/.bashrc $ENV /etc/zshrc $ZDOTDIR/.zshrc

Many shells source the same files regardless of whether they have been run in "login" contexts. This is not a consideration for users of these shells. However, Bash sources two separate locations.

vm$ cat ~/.bash_profile
# This file is sourced by "login" Bash shells, but "non-login"
# Bash shells source the `~/.bashrc` file. To promote
# consistency between those two contexts, manage configuration
# settings in `~/.bashrc`, and source that file from
# `~/.bash_profile`.
if [ -f ~/.bashrc ]
then
  . ~/.bashrc
fi
vm$

Bash users can account for this by writing a short ~/.bash_profile file that sources from their ~/.bashrc file.

vm$ echo "PS1='NEW$ '" > ~/.bashrc
vm$ exit
pc$ vagrant ssh
NEW$ ​

As we begin working with these files, remember that the shell sources them during initialization only. This means any changes we make will not effect the current shell process. In order to see the effect of our changes, we'll need to create a new shell (e.g. by logging out and logging back in) or explicitly source the file using the "dot" utility.

Shell customization: Paths

vm$ cd projects/dragonweb
vm$ ../scripts/my-component-scaffold.sh
Creating scaffolding for new component...
Done!
vm$

Chapter 13 - Scripting detailed how shell scripts can be created to automate repetitive tasks. Although traits like the "execute" permission and the "shebang" make scripts easier to use, invoking the scripts can still be cumbersome. We need to specify the full path to the script each time we want to invoke it.

vm$ cd projects/dragonweb
vm$ mv ../scripts/my-component-scaffold.sh ~
vm$ ~/my-component-scaffold.sh
Creating scaffolding for new component...
Done!
vm$

We can mitigate this somewhat by placing scripts in our HOME directory. That allows us take advantage of shell expansion and reference the script relative to the special tilde character (~). However, the approach tends to add clutter to the HOME directory, and it still requires a few more keystrokes than invoking a system-provided utility like grep.

vm$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin
vm$ which ls
/bin/ls
vm$ which man
/usr/bin/man
vm$

Recall from Chapter 6 - Command Invocation that the PATH environment variable is a colon-separated list of directories that the shell will reference when searching for applications.

vm$ cd projects/dragonweb
vm$ echo "PATH=$PATH:~/projects/scripts" >> ~/.bashrc
vm$ . ~/.bashrc
vm$ my-component-scaffold.sh
Creating scaffolding for new component...
Done!
vm$

By adding a directory to that list, we can designate a personal directory for our scripts. Any executable file placed in that directory may then be invoked without a complete path because the shell now has the information it needs to find it.

vm$ fetch webrtc
fetch: command not found
vm$ echo "PATH=$PATH:/opt/google/depot_tools" >> ~/.bashrc
vm$ . ~/.bashrc
vm$ fetch webrtc
Running: gclient root
Running: gclient sync --with_branch_heads
vm$

Some projects include a set of tools required for development. For instance, the Chromium and V8 projects both rely on Google's Depot Tools.

PATH security

vm$ ls
deploy.sh  src  test
vm$ echo "PATH=.:$PATH" >> ~/.bashrc
vm$ . ~/.bashrc
vm$ deploy.sh
Now running local "deploy" script
Deploy complete!
vm$

Many of the commands issued so far have been prefixed with the characters ./, which instructs the shell to search for the file in the current directory.

Some users place relative paths in their PATH variable. This is a convenience that makes the leading ./ unnecessary.

vm$ ls dangerous-directory
ls
vm$ cd dangerous-directory
vm$ ls
Because "." is at the beginning of your PATH, you have
just invoked the "local" script named `ls`. It might
delete your files, read your passwords, or any number
of other malicious things.
vm$

This practice is dangerous because it allows scripts in the current directory (which may have been created by any user) to take precedence over common system utilities.

Shell customization: Aliases

vm$ ls
change-prompt.sh
vm$ ls -l
-rw-rw-r-- 1 vagrant vagrant   12 Aug  2 18:30 change-prompt.sh
vm$

As you grow familiar with various utilities, you may find that you are consistently using certain options. For example, the -l option of ls enables a "long listing" format.

Aliases are user-defined commands that can be thought of as "shortcuts" for other commands.

vm$ help alias
alias: alias [-p] [name[=value] ... ]
    Define or display aliases.

    Without arguments, `alias' prints the list of aliases in
    the reusable form `alias NAME=VALUE' on standard output.

    Otherwise, an alias is defined for each NAME whose VALUE
    is given A trailing space in VALUE causes the next word
    to be checked for alias substitution when the alias is
    expanded.
vm$

Use the alias utility to define alias "names" (the command you wish to type) with alias "values" (the command you wish to be executed).

vm$ alias ll='ls -l'
vm$ ll
-rw-rw-r-- 1 vagrant vagrant   12 Aug  2 18:30 change-prompt.sh
vm$

When the shell encounters a command that has been defined as an alias, it substitutes the name of the alias with the definition originally provided to the alias utility.

vm$ alias ll='ls -l'
vm$ ll /tmp
total 4
-rw-rw-r-- 1 vagrant vagrant   0 Aug  2 21:50 a-temporary-file-01.txt
-rw-rw-r-- 1 vagrant vagrant   0 Aug  2 21:50 a-temporary-file-02.txt
-rw-rw-r-- 1 vagrant vagrant   0 Aug  2 21:50 a-temporary-file-03.txt
-rw-rw-r-- 1 vagrant vagrant   0 Aug  2 21:50 a-temporary-file-04.txt
vm$

Because aliases are expanded by the shell, additional options "pass through" to the underlying command.

vm$ alias desktop='cd ~/Desktop'
vm$ pwd
/home/sally
vm$ desktop
vm$ pwd
/home/sally/Desktop
vm$

Unlike shell scripts, aliases are executed in the current shell. This avoids the "sourcing" issue discussed previously, making aliases a good choice for environment-modifying tasks like changing directories.

vm$ git checkout dev
Switched to branch 'dev'
vm$ alias co='git checkout'
vm$ co master
Switched to branch 'master'
vm$

Another common use-case for aliases is shortening lengthy commands. For example, the "git" application is notorious for its complex interface. Aliases can significantly reduce the amount of typing required for common workflow tasks.

Application customizations

vm$ ls -a ~
.   .asunder  .gitconfig  .npmrc  .xinputrc
..  .bashrc   .hgrc       .vimrc  .zshrc
vm$

Many applications support customization through so-called "dot files." The abilities and syntax of these files vary greatly between applications, but the general convention is that they are "hidden" text files placed in your HOME directory.

For more details on a given application's configuration files, consult the documentation provided by man.

vm$ git add .npmrc
vm$ git commit -m 'Add config file for npm package manager.'
vm$ git push origin master
vm$

Your configuration is likely to change and grow over time. It can be a good idea to maintain these files in a version control system like git. This helps to recover from mistakes like typos and deleted files.

vm$ ssh our-dev-server.example.com
# Welcome to the dev server!
#
# This is your first time logging in to this server,
# so it's probably not set up the way you like. Feel
# free to customize the environment however you wish.
dev$ ​git clone git@github.com:sally/dotfiles.git .
Cloning into '.'...
Checking connectivity... done.
dev$ ​source .bashrc
vm$

Synchronizing your changes with a remote repository can greatly simplify the process of configuring a branch new system (or initially logging in to an existing system).

In Review

Exercise

In order to complete these exercises, configure your environment to apply these settings automatically.

  1. Customize your shell to greet you when you first log in (but not when you run a script or execute a new shell in the same session).

  2. Update your command prompt to include the current date, as provided by the date utility. This prompt should also be used if you execute bash from the command line.

  3. The ls utility supports an option named --color that causes the program to display certain kinds of files with fancy colors. Override the built-in ls utility with a custom alias that enables this option.

  4. Define an alias for the alias command that is named alias.

  5. You may find that you are commonly leaving "to-do" notes to yourself in your files. nano can help you avoid forgetting about them by highlighting special text. Customize nano to color the text "TODO" in black with a yellow background.

    A few hints:

    • Start with man nano.
    • Your final solution will need a "regular expression" for file names--use the value ".*" (including quotation marks) to enable this highlighting for all files.

Solution

  1. The virtual environment is using bash. As we've seen, that shell invokes the ~/.profile file only when users log in. If we place the "greeting" command (e.g. echo Hello) in that file, it will only be executed only then.

    We can verify this by logging out of the virtual machine and running vagrant up. The greeting should be displayed here. If we run bash, the greeting should not be displayed.

  2. We've seen how the PS1 value controls the contents of the command prompt. We also know that command substitution allows us to store the output of a command in an environment variable.

    We could place the following text in ~/.profile:

    PS1="$(date) $ "

    But this has a couple of problems.

    First, the value is "static." It describes the time when the file was sourced, but it doesn't update as time goes by (press the Enter key a few times to see this). We can address this by "escaping" the dollar sign character ($) in the command substitution syntax. This way, the shell will not expand it when the configuration file is first sourced:

    PS1="\$(date) $ "

    Now the value of PS1 should be re-interpreted every time a new prompt is displayed.

    The instructions specifically say that this prompt should also apply when we invoke bash from the terminal. Our current solution does not satisfy this requirement:

    Wed Aug  3 16:58:15 UTC 1970 $ bash
vm$

    When we start a new shell by invoking Bash, the new shell is not a "login shell." As we've seen, Bash only sources ~/.profile for login shells--otherwise it sources ~/.bashrc. We can start by moving the PS1 definition to a new file named ~/.bashrc, but then we would have the opposite problem: the command prompt would be modified only when we invoke bash from the command line. We could define the variable in both files, but maintaining the duplication would be a hassle. Instead, we'll source the ~/.bashrc file from the ~/.profile file by adding the following line to ~/.profile:

    [ -f ~/.bashrc ] && . ~/.bashrc

    We're using the "test" command (written here with the open bracket character) to be extra safe--we will only source that file if it is defined.

  3. We've seen that the system provides an executable file named ls:

    vm$ which ls
/bin/ls

    So it may be surprising that we are defining an alias with the same name. This is perfectly valid, though--when we issue commands, the alias will take precedence over the executable file.

    In this case, the alias "name" is ls, and the alias "value" is ls --color. Place the following alias definition in the new ~/.bashrc file.

    alias ls="ls --color"

  4. This alias is a little odd (and generally useless), but it's technically valid. The "name" is alias and the "value" is alias. Place the following alias definition in the ~/.bashrc file:

    alias alias=alias

  5. The man page for nano has a section named "INITIALIZATION FILE" that references a separate page named "nanorc". man nanorc is full of documentation for how a file named ~/.nanorc can modify the application's behavior. We'll want to create that file and insert the following text:

    syntax "todo" ".*"
color black,yellow "TODO"