Basic Kodiak Usage


Introduction

This document is an introduction to using the Kodiak high performance computing cluster. The intended audience of the first few sections are users with little to no experience with "Unix-like" enviroments such as Linux. Experienced users will probably end up skipping this and jumping over part 2 that explains how to compile and run programs on Kodiak. This document is not meant to be a comprehensive tutorial on Linux usage but hopefully it will be enough for you to get started.

Note: This document is a "work in progress". If anything is unclear, inaccurate, or even outright wrong please send email to Carl_Bell@baylor.edu.

Contents


Getting Started


Logging In

To log in to Kodiak, you will need to use ssh. (Telnet is not supported.)

Microsoft Windows users should use PuTTY. Faculty and staff can download PuTTY from Baylor's AppCenter page. Students can get PuTTY from their official site. When you run PuTTY, you should see the "PuTTY Configuration" window. Specify "kodiak.baylor.edu" as the Host Name (or IP address) and click Open to connect. Enter your username and password to continue.

Mac OS X user should connect with the ssh command in a Terminal window. The Terminal.app program can be found in your Utilities folder. (At the Finder, select the Go→Utilities menu.) Linux users should connect with the ssh command in an xterm terminal window or the console. The actual command is ssh username@kodiak.baylor.edu.

Note: In this document, we will use "bobby" as your username.

When you first log on to Kodiak, you will see several informational messages. Eventually you should end up with something like:

[bobby@n130 ~]$ 

What you are looking at is known as the command prompt or just prompt. When you see this, the system is waiting for you to tell it what to do. The default format of the prompt is:

[your_username@system_name current_working_directory]$

Why does the prompt say "n130" as the system name instead of "kodiak"? It just does. And what is meant by "current working directory" and why is it "~"? We'll get to that in a minute.

Note: To make things a bit easier to read, this document will usually omit the [user@host directory] bit and just use $ as the prompt.

General Info

So you are now sitting at the prompt. What next? To do something, type a command along with any any options and arguments that the command needs and press the return (or enter) key. For example, to see a list of users currently logged on, type who and press return.

$ who
betty    pts/2        2013-10-02 13:48 (172.16.8.20)
bobby    pts/3        2013-10-02 13:51 (172.16.8.24)
bubba    pts/1        2013-10-02 08:16 (93.184.216.119)

Note: You always need to press return to run a command, so from now on, pressing return is implied.

Commands often have options associated with them. For example, to add column headings to the output of the who command, add a -H option.

$ who -H
USER     LINE         TIME             COMMENT
betty    pts/2        2013-10-02 13:48 (172.16.8.20)
bobby    pts/3        2013-10-02 13:51 (172.16.8.24)
bubba    pts/1        2013-10-02 08:16 (93.184.216.119)

Some commands specify options with a "-" and a single letter. Some commands specify options with "--" and a word. Some commands can do both. Some options take extra arguments. How do you know what's what? Well, you could try running who with an invalid option such as -x and the command might give you some helpful information. (It helps if you already know that -x is, in fact, an invalid option.)

$ who -x
who: invalid option -- x
Try `who --help' for more information.

Okay then, let's try who --help for more information...

$ who --help
Usage: who [OPTION]... [ FILE | ARG1 ARG2 ]

  -a, --all         same as -b -d --login -p -r -t -T -u
  -b, --boot        time of last system boot
  -d, --dead        print dead processes
  -H, --heading     print line of column headings
  -l, --login       print system login processes
      --lookup      attempt to canonicalize hostnames via DNS
  -m                only hostname and user associated with stdin
  -p, --process     print active processes spawned by init
  -q, --count       all login names and number of users logged on
  -r, --runlevel    print current runlevel
  -s, --short       print only name, line, and time (default)
  -t, --time        print last system clock change
  -T, -w, --mesg    add user's message status as +, - or ?
  -u, --users       list users logged in
      --message     same as -T
      --writable    same as -T
      --help     display this help and exit
      --version  output version information and exit

If FILE is not specified, use /var/run/utmp.  /var/log/wtmp as FILE is common.
If ARG1 ARG2 given, -m presumed: `am i' or `mom likes' are usual.

Report bugs to <bug-coreutils@gnu.org>.

Not all commands support the --help option. The best way to get detailed information about commands is to read the on-line manual with the man command. Entries in the on-line manual also known as man pages.

$ man who
WHO(1)                           User Commands                          WHO(1)

NAME
       who - show who is logged on

SYNOPSIS
       who [OPTION]... [ FILE | ARG1 ARG2 ]

DESCRIPTION
       -a, --all
              same as -b -d --login -p -r -t -T -u

       -b, --boot
              time of last system boot

       -d, --dead
              print dead processes

       -H, --heading
              print line of column headings

       -l, --login
              print system login processes

       --lookup
              attempt to canonicalize hostnames via DNS

       -m     only hostname and user associated with stdin

       -p, --process
              print active processes spawned by init

       -q, --count
              all login names and number of users logged on

       -r, --runlevel
              print current runlevel

       -s, --short
              print only name, line, and time (default)

       -t, --time
              print last system clock change

       -T, -w, --mesg
              add user's message status as +, - or ?

       -u, --users
              list users logged in

       --message
              same as -T

       --writable
              same as -T

       --help display this help and exit

       --version
              output version information and exit

       If  FILE is not specified, use /var/run/utmp.  /var/log/wtmp as FILE is
       common.  If ARG1 ARG2 given, -m presumed: 'am i'  or  'mom  likes'  are
       usual.

AUTHOR
       Written by Joseph Arceneaux, David MacKenzie, and Michael Stone.

REPORTING BUGS
       Report bugs to <bug-coreutils@gnu.org>.

COPYRIGHT
       Copyright © 2006 Free Software Foundation, Inc.
       This  is  free  software.   You may redistribute copies of it under the
       terms      of      the      GNU      General       Public       License
       <http://www.gnu.org/licenses/gpl.html>.   There  is NO WARRANTY, to the
       extent permitted by law.

SEE ALSO
       The full documentation for who is maintained as a Texinfo  manual.   If
       the info and who programs are properly installed at your site, the com-
       mand

              info who

       should give you access to the complete manual.



who 5.97                         January 2009                           WHO(1)

Changing Your Password

Actually, one of the things you should do when logging on to Kodiak the first time is change your password from the one randomly generated when your account was created. Let's go ahead and get that out of the way. To change your password, use the passwd command. (Yes, the command is "passwd" and not "password".)

When you type your current password, you will not see it displayed. Same thing with your new password. If all goes well, you should see the following:

$ passwd
Changing password for user bobby.
(current) UNIX password: 
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.

It's possible that the system won't accept the password you want. For example, if you were to try to set your password to "secret" (don't set your password to "secret"...) or if you were to try to set it to something very simple, the passwd command will complain:

BAD PASSWORD: it is based on a dictionary word

or

BAD PASSWORD: it is too simplistic/systematic

Password formats on Kodiak are somewhat strict, so keep trying. You will get 3 attempts before the passwd command gives up. Just try again.

Logging Out

When you are finished working on Kodiak and are ready to log out type logout or exit. You can also press control-D (^D) to log out.

Note: It is common to use the "^" character as an abbreviation for "control" key sequences and we will use it from now on. Just remember that if you see something like "^X" it means "control-X".


Directories and Files


It is quite likely that you are familiar with a desktop operating system such as Microsoft Windows or Mac OS X. If so, then you are probably also familiar with how those operating systems usually organize things, namely, folders and documents:

Linux organizes things the same way except it uses the terms "directory" and "file" instead of "folder" and "document". Actually, Windows and OS X also use "directory" and "file". Folders and documents are just metaphors used in their graphical user interfaces (GUI). There are also GUIs avaialable for Linux that display directories and files as folders and documents. Because you will be using a command line interface on Kodiak, we'll stick with that.

Just about everything on Linux is a file. Even the commands you run are files. Files are located within directories. Directories can also contain other (sub-)directories. Directories and files are stored in file systems and there may be multiple file systems. (There are several file systems on Kodiak.)

You can visualize the file system as an upside-down tree, with the roots of the tree at the top, directories and subdirectories as the tree's branches, and files as the tree's leaves. Files (and directories) can be expressed as a path starting at the root, traversing the directories, and ending at the file itself. So using the example image above, the path to the "budget" file would be:

{ ROOT } → Bobby → Documents → Budget

Linux uses / as the root. It also uses / as the directory separator (or delimiter). So the path to the file above would be:

/Bobby/Documents/Budget

A path to a file or directory that begins with "/" (the root) is considered an absolute or full path and specifies a file or directory explicitly. If a path does not start with "/", it is considered a relative path and specifies a file or directory starting with the current working directory.

Something to keep in mind is that Linux file systems are case sensitive. In other words, the files "letter.txt" and "Letter.txt" are two distinct files. Also, some operating systems (e.g., VMS) support the concept of file versions. This is not the case with Kodiak.

Directories

When you first log on to Kodiak, you will be "in" your home directory. Whichever directory that you are currently "in" is considered your current working directory or working directory. To see the full path to your current working directory, use the pwd (print working directory) command.

[bobby@n130 documents]$ pwd
/home/bobby/documents

In this case, the working directory is "/home/bobby/documents". As you can see, the prompt displays the name of the current directory. But if your current directory were your home directory you would see:

[bobby@n130 ~]$ pwd
/home/bobby

So what's the deal with the "~" character in the prompt? That's just an abbreviation or shortcut for your home directory. You can often substitute "~" when referring to something within your home directory. For example, "~/pictures/family" would be equivalent to "/home/bobby/pictures/family".

There are two other common directory abbreviations:

.    - current directory
..   - "parent" directory

In the example above, the current directory (i.e., ".") is "documents" (full path /home/bobby/documents) and the parent directory (i.e., "..") is "bobby" (full path /home/bobby). You can usually refer to a file or directory within the current working directory just by its name. So if your working directory is "/home/bobby/pictures", you can refer to the family subdirectory as simply "family". But you could also refer to it as "./family". As you will see in a bit, there are times that it will be necessary to refer to a file using "./". Also, assuming your current working directory is still "/home/bobby/pictures", you can refer to the letter file as "../documents/letter.txt".

Navigating Directories

Okay, you've logged on to Kodiak and are in your home directory. Now what? To change your current working directory, use the cd command. You can see that you specify a directory with either a relative or full path.

[bobby@n130 ~]$ pwd
/home/bobby

[bobby@n130 ~]$ cd pictures/family

[bobby@n130 family]$ pwd
/home/bobby/pictures/family

[bobby@n130 family]$ cd ..

[bobby@n130 pictures]$ pwd
/home/bobby/pictures

[bobby@n130 family]$ cd /home/bobby/documents

[bobby@n130 documents]$ pwd
/home/bobby/documents

To change back to your home directory, you can use cd ~. Because your home directory is the default directory for the cd command and will be used if no directory is explicitly supplied, you can omit the ~ if you wish.

$ pwd
/home/bobby/downloads/foo/bar/baz

$ cd

$ pwd
/home/bobby

Just using cd to navigate directories is all well and good if you happen to have all of the files and directories memorized. Since you probably don't, it would help to see a listing of a directory's contents. You can do this with the ls command.

$ cd

$ pwd
/home/bobby

$ ls
documents  downloads  pictures

$ cd documents

$ ls
budget  letter.txt

What would be printed out if there was nothing in the directory? Nothing, then just the prompt. When viewing the directory listing generated by ls, how can you tell if an entry is a directory or a file? One way to do this is to add the -F (or --classify) option.

$ ls -F
documents/  downloads/	pictures/

$ ls -F documents
budget	letter.txt

With the -F option, the names of directories have a "/" after them. The -F option will append other characters for other types but we won't list them here. You may have noticed that by adding the optional argument "documents" to the second ls command, you didn't have to cd to the directory.

Another way to get more detailed (or long) information from the ls command is to add the -l option.

$ ls -l
total 2
drwxr-xr-x 2 bobby users 2 Oct  7 16:01 documents
drwxr-x--- 3 bobby users 1 Oct  7 15:02 downloads
drwxr-x--- 4 bobby users 2 Oct  7 14:14 pictures

$ cd documents

$ ls -l
total 3
-rw------- 1 bobby users 1169 Oct  7 15:54 budget
-rw-r--r-- 1 bobby users 1324 Oct  7 15:56 letter.txt

$ ls -l -F ..
total 2
drwxr-xr-x 2 bobby users 2 Oct  7 16:01 documents/
drwxr-x--- 3 bobby users 1 Oct  7 15:02 downloads/
drwxr-x--- 4 bobby users 2 Oct  7 14:14 pictures/

There's quite a bit of information here. Right now, we will just take a look at the line for "documents".

drwxr-xr-x 2 bobby users 2 Oct 7 16:01 documents

The first character tells what the entry actually is, in this case "d" for directory. An ordinary file would have a "-" as the first character. The next 9 gobbledygook looking letters (rwxr-xr-x) are the entry's permissions. We'll talk about permissions later. Just ignore the next number. The next two columns are the entry's owner (bobby) and group (users), and are related to the permissions. The remaining columns are the entry's size in bytes, the date it was last modified, and its name.

Notice how we got fancy with the last command above and used multiple options, -l and -F as well as the parent directory ".." as the argument. Another useful option is -a which tells ls to include entries that begin with a dot. When a file begins with a dot, it is considered hidden. This is not for security reasons (i.e., "security through obscurity") but is more of a way to help de-clutter things.

$ ls -laF
total 3
drwxr-x--x  5 bobby users  4 Oct  7 16:15 ./
drwxr-xr-x 11 bobby users  9 Oct  7 14:14 ../
-rw-------  1 bobby users 87 Jul  9 16:15 .bashrc
drwxr-xr-x  2 bobby users  2 Oct  7 16:01 documents/
drwxr-x---  3 bobby users  1 Oct  7 15:02 downloads/
drwxr-x---  4 bobby users  2 Oct  7 14:14 pictures/

You can see that both the current directory '.' and parent directory '..' are now included in the listing, along with some file named ".bashrc". Also, note that the options were combined, -laF instead of -l -a -F. Many commands can do this.

To create a directory, use the mkdir (make directory) command.

$ pwd
/home/bobby

$ ls -F
documents/  downloads/	pictures/

$ mkdir backups

$ ls -F
backups/  documents/  downloads/  pictures/

Working with Files

Now that you can navigate around directories and see what's actually in those directories, it's time to start working with files. If you want to see the contents of a file, use the cat command.

$ ls
budget	letter.txt

$ cat letter.txt
Lorem ipsum,

Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores
postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia
tollit definitiones qui in. Ea vis eripuit nusquam. Eos ceteros tibique
adversarium ne, nisl tota convenire at pri.

Nobis adolescens contentiones sed id, vis sonet neglegentur in, an tation
consetetur qui. Duo cu aliquip molestiae, decore numquam oporteat in sit. Ut
mea error option conclusionemque, in posse interesset ullamcorper eam, cum ex
assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi,
dictas corrumpit euripidis no vim.

An per habemus omittam philosophia. Pro ad verterem salutandi. Et ipsum oratio
consectetuer duo, mea ex vidit definitionem. Ad dolor albucius deleniti per,
postea ocurreret definitionem ex mea. Ei putent voluptua intellegat vel, pri
in liber quodsi lobortis, quis altera essent at his.

Laboramus elaboraret eu usu, pro labitur adipisci at, qui congue appetere
assueverit ea. Liber scribentur et mel. Ex pro nullam aliquam, in primis
hendrerit per, ea est inani quando. Quo ei quis sint dicit, te atqui quando
mel. An rebum sententiae incorrupte eum, sea timeam similique ne.

                                           Sincerely,


                                           Robert M. "Bobby" Baylor

Where did the name "cat" come from? It's an abbreviation of "catenate" or "concatenate". So what does that have to do with displaying a file? The cat command actually concatenates all of the requested files and prints them out. If you wanted to see both files, letter.txt and budget, you would have run cat letter.txt budget.

Sometimes, a file is too big to display all at once. Instead of cat, you can display a screen at a time with the more. When the output pauses, press the space bar to continue. There is also a similar but more powerful command, less, which you will probably end up using instead.

$ more letter.txt
Lorem ipsum,

Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores
postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia
tollit definitiones qui in. Ea vis eripuit nusquam. Eos ceteros tibique
adversarium ne, nisl tota convenire at pri.

Nobis adolescens contentiones sed id, vis sonet neglegentur in, an tation
consetetur qui. Duo cu aliquip molestiae, decore numquam oporteat in sit. Ut
mea error option conclusionemque, in posse interesset ullamcorper eam, cum ex
assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi,
dictas corrumpit euripidis no vim.

An per habemus omittam philosophia. Pro ad verterem salutandi. Et ipsum oratio
consectetuer duo, mea ex vidit definitionem. Ad dolor albucius deleniti per,
postea ocurreret definitionem ex mea. Ei putent voluptua intellegat vel, pri
in liber quodsi lobortis, quis altera essent at his.

Laboramus elaboraret eu usu, pro labitur adipisci at, qui congue appetere
assueverit ea. Liber scribentur et mel. Ex pro nullam aliquam, in primis
hendrerit per, ea est inani quando. Quo ei quis sint dicit, te atqui quando
mel. An rebum sententiae incorrupte eum, sea timeam similique ne.

--More--(90%)

                                           Sincerely,


                                           Robert M. "Bobby" Baylor

If you don't want to view the entire file, you can use the head command. This will display just the first 10 lines. To see the last 10 lines of a file, use the tail command. A very useful option for tail is -f which tells tail to output appended data as the file grows. We'll use this when we see how to run programs on Kodiak in part 2.

To copy a file, use the cp command. You can simply supply file names and it will copy the file in the current directory. If you want to copy the file into another directory, use the path to the destination as the second argument. If the destination is a directory without a file name, then the file will be copied to that directory and keep the same name.

$ pwd
/home/bobby/downloads

$ ls -l
total 13
-rw-r--r-- 1 bobby users 12424 Oct  8 13:17 report.pdf

$ cp report.pdf report.pdf.bak

$ ls -l
total 25
-rw-r--r-- 1 bobby users 12424 Oct  8 13:17 report.pdf
-rw-r--r-- 1 bobby users 12424 Oct  8 13:19 report.pdf.bak

$ cp report.pdf ../documents

$ ls -l ../documents
total 16
-rw-r--r-- 1 bobby users  1169 Oct  7 15:54 budget
-rw-r--r-- 1 bobby users  1324 Oct  7 15:56 letter.txt
-rw-r--r-- 1 bobby users 12424 Oct  8 13:46 report.pdf

What if, instead of copying a file somewhere, you wanted to move it there. In that case, you would use the mv command.

$ pwd
/home/bobby/downloads

$ ls
report.pdf

$ ls ../documents
budget	letter.txt

$ mv report.pdf ../documents

$ ls

$ ls ../documents
budget	letter.txt  report.pdf

The mv command is also used to rename a file. You can also move and rename a file at the same time by using a path to a file (as opposed to a path to a directory) as the destination. mv is also used to move or rename a directory.

Be careful. If you cp or mv a file to a file that already exists, you will overwrite the file. Both cp and mv have an option -i (--interactive) which will ask you to confirm if you want to overwrite an existing file.

$ ls
budget	letter.txt  report.pdf

$ mv report.pdf annual_report.pdf

$ ls
annual_report.pdf  budget  letter.txt

$ cd ../downloads

$ ls
report.pdf.bak

$ mv report.pdf.bak ../backups/annual_report.pdf

$ ls

$ ls ../backups
annual_report.pdf

To delete a file, use the rm (remove) command. To delete a directory, use the rmdir command. Note that rmdir will only delete a non-empty directory.

$ ls
figure_1.pdf  poster.pdf.gz  report.pdf.bak

$ rm figure_1.pdf

$ ls
poster.pdf.gz  report.pdf.bak

$ ls -F
backups/  documents/  downloads/  misc/  pictures/

$ ls misc

$ rm misc
rm: cannot remove `misc': Is a directory

$ rmdir misc

$ ls -F
backups/  documents/  downloads/  pictures/

Editing Files

How does one edit a text file on Kodiak? There are several text editors installed: vi, emacs, and nano. The simplest editor is nano. To edit an existing file, run nano with the name of the file as its argument. To create and edit a new file, run nano without an argument or with the name of the file you want to create.

$ nano letter.txt

You will then see something like the following. You should be able to use the arrow keys to move around the document and type to modify it. Available commands can be found at the bottom of the screen. For example, to get help for using nano, press ^G.

  GNU nano 1.3.12             File: letter.txt                                  

Lorem ipsum,

Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores
postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia
tollit definitiones qui in. Ea vis eripuit nusquam. Eos ceteros tibique
adversarium ne, nisl tota convenire at pri.

Nobis adolescens contentiones sed id, vis sonet neglegentur in, an tation
consetetur qui. Duo cu aliquip molestiae, decore numquam oporteat in sit. Ut
mea error option conclusionemque, in posse interesset ullamcorper eam, cum ex
assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi,
dictas corrumpit euripidis no vim.

An per habemus omittam philosophia. Pro ad verterem salutandi. Et ipsum oratio
consectetuer duo, mea ex vidit definitionem. Ad dolor albucius deleniti per,
postea ocurreret definitionem ex mea. Ei putent voluptua intellegat vel, pri
in liber quodsi lobortis, quis altera essent at his.

Laboramus elaboraret eu usu, pro labitur adipisci at, qui congue appetere
                               [ Read 27 lines ]
^G Get Help  ^O WriteOut  ^R Read File ^Y Prev Page ^K Cut Text  ^C Cur Pos
^X Exit      ^J Justify   ^W Where Is  ^V Next Page ^U UnCut Text^T To Spell

When you are finished editing your file, press ^X to quit and you should see the following. Press "Y" if you want to save any changes, or "N" to ignore any changes. To go back to editing the document, press ^C.

Save modified buffer (ANSWERING "No" WILL DESTROY CHANGES) ?                    
 Y Yes
 N No           ^C Cancel

If this is a new document, you will then be prompted to enter the name of the document. Just type in the new name and press return to quit.

File Name to Write: new_letter.txt                                              
^G Get Help         ^T To Files         M-M Mac Format      M-P Prepend
^C Cancel           M-D DOS Format	M-A Append          M-B Backup File

The vi and emacs editors are more powerful than nano but consequently are also more difficult to use. We won't describe how to use them here other than explain below how to quit them in case you do happen to run them. There also are graphical editors on Kodiak as well. These include gedit and the aforementioned emacs. To use them, you would need to be running an X11 server on your desktop system such as Cygwin/X or Xming for Microsoft Windows, and XQuartz for Mac OS X.

To quit vi, first make sure you are in command mode and not insert mode. If you see "-- INSERT --" at the bottom of the screen, you are in insert mode. Press the escape key to switch to command mode. Now press :q! to quit without saving. (You should see the ":q!" at the bottom of the screen.)

To quit emacs, press ^X then ^C. You might see C-x C-c at the bottom of the screen before it actually quits. It's possible that you will be asked if you want to save the file. To quit without saving, press "n". If you are told "Modified buffers exist; exit anyway?" answer "yes". You have to type "yes", not just "y".

Wildcards

Now, suppose that there are more than a few files in a directory...

$ pwd
/home/bobby/pictures/vacation

$ ls
2011Dec22_01.jpg  2012Aug20_27.jpg  2013Jul03_06.jpg  2013Jul08_01.jpg
2011Dec22_02.jpg  2012Aug20_28.jpg  2013Jul03_07.jpg  2013Jul08_02.jpg
2011Dec22_03.jpg  2012Aug20_29.jpg  2013Jul03_08.jpg  2013Jul08_03.jpg
2011Dec22_04.jpg  2012Aug20_30.jpg  2013Jul03_09.jpg  2013Jul08_04.jpg
2011Dec22_05.jpg  2012Aug21_01.jpg  2013Jul04_01.jpg  2013Jul08_05.jpg
2011Dec22_06.jpg  2012Aug21_02.jpg  2013Jul04_02.jpg  2013Jul08_06.jpg
2011Dec22_07.jpg  2012Aug21_03.jpg  2013Jul04_03.jpg  2013Jul08_07.jpg
2011Dec22_08.jpg  2012Aug21_04.jpg  2013Jul04_04.jpg  2013Jul08_08.jpg
2011Dec22_09.jpg  2012Aug21_05.jpg  2013Jul04_05.jpg  2013Jul08_09.jpg
2011Dec22_10.jpg  2012Aug21_06.jpg  2013Jul04_06.jpg  2013Jul08_10.jpg
2011Dec22_11.jpg  2012Aug21_07.jpg  2013Jul04_07.jpg  2013Jul08_11.jpg
2011Dec22_12.jpg  2012Aug21_08.jpg  2013Jul04_08.jpg  2013Jul08_12.jpg
2011Dec22_13.jpg  2012Aug21_09.jpg  2013Jul04_09.jpg  2013Jul09_01.jpg
2011Dec22_14.jpg  2012Aug21_10.jpg  2013Jul04_10.jpg  2013Jul09_02.jpg
2011Dec22_15.jpg  2012Aug21_11.jpg  2013Jul04_11.jpg  2013Jul09_03.jpg
2011Dec22_16.jpg  2012Aug21_12.jpg  2013Jul04_12.jpg  2013Jul09_04.jpg
2011Dec22_17.jpg  2012Aug21_13.jpg  2013Jul04_13.jpg  2013Jul09_05.jpg
2011Dec22_18.jpg  2012Aug21_14.jpg  2013Jul04_14.jpg  2013Jul09_06.jpg
2011Dec22_19.jpg  2012Aug21_15.jpg  2013Jul04_15.jpg  2013Jul09_07.jpg
2011Dec22_20.jpg  2012Aug21_16.jpg  2013Jul04_16.jpg  2013Jul09_08.jpg
2011Dec22_21.jpg  2012Aug21_17.jpg  2013Jul04_17.jpg  2013Jul09_09.jpg
2011Dec22_22.jpg  2012Aug21_18.jpg  2013Jul04_18.jpg  2013Jul09_10.jpg
2011Dec22_23.jpg  2012Aug21_19.jpg  2013Jul04_19.jpg  2013Jul09_11.jpg
2011Dec22_24.jpg  2012Aug21_20.jpg  2013Jul04_20.jpg  2013Jul09_12.jpg
2011Jul04_01.jpg  2012Aug21_21.jpg  2013Jul04_21.jpg  2013Jul09_13.jpg
2011Jul04_02.jpg  2012Aug21_22.jpg  2013Jul04_22.jpg  2013Jul09_14.jpg
2011Jul04_03.jpg  2012Aug21_23.jpg  2013Jul04_23.jpg  2013Jul09_15.jpg
2011Jul04_04.jpg  2012Aug21_24.jpg  2013Jul04_24.jpg  2013Jul09_16.jpg
2011Jul04_05.jpg  2012Aug21_25.jpg  2013Jul04_25.jpg  2013Jul09_17.jpg
2011Jul04_06.jpg  2012Aug21_26.jpg  2013Jul04_26.jpg  2013Jul09_18.jpg
2011Jul04_07.jpg  2012Aug21_27.jpg  2013Jul04_27.jpg  2013May28_01.jpg
2011Jul04_08.jpg  2012Aug21_28.jpg  2013Jul04_28.jpg  2013May28_02.jpg
2011Jul04_09.jpg  2012Aug21_29.jpg  2013Jul04_29.jpg  2013May28_03.jpg
2011Jul04_10.jpg  2012Aug21_30.jpg  2013Jul04_30.jpg  2013May28_04.jpg
2011Jul04_11.jpg  2012Aug21_31.jpg  2013Jul04_31.jpg  2013May28_05.jpg
2011Jun02_01.jpg  2012Aug21_32.jpg  2013Jul04_32.jpg  2013May28_06.jpg
2011Jun02_02.jpg  2012Aug21_33.jpg  2013Jul04_33.jpg  2013May28_07.jpg
2011Jun02_03.jpg  2012Aug21_34.jpg  2013Jul04_34.jpg  2013May28_08.jpg
2011Jun02_04.jpg  2012Aug21_35.jpg  2013Jul04_35.jpg  2013May28_09.jpg
2011Jun02_05.jpg  2012Aug21_36.jpg  2013Jul04_36.jpg  2013May28_10.jpg
2011Jun02_06.jpg  2012Aug21_37.jpg  2013Jul04_37.jpg  2013May28_11.jpg
2011Jun02_07.jpg  2012Aug21_38.jpg  2013Jul04_38.jpg  2013May28_12.jpg
2011Jun02_08.jpg  2012Aug21_39.jpg  2013Jul04_39.jpg  2013May28_13.jpg
2011Jun02_09.jpg  2012Aug21_40.jpg  2013Jul04_40.jpg  2013May28_14.jpg
2011Jun02_10.jpg  2012Aug21_41.jpg  2013Jul04_41.jpg  2013May28_15.jpg
2011Jun02_11.jpg  2012Aug21_42.jpg  2013Jul04_42.jpg  2013May28_16.jpg
2011Jun02_12.jpg  2012Aug21_43.jpg  2013Jul04_43.jpg  2013May28_17.jpg
2011Jun02_13.jpg  2012Aug21_44.jpg  2013Jul04_44.jpg  2013May28_18.jpg
2011Jun02_14.jpg  2012Aug21_45.jpg  2013Jul04_45.jpg  2013May28_19.jpg
2011Jun02_15.jpg  2012Mar11_01.jpg  2013Jul05_01.jpg  2013May28_20.jpg
2011Jun02_16.jpg  2012Mar11_02.jpg  2013Jul05_02.jpg  2013May28_21.jpg
2011Jun02_17.jpg  2012Mar11_03.jpg  2013Jul05_03.jpg  2013May28_22.jpg
2011Jun02_18.jpg  2012Mar11_04.jpg  2013Jul05_04.jpg  2013May28_23.jpg
2011Jun02_19.jpg  2012Mar11_05.jpg  2013Jul05_05.jpg  2013May28_24.jpg
2011Jun02_20.jpg  2012Mar11_06.jpg  2013Jul05_06.jpg  2013May28_25.jpg
2011Jun02_21.jpg  2012Mar11_07.jpg  2013Jul05_07.jpg  2013May28_26.jpg
2011Jun02_22.jpg  2012Mar11_08.jpg  2013Jul05_08.jpg  2013May28_27.jpg
2011Jun02_23.jpg  2012Mar11_09.jpg  2013Jul05_09.jpg  2013May28_28.jpg
2011Jun02_24.jpg  2012Mar11_10.jpg  2013Jul05_10.jpg  2013May28_29.jpg
2011Jun02_25.jpg  2012Mar11_11.jpg  2013Jul05_11.jpg  2013May28_30.jpg
2011Jun02_26.jpg  2012Mar11_12.jpg  2013Jul05_12.jpg  2013May28_31.jpg
2011Jun02_27.jpg  2012Mar11_13.jpg  2013Jul05_13.jpg  2013May28_32.jpg
2011Jun02_28.jpg  2012Mar11_14.jpg  2013Jul05_14.jpg  2013May28_33.jpg
2011Jun02_29.jpg  2012Mar11_15.jpg  2013Jul05_15.jpg  2013May28_34.jpg
2011Jun03_01.jpg  2012Mar11_16.jpg  2013Jul05_16.jpg  2013May28_35.jpg
2011Jun03_02.jpg  2012Mar11_17.jpg  2013Jul05_17.jpg  2013May28_36.jpg
2011Jun03_03.jpg  2012Mar11_18.jpg  2013Jul05_18.jpg  2013May29_01.jpg
2011Jun03_04.jpg  2012Mar11_19.jpg  2013Jul05_19.jpg  2013May29_02.jpg
2011Jun04_01.jpg  2012Mar11_20.jpg  2013Jul05_20.jpg  2013May29_03.jpg
2011Jun04_02.jpg  2012Mar11_21.jpg  2013Jul07_01.jpg  2013May29_04.jpg
2011Jun04_03.jpg  2012Mar11_22.jpg  2013Jul07_02.jpg  2013May29_05.jpg
2011Jun04_04.jpg  2012Mar11_23.jpg  2013Jul07_03.jpg  2013May29_06.jpg
2011Jun04_05.jpg  2012Mar11_24.jpg  2013Jul07_04.jpg  2013May29_07.jpg
2011Jun04_06.jpg  2012Mar11_25.jpg  2013Jul07_05.jpg  2013May29_08.jpg
2011Jun04_07.jpg  2012Mar11_26.jpg  2013Jul07_06.jpg  2013May29_09.jpg
2011Jun04_08.jpg  2012Mar11_27.jpg  2013Jul07_07.jpg  2013May29_10.jpg
2011Jun04_09.jpg  2012Mar11_28.jpg  2013Jul07_08.jpg  2013May29_11.jpg
2011Jun04_10.jpg  2012Mar11_29.jpg  2013Jul07_09.jpg  2013May29_12.jpg
2011Jun04_11.jpg  2012Mar11_30.jpg  2013Jul07_10.jpg  2013May29_13.jpg
2011Jun04_12.jpg  2012Mar11_31.jpg  2013Jul07_11.jpg  2013May29_14.jpg
2011Jun04_13.jpg  2012Mar11_32.jpg  2013Jul07_12.jpg  2013May29_15.jpg
2011Jun04_14.jpg  2012Mar11_33.jpg  2013Jul07_13.jpg  2013May29_16.jpg
2011Jun04_15.jpg  2012Mar11_34.jpg  2013Jul07_14.jpg  2013May29_17.jpg
2011Jun04_16.jpg  2012Mar11_35.jpg  2013Jul07_15.jpg  2013May29_18.jpg
2011Jun04_17.jpg  2012Mar12_01.jpg  2013Jul07_16.jpg  2013May29_19.jpg
2011Jun04_18.jpg  2012Mar12_02.jpg  2013Jul07_17.jpg  2013May29_20.jpg
2011Jun04_19.jpg  2012Mar12_03.jpg  2013Jul07_18.jpg  2013May29_21.jpg
2011Jun04_20.jpg  2012Mar12_04.jpg  2013Jul07_19.jpg  2013May29_22.jpg
2011Jun04_21.jpg  2012Mar12_05.jpg  2013Jul07_20.jpg  2013May29_23.jpg
2011Jun04_22.jpg  2012Mar12_06.jpg  2013Jul07_21.jpg  2013May29_24.jpg
2012Aug20_01.jpg  2012Mar12_07.jpg  2013Jul07_22.jpg  2013May29_25.jpg
2012Aug20_02.jpg  2012Mar12_08.jpg  2013Jul07_23.jpg  2013May29_26.jpg
2012Aug20_03.jpg  2012Mar12_09.jpg  2013Jul07_24.jpg  2013May29_27.jpg
2012Aug20_04.jpg  2012Mar12_10.jpg  2013Jul07_25.jpg  2013May29_28.jpg
2012Aug20_05.jpg  2012Mar12_11.jpg  2013Jul07_26.jpg  2013May29_29.jpg
2012Aug20_06.jpg  2012Mar12_12.jpg  2013Jul07_27.jpg  2013May29_30.jpg
2012Aug20_07.jpg  2012Mar12_13.jpg  2013Jul07_28.jpg  2013May29_31.jpg
2012Aug20_08.jpg  2012Mar13_01.jpg  2013Jul07_29.jpg  2013May29_32.jpg
2012Aug20_09.jpg  2012Mar13_02.jpg  2013Jul07_30.jpg  2013May29_33.jpg
2012Aug20_10.jpg  2012Mar13_03.jpg  2013Jul07_31.jpg  2013May29_34.jpg
2012Aug20_11.jpg  2012Mar13_04.jpg  2013Jul07_32.jpg  2013May29_35.jpg
2012Aug20_12.jpg  2012Mar13_05.jpg  2013Jul07_33.jpg  2013May29_36.jpg
2012Aug20_13.jpg  2012Mar13_06.jpg  2013Jul07_34.jpg  2013May30_01.jpg
2012Aug20_14.jpg  2012Mar13_07.jpg  2013Jul07_35.jpg  2013May30_02.jpg
2012Aug20_15.jpg  2012Mar13_08.jpg  2013Jul07_36.jpg  2013May30_03.jpg
2012Aug20_16.jpg  2012Mar13_09.jpg  2013Jul07_37.jpg  2013May30_04.jpg
2012Aug20_17.jpg  2012Mar13_10.jpg  2013Jul07_38.jpg  2013May30_05.jpg
2012Aug20_18.jpg  2012Mar13_11.jpg  2013Jul07_39.jpg  2013May30_06.jpg
2012Aug20_19.jpg  2012Mar13_12.jpg  2013Jul07_40.jpg  2013May30_07.jpg
2012Aug20_20.jpg  2012Mar13_13.jpg  2013Jul07_41.jpg  2013May30_08.jpg
2012Aug20_21.jpg  2012Mar13_14.jpg  2013Jul07_42.jpg  2013May30_09.jpg
2012Aug20_22.jpg  2013Jul03_01.jpg  2013Jul07_43.jpg  2013May30_10.jpg
2012Aug20_23.jpg  2013Jul03_02.jpg  2013Jul07_44.jpg  2013May30_11.jpg
2012Aug20_24.jpg  2013Jul03_03.jpg  2013Jul07_45.jpg
2012Aug20_25.jpg  2013Jul03_04.jpg  2013Jul07_46.jpg
2012Aug20_26.jpg  2013Jul03_05.jpg  2013Jul07_47.jpg

Okay. That list of files is somewhat unwieldy. But by using wildcards, it's possible to prune the list. When specifying file names or paths, you can use the special character "*" which matches zero or more characters. For example, "a*" would match "a", "at", "abacab", etc. In the pictures directory above, the names of the image files happen to be yearmonthday_num.jpg. So to get a list of just the pictures from 2011...

$ ls 2011*
2011Dec22_01.jpg  2011Dec22_24.jpg  2011Jun02_12.jpg  2011Jun04_02.jpg
2011Dec22_02.jpg  2011Jul04_01.jpg  2011Jun02_13.jpg  2011Jun04_03.jpg
2011Dec22_03.jpg  2011Jul04_02.jpg  2011Jun02_14.jpg  2011Jun04_04.jpg
2011Dec22_04.jpg  2011Jul04_03.jpg  2011Jun02_15.jpg  2011Jun04_05.jpg
2011Dec22_05.jpg  2011Jul04_04.jpg  2011Jun02_16.jpg  2011Jun04_06.jpg
2011Dec22_06.jpg  2011Jul04_05.jpg  2011Jun02_17.jpg  2011Jun04_07.jpg
2011Dec22_07.jpg  2011Jul04_06.jpg  2011Jun02_18.jpg  2011Jun04_08.jpg
2011Dec22_08.jpg  2011Jul04_07.jpg  2011Jun02_19.jpg  2011Jun04_09.jpg
2011Dec22_09.jpg  2011Jul04_08.jpg  2011Jun02_20.jpg  2011Jun04_10.jpg
2011Dec22_10.jpg  2011Jul04_09.jpg  2011Jun02_21.jpg  2011Jun04_11.jpg
2011Dec22_11.jpg  2011Jul04_10.jpg  2011Jun02_22.jpg  2011Jun04_12.jpg
2011Dec22_12.jpg  2011Jul04_11.jpg  2011Jun02_23.jpg  2011Jun04_13.jpg
2011Dec22_13.jpg  2011Jun02_01.jpg  2011Jun02_24.jpg  2011Jun04_14.jpg
2011Dec22_14.jpg  2011Jun02_02.jpg  2011Jun02_25.jpg  2011Jun04_15.jpg
2011Dec22_15.jpg  2011Jun02_03.jpg  2011Jun02_26.jpg  2011Jun04_16.jpg
2011Dec22_16.jpg  2011Jun02_04.jpg  2011Jun02_27.jpg  2011Jun04_17.jpg
2011Dec22_17.jpg  2011Jun02_05.jpg  2011Jun02_28.jpg  2011Jun04_18.jpg
2011Dec22_18.jpg  2011Jun02_06.jpg  2011Jun02_29.jpg  2011Jun04_19.jpg
2011Dec22_19.jpg  2011Jun02_07.jpg  2011Jun03_01.jpg  2011Jun04_20.jpg
2011Dec22_20.jpg  2011Jun02_08.jpg  2011Jun03_02.jpg  2011Jun04_21.jpg
2011Dec22_21.jpg  2011Jun02_09.jpg  2011Jun03_03.jpg  2011Jun04_22.jpg
2011Dec22_22.jpg  2011Jun02_10.jpg  2011Jun03_04.jpg
2011Dec22_23.jpg  2011Jun02_11.jpg  2011Jun04_01.jpg

The "*" character doesn't have to be at the end and you can also use multiple wildcard characters. Want to get a list of pictures from the 4th of July no matter what year?

$ ls *Jul04*
2011Jul04_01.jpg  2013Jul04_04.jpg  2013Jul04_18.jpg  2013Jul04_32.jpg
2011Jul04_02.jpg  2013Jul04_05.jpg  2013Jul04_19.jpg  2013Jul04_33.jpg
2011Jul04_03.jpg  2013Jul04_06.jpg  2013Jul04_20.jpg  2013Jul04_34.jpg
2011Jul04_04.jpg  2013Jul04_07.jpg  2013Jul04_21.jpg  2013Jul04_35.jpg
2011Jul04_05.jpg  2013Jul04_08.jpg  2013Jul04_22.jpg  2013Jul04_36.jpg
2011Jul04_06.jpg  2013Jul04_09.jpg  2013Jul04_23.jpg  2013Jul04_37.jpg
2011Jul04_07.jpg  2013Jul04_10.jpg  2013Jul04_24.jpg  2013Jul04_38.jpg
2011Jul04_08.jpg  2013Jul04_11.jpg  2013Jul04_25.jpg  2013Jul04_39.jpg
2011Jul04_09.jpg  2013Jul04_12.jpg  2013Jul04_26.jpg  2013Jul04_40.jpg
2011Jul04_10.jpg  2013Jul04_13.jpg  2013Jul04_27.jpg  2013Jul04_41.jpg
2011Jul04_11.jpg  2013Jul04_14.jpg  2013Jul04_28.jpg  2013Jul04_42.jpg
2013Jul04_01.jpg  2013Jul04_15.jpg  2013Jul04_29.jpg  2013Jul04_43.jpg
2013Jul04_02.jpg  2013Jul04_16.jpg  2013Jul04_30.jpg  2013Jul04_44.jpg
2013Jul04_03.jpg  2013Jul04_17.jpg  2013Jul04_31.jpg  2013Jul04_45.jpg

What about using "*" by itself as in ls *? Because "*" matches zero or more characters that should match any and all files in the directory, right? Well, not necessarily. It turns out that there is an exception to wildcard name matching. Files that begin with "." (hidden files) are not included. Wildcards are not just used with the ls command. Most commands that take files as arguments can use wildcards.

$ mkdir 4th_pics

$ mv *Jul04* 4th_pics/

$ ls *Jul04*
ls: cannot access *Jul04*: No such file or directory

$ cd 4th_pics

$ ls
2011Jul04_01.jpg  2013Jul04_04.jpg  2013Jul04_18.jpg  2013Jul04_32.jpg
2011Jul04_02.jpg  2013Jul04_05.jpg  2013Jul04_19.jpg  2013Jul04_33.jpg
2011Jul04_03.jpg  2013Jul04_06.jpg  2013Jul04_20.jpg  2013Jul04_34.jpg
2011Jul04_04.jpg  2013Jul04_07.jpg  2013Jul04_21.jpg  2013Jul04_35.jpg
2011Jul04_05.jpg  2013Jul04_08.jpg  2013Jul04_22.jpg  2013Jul04_36.jpg
2011Jul04_06.jpg  2013Jul04_09.jpg  2013Jul04_23.jpg  2013Jul04_37.jpg
2011Jul04_07.jpg  2013Jul04_10.jpg  2013Jul04_24.jpg  2013Jul04_38.jpg
2011Jul04_08.jpg  2013Jul04_11.jpg  2013Jul04_25.jpg  2013Jul04_39.jpg
2011Jul04_09.jpg  2013Jul04_12.jpg  2013Jul04_26.jpg  2013Jul04_40.jpg
2011Jul04_10.jpg  2013Jul04_13.jpg  2013Jul04_27.jpg  2013Jul04_41.jpg
2011Jul04_11.jpg  2013Jul04_14.jpg  2013Jul04_28.jpg  2013Jul04_42.jpg
2013Jul04_01.jpg  2013Jul04_15.jpg  2013Jul04_29.jpg  2013Jul04_43.jpg
2013Jul04_02.jpg  2013Jul04_16.jpg  2013Jul04_30.jpg  2013Jul04_44.jpg
2013Jul04_03.jpg  2013Jul04_17.jpg  2013Jul04_31.jpg  2013Jul04_45.jpg

Another wildcard character that is worth mentioning is "?" which matches exactly one character. There are also other wildcards that allow you to match ranges of characters, negate matches, etc. You will probably find yourself using wildcards quite often.

Permissions

Above we mentioned "permissions". Permissions are used to restrict (or allow) access to files and directories.

Each user on Kodiak has a username and is a member of default group. Here, the username is "bobby" and the default group is "users". Users can also be members of other groups, not just the default. You can use the id command to see your username and group membership information.

$ id
uid=520(bobby) gid=100(users) groups=100(users)

You can also use the whoami and groups commands to display your username and group membership. So what are those numbers, 520 and 100? Users and groups are actually identified by numeric values called user identifiers and group identifiers. These are often abbreviated as "uid" and "gid". The names "bobby" and "users" are really labels associated with uid 520 and gid 100 and are used to make things easier for humans to read. Yes, as far a Kodiak is concerned, you are just a number.

Each file (and directory) has a specific owner and group associated with it. That is, a file is owned by a specific user and assigned a group. Using ls -l we can see the files' owners and groups in the output. Want to see the numeric uid and gid instead of the username and group name? Use ls -n.

$ pwd
/home/bobby

$ ls -la
total 3
drwxr-x--x  6  bobby users   5 Oct  8 15:27 .
drwxr-xr-x 115 root  root  114 Oct  1 14:09 ..
drwxr-x---  2  bobby users   0 Oct  8 15:36 backups
-rw-------  1  bobby users  87 Oct  7 16:15 .bashrc
drwxr-xr-x  2  bobby users   3 Oct  8 15:23 documents
drwxr-x---  2  bobby users   1 Oct  8 15:36 downloads
drwxr-x---  4  bobby users   2 Oct  7 14:14 pictures

$ ls -n .bashrc
-rw------- 1 520 501 87 Oct  7 16:15 .bashrc

$ ls -l documents
total 16
-rw-r--r-- 1 bobby users 12424 Oct  8 13:17 annual_report.pdf
-rw------- 1 bobby users  1169 Oct  7 15:54 budget
-rw-r--r-- 1 bobby users  1324 Oct  7 15:56 letter.txt

As you can see, almost all of the items have as the owner "bobby" (that's you) and group "users". Note that the parent directory '..' (in this case, "/home") is owned by "root". The user "root" is the system administrator account and is sometimes called the "superuser".

Earlier, when describing the output from ls -l, we mentioned that rwxr-xr-x were the file's (or directory's) permissions. So what does that actually mean? There are three types of permissions:

PermissionAbbr. Needed to... (files) Needed to... (directories)
Read"r" Read a file's contents List the files in the directory
Write"w" Modify or delete the file Modify the directory, i.e., create/delete files within it
Execute"x" Execute the file/program Search the directory (see below)

Permissions on a file or directory apply to three different classes: 1) the file's or directory's owner or user; 2) members of the file's or directory's assigned group; and 3) others (sometimes called "world"). So putting the permissions for the file "letter.txt" (rw-r--r--) and classes together, we get:

User Group Others
rw- r-- r--

This tells us that the owner of letter.txt (bobby) has read and write access but not execute. Members of the group "users" have read access but not write or execute. Likewise, everyone else also has read access but not write or execute.

To modify permissions of a file or directory, use the chmod (change mode) command. To set or add a permission, that is, to allow that permission, you would use the argument class+permission. For example, to give everyone/others read permission you would use chmod o+r. The possible classes are u (user), g (group), o (others). There is also a special class a (all) which means all three classes. "All" is not a synonym for others. To clear or remove a permission, use class-permission (e.g., chmod o-r).

The method above uses the symbolic mode for modifying permissions. There is second absolute (or numeric) mode as well. The permissions are actually bits and the three types of permissions (read, write, execute) for a class (user, group, others) are sets of 3 bits. The permissions (rwx) map to bit values (100, 010, 001) and their corresponding octal values (4, 2, 1). To set a permission, add the corresponding octal value, e.g., r-x would be 4+1 = 5. You need to do this for all three classes, e.g., rwxr-x--- = 4+2+1,4+1,0 = 750. (You are actually adding 400+200+100 + 040+010 + 000, but it's easier to calculate them as separate digits.) So to set those permissions, you would use chmod 750.

Absolute/numeric mode can be somewhat confusing at first but it's really not too hard once you get the hang of it. There may be times where you would be required to use abosolute permissions.

The examples in this document will use symbolic mode.

File Permissions

So right now everyone can read the file letter.txt. Let's fix that.

$ ls -l letter.txt
-rw-r--r-- 1 bobby users 1324 Oct  7 15:56 letter.txt

$ chmod o-r letter.txt

$ ls -l letter.txt
-rw-r----- 1 bobby users 1324 Oct  7 15:56 letter.txt

Now what happens when another user, betty, tries to read the file?

$ whoami
betty

$ cat /home/bobby/documents/letter.txt
cat: /home/bobby/documents/letter.txt: Permission denied

What about the execute permission. What does that mean? Earlier, we stated that just about everything in Linux is a file, including the commands you run. The execute permission is what permits the file to be run as a command as well as who can run it. So commands are files, too? Most of them, yes. The ls command is located in the "/bin" directory. You can use the which command to find the full path of a command.

$ which ls
/bin/ls

$ ls -l /bin/ls
-rwxr-xr-x 1 root root 90168 Jan 21  2009 /bin/ls

Setting the execute permission does not make a file executable; it allows an executable file, such as a compiled program or shell script, to be run. (We'll talk about shell scripts later.) So happens when you try to run an executable program that doesn't have the execute permission set? Let's try it with the fictitious command howdy.

$ which howdy
/usr/local/bin/howdy

$ ls -l /usr/local/bin/howdy
-rw-r--r-- 1 root root 27 Apr 01 12:01 /usr/local/bin/howdy

$ howdy
-bash: /usr/local/bin/howdy: Permission denied

"Permission denied". That makes sense because the execute permission is not set. And if it were?

$ ls -l /usr/local/bin/howdy
-rwxr-xr-x 1 root root 27 Apr 01 12:01 /usr/local/bin/howdy

$ howdy
Howdy, bobby!

What happens when you set the execute permission on a non-executable file and try to run it?

$ ls -l letter.txt
-rw-r----- 1 bobby users 1324 Oct  7 15:56 letter.txt

$ chmod u+x letter.txt

$ ls -l letter.txt
-rwxr----- 1 bobby users 1324 Oct  7 15:56 letter.txt

$ ./letter.txt
./letter.txt: line 1: Lorem: command not found
./letter.txt: line 3: Lorem: command not found
./letter.txt: line 4: postulant: command not found
./letter.txt: line 5: tollit: command not found
./letter.txt: line 6: adversarium: command not found
./letter.txt: line 8: Nobis: command not found
./letter.txt: line 9: consetetur: command not found
./letter.txt: line 10: mea: command not found
./letter.txt: line 11: assum: command not found
./letter.txt: line 12: dictas: command not found
./letter.txt: line 14: An: command not found
./letter.txt: line 15: consectetuer: command not found
./letter.txt: line 16: postea: command not found
./letter.txt: line 17: syntax error near unexpected token `in'
./letter.txt: line 17: `in liber quodsi lobortis, quis altera essent at his.'

$ chmod -x letter.txt

Okay. Let's not do that again. By the way, did you happen to notice that when we tried to run the letter.txt file, we included a "./" in front of it? Sometimes you have to refer to a program using its path (full or relative) and not just by name. We'll explain why later.

Directory Permissions

Permissions have somewhat different meanings for directories. It might help to think of a directory as a special file that contains the list of all of the files and sub-directories within it.

We'll start with the execute permission. For directories, the execute permission is sometimes called "search" permission. It is needed in order to cd into the directory as well as access the files or sub-directories within it. It does not give access the actual list of files. To access a file within the directory, one would need to already know the name of the file as well as have read access to the file itself.

Below, user betty wants to allow bobby (you) to read the file "/home/betty/shared/misc.txt". You can see that although everyone can read misc.txt, the enclosing directory only has search permission for the owner. Bobby is unable to read the file.

[betty@n130 ~] $ whoami
betty

[betty@n130 ~]$ ls -l
total 16
drwx------ 2 betty users 4096 Oct 21 13:17 documents
drwx------ 2 betty users 4096 Oct 21 13:17 downloads
drwx------ 2 betty users 4096 Oct 21 13:17 pictures
drwx------ 2 betty users 4096 Oct 22 09:28 shared

[betty@n130 ~]$ ls -l shared
total 4
-rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt


[bobby@n130 ~]$ whoami bobby [bobby@n130 ~]$ cat /home/betty/shared/misc.txt cat: /home/betty/shared/misc.txt: Permission denied

Rather than give everyone access, betty can add group search permission. Both bobby and betty are members of the "users" group. Note that by adding the -d option to ls you can see the permissions for a directory itself rather than the directory's contents. So now that the directory "/home/betty/shared" has search permission set, bobby should be able to access the file within it. Correct?

[betty@n130 ~]$ chmod g+x shared

[betty@n130 ~]$ ls -ld shared
drwx--x--- 2 betty users 4096 Oct 22 09:28 shared


[bobby@n130 ~]$ cat /home/betty/shared/misc.txt cat: /home/betty/shared/misc.txt: Permission denied

Not yet. In order to access the file misc.txt, not only does its directory (/home/betty/shared) need search permission, but so does its directory's parent directory (/home/betty), and its parent directory's parent (/home) all the way up to root (/). It's turtles all the way up. The /home and / directories already have have search permissions set (which is good because you wouldn't be able to chmod those - they are owned by user root). So adding group search permission to /home/betty should allow bobby to access it. Also because we are adding group permissions and not others permissions, user bubba (who is not a member of the "users" group) will not have access.

[betty@n130 ~]$ ls -ld /home/betty
drwx------ 8 betty users 4096 Oct 21 13:57 /home/betty

[betty@n130 ~]$ chmod g+x /home/betty

[betty@n130 ~]$ ls -ld /home/betty
drwx--x--- 8 betty users 4096 Oct 21 13:57 /home/betty

[betty@n130 ~]$ ls -ld /home
drwxr-xr-x 6 root root 4096 Oct 17 14:05 /home

[betty@n130 ~]$ ls -ld /
dr-xr-xr-x 26 root root 4096 Oct 17 14:22 /


[bobby@n130 ~]$ ls /home/betty/shared ls: cannot open directory /home/betty/shared: Permission denied [bobby@n130 ~]$ cat /home/betty/shared/misc.txt Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia tollit definitiones qui in. Ea vis eripuit nusquam. Eos ceteros tibique adversarium ne, nisl tota convenire at pri.
[bubba@n130 ~]$ whoami bubba [bubba@n130 ~]$ groups grads [bubba@130 ~]$ ls /home/betty/shared ls: cannot access /home/betty/shared: Permission denied [bubba@n130 ~]$ cat /home/betty/shared/misc.txt cat: /home/betty/shared/misc.txt: Permission denied

Read permission allows one to list the contents of the directory. While there are times where you want to allow search but disallow reading on a directory, you typically will not see the reverse. That is, if you set read permission, you will likely set search permission as well.

[betty@n130 ~]$ ls -ld shared
drwx--x--- 2 betty users 4096 Oct 22 09:28 shared


[bobby@n130 ~]$ cd /home/betty/shared [bobby@n130 shared]$ ls ls: cannot open directory .: Permission denied
[betty@n130 ~]$ chmod g+r shared [betty@n130 ~]$ ls -ld shared drwxr-x--- 2 betty users 4096 Oct 22 09:28 shared
[bobby@n130 shared]$ ls flag.gif misc.txt

Write permission allows one to create, delete, or rename files in the directory. The touch command is a simple way to create an empty file. (By the way, if you touch an existing file, it updates the file's modification timestamp.) As you can see below, when betty adds group write permission to the /home/betty/shared directory, bobby is able to create a file within it.

[betty@n130 shared]$ ls -ld /home/betty/shared
drwxr-x--- 2 betty users 4096 Oct 22 10:29 /home/betty/shared


[bobby@n130 ~]$ cd /home/betty/shared [bobby@n130 shared]$ ls flag.gif misc.txt [bobby@n130 shared]$ touch new_file touch: cannot touch `new_file': Permission denied
[betty@n130 shared]$ chmod g+w /home/betty/shared [betty@n130 shared]$ ls -ld /home/betty/shared drwxrwx--- 2 betty users 4096 Oct 22 10:29 /home/betty/shared
[bobby@n130 shared]$ ls -l total 16 -rw------- 1 betty users 10628 Oct 22 10:28 flag.gif -rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt [bobby@n130 shared]$ touch new_file [bobby@n130 shared]$ ls -l total 16 -rw------- 1 betty users 10628 Oct 22 10:28 flag.gif -rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt -rw-r--r-- 1 bobby users 0 Oct 24 11:23 new_file

So bobby is now able to create a file within /home/betty/shared. But because bobby does not have any type of access (read nor write) to the flag.gif file, you shouldn't be able to do anything to that file, right?

[bobby@n130 shared]$ ls -l
total 16
-rw------- 1 betty users 10628 Oct 22 10:28 flag.gif
-rw-r--r-- 1 betty users   268 Oct 21 13:18 misc.txt
-rw-r--r-- 1 bobby users     0 Oct 24 11:23 new_file

[bobby@n130 shared]$ mv flag.gif huh.gif
[bobby@n130 shared]$ ls -l
total 16
-rw------- 1 betty users 10628 Oct 22 10:28 huh.gif
-rw-r--r-- 1 betty users   268 Oct 21 13:18 misc.txt
-rw-r--r-- 1 bobby users     0 Oct 24 11:23 new_file

[bobby@n130 shared]$ rm huh.gif
rm: remove write-protected regular file `huh.gif'? y

[bobby@n130 shared]$ ls -l
total 12
-rw-r--r-- 1 betty users   268 Oct 21 13:18 misc.txt
-rw-r--r-- 1 bobby users     0 Oct 24 11:23 new_file

So not only were you able to rename the file, you were even able to delete it as well. What's going on here? Recall that earlier it was said that you can think of a directory as a special file that contains the list of all of the files and sub-directories within it. So adding write permission to the directory means we can modify that list. This includes renaming the entries in that list thus renaming the files in the directory. It also includes removing entries from that list thus deleting the files the files (although we did have to confirm that we wanted to delete a write-protected file). Without write permissions on the file, we would not have been able to edit and modify the contents of the file, but the ability to delete a file that we don't own might be a problem.

But what if we don't want someone to be able to rename or delete files that they don't own? It was mentioned above that the three types of permissions for the three classes were three sets of three bits. There is actually a fourth set of permission-related bits. These extra bits are called the setuid (set user id), setgid (set group id), and sticky bits. The sticky bit is the one we are interested in. If set, it prevents a user from deleting or renaming a file in the directory unless they own the file or the directory. To set or clear the sticky bit, use chmod +t or chmod -t.

[betty@n130 ~]$ ls -ld shared
drwxr-xr-x 2 betty users 4096 Oct 24 15:17 shared

[betty@n130 ~]$ chmod +t shared

[betty@n130 ~]$ ls -ld shared
drwxr-xr-t 2 betty users 4096 Oct 24 15:17 shared

Notice that the world execute/search permission is now displayed as t. The sticky bit is set. If it were displayed as an uppercase T, that would have meant that the sticky bit is set but world execute/search permission was not. So now what happens when we try to delete or rename the file?

[bobby@n130 shared]$ pwd
/home/betty/shared

[bobby@n130 shared]$ ls -ld .
drwxr-xr-t 2 betty users 4096 Oct 24 15:17 .

[bobby@n130 shared]$ ls -l
total 16
-rw-r--r-- 1 betty users 10628 Oct 22 10:28 flag.gif
-rw-r--r-- 1 betty users   186 Oct 24 11:45 misc.txt
-rw-r--r-- 1 bobby users     0 Oct 24 11:23 new_file

[bobby@n130 shared]$ rm flag.gif
rm: remove write-protected regular file `flag.gif'? y
rm: cannot remove `flag.gif': Operation not permitted

[bobby@n130 shared]$ mv flag.gif other.gif
mv: cannot move `flag.gif' to `other.gif': Operation not permitted

[bobby@n130 shared]$ rm new_file

[bobby@n130 shared]$ ls -l
total 16
-rw-r--r-- 1 betty users 10628 Oct 22 10:28 flag.gif
-rw-r--r-- 1 betty users   186 Oct 24 11:45 misc.txt

Now that the sticky bit for /home/betty/shared is set, we are unable to delete or rename the flag.gif file because we aren't the file's or enclosing directory's owner. Note that because we were the owner of the file "new_file", we were able to delete it. One of the directories that has its sticky bit set is "/tmp", a directory on the system that is writable by all users and is used to hold temporary files.

We won't discuss the other bits (setuid and setgid) here. So where did the name "sticky" bit come from? If commands are actually files, then each time someone wants to run the ls command, the system has to read the ls file from disk and place it into memory before it can run. This takes time. Because ls is a popular command and is run often, it would be faster for everybody if the ls file just remained in memory, that is, it were stuck in memory all the time. Programs with their sticky bit set got stuck in memory. Okay, that was an over-simplification but it's fairly accurate. These days, most systems ignore the the sticky bit for files and only use it for directories instead. But the name "sticky" bit has stuck.


Nifty Stuff


A Few More Useful Commands

Below are some more commands worth mentioning. Some of them, such as echo may seem very simplistic, but combined with other commands or included in shell scripts, they can be quite useful. For more detailed information on these commands, check the on-line manual pages with man.

echo — Output a line of text.

$ echo Howdy!
Howdy!

date — Prints the system date and time.

$ date
Fri Oct 25 13:46:28 CDT 2013

w — Show who is logged on and what they are doing. This is similar to the who command but displays more information. At the very top you can see how long it has been since the system was rebooted (also known as uptime) as well as how busy the system is (load average). For each user, you can see when they logged in (and from where), how long they've been idle, and the command they are currently running.

$ w
 14:27:50 up 45 days, 44 min, 7 users,  load average: 1.27, 1.18, 1.11
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     tty1     -                10Sep13 45days  0.01s  0.01s -bash
root     pts/1    172.16.1.12      10:28   31:03   0.15s  0.15s -bash
betty    pts/2    172.16.8.20      14:12    4:08   1.05s  0.02s less ./test.f
root     :0       -                09Oct13 ?xdm?   2days  0.14s /usr/bin/gnome-
bobby    pts/3    172.16.8.24      13:50    1.00s  0.05s  0.00s w
bubba    pts/4    93.184.216.119   10:25   15:37   0.78s  0.04s man gcc
betty    pts/5    172.16.8.20      10:31   24:20   0.07s  0.07s -bash

top — Display and update sorted information about processes. This command will continuously display the most CPU intensive programs currently running on the system. If the system seems slow, you can use top to see if something (or someone!) is hogging the CPU. When you are ready to quit top, press q.

$ top
top - 13:50:56 up 45 days, 8 min, 32 users,  load average: 1.15, 1.14, 1.10
Tasks: 453 total,   1 running, 452 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.4%us,  0.9%sy,  0.0%ni, 97.6%id,  0.0%wa,  0.1%hi,  0.0%si,  0.0%st
Mem:  16435208k total,  5800124k used, 10635084k free,    33076k buffers
Swap: 16386292k total,   723220k used, 15663072k free,   206516k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
20054 bobby     15   0 12996 1360  816 R  1.0  0.0   0:00.06 top                
 3407 betty     15   0 68292 2308  956 S  0.3  0.0  15:02.57 less           
 4080 bubba     15   0 56408 2592 1844 S  0.3  0.0   0:04.69 man            
20064 betty     15   0 13776 2136  248 D  0.3  0.0   0:00.01 sshd           
20065 betty     17   0  3812  460  368 S  0.3  0.0   0:00.01 wc                 
20066 betty     16   0 68808 2040  260 S  0.3  0.0   0:00.01 ./a.out          
24945 bubba     15   0 92212 2264  840 S  0.3  0.0  35:50.09 sshd               
    1 root      15   0 10344  576  540 S  0.0  0.0   0:05.86 init               
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:22.58 migration/0        
    3 root      34  19     0    0    0 S  0.0  0.0   0:05.37 ksoftirqd/0        
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0         
    5 root      RT  -5     0    0    0 S  0.0  0.0   2:39.97 migration/1        
    6 root      34  19     0    0    0 S  0.0  0.0   0:07.07 ksoftirqd/1        
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1         
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:05.05 migration/2        
    9 root      34  19     0    0    0 S  0.0  0.0   0:01.36 ksoftirqd/2        
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2         

wc — Print line, word, and character counts for a file.

$ wc letter.txt
  27  189 1324 letter.txt

$ wc --lines letter.txt
27 letter.txt

file — Determine file type.

$ file *
annual_report.pdf: PDF document, version 1.3
budget:            ASCII text
letter.txt:        ASCII text

$ file /home/betty/shared/flag.gif
/home/betty/shared/flag.gif: GIF image data, version 89a, 68 x 50

$ file /tmp
/tmp: sticky directory

gzip, gunzip — Compress or expand files. The gzip (GNU Zip) command will compress a file, add a .gz suffix to the compressed file, and then remove the old (uncompressed) file. The gunzip command does the opposite.

$ ls
poster.pdf  report.pdf.bak

$ file *
poster.pdf:     PDF document, version 1.3
report.pdf.bak: PDF document, version 1.3

$ gzip poster.pdf

$ ls -l
total 28
-rw-r--r-- 1 bobby users 10956 Oct 17 14:30 poster.pdf.gz
-rw-r--r-- 1 bobby users 12424 Oct 17 14:30 report.pdf.bak

$ file poster.pdf.gz
poster.pdf.gz: gzip compressed data, was "poster.pdf", from Unix,
last modified: Thu Oct 17 14:30:39 2013

$ gunzip poster.pdf.gz

$ ls -l
total 32
-rw-r--r-- 1 bobby users 12424 Oct 17 14:30 poster.pdf
-rw-r--r-- 1 bobby users 12424 Oct 17 14:30 report.pdf.bak

hostname — Show the system's host name.

$ hostname
n130

ssh — Remote login program. You can use the ssh command to log in to the compute nodes if necessary.

[bobby@n130 ~]$ hostname
n130

[bobby@n130 ~]$ who
root     tty1         2013-09-10 13:48
betty    pts/2        2013-10-02 13:48 (172.16.8.20)
bobby    pts/3        2013-10-02 13:51 (172.16.8.24)
bubba    pts/1        2013-10-02 08:16 (93.184.216.119)
root     :0           2013-10-09 15:39

[bobby@n130 ~]$ ssh n007
Last login: Thu Sep 12 12:59:49 2013 from 192.168.1.130

[bobby@n007 ~]$ hostname
n007

[bobby@n007 ~]$ who
bobby    pts/0        2013-10-25 15:10 (192.168.1.130)

[bobby@n007 ~]$ exit
logout

Connection to n007 closed.
[bobby@n130 ~]$ 

diff — Compare files line by line. A useful option is -y which displays the two files side by side.

grep — Search a file for a string or pattern. The grep command has an odd name but is very useful.

$ cat budget
     Monthly Income & Expense Estimates

INCOME: 
  Salary                                         $3,000 
  Interest                                           50 
  Other Income                                       50
                                                 ------ 
  Total Income                                   $3,100

EXPENSES: 
  Mortgage Payment                     $700 
  Auto Loan Payment                     350 
  Student Load Payment                  200
  Food and Groceries                    500 
  Utilities                             250 
  Home insurance                        120 
  Auto insurance                         50 
  Life Insurance                         20 
  Clothing                               50 
  Retirement Fund                       150 
  Education Fund                        100 
  Auto Fuel and Maintenance             160 
  Entertainment                         120 
  Miscellaneous                          50 
                                     ------ 
  Total Expenses                     $2,820 

$ grep Auto budget
  Auto Loan Payment                     350 
  Auto insurance                         50 
  Auto Fuel and Maintenance             160 

With the -n option, grep will display line numbers as well. To tell grep to ignore case when searching, add the -i option.

$ grep -n Auto budget
12:  Auto Loan Payment                     350 
17:  Auto insurance                         50 
22:  Auto Fuel and Maintenance             160 

$ grep Insurance budget
  Life Insurance                         20 

$ grep -i Insurance budget
  Home insurance                        120 
  Auto insurance                         50 
  Life Insurance                         20 

For some commands, it's obvious where their names came from. The who command shows who is logged on. The wc command displays a word count. So how in the world did a command that searches a file for some text get the name "grep"? Well, at first, the original programmers wanted to call the command "find" but there was already a command with that name. While they were sitting around trying to come up with a different name, through an open window they heard a bullfrog outside make a "grrrrrepppp" croaking sound and just decided to use "grep" for the name.

Okay, not really. So where did the name come from? Search patterns are sometimes called "regular expressions" (abbreviated "regex", "regexp", or often "re"). Regular expressions can get complicated so we won't go into detail about them here. But a very simple regular expression can just be the actual text you want search for. To search for the string "Auto", the regular expression would just be "Auto". Easy enough. There was (still is!) a line-oriented text editor called ed and you would type commands within ed to do various things. For example, to print lines 1 through 4, you would enter the command 1,4p. To search for the next line that contains some text and print it out, you would enter the command /text/p. To be more accurate, the text there is actually a regular expression so the general command to search for and print the next line that matches the regular expression would be /re/p. What if you wanted to search for and display all of the lines in the file that matches a regular expression? You would use the "global" version of the command, i.e., g/re/p. That's where the name "grep" comes from.

$ ed letter.txt
1324

1,4p
Lorem ipsum,

Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores
postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia

/quo/p
assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi,

g/quo/p
postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia
assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi,
in liber quodsi lobortis, quis altera essent at his.

q
$

By the way, although the story above about the bullfrog was made up, there is a similar story about how another program got its name. On some unix-like systems (BSD/Berkeley unix derived systems) you could run a command to enable notification when you received email. When email arrived for you, you would get a line displayed on your terminal telling you that "You have new mail from betty" or something like that. At U.C. Berkeley, where early versions of BSD unix were being developed, there was a dog named Biff who would bark at the mailman whenever the (paper) mail was delivered. The email notification command, biff, was named for that dog. Mac OS X is derived from BSD and does, in fact, have the biff command. If you have a Mac, open a Terminal.app window and run biff or man biff.

Standard Output and Standard Input

Stdout

As we have seen, commands have the useful habit of displaying stuff on the terminal window or screen. This output is called "standard output", often abbreviated as "stdout". The default destination for stdout is the terminal but you can redirect stdout to a file if necessary. You do this by adding "> filename" when running the command. For example, if you wanted to save the output from who to a file named "current_users"...

$ ls
annual_report.pdf  budget  letter.txt

$ who > current_users

$ ls
annual_report.pdf  budget  current_users  letter.txt

$ cat current_users
bobby    pts/0        2013-10-30 13:20 (172.16.8.24)
betty    pts/1        2013-10-30 14:04 (172.16.8.20)
bubba    pts/2        2013-10-30 14:47 (93.184.216.119)

Be careful. If you redirect the output to an existing file with ">" you will overwrite the file with the new output. This is sometimes called "clobbering" and we'll show how to prevent it when we talk about scripts below.

So what if you need to append to the file instead? You would use ">> filename" (two ">"s).

$ ls
annual_report.pdf  budget  letter.txt

$ echo 'Users...' > current_users

$ cat current_users
Users...

$ date >> current_users

$ who >> current_users

$ echo 'And what they are doing...' >> current_users

$ w >> current_users

$ cat current_users
Users...
Thu Oct 31 09:23:31 CDT 2013
bobby    pts/0        2013-10-30 13:20 (172.16.8.24)
betty    pts/1        2013-10-30 15:09 (172.16.8.20)
bubba    pts/2        2013-10-30 14:47 (93.184.216.119)
And what they are doing...
 09:23:53 up 6 days, 23:01,  3 users,  load average: 0.33, 0.39, 0.27
USER     TTY      FROM           LOGIN@   IDLE   JCPU   PCPU WHAT
bobby    pts/0    172.16.8.24    Wed13    0.00s  0.05s  0.00s w
betty    pts/1    172.16.8.20    Wed15   16:18m  0.00s  0.00s -bash
bubba    pts/2    93.184.216.119 Wed14   52.00s  0.14s  0.13s top

Stdin

Similar to standard output is "standard input" or "stdin". This is often you typing away at the keyboard. Recall our fictional program howdy from above. If you add a -q optin, the program will query the user for a name.

$ howdy
Howdy, bobby!

$ howdy -q
Your name? 

Tim the Enchanter
Greetings, Tim the Enchanter!

The howdy program is reading input from stdin, in this case, the user typing a name. But like stdout, you can redirect from a file with < filename. When you redirect stdin from a file, it may or may not be echoed to stdout. That just depends on the program itself.

$ cat my_real_name
Robert M. Baylor

$ howdy -q < my_real_name
Your name? 
Greetings, Robert M. Baylor!

Often, programs that accept a file as a command line argument will use stdin if no file is specified. For example, we could use wc to count the number of characters typed at the keyboard. When entering text this way, you will need a way to tell the system that you are finished typing. Typically, you will press ^D to do this.

Below, we see an example of different ways of counting the number of characters in a sentence. The first method just specifies a file. The second method redirects stdin from that file. The commands look similar (the only difference is the < character) but way wc is reading the text is different. Notice that when you specify a file, wc includes the name of the file in the output. When stdin is redirected from the file, wc doesn't know from where the text originated so it can't print out the name of the file. The third method is using stdin from the keyboard.

$ cat sentence.txt
The quick brown fox jumped over the lazy dogs.

$ wc -c sentence.txt
47 sentence.txt

$ wc -c < sentence.txt
47

$ wc -c
The quick brown fox jumped over the lazy dogs.
^D
47

So a program that accepts a file name as an argument may also use standard input, e.g., the user typing at the keyboard, if the file is omitted. But users aren't going to type an entire document just to count the number of characters in it. So seriously... is this really all that useful?

Pipes and Piping

Why, yes. Yes it is. It is possible to chain multiple commands together so that one program's output (stdout) is another program's input (stdin). This is known as "piping" and is done using the "|" vertical bar character, aka "pipe".

The ps (process status) command will print out information about processes currently running on the system. There are many, many options to ps but we will just use -ef which means shows full info for processes for all users.

$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Oct24 ?        00:00:02 /sbin/init
root         2     0  0 Oct24 ?        00:00:00 [kthreadd]
root         3     2  0 Oct24 ?        00:00:00 [migration/0]
root         4     2  0 Oct24 ?        00:00:00 [ksoftirqd/0]
root         5     2  0 Oct24 ?        00:00:00 [migration/0]
root         6     2  0 Oct24 ?        00:00:00 [watchdog/0]
root         7     2  0 Oct24 ?        00:00:00 [migration/1]
root         8     2  0 Oct24 ?        00:00:00 [migration/1]
root         9     2  0 Oct24 ?        00:00:03 [ksoftirqd/1]
root        10     2  0 Oct24 ?        00:00:00 [watchdog/1]
root        11     2  0 Oct24 ?        00:00:00 [migration/2]
root        12     2  0 Oct24 ?        00:00:00 [migration/2]
root        13     2  0 Oct24 ?        00:00:00 [ksoftirqd/2]
root        14     2  0 Oct24 ?        00:00:00 [watchdog/2]
root        15     2  0 Oct24 ?        00:00:00 [migration/3]
root        16     2  0 Oct24 ?        00:00:00 [migration/3]
root        17     2  0 Oct24 ?        00:00:00 [ksoftirqd/3]
root        18     2  0 Oct24 ?        00:00:00 [watchdog/3]
root        19     2  0 Oct24 ?        00:00:12 [events/0]
root        20     2  0 Oct24 ?        00:00:10 [events/1]
root        21     2  0 Oct24 ?        00:00:12 [events/2]
root        22     2  0 Oct24 ?        00:00:13 [events/3]
root        23     2  0 Oct24 ?        00:00:00 [cgroup]
root        24     2  0 Oct24 ?        00:00:00 [khelper]
root        25     2  0 Oct24 ?        00:00:00 [netns]
root        26     2  0 Oct24 ?        00:00:00 [async/mgr]
root        27     2  0 Oct24 ?        00:00:00 [pm]
root        28     2  0 Oct24 ?        00:00:00 [sync_supers]
root        29     2  0 Oct24 ?        00:00:01 [bdi-default]
root        30     2  0 Oct24 ?        00:00:00 [kintegrityd/0]
root        31     2  0 Oct24 ?        00:00:00 [kintegrityd/1]
root        32     2  0 Oct24 ?        00:00:00 [kintegrityd/2]
root        33     2  0 Oct24 ?        00:00:00 [kintegrityd/3]
root        34     2  0 Oct24 ?        00:00:00 [kblockd/0]
root        35     2  0 Oct24 ?        00:00:00 [kblockd/1]
root        36     2  0 Oct24 ?        00:00:00 [kblockd/2]
root        37     2  0 Oct24 ?        00:00:00 [kblockd/3]
root        38     2  0 Oct24 ?        00:00:00 [kacpid]
root        39     2  0 Oct24 ?        00:00:00 [kacpi_notify]
root        40     2  0 Oct24 ?        00:00:00 [kacpi_hotplug]
root        41     2  0 Oct24 ?        00:00:00 [ata/0]
root        42     2  0 Oct24 ?        00:00:00 [ata/1]
root        43     2  0 Oct24 ?        00:00:00 [ata/2]
root        44     2  0 Oct24 ?        00:00:00 [ata/3]
root        45     2  0 Oct24 ?        00:00:00 [ata_aux]
root        46     2  0 Oct24 ?        00:00:00 [ksuspend_usbd]
root        47     2  0 Oct24 ?        00:00:00 [khubd]
root        48     2  0 Oct24 ?        00:00:00 [kseriod]
root        49     2  0 Oct24 ?        00:00:00 [md/0]
root        50     2  0 Oct24 ?        00:00:00 [md/1]
root        51     2  0 Oct24 ?        00:00:00 [md/2]
root        52     2  0 Oct24 ?        00:00:00 [md/3]
root        53     2  0 Oct24 ?        00:00:00 [md_misc/0]
root        54     2  0 Oct24 ?        00:00:00 [md_misc/1]
root        55     2  0 Oct24 ?        00:00:00 [md_misc/2]
root        56     2  0 Oct24 ?        00:00:00 [md_misc/3]
root        57     2  0 Oct24 ?        00:00:00 [khungtaskd]
root        58     2  0 Oct24 ?        00:00:00 [kswapd0]
root        59     2  0 Oct24 ?        00:00:00 [ksmd]
root        60     2  0 Oct24 ?        00:00:03 [khugepaged]
root        61     2  0 Oct24 ?        00:00:00 [aio/0]
root        62     2  0 Oct24 ?        00:00:00 [aio/1]
root        63     2  0 Oct24 ?        00:00:00 [aio/2]
root        64     2  0 Oct24 ?        00:00:00 [aio/3]
root        65     2  0 Oct24 ?        00:00:00 [crypto/0]
root        66     2  0 Oct24 ?        00:00:00 [crypto/1]
root        67     2  0 Oct24 ?        00:00:00 [crypto/2]
root        68     2  0 Oct24 ?        00:00:00 [crypto/3]
root        73     2  0 Oct24 ?        00:00:00 [kthrotld/0]
root        74     2  0 Oct24 ?        00:00:00 [kthrotld/1]
root        75     2  0 Oct24 ?        00:00:00 [kthrotld/2]
root        76     2  0 Oct24 ?        00:00:00 [kthrotld/3]
root        78     2  0 Oct24 ?        00:00:00 [kpsmoused]
root        79     2  0 Oct24 ?        00:00:00 [usbhid_resumer]
root       110     2  0 Oct24 ?        00:00:00 [kstriped]
root       154     2  0 Oct24 ?        00:00:00 [ttm_swap]
root       155     2  0 Oct24 ?        00:00:00 [kslowd000]
root       156     2  0 Oct24 ?        00:00:00 [kslowd001]
root       237     2  0 Oct24 ?        00:00:00 [scsi_eh_0]
root       238     2  0 Oct24 ?        00:00:00 [scsi_eh_1]
root       297     2  0 Oct24 ?        00:00:00 [scsi_eh_2]
root       298     2  0 Oct24 ?        00:00:00 [scsi_eh_3]
root       299     2  0 Oct24 ?        00:00:33 [scsi_eh_4]
root       300     2  0 Oct24 ?        00:00:33 [scsi_eh_5]
root       301     2  0 Oct24 ?        00:00:00 [scsi_eh_6]
root       302     2  0 Oct24 ?        00:00:00 [scsi_eh_7]
root       366     2  0 Oct24 ?        00:00:00 [kdmflush]
root       368     2  0 Oct24 ?        00:00:00 [kdmflush]
root       386     2  0 Oct24 ?        00:00:03 [jbd2/dm-0-8]
root       387     2  0 Oct24 ?        00:00:00 [ext4-dio-unwrit]
root       388     2  0 Oct24 ?        00:00:00 [ext4-dio-unwrit]
root       389     2  0 Oct24 ?        00:00:00 [ext4-dio-unwrit]
root       390     2  0 Oct24 ?        00:00:00 [ext4-dio-unwrit]
root       433     2  0 Oct24 ?        00:00:00 [kauditd]
root       483     1  0 Oct24 ?        00:00:00 /sbin/udevd -d
root       708  1925  0 11:26 ?        00:00:00 sleep 60
root       709  1781  0 11:26 ?        00:00:00 sshd: bobby [priv]
bobby      711   709  0 11:26 ?        00:00:00 sshd: bobby@pts/0
bobby      712   711  0 11:26 pts/0    00:00:00 -bash
root       735     2  0 Oct24 ?        00:00:00 [hd-audio0]
bobby      741   712  0 11:26 pts/0    00:00:00 ps -ef
root       985   483  0 Oct24 ?        00:00:00 /sbin/udevd -d
root       988     2  0 Oct24 ?        00:00:00 [kvm-irqfd-clean]
root      1030     2  0 Oct24 ?        00:00:00 [jbd2/sda1-8]
root      1031     2  0 Oct24 ?        00:00:00 [ext4-dio-unwrit]
root      1032     2  0 Oct24 ?        00:00:00 [ext4-dio-unwrit]
root      1033     2  0 Oct24 ?        00:00:00 [ext4-dio-unwrit]
root      1034     2  0 Oct24 ?        00:00:00 [ext4-dio-unwrit]
root      1140     2  0 Oct24 ?        00:00:03 [flush-253:0]
root      1415     1  0 Oct24 ?        00:00:00 /sbin/rsyslogd -i /var/run/syslo
root      1438     2  0 Oct24 ?        00:00:18 [kondemand/0]
root      1439     2  0 Oct24 ?        00:00:10 [kondemand/1]
root      1440     2  0 Oct24 ?        00:00:13 [kondemand/2]
root      1441     2  0 Oct24 ?        00:00:14 [kondemand/3]
root      1462     1  0 Oct24 ?        00:00:34 irqbalance
rpc       1481     1  0 Oct24 ?        00:00:00 rpcbind
dbus      1496     1  0 Oct24 ?        00:00:00 dbus-daemon --system
root      1506     1  0 Oct24 ?        00:00:01 NetworkManager --pid-file=/var/r
root      1513     1  0 Oct24 ?        00:00:00 /usr/sbin/modem-manager
avahi     1518     1  0 Oct24 ?        00:00:00 avahi-daemon: running [n130.l
avahi     1519  1518  0 Oct24 ?        00:00:00 avahi-daemon: chroot helper
rpcuser   1538     1  0 Oct24 ?        00:00:00 rpc.statd
root      1547  1506  0 Oct24 ?        00:00:00 /sbin/dhclient -d -4 -sf /usr/li
root      1553     1  0 Oct24 ?        00:00:00 /usr/sbin/wpa_supplicant -c /etc
root      1624     2  0 Oct24 ?        00:00:00 [rpciod/0]
root      1625     2  0 Oct24 ?        00:00:00 [rpciod/1]
root      1626     2  0 Oct24 ?        00:00:00 [rpciod/2]
root      1627     2  0 Oct24 ?        00:00:00 [rpciod/3]
root      1631     1  0 Oct24 ?        00:00:00 rpc.idmapd
root      1646     1  0 Oct24 ?        00:00:00 cupsd -C /etc/cups/cupsd.conf
root      1657     1  0 Oct24 ?        00:00:00 /usr/sbin/acpid
68        1666     1  0 Oct24 ?        00:00:02 hald
root      1667  1666  0 Oct24 ?        00:00:00 hald-runner
root      1705  1667  0 Oct24 ?        00:00:00 hald-addon-input: Listening on /
68        1706  1667  0 Oct24 ?        00:00:00 hald-addon-acpi: listening on ac
root      1707  1667  0 Oct24 ?        00:00:31 hald-addon-storage: polling /dev
root      1708  1667  0 Oct24 ?        00:00:33 hald-addon-storage: polling /dev
root      1732     1  0 Oct24 ?        00:00:00 pcscd
root      1748     1  0 Oct24 ?        00:00:02 automount --pid-file /var/run/au
root      1769     1  0 Oct24 ?        00:00:00 /usr/sbin/mcelog --daemon
root      1781     1  0 Oct24 ?        00:00:00 /usr/sbin/sshd
root      1789     1  0 Oct24 ?        00:00:00 xinetd -stayalive -pidfile /var/
ntp       1797     1  0 Oct24 ?        00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd
root      1879     1  0 Oct24 ?        00:00:01 /usr/libexec/postfix/master
postfix   1887  1879  0 Oct24 ?        00:00:00 qmgr -l -t fifo -u
root      1903     1  0 Oct24 ?        00:00:00 /usr/sbin/abrtd
root      1911     1  0 Oct24 ?        00:00:00 abrt-dump-oops -d /var/spool/abr
root      1925     1  0 Oct24 ?        00:00:13 /bin/bash /usr/sbin/ksmtuned
qpidd     1937     1  0 Oct24 ?        00:00:40 /usr/sbin/qpidd --data-dir /var/
root      1979     1  0 Oct24 ?        00:00:01 crond
root      1990     1  0 Oct24 ?        00:00:00 /usr/sbin/atd
root      2003     1  0 Oct24 ?        00:01:59 /usr/bin/python /usr/share/virt-
root      2018     1  0 Oct24 ?        00:01:35 libvirtd --daemon
root      2048     1  0 Oct24 ?        00:00:00 rhnsd
root      2102   483  0 Oct24 ?        00:00:00 /sbin/udevd -d
root      2107     1  0 Oct24 ?        00:00:00 /usr/bin/rhsmcertd
root      2135     1  0 Oct24 ?        00:00:00 /usr/sbin/gdm-binary -nodaemon
root      2142     1  0 Oct24 tty2     00:00:00 /sbin/mingetty /dev/tty2
root      2144     1  0 Oct24 tty3     00:00:00 /sbin/mingetty /dev/tty3
root      2148     1  0 Oct24 tty4     00:00:00 /sbin/mingetty /dev/tty4
root      2150     1  0 Oct24 tty5     00:00:00 /sbin/mingetty /dev/tty5
root      2152     1  0 Oct24 tty6     00:00:00 /sbin/mingetty /dev/tty6
nobody    2158     1  0 Oct24 ?        00:00:00 /usr/sbin/dnsmasq --strict-order
root      2183  2135  0 Oct24 ?        00:00:00 /usr/libexec/gdm-simple-slave --
root      2194  2183  0 Oct24 tty1     00:00:22 /usr/bin/Xorg :0 -nr -verbose -a
root      2242     1  0 Oct24 ?        00:00:00 /usr/sbin/console-kit-daemon --n
gdm       2312     1  0 Oct24 ?        00:00:00 /usr/bin/dbus-launch --exit-with
gdm       2313     1  0 Oct24 ?        00:00:00 /bin/dbus-daemon --fork --print-
gdm       2314  2183  0 Oct24 ?        00:00:00 /usr/bin/gnome-session --autosta
root      2317     1  0 Oct24 ?        00:00:00 /usr/libexec/devkit-power-daemon
gdm       2321     1  0 Oct24 ?        00:00:01 /usr/libexec/gconfd-2
gdm       2339  2314  0 Oct24 ?        00:00:01 /usr/libexec/at-spi-registryd
gdm       2340     1  0 Oct24 ?        00:10:16 /usr/libexec/gnome-settings-daem
gdm       2342     1  0 Oct24 ?        00:00:00 /usr/libexec/bonobo-activation-s
gdm       2349     1  0 Oct24 ?        00:00:00 /usr/libexec/gvfsd
gdm       2350  2314  0 Oct24 ?        00:00:02 metacity
gdm       2352  2314  0 Oct24 ?        00:00:00 /usr/libexec/polkit-gnome-authen
gdm       2353  2314  0 Oct24 ?        00:00:03 gnome-power-manager
root      2355     1  0 Oct24 ?        00:00:00 /usr/libexec/polkit-1/polkitd
gdm       2357  2314  0 Oct24 ?        00:00:11 /usr/libexec/gdm-simple-greeter
gdm       2379     1  0 Oct24 ?        00:00:00 /usr/bin/pulseaudio --start --lo
rtkit     2381     1  0 Oct24 ?        00:00:03 /usr/libexec/rtkit-daemon
root      2388     1  0 Oct24 ?        00:00:00 auditd
root      2404  2183  0 Oct24 ?        00:00:00 pam: gdm-password
root     22385  1781  0 Oct30 ?        00:00:00 sshd: bubba [priv]
bubba    22387 22385  0 Oct30 ?        00:00:00 sshd: bubba@pts/2
bubba    22388 22387  0 Oct30 pts/2    00:00:00 -bash
root     22629  1781  0 Oct30 ?        00:00:00 sshd: betty [priv]
betty    22631 22629  0 Oct30 ?        00:00:00 sshd: betty@pts/1
betty    22632 22631  0 Oct30 pts/1    00:00:00 -bash
bubba    31903 22388  0 09:23 pts/2    00:00:17 top
postfix  32714  1879  0 10:51 ?        00:00:00 pickup -l -t fifo -u

That's a lot of output. Let's say we want to get the processes related to user bubba. (There's actually a -u user option for ps but we're not going to use that here.) We could redirect the output from ps to a temporary file and grep that file for "bubba". We would need to be sure to remember to delete the file when we were finished with it.

$ ps -ef > /tmp/current_processes

$ grep bubba /tmp/current_processes
root     22385  1781  0 Oct30 ?        00:00:00 sshd: bubba [priv]
bubba    22387 22385  0 Oct30 ?        00:00:00 sshd: bubba@pts/2
bubba    22388 22387  0 Oct30 pts/2    00:00:00 -bash
bubba    31903 22388  0 09:23 pts/2    00:00:16 top

$ rm /tmp/current_processes

Fortunately, grep will use stdin if we don't specify a file. So instead let's simplify the above into a single command.

$ ps -ef | grep bubba
bobby      661 21591  0 11:21 pts/0    00:00:00 grep bubba
root     22385  1781  0 Oct30 ?        00:00:00 sshd: bubba [priv]
bubba    22387 22385  0 Oct30 ?        00:00:00 sshd: bubba@pts/2
bubba    22388 22387  0 Oct30 pts/2    00:00:00 -bash
bubba    31903 22388  0 09:23 pts/2    00:00:16 top

Notice that in addition to bubba's processes, this included two others - our grep command and some command running as user root. This is because the grep command is searching for the text "bubba" within stdin, not listing bubba's processes.

It's also possible to chain multiple commands using pipes. To count the number of "bubba related processes" pipe the above output into wc -l (line count).

$ ps -ef | grep bubba | wc -l
5

Stderr

We'll mention one more issue concerning input/output redirection. What if we wanted to save the word count of all the files in a directory to a file, i.e., redirect the output of wc * to the file "word_counts"? We know how to do that...

$ ls -l
total 36
-rw-r--r-- 1 bobby users 12424 Oct 17 14:30 annual_report.pdf
-rw------- 1 bobby users  1065 Oct 28 10:01 budget
-rw-r--r-- 1 bobby users   594 Oct 31 09:24 current_users
-rw-r----- 1 bobby users  1324 Oct 17 14:30 letter.txt
-rw-r--r-- 1 bobby users    18 Oct 30 13:23 my_name
-rw-r--r-- 1 bobby users    47 Oct 31 09:55 sentence.txt

$ wc * > word_counts

$ cat word_counts
  175   771 12424 annual_report.pdf
   27    65  1065 budget
   11    71   594 current_users
   27   189  1324 letter.txt
    1     3    18 my_name
    1     9    47 sentence.txt
  242  1108 15472 total

That worked as expected. Now it just so happens that the wc command does not like directories and it will let you know if an item is a directory. Let's create one and see what happens.

$ mkdir misc

$ ls -l
total 44
-rw-r--r-- 1 bobby users 12424 Oct 17 14:30 annual_report.pdf
-rw------- 1 bobby users  1065 Oct 28 10:01 budget
-rw-r--r-- 1 bobby users   594 Oct 31 09:24 current_users
-rw-r----- 1 bobby users  1324 Oct 17 14:30 letter.txt
drwxr-xr-x 2 bobby users  4096 Oct 31 14:08 misc
-rw-r--r-- 1 bobby users    18 Oct 30 13:23 my_name
-rw-r--r-- 1 bobby users    47 Oct 31 09:55 sentence.txt

$ wc * > word_counts
wc: misc: Is a directory

$ cat word_counts
    175     771   12424 annual_report.pdf
     27      65    1065 budget
     11      71     594 current_users
     27     189    1324 letter.txt
      0       0       0 misc
      1       3      18 my_name
      1       9      47 sentence.txt
    242    1108   15472 total

The wc command displayed an error message. But we were directing output to a file so why did the error message display on the terminal and not get included in the file? Because in addition to standard output, there is a separate output stream for errors, "standard error" or "stderr". To redirect stderr to a file, you would use "2> filename". Note that although you can specify the same file for both stdout and stderr, you shouldn't because the file will likely end up corrupted.

$ wc * > word_counts 2> word_counts.err

$ cat word_counts.err
wc: misc: Is a directory

Why "2"? Where did that come from? These input and output streams are actually unix file descriptors and file descriptors 0, 1, and 2, are reserved for stdin, stdout, and sterr respectively. When you redirect from the command line, you are really redirecting the data for those file descriptors. So by adding "2> filename" we are redirecting data from file descriptor 2 (stderr) to the file "filename". We could actually redirect stdout with "1> filename" if we wanted but the 1 is implied and is omitted.

What if we know that wc doesn't like directories and so we don't really care about error messages and just want to ignore stderr completely. We could always redirect stderr to some file but we would still to deal with leftover error files. A better way is to redirect stderr to a special file on the system, "/dev/null". The /dev/null file is sometimes called the "null device" and just discards any data written to it.

$ wc * 2> /dev/null
    175     771   12424 annual_report.pdf
     27      65    1065 budget
     11      71     594 current_users
     27     189    1324 letter.txt
      0       0       0 misc
      1       3      18 my_name
      1       9      47 sentence.txt
    242    1108   15472 total

The wc command actually did output the error message, but it was redirected to /dev/null and so it just disappeared. By the way, it's also possible to redirect stdout to /dev/null. If we wanted to ignore any word counts but did want to see error messages...

$ wc * > /dev/null
wc: misc: Is a directory

The Shell and Scripts

The Shell

We have seen references above to something called a "shell". A shell is the command-line interpreter UI for unix like systems. When you are typing commands at the command prompt, you are actually running a shell program. Above, we also discussed wildcards, I/O redirection, and pipes. It turns out that those features are actually provided through the shell and not individual programs.

Traditionally, there are two major shell programs - the "Bourne shell" (sh) and the "C shell" (csh). Over time, these shells have been replaced by other shell programs such as the "Bourne-Again shell" (bash). These newer shells provide compatibility support for the older shells but have many other added features as well. You will still see references to sh and csh.

The bash shell is the default shell for Kodiak users. One of the useful enhancements that bash provides is command autocompletion. As you are typing a command at the prompt, you can press the tab key and the shell will use the current context, such as the items in the current directory, and complete the word for you.

$ ls
backups  documents  downloads  pictures  shared

$ ls doc[tab]
$ ls documents/
annual_report.pdf  current_users  misc     sentence.txt
budget             letter.txt     my_name

If there are multiple possibilities for autocompletion, the shell will show them if you press the tab key twice. Type a bit more to make it unambiguous and press tab again to finish. (If it doesn't display anything after you press tab multiple times, there are no matches.)

$ ls do[tab][tab]
documents/ downloads/ 

$ ls dow[tab]
$ ls downloads/
poster.pdf  report.pdf.bak

You can also use the arrow keys to recall previous commands and edit the command line. Pressing the up arrow ↑ will step backwards through the history of previous commands that you have entered. Press the down arrow ↓ to step forward through previous commands. You can use the left ← and right → arrows to move the cursor back and forth to edit the current line. A quick way to move the cursor the beginning of the line is press ^A. Press ^E to move it to the end of the line. There are many more keyboard shortcuts besides these. To see them, along with a lot more info, run man bash.

Scripts

The shell is not just a command-line user interface that runs commands one at a time. It is possible to place multiple commands in a text file, called a "script" or "shell script" and run them all by running that script as a command. But wait! There's more! The shell doesn't just run commands. It also provides many features in "real" programming languages, such as variables, control-flow constructs and loops, subroutines, etc. Although shell scripts are executable, they are interpreted and not compiled. When you run your program on Kodiak (discussed in Part 2) you will actually be running a shell script which then runs your program.

Not all scripts are unix shell scripts. There are other scripting languages, e.g., Perl, Python, and Tcl, as well. But we are only going to discuss shell scripts here.

So let's take a look at a simple shell script. Below is a script that outputs the current date and time (date), current users (w) and a blank line (echo), then waits for 30 seconds before continuing (sleep 30). It does this three times and quits.

$ pwd
/home/bobby/scripts

$ cat list_users.sh
#!/bin/sh

date
w
echo
sleep 30

date
w
echo
sleep 30

date
w

$ sh list_users.sh
Thu Nov  7 09:47:28 CST 2013
bobby    pts/0    172.16.8.24      0.00s sh list_users.sh
betty    pts/1    172.16.8.20      4:33  -bash
bubba    pts/2    93.184.216.119   4:04  top

Thu Nov  7 09:47:58 CST 2013
bobby    pts/0    172.16.8.24     30.00s sh list_users.sh
bubba    pts/2    93.184.216.119   4:34  top

Thu Nov  7 09:48:28 CST 2013
bobby    pts/0    172.16.8.24     60.00s sh list_users.sh
bubba    pts/2    93.184.216.119   2.00s -bash

The first thing to point out is that the script file name's suffix is ".sh". But this is by convention and not really required. You may also see other suffixes for scripts as well, such as ".csh" and ".py". Ignore the first line of the script for now. The rest of the script are just commands that get run as if you had typed them at the prompt.

When we ran the script, we actually ran the sh command with the script file as an argument. Unsurprisingly, the command we ran (sh list_users.sh) is what is actually listed by the w command within the script. Now you may recall that earlier we said that the default shell on Kodiak was bash. So why did we use the Bourne shell sh instead if bash? Like many systems, on Kodiak the sh command is actually a symbolic link to the bash command. A symbolic link is special kind of file that points to or is linked to another file or directory, the target. When you access the symbolic link, the system automatically uses the target file or directory instead. There would have been no difference if we had used bash list_users.sh to run the script instead.

$ which sh
/bin/sh

$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Aug 21  2009 /bin/sh -> bash

$ file /bin/sh
/bin/sh: symbolic link to `bash'

$ ls -l /bin/bash
-rwxr-xr-x 1 root root 801512 Jan 21  2009 /bin/bash

Now, let's modify the script a bit. We want to print the date and users just once, not three times. We could delete the second and third sets of commands but we might want to go back to printing three times later. So instead we'll "comment out" the remaining lines. We do this by inserting a "#" character at the beginning of a line. When the shell sees a "#" it considers everything on the line after it a comment and ignores it.

$ cat list_users.sh
#!/bin/sh

date
w
echo

# Comment out the remaining lines...

#sleep 30
#date
#w
#echo

#sleep 30
#date
#w

Now we want to run the script as if it were a regular command. Notice below that when we first try to run the script from the prompt, bash (our command line interface shell) gives us a "Permission denied" error. The first thing we need to do is make sure that the file is executable. While we're at it, we're going to rename the file and get rid of the .sh suffix to show we really don't need it. Also note that to run the list_users script, we still must do so with a path to the file by putting a "./" in front of it.

$ ./list_users.sh
-bash: ./list_users.sh: Permission denied

$ ls -l
total 4
-rw-r--r-- 1 bobby users 135 Nov  7 13:40 list_users.sh

$ chmod u+x list_users.sh
-rwxr--r-- 1 bobby users 135 Nov  7 13:40 list_users.sh

$ mv list_users.sh list_users

$ ./list_users
Fri Nov  8 11:40:58 CST 2013
bobby    pts/0    172.16.8.24     0.00s /bin/sh ./list_users
bubba    pts/1    93.184.216.119 16.00s -bash
betty    pts/2    172.16.8.20     4.00s man make

You can see above that although we ran ./list_users, the system is actually running /bin/sh ./list_users. So how exactly did it know we wanted to use sh (really, bash)? Because of the first line of the script, "#!/bin/sh". This is a special comment, sometimes called a "shebang", that tells the system what program to use to interpret the script. You might be able to get by without including a shebang line because your current interactive shell will then try to interpret the script. But to prevent compatibility issues in the future, it's good practice to always include the shebang line. Because this script is very simple and doesn't take advantage of any bash specific features, we use #!/bin/sh (the "lowest common denominator" shell) for portability and compatibility in the future. If the script did require the bash shell, we would use #!/bin/bash as the shebang line.

Why is that called a "shebang"? We saw earlier that the vertical bar "|" character is called "pipe". The "#" character is sometimes called "hash" (you may be familiar with the term "#hashtags"...) and the "!" character is sometimes called a "bang". So "#!" is sometimes called hash-bang but that is usually shortened to shebang. By the way, the "*" character is sometimes called a "splat".

PATH and Other Environment Variables

We've seen where to run a program we sometimes were required to specify it with its absolute or relative path. To run the list_users script (which happens to be in the directory /home/bobby/scripts), we must explicitly include the path to it (e.g., /home/bobby/scripts/list_users or ./list_users). So why do some programs work with just their names while others require a path? Because of the $PATH environment variable.

Like "regular" programming languages, the shell provides and uses variables or "environment variables". You can echo them to see their values. Remember how "~" is a shortcut for your home directory? There is an environment variable for this as well: $HOME.

$ echo $HOME
/home/bobby

$ ls $HOME/documents
annual_report.pdf  current_users  misc     sentence.txt
budget             letter.txt     my_name

To see a list of variables use the printenv, env, or set commands. You can create and set your own variables which is especially useful within shell scripts.

$ echo $howdy

$ howdy=doody

$ echo $howdy
doody

So back to the $PATH variable. A path variable is a list of directory paths, both absolute and relative, that the system will search for programs you run when you don't include the path in the command. The directories are separated by colon (:) characters. Shown below is a default $PATH. (Actually, the default $PATH on Kodiak contains many more directories...)

$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:
/sbin:/home/bobby/bin

So when you run the command list_users, the system searches each of the above directories, in order, for the executable file "list_users". Once it finds one, it runs it. If it doesn't find one, it will fail with a "command not found" error. Right now, the list_users script is located in /home/bobby/scripts which is not in $PATH. But we can see that the directory /home/bobby/bin is there. (Notice that there are several "bin" directories. These are standard locations for executable programs or binaries.) So if we move list_users to /home/bobby/bin, it should work.

$ which list_users
/usr/bin/which: no list_users in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:
/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/bobby/bin)

$ mv /home/scripts/list_users /home/bin/

$ which list_users
~/bin/list_users

$ list_users
Mon Nov 11 14:26:21 CST 2013
bobby    pts/0    172.16.8.24  0.00s /bin/sh /home/bobby/bin/list_users

But suppose we don't want to move list_users to our ~/bin directory, but want to keep it in the ~/scripts directory. We can tell the system to include /home/bobby/scripts when searching by appending it to the current $PATH.

$ which list_users
/usr/bin/which: no list_users in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:
/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/bobby/bin)

$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:
/sbin:/home/bobby/bin

$ PATH=$PATH:/home/bobby/scripts

$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:
/sbin:/home/bobby/bin:/home/bobby/scripts

$ which list_users
~/scripts/list_users

Because any changes made to environment variables disappear once the shell script exits or when logging out, we must remember to set them back each time. This would be a major hassle, but fortunately there is a solution: startup scripts. These are hidden files, located in your home directory, that run when you first log in or start a shell. The bash man page actually calls them "personal initialization files". We'll stick with "startup scripts".

$ ls -la $HOME
total 100
drwx------. 13 bobby users  4096 Nov 11 14:49 .
drwxr-xr-x.  6 root  root   4096 Oct 17 14:05 ..
drwxr-x---   2 bobby users  4096 Oct 25 14:09 backups
-rw-r--r--.  1 bobby users    18 Jul  9 08:24 .bash_logout
-rw-r--r--.  1 bobby users   176 Jul  9 08:24 .bash_profile
-rw-r--r--   1 bobby users   150 Nov 11 14:49 .bashrc
drwxr-xr-x   2 bobby users  4096 Nov 11 14:36 bin
drwx--x--x.  3 bobby users  4096 Oct 31 14:47 documents
drwxr-x---.  2 bobby users  4096 Oct 25 14:10 downloads
drwxr-x---.  4 bobby users  4096 Oct 25 08:56 pictures
drwxr-xr-x   2 bobby users  4096 Nov 11 14:36 scripts
drwxr-x--x   2 bobby users  4096 Oct 21 12:40 shared

For bash the startup scripts are .bash_profile and .bashrc. Why two files? What's the difference? The .bash_profile script is executed for login shells, that is, you had to enter your username and password. The .bashrc script is executed for interactive non-login shells. Although these are mutually exclusive, as we'll soon see, the .bashrc script is run by the default .bash_profile so anything in .bashrc will get run you log in as well.

Right now, we'll look at .bash_profile, the script that runs whenever you log in.

$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

Notice that the first line is not a shebang, but just a regular comment line. This script is only run by bash so the shebang isn't needed. The second bit of code is what runs the .bashrc file (after first checking if it exists). Don't modify that part unless you know what you are doing.

The last part is the section that we care about. See the line "PATH=$PATH:$HOME/bin"? What that does is take the current, default, PATH variable (already set by system-wide startup scripts) and append ":$HOME/bin" to it. That's why we see /home/bobby/bin at the end when we echo $PATH. Recall that what we want is to be able to run scripts in the /home/bobby/scripts directory so we just need to append ":$HOME/scripts" to the end of that line. When changing a path variable like $PATH, always remember to separate directories with ':'. Also, you almost always want to modify PATH, either appending or prepending directories, so be sure and include the current value of $PATH there. If we were to leave out $PATH, (e.g., PATH=$HOME/bin:$HOME/scripts) then only those two directories would be in the current path and regular commands, such as ls, would not be found. Note that if you do this, you would still be able to run commands by using their full paths (e.g., /bin/ls). For any changes to .bash_profile to take affect, log out and log back in.

At some point, you might be tempted to add the current working directory "." to PATH. That way you wouldn't have to always use "./" in front of programs and scripts to run them. That sounds like it would be convenient, right? Don't do it. Adding your current directory to your PATH is potentially dangerous, especially if it appears before the system-wide directories. Remember that the directories in $PATH are searched in order. What if some ne'er-do-well were to write a script that deleted all of the files and directories in your account? And what if he named the script "ls" and placed it in the /tmp directory? If you were to cd /tmp and ls and if "." were at the beginning of your PATH, the system would find the malicious script first, run it instead of the real ls command, and all of your files would get deleted. It's safer to just type "./" before your scripts.

One more thing to mention before we do so. Note the last line in the script export PATH. By default, variables are only set and usable in the environment (shell or script) where they are created. To make them usable in sub- or child processes, that is, programs and scripts run by the current environment, they must be exported. If you make multiple changes to a variable, you can do that with multiple lines and export it after the final change. If you just need to set a varible or make one change, you could set and export it on the same line.

PATH=$PATH:$HOME/bin
PATH=$PATH:$HOME/scripts
export PATH

export PRINTER="Some_Printer"
$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:$HOME/scripts

export PATH

[Log out and log back in...]

$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:
/sbin:/home/bobby/bin:/home/bobby/scripts

$ which list_users
~/scripts/list_users

The .bash_profile script is used to set default settings such as your $PATH variable whenever you log in. You can run other commands as well. For example, if you wanted to list the current users when you log in, add a who or w to the script. Or if you want to keep track of when you log in, add something like date >> ~/login_times.dat.

Remember how redirecting stdout with ">" to an existing file would overwrite or clobber the file? If you want to protect yourself from doing that accidentally, set the bash "noclobber option" by adding a "set -o noclobber" line to your .bash_profile. If you try to redirect output to an existing file, the command will fail. To ignore the noclobber option and force the shell to overwrite an existing file, use ">| filename".

$ who > current_users
-bash: current_users: cannot overwrite existing file

$ who >| current_users

So what is the .bashrc script used for? As we said above, it's for interactive non-login shells. What does that mean? There may be times where you may need to start a separate shell from the current one, for example, by explicitly running bash. Or you may be editing a file with vi and need to temporarily escape the editor (without quitting) to run a new shell to search for some other file or something. (In vi you do that by entering ":!bash".) You can run a single command on a compute node without logging in to it with ssh, e.g., ssh n001 'ps -fu bobby'. In all these cases, only .bashrc gets run, not .bash_profile.

Some of the things that are often placed in .bashrc are "aliases" and functions. We'll just describe aliases here. An alias is just a "shortcut" command that replaces a real one. For example, if you find yourself running the command ls -laF often, you could create an alias lf for this.

$ alias lf="ls -laF"

$ lf
[bobby@n130 documents]$ lf
total 48
drwx--x--x.  3 bobby users  4096 Oct 31 14:47 ./
drwx------. 13 bobby users  4096 Nov 13 09:55 ../
-rw-r--r--   1 bobby users 12424 Oct 17 14:30 annual_report.pdf
-rw-------   1 bobby users  1065 Oct 28 10:01 budget
-rw-r--r--   1 bobby users   594 Oct 31 09:24 current_users
drwxr-xr-x   2 bobby users  4096 Oct 31 14:08 misc/
-rw-r--r--   1 bobby users    18 Oct 30 13:23 my_name
-rw-r-----   1 bobby users  1324 Oct 17 14:30 letter.txt
-rw-r--r--   1 bobby users    47 Oct 31 09:55 sentence.txt

If you find yourself often (mis)typing "mroe" instead of "more", embrace your inner typo and add alias mroe="more". Or if you prefer to use less instead of more but keep typing more out of habit, alias more="less".

If you are unsure which startup script gets called when, you can place a echo 'This is bashrc/bash_profile' command at the beginning of them. Be sure to remove or comment out those lines when finished. Extraneous output, especially from .bashrc, can sometimes cause odd problems.


Summary of Commands


Below is list of commands mentioned in this document. For more information, use the man command.

Command Description
who Show who is logged on
man Display the on-line manual pages
passwd Modify your password
pwd Print name of current working directory
cd Change current directory
ls List directory contents
mkdir Make directory
cat Print file on standard output
more Display a file, one screen at a time
less Less is more (more or less)
head Display the first part of a file
tail Display the last part of a file
cp Copy files and directories
mv Move or rename files
rm Remove/delete a file
nano Simple text editor
vi Text editor
emacs Text editor
gedit Graphical text editor
id Print user identity
whoami Print username
groups Print the groups a user is in
chmod Change file access permissions
which Show the full path of a command
touch Change file access and modification time
echo Output a line of text
date Prints the system date and time
w Show who is logged on and what they are doing
top Display and update sorted information about processes
wc Print line, word, and character counts for a file
file Determine file type
gzip,gunzip Compress or expand files
hostname Show the system's host name
ssh Remote login program
diff Compare files line by line
grep Search a file for a string or pattern
ed Line-oriented text editor
ps Process status
sleep Delay for a specified amount of time
env,printenv,set Print environment varibles

More Information


You can get more information on the topics above at the following link...

http://www.google.com/

When you are ready to learn how to compile and run programs on Kodiak, you can continue on to Part 2.