S603: UNIX Intermediate Workshop

Indiana University - Bloomington
First Summer Session 2013 (13691)
Meeting Thursdays, 9:00a-12:00p in PV 151

Steven W. Cox
Bloomington campus
Main Library
Office: 002F
Email

Purpose

The Unix Operating System plays a critical role in the contemporary global information technology infrastructure. With at least half the web sites on the Internet hosted by computers running Unix, a similarly broad deployment in underlying database and network infrastructure, a well established role in scientific computing, and a recent increased presence in desktop computing, it almost certain that contemporary information professionals will encounter Unix based systems in their work.

This workshop is an intermediate level look at the Unix operating system as compared to the Unix introduction that our S401 students receive. We will have a short review but it will be essential to have the S401-UNIX material internalized as much as possible. The original course materials for the L401-UNIX module (as opposed to the newer S401) covered more UNIX topics but the original slides are linked below. Please review this material before coming to class. Feel free to email me with questions regarding these extra topics.

We will first outline the basic purpose and functions of an operating system. I will then provide a base from which to explore many facets of using a Unix operating system: shell environment and customization, a robust text editor supporting regular expressions, more Unix commands (utilities and filters), variable and command substitution, advanced I/O redirection, process management and job control, resource management, TCP/IP networking, file security mechanisms, and an introduction to shell programming. While some Unix system administration topics will naturally arise, this is not intended as a Unix administration course.

Objectives

Developed over 30 years ago, the Unix operating system has become a central fixture in todays information technology infrastructure. Unix did not arrive at this position through clever marketing or slick user interface design. Unix has survived through sound technical engineering and an ability to adapt to changes in the information technology environment.

You should come away from this workshop with experience using a range of Unix tools to solve problems but more importantly an understanding of the the design philosophy of Unix. The Philosophy of Unix is to provide the user with a very large set of simple tools and a mechanism for combining them to accomplish a task. This building block approach differs substantially from the monolithic application that tries to do everything typically seen in desktop computer software. The simple Unix tools rarely solve a user's needs completely by themselves, but when combined with other tools they can solve problems never conceived of by the developers of monolithic applications.

Understanding the design and traditions of Unix is also essential to self sufficiency when encountering previously unexplored aspects of the operating system. Just as a graphical user interface provides common interface elements to streamline a user's experience learning a new program, there are similar common tool behaviors in Unix which if understood can make new tools relatively easy.

By the end of the course, the student should be familiar enough with Unix to utilize many constructive utilities and filters, customize the shell environment, understand and modify system variables, use regex effectively with various tools, edit text files interactively with vi and non-interactively with sed, understand command line parsing and tokenizing, fully grasp I/O redirection and piping to accomplish various tasks, use variable and command substitution, understand how to control metacharacter expansion with quoting, and effectively manage user system resources (files and processes). The student should also be able to write basic shell programs as well as understand and modify more complex ones.

Requirements

It is assumed that students have either taken L401-UNIX (or the newer S401-UNIX) or are familiar with Unix topics covered in that module. If it has been awhile since you've reviewed this material, please do so before attending class. The UNIX module to the older L401 consisted of three weeks so any newer SLIS students will need to go through the slides linked below:

L401-UNIX, Week 1 - L401-UNIX, Week 2  - L401-UNIX, Week 3

Grading

There will be three assignments worth a total of 100 points. Assignment One is worth 30 points, Assignment Two is worth 40 points, and Assignment Three is worth 30 points. In fairness to students who turn in assignments on time, late assignments will be penalized. I will deduct 10% of the total possible assignment points for each day that they are late, unless prearranged with the instructor 24 hrs prior to the due date. Exceptions will be granted only in extreme circumstances. Please refer to the grading policy for this workshop.

 

Outline

Day 1: History, Architecture & Standards, and Unix Essentials

We will start out by discussing what makes an operating system and what is its purpose. From there we will look at the evolution of Unix and the events that shaped it and created the various implementations. We will then discuss the events that took place that led to the need for standards and see how they are often derived from implementations. By the end of this section it should be clear as to why Unix has survived so long when many operating systems have come and gone.

We will then look at the practical application of this OS with a short review of some important Unix concepts and commands. We will also look into some essential Unix tools and some of their useful options which will be relevent to the topic areas I'll be covering in this workshop. We'll continue by looking at our command shell and how it will tokenize, interpret and process your command line. This will do many things for you such as; expand your metacharacters, perform command substitution, variable substitution, etc. It is important to understand how the shell will see your command line and how it will process your commands. Finally, we will look into some timesaving tips of our shell's editing features. When we work with more complex command lines, especially when we are building pipelines, it is important to be able to add or remove filters, command options, or change the syntax without having to duplicate your efforts.

Slides from Day 1 (PDF)

Day 2: The Vi and Ex Editors & Batch Text Filtering with Sed

Unix is a very text-centric system. A text editor is one of the Unix professional's most important tools. To be proficient with a local Unix editor is a valuable tool whether you are editing your local initialization files, writing programs or modifying your html content.

The editor we will concentrate on today is vi, a screen oriented visual editor which is built on the ex superset. The vi editor has been around since 1976 and was developed by Bill Joy (co-founder Sun Microsystems). The editing tools vi and ex are widely available on virtually all Unix/Linux platforms and use familiar regular expressions (regex) to make your text processing much more powerful. We will use vi for the remaining part of this workshop to create and edit files.

Finally, we look into sed (stream editor) and see the power that this utility has to offer for batch processing. This utility also supports regular expressions and is often used as a filter in a pipeline. Again, this tool is widely available on Unix/Linux derivatives as well as other operating systems with a command line. Combining sed and regular expressions allows you to non-interactively search and modify bodies of text (in many files at once) based on certain expressions.

Slides from Day 2 (PDF), vi Reference (PDF), sed Reference (PDF)

Assignment One (HTML), Sed Exercise (HTML)

Day 3: Bash Init Files, Options, and Variables (Local and Environment)

The shell is the outer layer of the Unix Operating System. It is an interface between the user and the internal components of the operating system. We will focus on the bash shell for this workshop.

There are two ways that you will utilize this shell. One is your day to day interactive use and the other is non-interactive use via a shell program (script). Today, we will concentrate on the interactive use of this shell and how to configure and customize your work environment with initialization files. We will then discuss some shell options and how to set them as well as some useful built-in shell commands. We will also discuss the command line history mechanism in-depth so working in your environment can be more efficient.

We will then be learning some concepts that will be helpful when we discuss shell programming later in this workshop. We will discuss two types of variables, local and environment (global), and how to set and export these types of parameters. We will also learn about system parameters; we will focus on the PATH and PS1 (prompt) variables. We will then look into the shell hash table and demonstrate how our commands are located. We will revisit the shell history mechanism and discover new ways to utilize this useful feature of the bash shell. We will end the day by looking into some essential Unix filters and their options. Many of them are highly specialized utilities and are often used as part of a pipeline when filtering or extracting data from text files.

Slides from Day 3 (PDF)

Day 4: Metacharacters & I/O Redirection; Credentials & Permissions

We will begin our day with further shell topics such as understanding and creating command aliases, metacharacters in-depth, and advanced I/O redirection.

We will then look at files which are a fundamental concept in Unix and are used to represent not only storage of data, but also physical and virtual devices such as displays, keyboards, and communication links between running programs. Topics covered today include an anatomy of a file, types of files, the hierarchical file and directory structure, how the physical and logical storage structures integrate, and Unix commands relevant to file and disk management.

Being a multi-user system, Unix has a representation of users, and ways of controlling access to files based on user credentials. The representation of users and how file access control is managed will be covered.

Finally, we return to the shell to look more concretely at a number of specific commands and how to use simple commands as building blocks to accomplish more complicated tasks. This will be accomplished with more complex pipelines. As we learn more about column and row filtering tools we begin to understand more about this core Unix philosophy.

Slides from Day 4 (PDF)

Assignment Two (HTML)

Day 5: Command Substitution, Positional Parameters, Functions, etc.

Today we will have a demonstration of one of our device files, our terminal. We will demonstrate how this core Unix design is implemented and why. We will then talk about more concepts that will be useful to the shell programmer; positional parameters, command substitution, and the command exit status.

We will then discuss what a bash function is and how to define and export them. Functions are memory resident shell code and will serve as a good introduction to the syntax necessary for creating shell programs.

We will continue by discussing the importance of quoting now that we have used more powerful metacharacters (variable and command subsititution) and need to control their expansion (interpretation). Finally, we will end our day revisiting the Unix find utility and looking at some advanced uses for this indispensable tool.

Slides for Day 5 (PDF)

Assignment Three (HTML)

Day 6: Processes, Networking Overview, & Programming with the Shell

A program running in Unix exists as a process. We will cover exactly what a process is, and Unix tools for examining and controlling processes.

Unix systems provide a bulk of the infrastructure for the Internet. We will cover the basics of TCP/IP networking and some common network related tools in Unix.

Finally, we return to the shell and explore it not as an interactive user interface, but as a programming language. We will learn the basics of variables, operators, expressions, statements, and control structures. You will leave the class with a menu-based home directory management script that I wrote for you to modify and enhance.

Slides for Day 6 (PDF), Iteration Exercise (HTML), Script Exercise (HTML)

Resources and Readings

All Resources

Recommended: Peek, Jerry, Grace Todino-Gonquet and John Strang (2002). Learning the UNIX operating System, 5th ed. O'Reilly & Associates

Recommended: Newham, Cameron and Bill Rosenblatt (1998). Learning the bash shell, 2nd ed. O'Reilly & Associates