Xkcd/1508/

One of the survivors, poking around in the ruins with the point of a spear, uncovers a singed photo of Richard Stallman. They stare in silence. "This," one of them finally says, "This is a man who BELIEVED in something."

Final projects

Example project from last semesterhttps://youtu.be/tk71yPer5Uk (These guys won the “Best Application” award at last year’s CIS course fair)

Earlierhttps://youtu.be/0b1GRyJXYuAhttps://youtu.be/E3p-oPNqL_E

Piazza

• 24 of you enrolled• Sign up today• piazza.com/upenn/spring2016/cis191

CIS 19x Shared Lectures

• Jan 19 - Basics of Command Line• Jan 26 - Version Control and git• Feb 2 - Brief history of the internet, the web, and how

modern web servers work

• Also posted on Piazza

On late homework

• After the 3pm deadlines, submission to hwx will be closed.

• Late homework should be submit tohwx-late instead (e.g hw0-late).

• After 48 hours, no more submission is accepted.

• Late submissions have a 25% penalty

CIS 191: Linux and Unix

Class 1Janurary 21st , 2016

Outline

Permissions

More Command Line Tools

Text Editors

Shell and IO

Unix Permissions

• Three categories of what we can do…1. Read (r)2. Write (w)3. Execute (x)

• Three categories for who is using the file…1. User (u)2. Group (g)3. Other (o)

• This leads naturally to a 9 bit permission scheme• Use ‘a’ to refer to all categories of users (u + g + o)

What do the permissions mean…?

• Read– I can cat the file, or open it in a text editor in read-only mode

• Write– I can modify the file using a text editor or some other file

manipulation tool (like cat in append mode).– I can delete the file if I want to.

• Execute– I can execute the file if it is an executable binary file– In other words, I can load the program into memory and run it!

Okay, but what about directories?

• Read– I can list the directory’s contents

• Write– I can add files to or remove files from the directory

• Execute– I can enter the directory (such as with cd)

• Non-intuitive, but sometimes we might want to have write-only permissions…– Why?

What does this mean?

• The User is the owner– They own the file

• The Group is the group in which the owner is in• Others is everyone else!

What is a group?

• A list of users, for the purpose of the permissions scheme• Useful if you are doing a class project• Or if you are working in a company

Which permission?

• Prioritized resolution of differences…• if the user is the owner

– then use the User permission• otherwise, if the user is in the owner’s group

– then use the Group permission• otherwise use the Other permission

Permission scheme breakdown

rwxUser

rwxOther

rwxGroup

File’s permissions; set with chmod

• Represented as a bit-vector in Unix• Here, 000 111 111 111• Or, 0777

-Type

Changing permissions and owners

• Change the owner of a file with chown• Change the group that owns a file with chgroup• Change the file permission scheme

– with chmod– This is also known as changing the file’s mode

• These operations may require root access…• Or they may require you to be the file owner• See man for more details!

More details on chmod

• Two modes of operation that we are concerned with– Relative

• $ chmod u+rx file– Adds read and execute permissions to the user/owner of the file

– Absolute• $ chmod 755 file

– Set the file’s permission to be rwx r-x r-x

chmod relative calls

• chmod o+r myfile.txt• Gives the “other” group read permissions on myfile.txt; can also

specify ‘-’ to remote a permission

• Important characters:a – all r – readu – owner w – writeg – group x – executeo – others

The superuser

• Also known as root• Root can do anything.

– Modify users– Change permissions on files– Kill any process– Delete any directory

Changing users

• It is possible and sometimes useful to switch to another user

• This can be done using the su command• $ su user

– This will prompt for the user’s password, and then act as though the user had logged in

– The working directory will remain the same• If the superuser uses su, then no password is required

sudo

• Execute commands as another user without knowing the other user’s password!– Typically used to execute commands as root.

• Users in the sudoers file can make use of sudo• sudo –i opens up an interactive root session

– Allows the user to execute commands as root in a running session

– Useful for running multiple commands– Also very dangerous.

Remember: Careful what you ask for!

• If you sudo something it will happen, almost assuredly.

• Make sure that what you are asking for is what you want.

• Root can make a lot of dangerous system modifications that basic users cannot – often for their protection!

xkcd.com/149

Outline

Permissions

More Command Line Tools

Text Editors

Shell and IO

pwd – Where you are in the file tree

• This simply prints the absolute path of the current working directory to the command line

• Useful for remembering where you are (because if you’re anything like me you’ll jump around and kinda get lost)

less and more – view files

• less – browse through files– Features forward and backward browsing– Can open large files by only loading the necessary parts into the

buffer for display• For large files, using less can be faster than an editor like vim!

– In fact, the man command uses less to display file contents!– Quite massive these days. Don’t be surprised if it’s not installed

on an embedded system.• more is worse (less is more – haha get it??)

– Doesn’t let you move backward– Has to load the entire file into the buffer before starting

• This can be very slow for big files

• With less and more, how about most? You betcha.

head and tail – see the top/end

• head lets you see the top 10 lines of a file– Specify more or less with the –n option

• tail lets you see the bottom 10 lines of a file– Specify more or less with the –n option– In BSD (like Mac OSX) systems, tail has a –r option which lets

you print the output in reverse order• This does not hold true in debian (like Ubuntu) based systems!• Debian-based systems have tac (see man pages)

file – Diagnose File Type

• Guesses the file type of a given file (passed in as command-line arguments)– This is done by looking at the “magic number” of a given file– “Magic numbers” are essentially a marker at the beginning of a

file in Unix systems which indicate the type of the file– Think of them as the Unix equivalent of file extensions in

Microsoft Word, but a little different

wc – count the number of words

• wc counts the number of words in a given input file or string

• It can also count number of characters, number of lines, etcetera

• See the man page for more details

nl – number lines

• Takes the lines in an input file, or as passed in on standard input, and numbers each line in increasing order– Starting from 1

sort – sort input

• sort is a simple tool for sorting lines of text files or text given in standard input

• See the man page for more details

find – Search Recursively for a File

• Executing this command on a directory will search for a file with constraints passed in as flags/arguments– There are many of these! See the man page for more details

• Some examples– Find files in dir with the name file.txt

• $ find dir –name ‘file.txt’• Use the –iname flag instead for case insensitive searches

– Find files ending with .jpg in both dir1 and dir2• find dir1 dir2 –iname ‘.jpg’

– Find files in dir that were modified in the last 7 days• find dir –mtime -7

Aside – Regular Expression Preview

• A means of specifying a formula for a language…– Regular expressions are to a grammar what a class is to an

object (kindasorta)• * means any number of any character in a language…

– So *.jpg can be cat.jpg or cats.jpg or fatpants.jpg– Or we can do cat*s.jpg

• Can be catdogs.jpg or catfs.jpg or…

• More on regular expressions in a later lecture• For now, make use of the useful * in your file commands!

grep – Search a File

• grep allows you to apply a basic regular expression to the contents of one or more input files, as specified in the arguments passed in to grep

• The given regular expression will be applied to each line in the file, and matching lines will be printed to standard output– One match on the line should be enough to make it print

• Can specify case insensitive with the –i flag• Example

– grep ‘*cheezburger’ lolcats.txt

Wildcards

• In the shell, you can refer to multiple files or directories at once by using “wildcards”

• In particular, you can use– *

• If you want to refer to “any amount of any character”– ?

• If you want to refer to “any character, but just one”

Outline

Ubuntu Basics

More Command Line Tools

Text Editors

Shell and IO

The Age-Old Debate

xkcd.comm/378

Vi vs. Emacs

• Two of the most popular text editors• These have sparked much debate over the years• Originally…

– VI was a lightweight mode-based text editor– emacs was designed to be easy to use from the start and

feature-complete while being easily extensible• Each now incorporates many features found in the other

Why use a Text Editor?

• To save time!– Save seconds to do small tasks

• If you spend a lot of time editing text (you will!), this adds up…

• Impressive macro support in both vi and emacs• Many libraries available to include in both editors as well• Not about typing text; about editing text• Customization through add-ons and configuration files

– .vimrc or .emacs

Emacs basics

• Uses modifier keys to tell it explicit commands– Control (abbreviated C): Ctrl– Meta (Abbreviated M): Alt or Esc

• Hold Alt and press key or press and release Esc, then press key

• Common tasks:– Type to enter text– C-f to move forwards, C-b for back, C-p to prev line, C-n to next– C-_ to undo– C-x C-s saves current buffer– C-x C-c to quit emacs

• Get to tutorial with C-h t

Why Emacs?

• Has the default keybindings for many command line programs– C-r will do a reverse search of your bash history, for example

• Emacs is non-modal– The same command will have the same behavior, always

• Emacs commands have a hierarchical structure, kind of• Emacs commands are named• Emacs contains a lisp implementation• Running code from within emacs is easy

In contrast - Vim

• Vim is very much a modal text editor– All commands vary based on which mode you are in!

• Keybindings arranged to make actions easy to type– Typical actions require typing one key in normal mode

• Available by default on highly slimmed-down distributions (more commonly than emacs)

• man, less, and other programs use vi-like keybindings• vi-like keybindings are available on several web services

– Gmail– Duck duck go

Vi modes

• Four modes (actually more, but we’ll stick with these)– Normal, awaiting commands (movement, pasting, deleting…)– Insert, awaiting text to be inserted– Visual, used for selecting blocks of text (like highlighting)– Replace, used to enter text without affecting formatting

• “A good way to get gibberish is to give a vi window to someone who doesn’t know how to use it”

• Press Esc or Ctrl-[ to return to Normal mode– If you are ever unsure of what’s happening, press Esc a bunch of

times!

Vim Philosophy

• Your hands should move as little as possible– Move with h,j,k,l

• Remain in Normal mode as much as possible– You spend most of the time moving around a document…– You’ll see this is true if you pay attention!

Learning text editors

• Official vim tutor program can be invoked from the command line with vimtutor.

• Access vim help system by typing :help [topic] within vim• There are even games to help you learn vim keybindings!

– vim-adventures.com, for example (in your homework this week)

• Official emacs tutorial: C-h t• Access emacs help: C-h

• Practice practice practice

Outline

Ubuntu Basics

Text Editors

More Command Line Tools

Shell and IO

A bit about shells• The Shell is responsible for executing and managing

programs, as well as displaying their output– You can think of it as a line of communication between user and

Operating System• Provides an environment and parameters to each

program it runs/is running

Terminology

• Operating System, Kernel, Shell, Terminal• An operating system has 3 major parts

– Kernel– File system– Shell

• Kernel manages the tasks of the computer and the hardware - most notably memory and CPU time.

• Shell acts as an interface between the user and the kernel.

• Terminal runs the shell. In the beginning terminal refers to physical hardware

Shells – A history

• In the beginning, users could only interact with the operating system via the shell!

• Early day standard shell on Unix is Bourne Shell (sh)• Nowadays, we use bash (Bourne-Again Shell)• Bash is the default shell on Linux and Mac OS• In a graphical user interface (GUI), you interact with the

shell via a terminal (e.g. gnone-terminal or xterm)

Lines of communication

• Four main channels of communication between processes– Not including mutexes, semaphores, and signals!

1. Command Line Arguments– Provided only when a program starts up

2. Standard Input– A source of input requested at runtime

3. Standard Output– Where to send (most) output

4. Standard Error– Reserved by convention for error-related output

More on CLI Arguments

• Programs can also accept file paths or an input string as a command line argument

• Command line arguments can be thought of as strings that are passed in to a program when it is invoked

• Ever write a program that accepts command line arguments, and then if they are not available, ask for input from the user via the keyboard?– Maybe in Java?

• Same concept applies with unix executable programs

Aside: An important signals

• Signals – like a message sent to a running process (more on these later)

• Two important signals:– C-c sends SIGINT, which asks a program to stop executing– C-\ sends QUIT, which tells a program to stop executing and

dump its core• If C-c is a baseball bat, C-\ is a medieval mace with spikes on it

We’ve seen these in action already

• CLI Arguments– ls me.html treats me.html as a command line argument

• Standard Output– echo prints on standard output…

• Standard Input– Invoking cat without any arguments opens up an interaction

session which reads from standard input• C-d (EOF character) to stop the session

• Standard Error– $ rm nonexistant

• rm: cannot remove ‘nonexistant’:• No such file or directory

Pipes to connect two programs

• The concept of pipes is a fundamental one…• Connects the Standard Output of one program into the

Standard Input of another!• Consider the Unix hypothesis

– Have a bunch of simple command line tools that do one thing well• grep, find, cat, wc (word count), cut (cut out bits of an input), sort

(sort input)

• Consider connecting many simple pieces together to get an infinitely complex pipeline of jobs…

Pipes: An example

• The pipe character is |, the vertical bar above the \ on standard QWERTY keyboards

• To concatenate two files list1 and list2, then sort the result…– $ cat list1 list2 | sort

• This literally concatenates list1 and list2 into a single string, which is then given to sort as standard input– To reiterate – this is not the same as a command-line argument!

sort receives its input via standard input, not via command-line arguments

Another piping example

• Count the number of visible files in a directory• $ ls | wc –l

– ls outputs a list of files to standard output, which is then passed to wc as standard input

– wc then counts the number of words on standard input, and the –l option prints only the number of newlines found

• Viola! The number of visible directories.

Redirection

• The standard streams (input, output, and error) can all be redirected to or from files

• More on this in the following slides

Redirecting Standard Output

• This can be accomplished with >• Sometimes it’s useful to place program output in a file…• $ echo “in here” > file

– This will overwrite file if it exists, and will create it if it does not– The contents of file will now be “in here”

• If we want to append instead of overwrite, we can use the >> operator

Redirecting Standard Input

• This can be accomplished with <• Rather than typing input, it can come from a file!• Redirecting standard input substitutes the contents of

the source file for what would be typed on standard input– Standard input does not affect command-line arguments!

• Example:– sort < file1.txt behaves the same as sort file1.txt

• Except that in the redirection example, sort does not know which file it is reading from…

Redirecting Standard Error

• This can be accomplished with 2>• Sometimes it’s useful to separate error messages from

normal program output– Especially when you don’t want to see any error messages!

• $ make 2> /dev/null• This sends anything directed as standard error to

/dev/null• Alternatively, redirect standard error to standard output

with 2>&1– Literally, redirects file descriptor 2 (standard error) to file

descriptor 1 (standard output)

/dev/null

• The null device…• Think of it like a black hole that swallows up everything!• http://devnull-as-a-service.com/

Some More Examples

• The following are (mostly) equivalent– cat file | wc– wc < file– wc file

• We can also use pipes in conjunction with redirects– head –n 17 < file | tail –n 7 – ls | sort >> filelist– wc < file > my_wc– grep hello file | nl > my_hello

A note on shell programming

• Almost always, fundamental data types are numbers and compound structures

• In shell scripting, fundamentals are strings, lists of strings, and files!– Bad news: arithmetic is weird…– More on that later

• Because of this, certain programs lend themselves to shell scripting!

• Connecting input and output of multiple programs is also very simple because of the piping and redirection tools

Mapping Input and Output

• In the following graph, to convert from the left side to the right side– Redirections are in black– Other shell constructs are in blue– And programs are in green

• The following diagram is borrowed from Sam Panzer

Mapping Input and Output