Topic 7: File and directory I/O CSE2395/CSE3395 Perl Programming Llama3 chapter 6, pages 86-97, chapters 11-14, pages 148-207 Camel3 pages 20-22, 28-29,

Topic 7: File and directory I/OTopic 7: File and directory I/O

CSE2395/CSE3395Perl Programming

CSE2395/CSE3395Perl Programming

Llama3 chapter 6, pages 86-97, chapters 11-14, pages 148-207

Camel3 pages 20-22, 28-29, 97-100, 426-428, 747-755, 770

perlfunc, perlopentut manpages

2Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University

In this topicIn this topic

Using standard error Opening and closing files Writing to and reading from files Manipulating files Manipulating directories Scanning directories Communicating with other processes

Using standard error Opening and closing files Writing to and reading from files Manipulating files Manipulating directories Scanning directories Communicating with other processes


Standard I/OStandard I/O

All programs have three filehandles open by default► STDIN (standard input)

– buffered input, defaults to keyboard► STDOUT (standard output)

– buffered output, defaults to terminal► STDERR (standard error)

– unbuffered output, defaults to terminal

Each may be independently redirected to a file or process by shell redirection

All programs have three filehandles open by default► STDIN (standard input)

– buffered input, defaults to keyboard► STDOUT (standard output)

– buffered output, defaults to terminal► STDERR (standard error)

– unbuffered output, defaults to terminal

Each may be independently redirected to a file or process by shell redirection


Standard I/OStandard I/O

% program < data | less% program < data | less

programdata less

screen

file processprocess

tty

pipe

STDIN STDOUT

STDERR

redirect


Writing to STDERRWriting to STDERR

STDOUT is the default filehandle for print print can use other filehandles

► put filehandle name immediately after print► no comma between filehandle and first parameter► print STDERR "Invalid data\n";► same syntax is used to print to manually-opened

filehandles► warn function is another way of printing to STDERR

STDOUT is the default filehandle for print print can use other filehandles

► put filehandle name immediately after print► no comma between filehandle and first parameter► print STDERR "Invalid data\n";► same syntax is used to print to manually-opened

filehandles► warn function is another way of printing to STDERR

Llama3 pages 91-94; Camel3 pages 21-22; perlfunc manpage


warn and diewarn and die

warn function prints a message to STDERR► warn "Near critical temperature" if $temp > 2500;

► appends the program’s name and line number unless the warning message ends in newline

die function prints a message to STDERR and then exits the program► behaviour is otherwise the same as warn► die "Reactor meltdown!" if $temp > 10000;

warn function prints a message to STDERR► warn "Near critical temperature" if $temp > 2500;

► appends the program’s name and line number unless the warning message ends in newline

die function prints a message to STDERR and then exits the program► behaviour is otherwise the same as warn► die "Reactor meltdown!" if $temp > 10000;

Llama3 pages 152-155; Camel3 pages 827-828, 700-701; perlfunc manpage


Opening filesOpening files

open function opens a file for reading or writing► associates a filehandle variable with the file

– filehandle variables don’t start with symbol like $, @, %, so usually written in capitals to not clash with any builtin name

► returns false and sets $! if open fails– $! contains last error message from failed open, close, system function

close function closes the filehandle► commits any unfinished changes to disk► all open filehandles are closed automatically at end of

program► many Perl programs don’t bother to close files

explicitly

open function opens a file for reading or writing► associates a filehandle variable with the file

– filehandle variables don’t start with symbol like $, @, %, so usually written in capitals to not clash with any builtin name

► returns false and sets $! if open fails– $! contains last error message from failed open, close, system function

close function closes the filehandle► commits any unfinished changes to disk► all open filehandles are closed automatically at end of

program► many Perl programs don’t bother to close files

explicitlyLlama3 pages 150-152; Camel3 pages 20-22, 747-755, 693; perlfunc manpage


Opening filesOpening files

open HANDLE, filenameopen HANDLE, filename

Llama3 pages 150-151; Camel3 pages 747-755; perlfunc, perlopentut manpages

filehandle name, conventionally written in capitals

filename is a string:

"file" (read from file)

"<file" (read from file)

">file" (write to file)

">>file" (append to file)

other forms exist too


Using filehandlesUsing filehandles

To write to a filehandle, use print HANDLE► no comma between filehandle and arguments to print► print OUTFILE "Some text\n";► print STDOUT "This is the default\n";

To read from a filehandle, use <HANDLE>► $line = <INFILE>;► while (<INFILE>) { # ... use $_ ... }► <HANDLE> returns undef at end of file

To write to a filehandle, use print HANDLE► no comma between filehandle and arguments to print► print OUTFILE "Some text\n";► print STDOUT "This is the default\n";

To read from a filehandle, use <HANDLE>► $line = <INFILE>;► while (<INFILE>) { # ... use $_ ... }► <HANDLE> returns undef at end of file

Llama3 pages 155-156; Camel3 pages 20-22, 80-83, 765-766; perlop, perlfunc manpages


TimeoutTimeout

# "or" keyword is a very low-precedence version of "||" # short-circuiting OR operator. If we used ||, would have# to use parentheses around open call.open OUTFILE, ">primes" or die "Cannot open file for writing: $!";

# Sieve of Eratosthenes algorithm.# "x" operator replicates a string or list. @isPrime = (1) x 100; # Construct array of 100 ones.for ($i = 2; $i < 100; $i++){ if ($isPrime[$i]) { # Number $i is prime, print it. print OUTFILE "$i\n"; # All multiples of $i are not prime. for ($j = $i * 2; $j < 100; $j += $i) { $isPrime[$j] = 0; } }}

# "or" keyword is a very low-precedence version of "||" # short-circuiting OR operator. If we used ||, would have# to use parentheses around open call.open OUTFILE, ">primes" or die "Cannot open file for writing: $!";

# Sieve of Eratosthenes algorithm.# "x" operator replicates a string or list. @isPrime = (1) x 100; # Construct array of 100 ones.for ($i = 2; $i < 100; $i++){ if ($isPrime[$i]) { # Number $i is prime, print it. print OUTFILE "$i\n"; # All multiples of $i are not prime. for ($j = $i * 2; $j < 100; $j += $i) { $isPrime[$j] = 0; } }}


TimeoutTimeout

print "Enter a number: ";chomp ($number = <STDIN>);

open PRIMES, "<primes" or die "Cannot open file for reading: $!";

# Try each prime in turn.while (<PRIMES>){ chomp; # Apply this prime as a factor as often as possible. while ($number % $_ == 0) { push @factors, $_; $number /= $_; } last if $_ > $number;}close PRIMES; # Rarely done explicitly.print join ("*", @factors), "\n";

print "Enter a number: ";chomp ($number = <STDIN>);

open PRIMES, "<primes" or die "Cannot open file for reading: $!";

# Try each prime in turn.while (<PRIMES>){ chomp; # Apply this prime as a factor as often as possible. while ($number % $_ == 0) { push @factors, $_; $number /= $_; } last if $_ > $number;}close PRIMES; # Rarely done explicitly.print join ("*", @factors), "\n";


Manipulating filesManipulating files

rename function renames/moves a file► like Unix mv command► rename "old", "new";

unlink function removes a file► like Unix rm command► unlink "victim";

link function creates hard link (alias for file)► like Unix ln command► link "oldname", "newname";

symlink function creates symbolic (soft) link► like Unix ln -s command► symlink "oldname", "newname";

Functions return false and set $! on failure

rename function renames/moves a file► like Unix mv command► rename "old", "new";

unlink function removes a file► like Unix rm command► unlink "victim";

link function creates hard link (alias for file)► like Unix ln command► link "oldname", "newname";

symlink function creates symbolic (soft) link► like Unix ln -s command► symlink "oldname", "newname";

Functions return false and set $! on failureLlama3 pages 174-182; Camel3 pages 773-774, 819, 736, 806; perlfunc manpage


Testing filesTesting files

File system keeps information about files► this information is what Unix ls -l prints► owner► type (file, directory, symlink, etc.)► size in bytes► permissions (e.g., owner can read and write)► modification and access times

stat function returns information about a file► returns a list of information► $size = (stat "file")[7];► lstat function is identical, but it does not follow

symbolic links

File system keeps information about files► this information is what Unix ls -l prints► owner► type (file, directory, symlink, etc.)► size in bytes► permissions (e.g., owner can read and write)► modification and access times

stat function returns information about a file► returns a list of information► $size = (stat "file")[7];► lstat function is identical, but it does not follow

symbolic links



Testing filesTesting files

Commoner file tests have a shorthand that does not require using stat► all tests consist of hyphen followed by single

character► $size = -s "file" gets length of file in bytes► if (-f "file") tests if file exists and is normal file► if (-r "file") tests if file exists and is readable► if (-d "dir") tests if dir is a directory► many, many other tests

Commoner file tests have a shorthand that does not require using stat► all tests consist of hyphen followed by single

character► $size = -s "file" gets length of file in bytes► if (-f "file") tests if file exists and is normal file► if (-r "file") tests if file exists and is readable► if (-d "dir") tests if dir is a directory► many, many other tests

Llama3 pages 157-162; Camel3 pages 28-29, 97-100; perlop, perlfunc manpages


TimeoutTimeout

# An implementation of Unix rm -i (interactive)

# Print usage message if called with no arguments.die "Usage: $0 file [...]\n" if @ARGV == 0;

foreach $file (@ARGV){ if (-d $file) { warn "$file is a directory, skipping\n"; next; } print "remove $file: are you sure? (y/n) "; $confirm = <STDIN>; if ($confirm =~ /^y/i) { # Try to remove file. unlink $file or warn "Couldn't remove $file: $!\n"; }}

# An implementation of Unix rm -i (interactive)

# Print usage message if called with no arguments.die "Usage: $0 file [...]\n" if @ARGV == 0;

foreach $file (@ARGV){ if (-d $file) { warn "$file is a directory, skipping\n"; next; } print "remove $file: are you sure? (y/n) "; $confirm = <STDIN>; if ($confirm =~ /^y/i) { # Try to remove file. unlink $file or warn "Couldn't remove $file: $!\n"; }}


Manipulating directoriesManipulating directories

chdir function changes the current directory► like Unix cd shell builtin► chdir ".."; # Up a level► chdir $ENV{"HOME"}; # Go home

mkdir function creates a directory► mkdir "new", 0600; # Mode a la chmod

rmdir function removes an empty directory► rmdir "victim";


chdir function changes the current directory► like Unix cd shell builtin► chdir ".."; # Up a level► chdir $ENV{"HOME"}; # Go home

mkdir function creates a directory► mkdir "new", 0600; # Mode a la chmod

rmdir function removes an empty directory► rmdir "victim";


Llama3 pages 168-169; Camel3 pages 688, 741, 777; perlfunc manpage


Scanning directoriesScanning directories

To get the contents of a file► open the file► read each line in turn► close the file when no more lines► use open, readline and close

– readline function is a Perl synonym for <...>

A directory is just a special file which contains the names of other files

To get the contents (file list) of a directory► open the directory► read each filename in turn► close the directory when no more filenames► use opendir, readdir and closedir

To get the contents of a file► open the file► read each line in turn► close the file when no more lines► use open, readline and close

– readline function is a Perl synonym for <...>

A directory is just a special file which contains the names of other files

To get the contents (file list) of a directory► open the directory► read each filename in turn► close the directory when no more filenames► use opendir, readdir and closedir


Scanning directoriesScanning directories

opendir function opens a directory for scanning► associates a directory handle with the directory

readdir function returns filename in directory► in scalar context, next filename

– undef returned when no more filenames► in list context, all (remaining) filenames

closedir function closes the directory handle► as with close, done automatically at program end

opendir function opens a directory for scanning► associates a directory handle with the directory

readdir function returns filename in directory► in scalar context, next filename

– undef returned when no more filenames► in list context, all (remaining) filenames

closedir function closes the directory handle► as with close, done automatically at program end

Llama3 pages 171-173; Camel3 pages 755, 770, 694; perlfunc manpage


TimeoutTimeout

# Print files in a directory, sorted by size.

$dir = $ARGV[0] || ".";

opendir HERE, $dir or die "Cannot open directory: $!";

while (defined ($name = readdir HERE)){ $size{$name} = (-d "$dir/$name") ? -1 : -s "$dir/$name";}closedir HERE;

foreach $file ( sort { $size{$a} <=> $size{$b} || $a cmp $b } keys %size){ printf "%40s%10d\n", $file, $size{$file};}

# Print files in a directory, sorted by size.

$dir = $ARGV[0] || ".";

opendir HERE, $dir or die "Cannot open directory: $!";

while (defined ($name = readdir HERE)){ $size{$name} = (-d "$dir/$name") ? -1 : -s "$dir/$name";}closedir HERE;

foreach $file ( sort { $size{$a} <=> $size{$b} || $a cmp $b } keys %size){ printf "%40s%10d\n", $file, $size{$file};}


TimeoutTimeout

# Rename all files in the current directory.

opendir HERE, "." or die "Cannot open directory: $!";

# Read each directory entry (scalar context).while (defined ($name = readdir HERE)){ # Skip files that already have .bak suffix. next if $name =~ /.bak$/; # Skip anything not a normal file. next unless -f $name;

rename $name, "$name.bak" or warn "Cannot rename $name: $!";}


opendir HERE, "." or die "Cannot open directory: $!";

# Read each directory entry (scalar context).while (defined ($name = readdir HERE)){ # Skip files that already have .bak suffix. next if $name =~ /.bak$/; # Skip anything not a normal file. next unless -f $name;



TimeoutTimeout


opendir HERE, "." or die "Cannot open directory: $!";# Read each directory entry (list context).@names = readdir HERE;

foreach $name (@names){ # Skip files that already have .bak suffix. next if $name =~ /.bak$/; # Skip anything not a normal file. next unless -f $name;



opendir HERE, "." or die "Cannot open directory: $!";# Read each directory entry (list context).@names = readdir HERE;

foreach $name (@names){ # Skip files that already have .bak suffix. next if $name =~ /.bak$/; # Skip anything not a normal file. next unless -f $name;



Communicating with othersCommunicating with others

Perl has ability to communicate with other programs► invoke other programs (a new process)► send input to them► capture their output

Perl has versions of all the normal process-related system calls► fork: clone this process into two (parent/child)► exec: replace this process with another one► waitpid: wait for a child process to finish

Perl also has higher-level ways of doing the same thing

Perl has ability to communicate with other programs► invoke other programs (a new process)► send input to them► capture their output

Perl has versions of all the normal process-related system calls► fork: clone this process into two (parent/child)► exec: replace this process with another one► waitpid: wait for a child process to finish

Perl also has higher-level ways of doing the same thing


Running a program from PerlRunning a program from Perl

system function invokes another program (process)► system "date";► system "ls -l $ENV{'HOME'}";

Perl program waits for invoked process to finish before continuing

Returns exit status of invoked process Invoked process inherits from Perl program

► filehandles (STDIN, STDOUT, STDERR)► environment (current directory, %ENV)

system function invokes another program (process)► system "date";► system "ls -l $ENV{'HOME'}";

Perl program waits for invoked process to finish before continuing

Returns exit status of invoked process Invoked process inherits from Perl program

► filehandles (STDIN, STDOUT, STDERR)► environment (current directory, %ENV)



Capturing output from programsCapturing output from programs

Backquote `...` operator used to capture output of another process

Like system, except return value is the standard output of the command► $now = `date`;► trailing newline character of process’ output is kept

Variables are interpolated inside backquotes as with double-quoted strings

In list context, returns each line of output as a separate element► @homeFiles = `/bin/ls -1 $ENV{'HOME'}`;

Backquote `...` operator used to capture output of another process

Like system, except return value is the standard output of the command► $now = `date`;► trailing newline character of process’ output is kept

Variables are interpolated inside backquotes as with double-quoted strings

In list context, returns each line of output as a separate element► @homeFiles = `/bin/ls -1 $ENV{'HOME'}`;

Llama3 pages 197-201; Camel3 page 80; perlop manpage


Getting more controlGetting more control

With both system and backquotes, Perl program must wait until invoked process finishes► possibly a long time

With backquotes, output of invoked process is returned all at once► possibly a lot of data

Sometimes better to deal with each line of process’ output as it comes

Solution is to invoke the process with a pipe

With both system and backquotes, Perl program must wait until invoked process finishes► possibly a long time

With backquotes, output of invoked process is returned all at once► possibly a lot of data

Sometimes better to deal with each line of process’ output as it comes

Solution is to invoke the process with a pipe


PipesPipes

A pipe is a file-like structure managed by the operating system

Consists of two filehandles► one (“ingress”) can only be written to► one (“egress”) can only be read from► all data written to ingress is read at egress

Each filehandle may be part of a different process► common trick is to create a pipe, then hand one end

off to another process using fork/exec

A pipe is a file-like structure managed by the operating system

Consists of two filehandles► one (“ingress”) can only be written to► one (“egress”) can only be read from► all data written to ingress is read at egress

Each filehandle may be part of a different process► common trick is to create a pipe, then hand one end

off to another process using fork/exec

ingress egress


TimeoutTimeout

# This code is not examinable!

pipe PIPEREAD, PIPEWRITE or die;

defined ($pid = fork()) or die; # Clone this process.unless ($pid){ # Child process. close PIPEREAD; # Don't need reader in child. open STDOUT, ">&PIPEWRITE" or die; # Redirect STDOUT. exec "/bin/ls -1 $ENV{'HOME'}" or die; # Never returns.}

# Parent process.close PIPEWRITE; # Don't need writer in parent. while (<PIPEREAD>) { # Read from pipe push @homeFiles, $_; # Do something with each line.} waitpid $pid, 0; # Clean up.

# This code is not examinable!

pipe PIPEREAD, PIPEWRITE or die;

defined ($pid = fork()) or die; # Clone this process.unless ($pid){ # Child process. close PIPEREAD; # Don't need reader in child. open STDOUT, ">&PIPEWRITE" or die; # Redirect STDOUT. exec "/bin/ls -1 $ENV{'HOME'}" or die; # Never returns.}

# Parent process.close PIPEWRITE; # Don't need writer in parent. while (<PIPEREAD>) { # Read from pipe push @homeFiles, $_; # Do something with each line.} waitpid $pid, 0; # Clean up.


Creating pipes with openCreating pipes with open

Perl has higher-level constructs to facilitate reading from and writing to processes

To open a process to read from, place vertical bar (|) at the end of the command

To open a process to write to, place vertical bar at the beginning of the command

Then place this string in an open function call Syntax is inspired by use of pipes in shell

► ls -1 $HOME | yourcode► yourcode | lpr -Plp

Perl has higher-level constructs to facilitate reading from and writing to processes

To open a process to read from, place vertical bar (|) at the end of the command

To open a process to write to, place vertical bar at the beginning of the command

Then place this string in an open function call Syntax is inspired by use of pipes in shell

► ls -1 $HOME | yourcode► yourcode | lpr -Plp

Llama3 pages 201-203; Camel3 page 750-752; perlfunc, perlopentut manpages


Reading from a processReading from a process

To open a process to read from, place vertical bar (|) at end of open string► open FILELIST, "ls -1 |"► “ls -1” is not a file to read, but a program to run

Perl invokes command (ls -1) and pipes its standard output to filehandle (FILELIST)

Reading from filehandle reads lines from output of invoked process► $filename = <FILELIST>; # Get a file.

To open a process to read from, place vertical bar (|) at end of open string► open FILELIST, "ls -1 |"► “ls -1” is not a file to read, but a program to run

Perl invokes command (ls -1) and pipes its standard output to filehandle (FILELIST)

Reading from filehandle reads lines from output of invoked process► $filename = <FILELIST>; # Get a file.



Writing to a processWriting to a process

To open a process to write to, place vertical bar (|) at beginning of open string► open PRINTER, "| lpr -Plp"► “lpr -Plp” is not a file to read, but a program to run

Perl invokes command (lpr -Plp) and pipes anything sent to filehandle (PRINTER) to that command’s standard input

Writing to filehandle writes to input of invoked process► print PRINTER "Hello, printer lp!";

To open a process to write to, place vertical bar (|) at beginning of open string► open PRINTER, "| lpr -Plp"► “lpr -Plp” is not a file to read, but a program to run

Perl invokes command (lpr -Plp) and pipes anything sent to filehandle (PRINTER) to that command’s standard input

Writing to filehandle writes to input of invoked process► print PRINTER "Hello, printer lp!";



TimeoutTimeout

# Send a quotation by email.

# Invoke fortune, a random fortune-cookie-generator.open QUOTATION, "/usr/games/fortune |" or die;# Invoke mail, a simple mail-sending program.open MAILER, "| /bin/mail -s Funny foo\@bar.com" or die;

print MAILER "Here's something funny:\n";# Read lines of output from fortune.while (<QUOTATION>){ # Print the line, indented with "> ", to the mailer. print MAILER "> $_";}print MAILER "Love, me.\n";

# Explicitly closing a pipe waits for the process to end.close QUOTATION;close MAILER;

# Send a quotation by email.

# Invoke fortune, a random fortune-cookie-generator.open QUOTATION, "/usr/games/fortune |" or die;# Invoke mail, a simple mail-sending program.open MAILER, "| /bin/mail -s Funny foo\@bar.com" or die;

print MAILER "Here's something funny:\n";# Read lines of output from fortune.while (<QUOTATION>){ # Print the line, indented with "> ", to the mailer. print MAILER "> $_";}print MAILER "Love, me.\n";

# Explicitly closing a pipe waits for the process to end.close QUOTATION;close MAILER;


Covered in this topicCovered in this topic

Opening and closing files► open, close

Reading from and writing to filehandles► <HANDLE>, print HANDLE

File operations File tests Directory operations Scanning directories Processes

► system, `command`► pipes

Opening and closing files► open, close

Reading from and writing to filehandles► <HANDLE>, print HANDLE

File operations File tests Directory operations Scanning directories Processes

► system, `command`► pipes


Going furtherGoing further

Exceptions► catching error conditions with eval and die► Camel3 pages 700, 705-707

IO::File and IO::Dir► object-oriented interfaces to files and directories► man IO::File, man IO::Dir

IO::Socket► communicating across a network► Camel3 pages 899-900

fork, exec, waitpid► getting dirty with processes► perlfunc manpage

Exceptions► catching error conditions with eval and die► Camel3 pages 700, 705-707

IO::File and IO::Dir► object-oriented interfaces to files and directories► man IO::File, man IO::Dir

IO::Socket► communicating across a network► Camel3 pages 899-900

fork, exec, waitpid► getting dirty with processes► perlfunc manpage


Next topicNext topic

Modules► use Module;► Perl standard library► CPAN

Databases► persistent data► Data::Dumper► DBM

– databases in files► DBI

– interfacing with SQL databases

Modules► use Module;► Perl standard library► CPAN

Databases► persistent data► Data::Dumper► DBM

– databases in files► DBI

– interfacing with SQL databases

Llama3 chapter xxxCamel3 pages xxxperlmod manpage

Documents

Topic 7: File and directory I/O CSE2395/CSE3395 Perl Programming Llama3 chapter 6, pages 86-97, chapters 11-14, pages 148-207 Camel3 pages 20-22, 28-29,