How to Write Awk and Perl Scripts to Enable your EDA Tools to Work Together

Robert C. Hutchins SMC (Standard Microsystems) 6 Hughes Irvine, CA 92718 Abstract1

Two tutorials are presented introducing the basic syntax and semantics of Awk and Perl. The languages are taught by example, and the same example EDA applications are developed in both languages so that a comparison of the two languages is facilitated. The examples include file modification, data extraction, EDA tool connection, and system administration.

Shankar Hemmady Guru Technologies Inc. 10750 Farallone Dr. Cupertino, CA 95014
by record. The simplest record is a single line from the input. One can make $x$ consecutive lines from the input to be a record by changing the value of the special variable called a record separate, $RS$. In other words, $RS$ indicates the pattern/symbol which separates two consecutive records. For example, the default value of $RS$ is \\n (a newline). Similarly, a record is made-up of fields. Each field is separated by a field separator, $FS$. For example, the default value of $FS$ is (a blank). Each field within a record is number from $1$ to $NF$, where $NF$ is set by Awk after reading in a record. The value of $NF$ indicates the number of fields present in the current record being processed. The first (last field) is referred to as $1 ($NF), while the whole record is referred to as $0. Example Let the input file contain the following lines: This is just-a test. Wish you-were here. With $RS$ and $FS$ set to the default values, the above file has two records. The first (second) record consists of four (three) fields. However, if $RS$ and $FS$ is set to - and ., respectively, the above file has three records. The first record has only one field (This is just), the second record has two fields (a test and Wish you), while the third record has one field (were here.). Standard loop construct to read input from all fields in command line: BEGIN { # This part is executed **before** the INPUT is processed while (getline < file1.tex) { print $0 ; # Comment Print line to STDOUT } { # INPUT is processed here print $0 ; # Comment Print line (from INPUT) to STDOUT } END { # This part is executed after the INPUT is processed file_name = sprintf(%s, file2.tex); while (getline < file_name) { print $0 >> /dev/stderr ; # Comment Print line) to STDERR } } Notice that the INPUT is processed line-by-line. Therefore, there is no loop like (while (getline < STDIN)) present in the middle. This construct allows for pre as well as post processing using files that are different from the INPUT file. Additionally, one can also scan a file in the middle portion up on seeing some pattern. For example:

1 Introduction
This tutorial will teach Awk in section 2, concentrating on the GNU implementation, Gawk, and Perl, version 4, in section 3. Language constructs are taught in terms of examples which perform EDA tasks. The assumption is that the target audience is made-up of Design Automation Engineers who have some familiarly with Unix and the C language. Common uses of these languages to tie EDA tools together, such as altering a file output from one EDA tool so that another EDA tool can input it are shown. The examples focus on cases where programs in these languages can be significantly shorter than programs written in traditional languages such as C.

2 Awk
With Awk you can manipulate records of data/input to: manage small personal databases compile reports validate data perform editing tasks prototype algorithms History of Awk and Gawk The name Awk comes from the initials of its designers: Alfred V. Aho, Peter J. Weinberger, and Brian W. Kernighan. The GNU implementation, Gawk, was written by Paul Rubin and Jay Fenlason John Woods, with advice from Richard Stallman. This original version was thoroughly reworded for compatibility with the Awk by David Trueman and Arnold Robbins. Basics Awk and Gawk include special variables, arrays, and strings Input can be read from STDIN, files, or pipes. Output can be STDOUT, STDERR, files or pipes. The input is read in record
1

33rd Design Automation Conference Permission to make digital/hard copy of all or part of this work forpersonal or class-room use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage, thecopyright notice, the title of the publication and its date appear,and notice is given that copying is by permission of ACM, Inc. Tocopy otherwise, to republish, to post on servers or to redistribute tolists, requires prior specific permssion and/or a fee. DAC 96 - 06/96 Las Vegas, NV, USA 1996 ACM, Inc. 0-89791-833-9/96/0006..$3.50

{ if ($1 == AND_gate) { # Read the truth table for AND_gate from AND.tb # and store each line into an array AND.func line_ct = 0; while (getline < AND.tb) { if ($NF > 0) { # Store only lines that are non-empty AND.func[line_ct++] = $0; } } } } Data Types are similar to the ones in C. Standard functions in Awk include: index(in, find) length(string) match(string, regexp) split(string, array, fieldstep) sprint(format, expression1, ...) sub(regexp, replacement, target) gsub(regexp, replacement, target) # Not available in awk Is similar to sub, except it replaces all the longest, leftmost, non-overlapping matching substrings in target substr(string, start, length) Standard functions for Input/Output (in Awk) include: close(filename) system(command) # Example: # END { # tmp = rm all.tex ; # system(tmp); # system(ls -la all.tex | wc -l > check.all); # } Example Programs: 2.1. Audit a File Simple Grep - print out all lines in a file that match a pattern. Example call: gawk -f grep.awk -v str="VHDL" fileA fileB BEGIN { while (getline) { if (index($0, str) == 0) continue; print $0; } exit 0; } 2.2. Modify a file Program modifies VHDL file to add library declarations, and change others #!/usr/local/bin/gawk { if (index($0, use COMPASS_LIB.COMPASS_ETC.ALL) != 0) { print $0; print library cb60hp230d_nohold;\n;

print use cb60hp230d_nohold.all;\n; print library cb60io220d;\n; print use cb60io220d_nohold;\n; } else if (index($0, cb60hp230d) != 0) { gsub(/cb60hp230d/, cb60hp230d_nohold, $0); print $0; } else { print $0; } } 2.3. Extract data from a file Parse a flattened VHDL netlist to extract a sorted count of all cells used. The line below is an example of the VHDL to be parsed. Note this program can easily be modified to count cell usage for Edif and Verilog. The line below is an example of the VHDL to be parsed for I_365 : NI01D1 use configuration cb60hpd.NI01D1_CON #!/usr/local/bin/gawk { if (index($0, use configuration) != 0) { if ($4 in cell_list) { cell_list[$4]++; else { cell_list[$4] = 1; # Initialize entry for $4 -- saw for the # first time } } } END { system(rm .tmp.file); for (cell_index in cell_list) { print cell_index cell_list[cell_index] >> .tmp.file; } close(.tmp.file); system(sort .tmp.file); system(rm -f .tmp.file); } 2.4. In place editing like SED but creates a backup file. The program would be called as follows: gawk -v backup_file_name=file1 -f mysed.awk < edit_file; BEGIN { tmp = sprintf(rm -f %s, backup_file_name); system(tmp); } { tmp_line = $0; if (index($0, #define global) != 0) { gsub(/#define global/, define global, tmp_line); } print tmp_line >> backup_file_name; print $0; # Output to STDOUT } END { close(backup_file_name); }

2.5. Loop through all items in a list. In this example, write out in groups of 6. BEGIN { item_ct = 1; } { for (i = 1; i <= NF; i++) { # Scan each line (having NF field) printf $i; if (!(item_ct % 6)) { printf "\n"; } else { printf " "; } } } 2.6. System manipulation i n v o l v i n g r e n a m i n g o f files/directories is quite difficult in Awk. 2.7. Take a list of files and create a script. A Makefile can be created using the same technique. This Awk program creates a script that compiles all VHDL files in a directory and then makes a backup copy of each file. #!/usr/local/bin/gawk BEGIN { system(rm -f .tmp.file); system(ls -1 *.vhdl > .tmp.file); close(.tmp.file); while (getline < .tmp.file) { tmp = sprintf(vcom %s ; cp %s %s.bak, $1, $1, $1); system(tmp); } exit 0; # Exit without processing any INPUT } { } 2.8. Opening a compressed file as an alternative . If file.xyz does not exits, try opening its compressed version file.xyz.Z. This program uses Unix pipes and the zcat utility. BEGIN { system(rm -f .tmp.file); tmp = sprintf(ls -1 %s > .tmp.file, file.xyz); system(tmp); getline test_var < .tmp.file; if (index($0, No such file) != 0) { # Cannot find the file ... Try the compressed version system(rm -f .tmp.file); tmp = sprintf(ls -1 %s > .tmp.file, file.xyz.Z); system(tmp); getline test_var < .tmp.file; if (index($0, No such file) != 0) { print Error! file.xyz is not here > /dev/stderr; exit 0; } system(zcat file.xyz.Z); # Print the uncompressed file exit 0;

} print Error! file.xyz is not here > /dev/stderr; } 2.9. Split a full name into its directory path and file name. BEGIN { FS=/; # Change field separator from blank to back-slash } { for (i = 1; i < NF; i++) { printf $i/; } print $NF; } USEFUL "ONE-LINERS": awk {if NF > max) max = NF } END { print max } This program prints the maximum number of fields on any input line. awk length($0) > 80 Print out all lines whose length is greater than 80 characters. awk NF > 0 Print every line with at-least one field. With default options, all empty lines, or lines with all spaces, are skipped. awk { if (NF > 0) print } Similar to the previous example. Only this time, the action of printing the line decided after comparing the value of NF to 0. awk BEGIN { for (i = 1; i <= 7; i++) print int(101 * rand()) } Print out 7 random numbers between 1 and 100, inclusive, WITHOUT receiving any INPUT. The reason is all the code is inside the BEGIN block. ls -l files | awk { x+= $4 } ; END { print "total bytes: "x } Print out the total bytes across all files in the current directory. awk BEGIN { FS = : } { print $1 | "sort" } /etc/passwd Print out the sorted list of login names of all users (in the/etc/passwd) file. awk END { print NR } Print out the total number of lines in the file. awk {print NR, $0} Append the lines number in front of each line, like "cat -n" BOOKS: sed & awk by Dale Dougberty, O'Reilly and Associates, Inc. ISBN 0-937175-59-5. This books only covers the awk language. The Gawk Manual by Diane Barlow Close, Arnold D. Robbins, Paul H. Rubin, and Richard Stallman. Published by the Free Software Foundation. This manual is both a tutorial and a reference on Gawk. FTP SITES:

Gawk and most (if not all) gnu software is available at anonymous ftp sites at many, if not most, universities. In particular, the site and path for the University of Minnesota is: ftp.cs.umn.edu pub/....

print STDOUT $_ ; is equivalent to print $_ ; is equivalent to print ; Standard loop construct to read input from all files in command line: WHILE ( $_ = <ARGV>) { print $_; # Comment Print to STDOUT } can be shortened in Perl to WHILE (<>) { print ; } because $_ is assumed to be what many constructs operate on if they are given no argument and ARGV is the default file handle For example, if the above loop is in a file called perlcat, then perlcat A B C will produce line by line all lines from file A, then B, and then C, just like the Unix utility cat. Here @ARGV = (A B C) Perl Data Types Scalar - numeric or string, starts with $. Examples $x $Cell_name $Instance_name $ARGV[0] $whole_list[2] $Cell{"string"} $score{$studentno, $examno} = 89; $_ Arrays of Scalars (Lists) - Indexed by number, starts with @. Examples @whole_list @ARGV Associative Arrays {Hashes} - Indexed by string, starts with % %Cell %score Standard functions covered in this paper print - print to standard output // - match a regular expression between slashes s/// - substitute a pattern shift - shift off an element of a list chop - chop off the last character of a string split - split a string based on a pattern delimiter Example Programs 3.1 Audit a File Simple Grep - print out all lines in a file or files that match a pattern. This shows an example of Perl pattern matching. example call: grep VHDL A B #!/usr/local/bin/perl

Major differences between Awk and Gawk: Gawk has the following additional features: AWKPATH environment variable for specifying a path to search for the -f command line. ARGIND variable which tracks the FILENAME through ARGV IGNORCASE variable which enables case-insensitive string matching. FIELDWIDTHS variable to specify the width of fields. "next file" statement for skipping files, and going on to the next file. /dev/stdin, /dev/stdout, /dev/stderr, and /dev/fd/n} file name interpretation. ACKNOWLEDGMENT: Dr. Minesh B. Amin of Honeywell Tech. Center

3 Perl
Perl4 only will be covered. For information about Perl5 and its object oriented features, refer to the Perl5 release and the man pages perlref, perlobj, and perlbot. Perl features : Parsing and string manipulation Supports all Unix regular expressions plus extensions File system manipulation Associative arrays for symbol tables Fills a niche between shell and C System Administration CGI programming. Perl Features for EDA Most programs can be done in 1 or 2 pages, so programs can be written rapidly and if a program must be modified later, the program can be relearned quickly. Programs can be created very rapidly once Perl shorthand is understood. Both contents of files and the file system are easily manipulated Perl History A gift to the world created and maintained by Larry Wall Personal Aside - I estimate that I am 30% more productive because of Perl. Before Perl, I had a group of subroutines, in Pascal and then in C that aided parsing and for symbol table support. These are no longer needed because of all the built in subroutines and variables in Perl Important Special variables, arrays, and filehandles. A filehandle can be a file, device, pipe, or socket. $_ - if an argument to a function is omitted, this is used @ARGV - array that holds command line file names ARGV - filehandle for all files in command line <ARGV> STDIN - input filehandle to read from <STDIN> STDOUT - DEFAULT output stream filehandle

$pattern = shift(@ARGV); # shift removes the # first element of an array and returns it while (<>) { # Treat the rest of @ARGV as filenames if (/$pattern/) { print ; } } 3.2 Modify a file Program modifies VHDL file to add library declarations, and change others. This can happen when one vendors EDA tool generates a file to be used by a second vendors tool. This example shows the use of the Perl substitute command. #!/usr/local/bin/perl while (<>) { if (/use COMPASS_LIB.COMPASS_ETC.ALL/) { print $_ ; print "library cb60hp230d_nohold;\n" ; print " use cb60hp230d_nohold.all;\n" ; print "library cb60io220d;\n" ; print " use cb60io220d.all;\n" ; } elsif (/cb60hp230d/) { s/cb60hp230d/cb60hp230d_nohold/ ; print $_ ; } else { print $_ ; } } 3.3 Extract data from a file Parse a flattened VHDL netlist to extract a sorted count of all cells used. The line below is an example of the VHDL to be parsed Note this program was easily modified to count cell usage for Edif and Verilog. This example shows the use of associative arrays and format statements. The line below is an example of the VHDL to be parsed for I_365 : NI01D1 use configuration cb60hpd.NI01D1_CON #!/usr/local/bin/perl while (<>) { # count cell calls if (/use configuration/) { chop; ($foritem, $instance, $colon, $cell_Name) = split; $cell_Name_UC = "\U$cell_Name" ; $cell_cnt{$cell_Name_UC}++ ; $totcnt++; } } print "Cell Usage for @ARGV[0]\n\" ; print "Total cells = $totcnt\n\"; foreach $key (sort keys %cell_cnt) { write ; } format STDOUT = @<<<<<<<<<<<<<<<

$key, $cell_cnt{$key}
.

Example output from Cellcntvhdl

Cell Usage for test_06.vhd Total cells = 28 AN02D1 1 NI01D1 9 NR02D2 1 PT5B01 2 PT5B01R 1 PT5B02R 1 PT5D01 4 PT5D01R 1 PT5D01U 1 PT5T02 7

3.4 In place editing like SED but creates a backup file. This is shorthand for the Perl standard loop using the -p option which places the standard loop around the script. The -i option says to create a backup file. All files ending in .boo are modified. perl -p -i.bak -e "s/#define global/define global/" *.boo Read a file into an array $a = <STDIN>; # reads one line into $a @lines = <STDIN>; # reads all lines of a file into @lines 3.5 Loop through all items in a list. In this example, write out in groups of 6. This example shows Perl automatically handling the test for end of array. foreach $cell (@lines) { chop ($cell) ; print "$cell " ; $i++; if (($i % 6) == 0) { print "\n" ; } } 3.6 System Manipulation Search for a pattern and modify for all files in a directory and subdirectories. Example of use of the Unix find command piped being treated as a file. This Perl program changes all patterns /home/projxa to /home/projxb for all files in a directory and all subdirectories. This technique is useful when a new revision of a chip is being created. #!/usr/local/bin/perl open (FIND, "find . -print |") || die "Can't run find . -print\n; while ($filename = <FIND>) {

@####

FILE: while ($filename = <FIND>) { chop $filename ; next FILE unless -T $filename; if (!open(TEXTFILE, $filename)) {

print STDERR "Cant open $filename -continuing...\n"; next FILE; } while (<TEXTFILE>) { s/\/home\/projxa/\home\/projxb/) ; } } 3.7 - Take a list of files and create a script. A Makefile can be created using the same technique. This Perl program creates a script that compiles all VHDL files in a directory and then makes a backup copy of each file. #!/usr/local/bin/perl $unix_cmd = "ls *.vhd | " ; open (V, "$unix_cmd") || die "Cant open $unix_cmd for input\n" ; while (<V>) { chop; print "vcom $_ ; cp $_ ${_}.bak \n" ; } close (V) ; Write a mail message showing an example of piping to Unix sendmail with a list of mail recipients. Note the list of recipients could be set up in another part of the Perl program, such as a list of users who have exceeded disk quotas. Just like in shell scripts using in-line data print <<EOM will print the string until EOM is seen. open (MAIL, "| /usr/lib/sendmail -oi -t"); print <<EOM; ~From: the Disk Monitor Daemon ($0) <daemon> To: $rcpts Cc: root ~Subject: $fs on $hostname is $percen% full Each of you is taking up at least $MIN_WHINE% of the space on $hostname:$fs.Please do what you can to reduce this. your friend, the daemon EOM close MAIL; } Test for file existence before opening, so that an a file is not overwritten. if ( -e $file_name) { print (STDERR "Error $file_name already exists\n"); exit ; } Make a system call during the running of a Perl program. This uses the Unix grave character,, surrounding a command to cause immediate execution of the command. The example below removes the file xxxtempx. $rm_call = "\\rm xxxtempx" ;

($rm_call) ; 3.8 Opening a compressed file as an alternative. IF file.xyz does not exist, try opening its compressed version file.xyz.Z. Since EDA files tend to be huge, by compressing files and writing Perl programs to read compressed files, huge amounts of disk space can be saved. This program uses Unix pipes and the zcat utility. if (!(open (V, "$input_file"))) { open (V, "zcat ${input_file}.Z |") || die "Cant open $input_file for input\n" ; } 3.9 Split a full file name into directory path and file name. This works because of Perls greedy regular expression matching which does a match for the maximal pattern length. #!/usr/local/bin/perl $fullspec = /usr/local/etc/httpd/abc/def.ghi/testfile.html; $file = $fullspec unless ($path, $file) = $fullspec =~ m:(.*/)(.*):; print "Path is $path, file is $file\n"; Output of the program is: Path is /usr/local/etc/httpd/abc/def.ghi/, file is testfile.html Perl program to find the difference between two lists. This example uses the Perl grep command, which evaluates the first argument against each element of the second grep argument. #!/usr/local/bin/perl local(%mark1); grep($mark1{$_}++,@listd); @result = grep(!$mark1{$_}, @listm); #print "List result: @result\n"; @result = sort( @result ); print "List result M-D:\n @result\n"; Perl Bibliography Books Larry Wall and Randal L. Schwartz, Programming Perl, 1991, 465 pages, OReilly and Associates, Inc., ISBN 0-937175-64-1, $29.95. The "Camel Book". A thorough reference with plenty of examples. This is the best language documentation I have seen. Randal L. Schwartz, Learning Perl, 1993, 274 pages, OReilly and Associates, Inc., ISBN 1-56592-042-2, $24.95 The "Llama Book". Ellie Quigley, Perl by Example, 385 pages, ISBN 0-13122839-0. $26.96. Internet Sites http://www.phlab.missouri.edu/cgi-bin/perlman http://www.metronet.com.70/1/perlinfo/faq