Industry guru Dave Taylor offers tech support on technical and business topics, including iPhone, iPod, Microsoft Windows, Sony PSP, cellphones, online advertising, CSS, Web design, business, Unix, Linux, SEO, Mac OS X, and shell script programming.     


How can I generate reports in Linux with an Awk script?

I'm trying to write an awk script that will take a text file phone report from a windows box and break it up into smaller text files. I know it'll only take a few minutes, but I'm not all that familiar with AWK and slogging through this O'Reilly book is really starting to tick me off. Any chance you can help?


Dave's Answer:

Though I realize that Perl is far more popular nowadays, I'm an old-time Unix guy, so I really still have a soft spot for awk, and am always glad to get a question about how to accomplish something in this simple, lightweight, and powerful scripting language. In a lot of ways, awk is really the perfect sidekick to shell script programming too, and in my book Wicked Cool Shell Scripts I delve into the world of awk more than once.

First off, to clear up a common misconception, it's not called awk as a shorthand for "awkward"! In fact, the program gets its name from the three programmers who wrote it: Alfred Aho, Brian Kernighan, and Peter Weinberger, all from AT&T Bell Telephone Labs at the time.

You were kindly enough to send along a snippet of your data file, so let's start by having a look:

Department: Undefined                
 Ext. 5648                          
 10/10  3:18p 014153               INCOMING      312-555-2922 IL 0:00:24   0.00
 10/30 12:10p 014148               INCOMING      773-555-7473 IL 0:03:43   0.00
 Totals for ext. 5648:  2 call(s).  0 hours, 4 min.   $0.00
 Ext. 5724                          
 Totals for ext. 5724:  0 call(s).  0 hours, 0 min.   $0.00
 Dept. Undefined had 2 call(s).
 Totals: 0 hours, 4 min.  $0.00   Averages: 2 min., 3 sec.  $0.00
Department: ACCOUNTING               
 Ext. 5590 ORTIZ, RON               
 10/05  9:38a 012144               LOCAL             555-7800 IL 0:01:13   0.00
 10/05 10:38a 014157               INCOMING      618-555-6768 IL 0:04:57   0.00
 10/05  2:44p 010108               LOCAL         630-555-8200 IL 0:01:00   0.00
....
 10/26  4:30p 010112               LOCAL         708-555-6436 IL 0:00:00   0.00
 Totals for ext. 5592:  25 call(s).  0 hours, 39 min.   $0.00
 Dept. ACCOUNTING had 35 call(s).
 Totals: 0 hours, 59 min.  $0.00   Averages: 1 min., 41 sec.  $0.00
Department: ACCOUNTS RECEIVABLE      
 Ext. 5214 BIGA, PAT                
 10/03 11:30a 010113               LONGDIST      805-555-5100 CA 0:00:30   0.00
 10/03  3:38p 010101               LONGDIST      815-555-7091 IL 0:00:10   0.00
 10/03  4:09p 010120               LONGDIST      617-555-7449 MA 0:00:55   0.00
 10/04  9:43a 014151               INCOMING      815-555-7769 IL 0:02:09   0.00
 10/04 11:14a 010114               LONGDIST      617-555-7449 MA 0:01:12   0.00

What you need to do is split this one data file into a set of separate information files, one per department, where the departments are delineated by the line Department: xxx, where 'xxx' is the name of the department.

This is surprisingly simple to accomplish in awk once you have a familiarity with the language. The key is to realize that you can redirect the output of a script by simply using the shell-like > symbol. Here's my entire script, saved as "test.awk":

{  if ( $1 == "Department:" ) {

if (NF > 3) {
department = $2"-"$3
} else {
department = $2
}

print "Starting new file for Department " department
outputfile=department ".dat"
print $0 > outputfile

}
else {
print $0 > outputfile
}

}

My code tends to be very open, with lots of white space for readability, but you really could trim this down to four or five lines if you preferred.

The trick here is that we check the first word of every line for the word "Department:" and when that's matched, we save the new department name in variable department and then tack on a ".dat" suffix to create the output file name.

Run this on the sample data file and in less than a second the task is accomplished:

$ awk -f test.awk < SampleInput
Starting new file for Department Undefined
Starting new file for Department ACCOUNTING
Starting new file for Department ACCOUNTS-RECEIVABLE

That should help you move along in the right direction.

In terms of your confusion with the O'Reilly book, I strongly encourage you to write to the author and publisher detailing why you find the book confusing, so they can fix it in the next edition.


More Useful Unix and Linux Help Articles:
✔   Copy and Paste from the Mac OS X Command Line?
I am constantly running commands in Terminal.app on my MacBook and then copying and pasting the results into email messages or documents. Yes,...
✔   Shell script to convert lowercase to title case?
As part of a project I'm working on, I find myself deep in a Linux shell script, needing to have a subroutine that...
✔   Can I script renaming files based on an XML data map?
I have a folder full of files which are named with four digits and a file extension e.g. 0312.file and an XML-file describing...
✔   Test for valid numbers in a Bash shell script?
In a different discussion on this site [see Redirecting input in a shell script] a visitor commented that "I was too busy trying...
✔   Review: iSSH for the iPad/iPhone
If you're running an online business like I am, there are times when you need to connect and log in to the server...

Let's stay in touch!
Sign up for my weekly AskDaveTaylor Newsletter and you'll receive even more tech and gadget help right to your inbox, along with exclusive news and industry updates. It's good stuff. I promise!
    Enter your name: and your email addr:  





Categorized: Unix and Linux Help   (Article 4285, Written by )
Tagged:
Previous: What can I do with a Sony PSP?
Next: Log in to America Online (AOL) as a guest?




Reader Comments To Date: 1

nadeem said, on October 2, 2006 1:10 PM:

How could I generate a list of 10 files with just modification at one single place.

Starbucks coffee cup I do have a lot to say, and questions of my own for that matter, but first I'd like to say thank you, Dave, for all your helpful information by buying you a cup of coffee!

I do have a comment, now that you mention it!











I will never send you any unsolicited email. Ever.






Check This Out Too...

 
Look for Answers
Need Help? Ask Dave Taylor!
Powered By
Linux Journal: Free Issue!


Follow Me on Pinterest

Find Me on Google+
ADT on G+
© 2002 - 2013 by Dave Taylor. All Rights Reserved.

Note: This web site is for the purpose of disseminating information for educational purposes, free of charge, for the benefit of all visitors. We take great care to provide quality information. However, we do not guarantee, and accept no legal liability whatsoever arising from or connected to, the accuracy, reliability, currency or completeness of any material contained on this web site or on any linked site. Further, please note that by submitting a question or comment you're agreeing to my terms of service, which are: you relinquish any subsequent rights of ownership to your material by submitting it on this site. My lawyer says "Thanks".
"Ask Dave Taylor®" is a registered trademark of Intuitive Systems, LLC.