Industry guru Dave Taylor offers free tech support on a wide variety of technical and business topics, including HTML, Apple iPhone, online advertising, Cascading Style Sheets, Web design, management, Unix, Linux, search engine optimization, online dating, Mac OS X, shell script programming and Microsoft Windows.

How do I read lines of data in a shell script?

Dave, where can I find a bash script that can read data from a file; the information should be separated by tabs or commas for easy pickup, and can only be accesed by a row.

Dave's Answer:

This sounds suspiciously like a homework assignment, something I generally don't offer assistance with since I think students should do their own work, but this particular question appears in my mailbox often enough that I thought it would be valuable to address it here.

There's a very easy way to solve this:

while read myline
do
echo $myline
done < inputfile

If the fields in a given line are separated by a known delimiter, either a tab or a comma, for example, then I suggest that you could use the cut command to extract specific values.

To demonstrate, let's pull some useful data out of the /etc/passwd file, a file that has lines of data in known fields, separated with a ":" as the deilmiter. Here's a typical line of data:

unknown:*:99:99:Unknown User:/var/empty:/usr/bin/false

The first field (remember, they're separated by colons) is the account name, the second the encrypted password (not shown because it's in a separate 'shadow' file for security), then the remaining fields are account ID, group ID, full user name, home directory and login shell.

Let's just pull out login and full name to see what that looks like:

#!/bin/sh

while read inputline
do
login="$(echo $inputline | cut -d: -f1)"
fulln="$(echo $inputline | cut -d: -f4)"
echo login = $login and fullname = $fulln
done < /etc/passwd

exit 0

You can see how the cut program makes this a straightforward task, albeit one that can be done more quickly in other scripting languages like Perl. But if you want to work with shell scripts, the combination of a while read loop with the input redirected and the great cut command should give you all the data parsing capabilities you need.

Hope that helps you out with your homework. :-)

Help others find this article at Del.icio.us, Digg, Netscape, Reddit, and Stumble Upon

Categorized: Shell Script Programming (Article 4282)
Tagged:
Previous: How do I read a Google AdSense report?
Next: Automating SSH with a shell script

Subscribe!

Never miss another useful Q&A article again! Subscribe to AskDaveTaylor with Google Reader.

Comments

how can i read an input stream i.e. if piped from another command into my shell script ?

Posted by: me at December 14, 2005 5:29 PM

The great thing about Unix is that your shell script basically can't differentiate between you typing in lines directly and that input coming from a file redirect or even a pipeline of commands. So just write your script to read "stdin", as I show above, and you'll be good to go!

Posted by: Dave Taylor at December 14, 2005 11:07 PM

You know, there's more than just students out there looking for relatively basic scripting questions. I am not a student but am a Linux amateur, and spent probably 45 minutes figuring out how to make my more sophisticated shell scripts read from configuration files (to increase portability and share value of the scripts), before I finally found your tip.

At first I used piping (i.e. cat myfile | while read myvar), but the variables don't survive when the pipe is done. Your answer was just what I needed. Thanks.

Posted by: Joshua Curtiss at January 27, 2007 4:49 PM

How should I change the script if in case the file to be read is remotely located ? I want to read a DNS server password file which is located on my PC .

Posted by: Pooja at February 27, 2007 10:26 PM

Dave, excellent stuff, thanks. I've been spending days trying to figure out how to use a configuration file and break it down like this - your script is just what I was after.
I'm with Joshua on this one. Now that we've just got a mac, I'm trying to understand bash a lot more. Thanks again.

Posted by: Baz at January 18, 2008 5:14 PM

Good piece of code. I like it. I got some new ideas from such style of shell scripting.

Posted by: Good Job Man at June 5, 2008 12:13 AM

Slight mod if you wish to use Bourne shell instead on Solaris system (replace " with `, remove $ before (echo.., and field 5 has the fullname:

#!/bin/sh

while read inputline
do
login=`(echo $inputline | cut -d: -f1)'
fulln=`(echo $inputline | cut -d: -f5)`
echo login = $login and fullname = $fulln
done < /etc/passwd

exit 0

Posted by: Paul Johnston at July 8, 2008 9:10 AM

If it makes you feel any better Dave, this helped me tremendously and I haven't had a homework assignment in 15 years.

Posted by: Ryan at December 12, 2008 7:22 AM

Shell scripting is fun! Here's one of my favorite techniques with the 'read' command when you have delimited data (like the passwd file...)

#!/bin/sh
#
#
#RUNDATE="`date +%Y%m%d`.txt" # only for a given date
PF=/etc/passwd
OFS=$IFS
IFS=':' # internal file separator are now colons, no longer whitespace
# the read command will populate each paramter of the /etc/passwd file
# if there are too-few parameters, the rest gets thrown into the last one
# here we print the login id and the name.
# a comma is printed as if we wanted to make this a CSV file..
#
while read p1 p2 p3 p4 p5 p6 junk
do
#
echo $p1,$p5
#
done < $PF
OFS=$IFS # reset separator for the remainder of the script
#

Posted by: Terry at May 27, 2009 7:24 AM

Hi All,

I have task to automate using Shell scripting,please help me in doing the same.

In my task, i need to :
1.Log into the sharepoint server, access the file and the data inside the file should be copied to a output file.
2.Now the output file should be used as an input file todo the configuration settings using telnet.

Can somebody help me in this..

Thanks & Regards,
Sri

Posted by: Srinivas at June 8, 2009 3:25 AM

I have something to say, now that you mention it, but ...

I do have a lot to say, and questions of my own for that matter, but first I'd like to say thank you for all your efforts on this Web site by buying you a cup of coffee!

I do have a comment, now that you mention it!