How do I read lines of data in a shell script?

how can i read an input stream i.e. if piped from another command into my shell script ?

The great thing about Unix is that your shell script basically can't differentiate between you typing in lines directly and that input coming from a file redirect or even a pipeline of commands. So just write your script to read "stdin", as I show above, and you'll be good to go!

You know, there's more than just students out there looking for relatively basic scripting questions. I am not a student but am a Linux amateur, and spent probably 45 minutes figuring out how to make my more sophisticated shell scripts read from configuration files (to increase portability and share value of the scripts), before I finally found your tip.

At first I used piping (i.e. cat myfile | while read myvar), but the variables don't survive when the pipe is done. Your answer was just what I needed. Thanks.

How should I change the script if in case the file to be read is remotely located ? I want to read a DNS server password file which is located on my PC .

Dave, excellent stuff, thanks. I've been spending days trying to figure out how to use a configuration file and break it down like this - your script is just what I was after.
I'm with Joshua on this one. Now that we've just got a mac, I'm trying to understand bash a lot more. Thanks again.

Good piece of code. I like it. I got some new ideas from such style of shell scripting.

Slight mod if you wish to use Bourne shell instead on Solaris system (replace " with `, remove $ before (echo.., and field 5 has the fullname:

#!/bin/sh

while read inputline
do
login=`(echo $inputline | cut -d: -f1)'
fulln=`(echo $inputline | cut -d: -f5)`
echo login = $login and fullname = $fulln
done < /etc/passwd

exit 0

If it makes you feel any better Dave, this helped me tremendously and I haven't had a homework assignment in 15 years.

Shell scripting is fun! Here's one of my favorite techniques with the 'read' command when you have delimited data (like the passwd file...)

#!/bin/sh
#
#
#RUNDATE="`date +%Y%m%d`.txt" # only for a given date
PF=/etc/passwd
OFS=$IFS
IFS=':' # internal file separator are now colons, no longer whitespace
# the read command will populate each paramter of the /etc/passwd file
# if there are too-few parameters, the rest gets thrown into the last one
# here we print the login id and the name.
# a comma is printed as if we wanted to make this a CSV file..
#
while read p1 p2 p3 p4 p5 p6 junk
do
#
echo $p1,$p5
#
done < $PF
OFS=$IFS # reset separator for the remainder of the script
#

Hi All,

I have task to automate using Shell scripting,please help me in doing the same.

In my task, i need to :
1.Log into the sharepoint server, access the file and the data inside the file should be copied to a output file.
2.Now the output file should be used as an input file todo the configuration settings using telnet.

Can somebody help me in this..

Thanks & Regards,
Sri

Well reading seems quite straight forward, but it has its annoying points... for instance, reading the tab char (\t) with 'read' in ksh.

imagine the file like (I'll write separators as literal\)

1[tab]0[space]a[tab]99[tab]8
2[tab]1[space]b[tab]88[tab]7
3[tab]2[space]c[tab]77[tab]6

a simple loop like:

while read lin ; do
echo $lin
done < thatfile.txt

will output:

1[space]0[space]a[space]99[space]8
2[space]1[space]b[space]88[space]7
3[space]2[space]c[space]77[space]6

It means that if that's a tab-delimited file, you are kind of lost to read the second field which should be '0 a', '1 b' and so on...

I haven't figured out yet how to solve this, I always have to use some kind of workaround.

f.

not found error

$(echo $line|cut -d',' -f1)

Hi,

Thanks for the great how-to!

Your example seems to work while the file "has next line". How does one deal with text if it's on the last line of the file? How does one echo the line number? what causes the automatic incrementation of the line number?

As previously mentioned,there are plenty of us out there who aren't working on a HW assignment. I'm an amateur java programmer trying to learn shell scripting for work because getting Java to handle text files is cumbersome and using an interpretive language just makes more sense for my data handling. Using Dr Java or learning Ruby was tempting, but I want the scripts to run natively on my CEO's mac as well as my linux boxen. Further, shell scripting will hopefully allow me to give back and help patch GNU/linux bugs.

how to count number of free spaces in a file using a cell script

Partha, I don't know what you mean by "free spaces"?

While while-read loops made good sense back in the days when we only had single core CPUs it is often worth the effort to rewrite your script to use more CPU cores today.

GNU Parallel http://www.gnu.org/software/parallel/ is a help in doing that.

while read lin ; do
echo $lin
done < thatfile.txt

can be written as:

cat thatfile.txt | parallel echo {}

or simply as:

cat thatfile.txt | parallel echo

For CPU heavy jobs or jobs waiting for reply from the network this can make a tremendous speed up of your scripts.

Watch the introduction video at http://www.youtube.com/watch?v=LlXDtd_pRaY

Hello Dave,
In this script:
while read myline
do
echo $myline
done < inputfile

How can I make it to read record from TWO input files? I want to read 1st record of inputfile1 and inputfile2 then make some processing out of that, then read next record, process, and so on..

That's an interesting question, Athan, and I'm not sure of the answer. I think I would pre-process the two files to create a new temporary file that had a line from file 1, a line from file 2, another line from file 1, and so on. I further surmise that might be something done 100x easier in Perl...

Here's a way to read from two files:

paste -d'\n' file1 file2 | while read line1 && read line2;
do
echo "$line1 $line2"
done

How to get the date and time when a user was added using shell command ?

cat /etc/passwd |\
while IFS=: read PLOGIN PHASH PUID PGID PGECOS PHOME PSHELL PREST ; do

do_something_here

done

# Test case - no new line character on last line
echo -n abc > inputfile
# error no output generated
while read myline;do echo $myline;done < inputfile
# possible solution - does not check empty lines
while true;do read myline;eof=$?;echo $myline;if [[ $eof != 0 ]];then break;fi;done<inputfile

hi
i am completely new in bash script. here is my question. i have text file which contains list of ips and status code. i want to extract for example the ip with most successful response. how can i do it? thanks to all

Sounds like a case for sort | uniq -c | sort -rn actually, Roshaan. Check the 'sort' and 'uniq' man pages.

how to get inputs from user for FOR LOOP?...

Example :
print "Enter number:"
read a #to get input from user
for i in $a #checking condition
do
print "$a\n"; ##HERE I NEED TO GET ITERATION WHICH I GONNA GIVE AS INPUT AND WANNA CHECK CONDITION FROM FOR LOOP:

done

i have to read an xml file that too certain attributes of a field please tell me how can i do that

and, also please explain me in cut function
login=`(echo $inputline | cut -d: -f1)'
fulln=`(echo $inputline | cut -d: -f5)`

what does that -f1 and -f5 mean and y is it used

Come on, guys. If you have a homework assignment, figure out the answer, don't post your question here. :-)

Oh, and Narain? Type "man cut" and you'll learn what the '-f' flag does...

Okay - many of you do not want to help a struggling student with a homework assignment, stating you dont want to do his homework for him. - YET, you have no problem helping a Linux Admin making 80k a year. Isn't that DOING HIS WORK for him? I mean, the student is not getting paid, but is looking for some direction in his linux quest. The 80k guy is getting paid cuz he (supposedly) already knows this stuff.

Interesting line of logic, Ro. My counterpoint: when you're a student, nothing you do is mission critical, so it's the time to make mistakes and struggle, trying to figure out how to fit things together and see the "big picture". Once you've got a job, however, then a mistake can cost the company money, can cause customers to have dangerous problems or even lose their lives (imagine the firmware in a heart monitor, for example). Stakes are higher, so it's critical that a savvy IT professional taps into any and every resource they have.

Asking me and my community to do a homework assignment for you isn't teaching you anything. Narain asks "what does -f1 mean" when a ten second perusal of the 'cut' man page will explain exactly what it means. That's not "tapping into the community", that's just lazy.

Dave, i'm a java developer who had the lack of sense to suggest a better way scraping our content managed data into our portal. so now it's on me to implement this solution.:)

anyhow i need a script to get all the files in a directory, read the file, which will contain a delimited String and parse the 2 values into variables which i pass to another function. i came across this thread on google and reading the files seemed easy enough and i wrote my script, but for some reason my while loop doesn't seem to be executing, as if the file where empty, but i know it is not empty.

this is the script

javaCmd=/usr/java/jdk1.5.0_22/bin/java
fetchCmd=com.iplanet.portalserver.providers.common.FetchUrl
desktopDir=/var/opt/sun/portal/portals/EatonPortal/desktop
refreshAllFilesDir=/opt/eaton/portal/properties/common/refreshAllFiles/*
input=i
output=o
echo $refreshAllFilesDir
for file in $refreshAllFilesDir
do
echo "FILE"
echo $file
while read inputline
do
echo "INPUT LINE"
echo $inputline
$input="$(echo $inputline | cut -d| -f1)"
$output="$(echo $inputline | cut -d| -f2)"
done < $file
echo "INPUT"
echo $input
echo "OUTPUT"
echo $output
${javaCmd} ${fetchCmd} ${input} ${desktopDir}${output} true 20 20000 20000

done
when i echo $input and $output at the end they are still set to i and o respectively. any thoughts? nothing is jumping out at me. when i echo $file i get the full path to the file as i would expect and there is a file with data in that location.

thx Mike

Hi, I am new to shell scripting. I have two input files i want to compare them and print if they fileds are same. Could you please suggest solution?
Thanks in advance.

echo "INPUT LINE"
echo $inputline
$input="$(echo $inputline | cut -d| -f1)"
$output="$(echo $inputline | cut -d| -f2)"

Remove "$" from the assignments
input="$(echo $inputline | cut -d| -f1)"
output="$(echo $inputline | cut -d| -f2)"
echo $input
...

Hi,

I'm writing a perl script in which I need to login to a remote server and thereafter execute the rest of the commands present in the script at that server..
The problem is I can't execute the rest of the cmds unless I logout of the server and doing so defeats the purpose...so plz advice regarding the same!!