|
|
How do I read lines of data in a shell script?Dave, where can I find a bash script that can read data from a file; the information should be separated by tabs or commas for easy pickup, and can only be accesed by a row. This sounds suspiciously like a homework assignment, something I generally don't offer assistance with since I think students should do their own work, but this particular question appears in my mailbox often enough that I thought it would be valuable to address it here. There's a very easy way to solve this: while read myline
do echo $myline done < inputfile If the fields in a given line are separated by a known delimiter, either a tab or a comma, for example, then I suggest that you could use the cut command to extract specific values. To demonstrate, let's pull some useful data out of the /etc/passwd file, a file that has lines of data in known fields, separated with a ":" as the deilmiter. Here's a typical line of data: unknown:*:99:99:Unknown User:/var/empty:/usr/bin/false
The first field (remember, they're separated by colons) is the account name, the second the encrypted password (not shown because it's in a separate 'shadow' file for security), then the remaining fields are account ID, group ID, full user name, home directory and login shell. Let's just pull out login and full name to see what that looks like: #!/bin/sh
while read inputline do login="$(echo $inputline | cut -d: -f1)" fulln="$(echo $inputline | cut -d: -f4)" echo login = $login and fullname = $fulln done < /etc/passwd exit 0 You can see how the cut program makes this a straightforward task, albeit one that can be done more quickly in other scripting languages like Perl. But if you want to work with shell scripts, the combination of a while read loop with the input redirected and the great cut command should give you all the data parsing capabilities you need. Hope that helps you out with your homework. :-)
More Useful Shell Script Programming Articles:
✔ Secretly capture screenshots on my Mac?
When I used to work on a Linux system, there was a utility we had that would let me take screen captures every...
✔ Parsing "id" strings in a Shell Script?Hello Dave. I need a Bash shell script that creates a directories with the group names automatically when user logs in to the...
✔ Copy and Paste from the Mac OS X Command Line?I am constantly running commands in Terminal.app on my MacBook and then copying and pasting the results into email messages or documents. Yes,...
✔ Script to test line lengths for Twitter compatibility?I've been tasked with writing a series of tweets for a Black Friday marketing campaign and am finding it a bit tricky because...
✔ Shell script to convert lowercase to title case?As part of a project I'm working on, I find myself deep in a Linux shell script, needing to have a subroutine that...
Let's stay in touch!
Sign up for my weekly AskDaveTaylor Newsletter and you'll receive even more tech and gadget help
right to your inbox, along with exclusive news and industry updates. It's good stuff. I promise!
Categorized:
Shell Script Programming
(Article 4282,
Written by Dave Taylor)
Tagged: Previous: How do I read a Google AdSense report? Next: Automating SSH with a shell script Reader Comments To Date: 33Dave Taylor said, on December 14, 2005 11:07 PM:
The great thing about Unix is that your shell script basically can't differentiate between you typing in lines directly and that input coming from a file redirect or even a pipeline of commands. So just write your script to read "stdin", as I show above, and you'll be good to go! Joshua Curtiss said, on January 27, 2007 4:49 PM:
You know, there's more than just students out there looking for relatively basic scripting questions. I am not a student but am a Linux amateur, and spent probably 45 minutes figuring out how to make my more sophisticated shell scripts read from configuration files (to increase portability and share value of the scripts), before I finally found your tip. At first I used piping (i.e. cat myfile | while read myvar), but the variables don't survive when the pipe is done. Your answer was just what I needed. Thanks. Pooja said, on February 27, 2007 10:26 PM:
How should I change the script if in case the file to be read is remotely located ? I want to read a DNS server password file which is located on my PC . Baz said, on January 18, 2008 5:14 PM:
Dave, excellent stuff, thanks. I've been spending days trying to figure out how to use a configuration file and break it down like this - your script is just what I was after. Good Job Man said, on June 5, 2008 12:13 AM:
Good piece of code. I like it. I got some new ideas from such style of shell scripting. Paul Johnston said, on July 8, 2008 9:10 AM:
Slight mod if you wish to use Bourne shell instead on Solaris system (replace " with `, remove $ before (echo.., and field 5 has the fullname: #!/bin/sh while read inputline exit 0 Ryan said, on December 12, 2008 7:22 AM:
If it makes you feel any better Dave, this helped me tremendously and I haven't had a homework assignment in 15 years. Terry said, on May 27, 2009 7:24 AM:
Shell scripting is fun! Here's one of my favorite techniques with the 'read' command when you have delimited data (like the passwd file...) #!/bin/sh Srinivas said, on June 8, 2009 3:25 AM:
Hi All, I have task to automate using Shell scripting,please help me in doing the same. In my task, i need to : Can somebody help me in this.. Thanks & Regards, filippo said, on December 7, 2009 8:25 AM:
Well reading seems quite straight forward, but it has its annoying points... for instance, reading the tab char (\t) with 'read' in ksh. imagine the file like (I'll write separators as literal\) 1[tab]0[space]a[tab]99[tab]8 a simple loop like: while read lin ; do will output: 1[space]0[space]a[space]99[space]8 It means that if that's a tab-delimited file, you are kind of lost to read the second field which should be '0 a', '1 b' and so on... I haven't figured out yet how to solve this, I always have to use some kind of workaround. f. john said, on January 28, 2010 5:45 AM:
not found error $(echo $line|cut -d',' -f1) Jason said, on February 8, 2010 6:00 PM:
Hi, Thanks for the great how-to! Your example seems to work while the file "has next line". How does one deal with text if it's on the last line of the file? How does one echo the line number? what causes the automatic incrementation of the line number? As previously mentioned,there are plenty of us out there who aren't working on a HW assignment. I'm an amateur java programmer trying to learn shell scripting for work because getting Java to handle text files is cumbersome and using an interpretive language just makes more sense for my data handling. Using Dr Java or learning Ruby was tempting, but I want the scripts to run natively on my CEO's mac as well as my linux boxen. Further, shell scripting will hopefully allow me to give back and help patch GNU/linux bugs. partha said, on April 30, 2010 1:46 PM:
how to count number of free spaces in a file using a cell script Dave Taylor said, on April 30, 2010 4:56 PM:
Partha, I don't know what you mean by "free spaces"? Ole Tange said, on June 13, 2010 5:06 PM:
While while-read loops made good sense back in the days when we only had single core CPUs it is often worth the effort to rewrite your script to use more CPU cores today. GNU Parallel http://www.gnu.org/software/parallel/ is a help in doing that. while read lin ; do can be written as: cat thatfile.txt | parallel echo {} or simply as: cat thatfile.txt | parallel echo For CPU heavy jobs or jobs waiting for reply from the network this can make a tremendous speed up of your scripts. Watch the introduction video at http://www.youtube.com/watch?v=LlXDtd_pRaY athan said, on July 21, 2010 5:23 AM:
Hello Dave, How can I make it to read record from TWO input files? I want to read 1st record of inputfile1 and inputfile2 then make some processing out of that, then read next record, process, and so on.. Dave Taylor said, on July 21, 2010 8:18 AM:
That's an interesting question, Athan, and I'm not sure of the answer. I think I would pre-process the two files to create a new temporary file that had a line from file 1, a line from file 2, another line from file 1, and so on. I further surmise that might be something done 100x easier in Perl... Tom Hedley said, on July 23, 2010 2:47 PM:
Here's a way to read from two files: paste -d'\n' file1 file2 | while read line1 && read line2;
upraj said, on September 3, 2010 8:52 PM:
How to get the date and time when a user was added using shell command ? michal said, on September 8, 2010 3:32 PM:
cat /etc/passwd |\ do_something_here done TomasN said, on October 12, 2010 11:30 AM:
# Test case - no new line character on last line roshaan said, on September 12, 2011 8:34 AM:
hi Dave Taylor said, on September 12, 2011 12:18 PM:
Sounds like a case for sort | uniq -c | sort -rn actually, Roshaan. Check the 'sort' and 'uniq' man pages. Crazy said, on December 12, 2011 2:29 PM:
how to get inputs from user for FOR LOOP?... Example : done narain said, on December 13, 2011 5:08 AM:
i have to read an xml file that too certain attributes of a field please tell me how can i do that and, also please explain me in cut function what does that -f1 and -f5 mean and y is it used Dave Taylor said, on December 13, 2011 9:35 AM:
Come on, guys. If you have a homework assignment, figure out the answer, don't post your question here. :-) Oh, and Narain? Type "man cut" and you'll learn what the '-f' flag does... Ro said, on December 13, 2011 9:34 PM:
Okay - many of you do not want to help a struggling student with a homework assignment, stating you dont want to do his homework for him. - YET, you have no problem helping a Linux Admin making 80k a year. Isn't that DOING HIS WORK for him? I mean, the student is not getting paid, but is looking for some direction in his linux quest. The 80k guy is getting paid cuz he (supposedly) already knows this stuff. Dave Taylor said, on December 13, 2011 9:50 PM:
Interesting line of logic, Ro. My counterpoint: when you're a student, nothing you do is mission critical, so it's the time to make mistakes and struggle, trying to figure out how to fit things together and see the "big picture". Once you've got a job, however, then a mistake can cost the company money, can cause customers to have dangerous problems or even lose their lives (imagine the firmware in a heart monitor, for example). Stakes are higher, so it's critical that a savvy IT professional taps into any and every resource they have. Asking me and my community to do a homework assignment for you isn't teaching you anything. Narain asks "what does -f1 mean" when a ten second perusal of the 'cut' man page will explain exactly what it means. That's not "tapping into the community", that's just lazy. mike said, on January 20, 2012 2:27 PM:
Dave, i'm a java developer who had the lack of sense to suggest a better way scraping our content managed data into our portal. so now it's on me to implement this solution.:) anyhow i need a script to get all the files in a directory, read the file, which will contain a delimited String and parse the 2 values into variables which i pass to another function. i came across this thread on google and reading the files seemed easy enough and i wrote my script, but for some reason my while loop doesn't seem to be executing, as if the file where empty, but i know it is not empty. this is the script javaCmd=/usr/java/jdk1.5.0_22/bin/java done thx Mike Pooja said, on January 21, 2012 4:21 AM:
Hi, I am new to shell scripting. I have two input files i want to compare them and print if they fileds are same. Could you please suggest solution? Bryan said, on January 31, 2012 8:10 AM:
echo "INPUT LINE" Remove "$" from the assignments Virtuoso said, on February 1, 2012 9:20 PM:
Hi, I'm writing a perl script in which I need to login to a remote server and thereafter execute the rest of the commands present in the script at that server..
I do have a comment, now that you mention it!Check This Out Too... |
Recent Entries
Look for Answers
Recommended
All Our Categories
Apple iPad Help
Articles and Reviews Auctions and Online Shopping Blogs and Blogging Building Web Site Traffic Business and Management Computer and Internet Basics d) None of the Above Facebook Help Google Gmail Help Google Plus Help HTML, JavaScript and Web Site Programming Industry News and Trade Shows iPhone and Cell Phone Help iPod, Sony PSP and MP3 Player Help Kindle Fire Help Mac OS X Help Pay Per Click (PPC) Advertising Pinterest Help Search Engine Optimization (SEO) Shell Script Programming Tech Support Video Help The Writing Business Twitter, LinkedIn and Social Network Help Unix and Linux Help Video Game Tips and Help Windows PC Help Find Me on Google+ ADT on G+ |
how can i read an input stream i.e. if piped from another command into my shell script ?