Industry guru Dave Taylor answers free tech support questions about a wide variety of business and technical topics, including blogging, Google AdSense, MySpace, Sony PSP, Apple iPod, Mp3 players, management, Linux, SEO, Mac OS X, Facebook, Twitter, LinkedIn and Microsoft Windows.

How do I strip leading zeroes for math in a shell script?

I have a file containing lines of data that are amounts padded with leading zeros, similar to the snip below.

  0000000004
  0000000016
  0000000012
  0000000008

Using a shell script, how can I add up a column of numbers contained in lines when BASH interprets the numbers as octal when I do in-line math?


Dave's Answer:

What an interesting puzzle you present to me! My first thought was that there's a cool way I can solve this using a shell function, one that stripped a single leading zero then compared its results to the pre-truncated version of the value, until they matched (e.g., all the leading zeroes were deleted).

It'd look something like this:

function stripzeroes
{
  myvalue=$1
  newvalue="0"
  while [ "$newvalue" != "$lastvalue" ]
    do
    newvalue="$(echo $myvalue | sed 's/^0//')"
    lastvalue=$myvalue
    myvalue=$newvalue
  done
  # return value is the global variable 'newvalue'
}

This would then be called as stripzeroes "0000000003434" (or whatever value you'd read in from the data file) and the result would be returned as the value newvalue without the leading zeroes.

Nice solution, classic little shell script function, but there's one problem.

With the right regular expression, you can strip all the leading zeroes from your data fields with just a few characters:

valueWithoutZeroes="$(echo $valueWithZeroes | sed 's/0*//')"

That's right, but using the proper regular expression to the sed command, you can very easily strip off all the leading zeroes, making your math far, far simpler and all without complicating your shell script with useless functions!

Hope that helps you out.



Help others find this article at Del.icio.us, Digg, Netscape, Reddit, and Simpy.

Subscribe!

Never miss another useful Q&A article again! Subscribe to AskDaveTaylor with Google Reader.

Comments

Not to be confused with this Dave Taylor, I'm the Dave Taylor from the game industry, and I'm visiting because my uncle Paul Taylor thought I might have been doing this as I share your penchant for answering things.

It's a neat site, and I think you're providing a darned handy service, btw.

I think there's a small bug with both solutions. You've got an issue if one of the lines reads thusly:

0000000000

You'd want to strip that down to just "0", but both algorithms you use basically delete the line, which could cause some pretty serious issues, depending on what's eating the output.

Posted by: Dave Taylor at April 27, 2006 11:50 PM

Darn, you're right, Dave! Using the 'sed' solution, simply add a test after the conditional:

if [ -z "$valueWithoutZeroes" ] ; then
valueWithoutZeroes="0"
fi

that should do the trick!

Posted by: Dave Taylor at April 28, 2006 12:44 AM

I figure out a much simpler way of doing it:

newValue=`expr $oldValue + 0`

Note that this only works with "data that are amounts", as the question says.

Posted by: Alejandro Biasin at November 2, 2006 12:04 PM

Here's my usual solution, pretty much equivalent to Alejandro's use of expr:
newValue=`echo $oldValue | awk '{print $1 + 0}'`
or
newValue=`echo $oldValue | awk '{printf "%d\n", $1}'


Posted by: Garrett Nievin at January 7, 2007 12:59 PM

I'm fairly new to this and I do follow the logic. However, I have problems trying to tie it all up together. For instance I have a file 'benshrs' with figures I need to add up in columns 21 thru 27. Some are all zeroes. What's the best way (complete script)that combines a. the cut -c 21-27 b. strip the leading zeroes except when it is all zeroes, c. add up the figures in do ... done loop, and lastly print the final sum.

My crude attempt (below) has failed miserably!
hrs_total=0.00
for line_num in `cut -c 21-27 $0`
do
line_hrs= `expr `sed 's/0*//' $line_num` + 0`
(hrs_total=$hrs_total + $line_hrs)
done
hrs_total=`echo "scale=2;$hrs_total/100" | bc`
echo "Hours Total = $hrs_total"

Posted by: Ben at April 26, 2007 8:58 AM

I'm not surprised this isn't working, Ben: you can't use floating point / real numbers in a shell script, so even your initial assignment of hrs_total=0.00 is going to fail before you even get into the for loop. It's possible that you need to do this in awk or, better, Perl, to get it working.

Posted by: Dave Taylor at May 7, 2007 7:55 PM

Not that your option wouldn't work, Dave, but I'm curious if it isn't a bit of an overkill solution. Why not simply employ something like:

$ typeset -i example1="00001"
$ typeset -i example2="a0001"
$ typeset -i example3="00000"
$ echo $example1 + 10 |bc
11
$ echo $example2 + 10 |bc
10
$ echo $example3 + 10 |bc
10
$ print $example1
1
$ print $example2
0
$ print $example3
0
===

This approach is available to both ksh and bash, and would immediately strip out the zeroes or otherwise negate the value to "0" if there were some alpha values. Same applies to a fully zero-filled value. This eliminates the need for any additional function definitions at all...but is this a matter for portability? Sure it restricts the math to integers, but sed would totally skew the numbers if it was used to massage the value.

[Your page doesn't seem to restrict the discussion to a particular shell that I can see.]

Posted by: curleb at May 25, 2007 1:55 PM

Cool. Never knew about the "typeset" built-in to the shell. Very interesting solution, thanks!

Posted by: Dave Taylor at May 25, 2007 8:13 PM

The typeset -i (or declare -i in more recent versions of bash) doesn't work if the value is greater than 7 -- it's interpreted as an octal number.

$ declare -i x=009
bash: declare: 009: value too great for base (error token is 009)
$ declare -i x=010
$ echo $x
8

The most reliable solution so far is using expr + 0 I think.

Posted by: Alexandre H at October 3, 2007 9:02 AM

I have a lot to say, but ...
Starbucks coffee cup I have a lot to say, and questions of my own for that matter, but most of all I'd like to say thank you for all your efforts on this Web site by buying you a chai!

I do have a comment, now that you mention it!









Remember personal info?


Please note that I will never send you any unsolicited commercial email. Ever.

While I'm at it, please note that by submitting a question or comment you're agreeing to my terms of service, which are: you relinquish any subsequent rights of ownership to your material by submitting it on this site.









Search
Find just the answers you seek from among our 1700+ free tech support articles by using our Lijit search engine.


Help!





Subscribe to
Ask Dave Taylor!

Add to Google Reader
Add to My Yahoo!
Subscribe in NewsGator Online

RDF   XML

Free Updates!
Sign up and get free weekly updates and special offers on books, seminars, workshops and more.


Recent Entries
Join the List!
Join my author info mailing list, where you'll learn about my upcoming books, speaking gigs, and more!


Book Links
© 2002 - 2008 by Dave Taylor. All Rights Reserved.

Note: This web site is for the purpose of disseminating information for educational purposes, free of charge, for the benefit of all visitors. We take great care to provide quality information. However, we do not guarantee, and accept no legal liability whatsoever arising from or connected to, the accuracy, reliability, currency or completeness of any material contained on this web site or on any linked site.

[whiteboard marker tray]