Industry guru Dave Taylor offers tech support on technical and business topics, including iPhone, iPod, Microsoft Windows, Sony PSP, cellphones, online advertising, CSS, Web design, business, Unix, Linux, SEO, Mac OS X, and shell script programming.     


How do I strip leading zeroes for math in a shell script?

I have a file containing lines of data that are amounts padded with leading zeros, similar to the snip below.

  0000000004
  0000000016
  0000000012
  0000000008

Using a shell script, how can I add up a column of numbers contained in lines when BASH interprets the numbers as octal when I do in-line math?


Dave's Answer:

What an interesting puzzle you present to me! My first thought was that there's a cool way I can solve this using a shell function, one that stripped a single leading zero then compared its results to the pre-truncated version of the value, until they matched (e.g., all the leading zeroes were deleted).

It'd look something like this:

function stripzeroes
{
  myvalue=$1
  newvalue="0"
  while [ "$newvalue" != "$lastvalue" ]
    do
    newvalue="$(echo $myvalue | sed 's/^0//')"
    lastvalue=$myvalue
    myvalue=$newvalue
  done
  # return value is the global variable 'newvalue'
}

This would then be called as stripzeroes "0000000003434" (or whatever value you'd read in from the data file) and the result would be returned as the value newvalue without the leading zeroes.

Nice solution, classic little shell script function, but there's one problem.

With the right regular expression, you can strip all the leading zeroes from your data fields with just a few characters:

valueWithoutZeroes="$(echo $valueWithZeroes | sed 's/0*//')"

That's right, but using the proper regular expression to the sed command, you can very easily strip off all the leading zeroes, making your math far, far simpler and all without complicating your shell script with useless functions!

Hope that helps you out.


More Useful Shell Script Programming Articles:
✔   Secretly capture screenshots on my Mac?
When I used to work on a Linux system, there was a utility we had that would let me take screen captures every...
✔   Parsing "id" strings in a Shell Script?
Hello Dave. I need a Bash shell script that creates a directories with the group names automatically when user logs in to the...
✔   Copy and Paste from the Mac OS X Command Line?
I am constantly running commands in Terminal.app on my MacBook and then copying and pasting the results into email messages or documents. Yes,...
✔   Script to test line lengths for Twitter compatibility?
I've been tasked with writing a series of tweets for a Black Friday marketing campaign and am finding it a bit tricky because...
✔   Shell script to convert lowercase to title case?
As part of a project I'm working on, I find myself deep in a Linux shell script, needing to have a subroutine that...

Let's stay in touch!
Sign up for my weekly AskDaveTaylor Newsletter and you'll receive even more tech and gadget help right to your inbox, along with exclusive news and industry updates. It's good stuff. I promise!
    Enter your name: and your email addr:  





Categorized: Shell Script Programming   (Article 6487, Written by )
Tagged: shell script programming, unix programming, wicked cool shell scripts
Previous: How do I get a cool MySpace URL?
Next: How do I configure a DMZ on my local LAN?




Reader Comments To Date: 20

Dave Taylor said, on April 27, 2006 11:50 PM:

Not to be confused with this Dave Taylor, I'm the Dave Taylor from the game industry, and I'm visiting because my uncle Paul Taylor thought I might have been doing this as I share your penchant for answering things.

It's a neat site, and I think you're providing a darned handy service, btw.

I think there's a small bug with both solutions. You've got an issue if one of the lines reads thusly:

0000000000

You'd want to strip that down to just "0", but both algorithms you use basically delete the line, which could cause some pretty serious issues, depending on what's eating the output.

Dave Taylor said, on April 28, 2006 12:44 AM:

Darn, you're right, Dave! Using the 'sed' solution, simply add a test after the conditional:

if [ -z "$valueWithoutZeroes" ] ; then
valueWithoutZeroes="0"
fi

that should do the trick!

Alejandro Biasin said, on November 2, 2006 12:04 PM:

I figure out a much simpler way of doing it:

newValue=`expr $oldValue + 0`

Note that this only works with "data that are amounts", as the question says.

Garrett Nievin said, on January 7, 2007 12:59 PM:

Here's my usual solution, pretty much equivalent to Alejandro's use of expr:
newValue=`echo $oldValue | awk '{print $1 + 0}'`
or
newValue=`echo $oldValue | awk '{printf "%d\n", $1}'


Ben said, on April 26, 2007 8:58 AM:

I'm fairly new to this and I do follow the logic. However, I have problems trying to tie it all up together. For instance I have a file 'benshrs' with figures I need to add up in columns 21 thru 27. Some are all zeroes. What's the best way (complete script)that combines a. the cut -c 21-27 b. strip the leading zeroes except when it is all zeroes, c. add up the figures in do ... done loop, and lastly print the final sum.

My crude attempt (below) has failed miserably!
hrs_total=0.00
for line_num in `cut -c 21-27 $0`
do
line_hrs= `expr `sed 's/0*//' $line_num` + 0`
(hrs_total=$hrs_total + $line_hrs)
done
hrs_total=`echo "scale=2;$hrs_total/100" | bc`
echo "Hours Total = $hrs_total"

Dave Taylor said, on May 7, 2007 7:55 PM:

I'm not surprised this isn't working, Ben: you can't use floating point / real numbers in a shell script, so even your initial assignment of hrs_total=0.00 is going to fail before you even get into the for loop. It's possible that you need to do this in awk or, better, Perl, to get it working.

curleb said, on May 25, 2007 1:55 PM:

Not that your option wouldn't work, Dave, but I'm curious if it isn't a bit of an overkill solution. Why not simply employ something like:

$ typeset -i example1="00001"
$ typeset -i example2="a0001"
$ typeset -i example3="00000"
$ echo $example1 + 10 |bc
11
$ echo $example2 + 10 |bc
10
$ echo $example3 + 10 |bc
10
$ print $example1
1
$ print $example2
0
$ print $example3
0
===

This approach is available to both ksh and bash, and would immediately strip out the zeroes or otherwise negate the value to "0" if there were some alpha values. Same applies to a fully zero-filled value. This eliminates the need for any additional function definitions at all...but is this a matter for portability? Sure it restricts the math to integers, but sed would totally skew the numbers if it was used to massage the value.

[Your page doesn't seem to restrict the discussion to a particular shell that I can see.]

Dave Taylor said, on May 25, 2007 8:13 PM:

Cool. Never knew about the "typeset" built-in to the shell. Very interesting solution, thanks!

Alexandre H said, on October 3, 2007 9:02 AM:

The typeset -i (or declare -i in more recent versions of bash) doesn't work if the value is greater than 7 -- it's interpreted as an octal number.

$ declare -i x=009
bash: declare: 009: value too great for base (error token is 009)
$ declare -i x=010
$ echo $x
8

The most reliable solution so far is using expr + 0 I think.

Cephi said, on September 8, 2008 4:51 PM:

if the variables are all of the same length, then:

(example: variables with eight characters, var1=00005555)

var1=$((1$var1-100000000))

Now var1 is 5555. Easy peasy gumdrops.

jesse said, on December 5, 2008 2:20 PM:

valueWithoutZeroes=`echo ${valueWithZeroes#0}`

Erik De Sonville said, on February 15, 2009 4:14 AM:

To remove leading zeroes, while keeping the last one for an all-zero input string:

sed 's/^0*//;s/^$/0/'

Use it e.g. as
NEW=$(echo $INPUT | sed 's/^0*//;s/^$/0/')

As a side-effect (or rather as interesting advantage), an empty (null) input string is also converted into a single zero.

123456 -> 123456
000123 -> 123
000000 -> 0
(empty) -> 0

Ryan B. Lynch said, on June 25, 2009 10:58 AM:

Seems like your (Dave) solutions and the comments fall into one of two categories: Heavyweight solutions that rely on external tools (bc, awk, sed), or incomplete solutions (try Jesse's with multiple leading zeros).

In pure Bourne (/bin/sh), I don't actually have an answer. But BASH itself has enough functionality to handle this without *any* external programs:

`shopt -s extglob
echo -e "${my_var/#+(0)/}"
shopt -u extglob`

There are two advanced BASH tricks, here.

First, `shopt -s extglob` turns on BASH's "extended globbing" functionality ('-u' turns it off). In addition to the usual BASH globbing of wildcards, you can specify '+(0)' to indicate "a sequence one or more '0' characters". (More info here: http://aplawrence.com/Words2005/2005_05_25.html)

Second, the '${my_var/#+(0)/}' construction, which applies BASH's string manipulation operators. '${var/seq/rep}' takes the value of the variable '$var' and replaces any instances of the literal string 'seq' with the literal string 'rep'. I added that '#' character, which restricts the match-replacement to the beginning of the string, only.

I get the impression that almost nobody knows about and uses BASH's string manipulation and extended globbing functionality. They're not intuitive to me, so I generally check the docs when using them.

Still, I'd prefer this to 'echo'ing the original string to 'sed', and using a regex substition. (Coincidentally, that's how I'd handle this, if I couldn't use the BASH techniques I've outlined, here.)

Peter said, on January 11, 2010 6:52 AM:

I've just written my own SED command for this, which is slightly purer than Erik's one in that it will leave an empty string as is:

sed 's/^0\+\([0-9]\)/\1/'

The trick here is to check that there is at least one digit after the zeros, and put that back into the output. (The bracketed expression becomes \1 in the replace pattern.)

Antriksh Pany said, on March 9, 2010 10:24 AM:

Interesting set of comments.. fairly wide array of solutions.

This is what I tried and got to work:
echo $(for i in `cat a`; do echo -n $i | sed 's/^0*\([0-9]\)/\1/g'; echo -n " + "; done; echo "0") | bc

['a' is the file with the numbers.]

However, I don't know why I had to do the echo of the entire for loop output. I would have thought that the following would also work:
for i in `cat a`; do echo -n $i | sed 's/^0*\([0-9]\)/\1/g'; echo -n " + "; done; echo "0" | bc

Any suggestions?

PowerPaul said, on August 25, 2010 4:57 AM:

Instead of manipulating the actual number by stripping the leading zeros you can force bash to use decimal representation by using:

$ echo $[000000000200-1]
127
$ echo $[10#000000000200-1]
199
echo $[10#000000000200]
200

jice said, on November 14, 2010 12:08 PM:

indeed I'm using
myvalue=$((10#$myvalue))
(see previous comment for explanations)
must be much faster than calling an external program (e.g. sed)

Raj Singh said, on December 20, 2011 4:03 AM:

The answer is rather a single line of code.

For e.g. echo "1000000001111111" | tr -s [0-9]

would squeez all the repeating numbers in the string.

Fred said, on January 3, 2012 1:47 PM:

(( var1 = $var + 0 ))

just another way to do it

Zoran said, on March 2, 2012 1:24 AM:

I have used this way:
MYVAR=`echo ${MYVAR}|sed 's/^0*//'` # stip leading zeroes so it can be assigned to integer variable
declare -i IMYVAR=`expr $MYVAR + 0` # integer variable get the value so it can be incremented
(( IMYVAR = ${MYVAR} + 1 )) # do the arithmetic operations needed

Fred's and RAj Singh's way is more elegant although :)

Starbucks coffee cup I do have a lot to say, and questions of my own for that matter, but first I'd like to say thank you, Dave, for all your helpful information by buying you a cup of coffee!

I do have a comment, now that you mention it!











I will never send you any unsolicited email. Ever.






Check This Out Too...

 
Look for Answers
Need Help? Ask Dave Taylor!


Follow Me on Pinterest

Find Me on Google+
ADT on G+
© 2002 - 2013 by Dave Taylor. All Rights Reserved.

Note: This web site is for the purpose of disseminating information for educational purposes, free of charge, for the benefit of all visitors. We take great care to provide quality information. However, we do not guarantee, and accept no legal liability whatsoever arising from or connected to, the accuracy, reliability, currency or completeness of any material contained on this web site or on any linked site. Further, please note that by submitting a question or comment you're agreeing to my terms of service, which are: you relinquish any subsequent rights of ownership to your material by submitting it on this site. My lawyer says "Thanks".
"Ask Dave Taylor®" is a registered trademark of Intuitive Systems, LLC.