I have a file containing lines of data that are amounts padded with leading zeros, similar to the snip below.
0000000004
0000000016
0000000012
0000000008
Using a shell script, how can I add up a column of numbers contained in lines when BASH interprets the numbers as octal when I do in-line math?
What an interesting puzzle you present to me! My first thought was that there’s a cool way I can solve this using a shell function, one that stripped a single leading zero then compared its results to the pre-truncated version of the value, until they matched (e.g., all the leading zeroes were deleted).
It’d look something like this:
function stripzeroes
{
myvalue=$1
newvalue="0"
while [ "$newvalue" != "$lastvalue" ]
do
newvalue="$(echo $myvalue | sed 's/^0//')"
lastvalue=$myvalue
myvalue=$newvalue
done
# return value is the global variable 'newvalue'
}
This would then be called as stripzeroes “0000000003434” (or whatever value you’d read in from the data file) and the result would be returned as the value newvalue without the leading zeroes.
Nice solution, classic little shell script function, but there’s one problem.
With the right regular expression, you can strip all the leading zeroes from your data fields with just a few characters:
valueWithoutZeroes="$(echo $valueWithZeroes | sed 's/0*//')"
That’s right, but using the proper regular expression to the sed command, you can very easily strip off all the leading zeroes, making your math far, far simpler and all without complicating your shell script with useless functions!
Hope that helps you out.
I have used this way:
MYVAR=`echo ${MYVAR}|sed ‘s/^0*//’` # stip leading zeroes so it can be assigned to integer variable
declare -i IMYVAR=`expr $MYVAR + 0` # integer variable get the value so it can be incremented
(( IMYVAR = ${MYVAR} + 1 )) # do the arithmetic operations needed
Fred’s and RAj Singh’s way is more elegant although 🙂
(( var1 = $var + 0 ))
just another way to do it
The answer is rather a single line of code.
For e.g. echo “1000000001111111” | tr -s [0-9]
would squeez all the repeating numbers in the string.
indeed I’m using
myvalue=$((10#$myvalue))
(see previous comment for explanations)
must be much faster than calling an external program (e.g. sed)
Instead of manipulating the actual number by stripping the leading zeros you can force bash to use decimal representation by using:
$ echo $[000000000200-1]
127
$ echo $[10#000000000200-1]
199
echo $[10#000000000200]
200
Interesting set of comments.. fairly wide array of solutions.
This is what I tried and got to work:
echo $(for i in `cat a`; do echo -n $i | sed ‘s/^0*\([0-9]\)/\1/g’; echo -n ” + “; done; echo “0”) | bc
[‘a’ is the file with the numbers.]
However, I don’t know why I had to do the echo of the entire for loop output. I would have thought that the following would also work:
for i in `cat a`; do echo -n $i | sed ‘s/^0*\([0-9]\)/\1/g’; echo -n ” + “; done; echo “0” | bc
Any suggestions?
I’ve just written my own SED command for this, which is slightly purer than Erik’s one in that it will leave an empty string as is:
sed ‘s/^0\+\([0-9]\)/\1/’
The trick here is to check that there is at least one digit after the zeros, and put that back into the output. (The bracketed expression becomes \1 in the replace pattern.)
Seems like your (Dave) solutions and the comments fall into one of two categories: Heavyweight solutions that rely on external tools (bc, awk, sed), or incomplete solutions (try Jesse’s with multiple leading zeros).
In pure Bourne (/bin/sh), I don’t actually have an answer. But BASH itself has enough functionality to handle this without *any* external programs:
`shopt -s extglob
echo -e “${my_var/#+(0)/}”
shopt -u extglob`
There are two advanced BASH tricks, here.
First, `shopt -s extglob` turns on BASH’s “extended globbing” functionality (‘-u’ turns it off). In addition to the usual BASH globbing of wildcards, you can specify ‘+(0)’ to indicate “a sequence one or more ‘0’ characters”. (More info here: http://aplawrence.com/Words2005/2005_05_25.html)
Second, the ‘${my_var/#+(0)/}’ construction, which applies BASH’s string manipulation operators. ‘${var/seq/rep}’ takes the value of the variable ‘$var’ and replaces any instances of the literal string ‘seq’ with the literal string ‘rep’. I added that ‘#’ character, which restricts the match-replacement to the beginning of the string, only.
I get the impression that almost nobody knows about and uses BASH’s string manipulation and extended globbing functionality. They’re not intuitive to me, so I generally check the docs when using them.
Still, I’d prefer this to ‘echo’ing the original string to ‘sed’, and using a regex substition. (Coincidentally, that’s how I’d handle this, if I couldn’t use the BASH techniques I’ve outlined, here.)
To remove leading zeroes, while keeping the last one for an all-zero input string:
sed ‘s/^0*//;s/^$/0/’
Use it e.g. as
NEW=$(echo $INPUT | sed ‘s/^0*//;s/^$/0/’)
As a side-effect (or rather as interesting advantage), an empty (null) input string is also converted into a single zero.
123456 -> 123456
000123 -> 123
000000 -> 0
(empty) -> 0
valueWithoutZeroes=`echo ${valueWithZeroes#0}`
if the variables are all of the same length, then:
(example: variables with eight characters, var1=00005555)
var1=$((1$var1-100000000))
Now var1 is 5555. Easy peasy gumdrops.
The typeset -i (or declare -i in more recent versions of bash) doesn’t work if the value is greater than 7 — it’s interpreted as an octal number.
$ declare -i x=009
bash: declare: 009: value too great for base (error token is 009)
$ declare -i x=010
$ echo $x
8
The most reliable solution so far is using expr + 0 I think.
Cool. Never knew about the “typeset” built-in to the shell. Very interesting solution, thanks!
Not that your option wouldn’t work, Dave, but I’m curious if it isn’t a bit of an overkill solution. Why not simply employ something like:
$ typeset -i example1=”00001″
$ typeset -i example2=”a0001″
$ typeset -i example3=”00000″
$ echo $example1 + 10 |bc
11
$ echo $example2 + 10 |bc
10
$ echo $example3 + 10 |bc
10
$ print $example1
1
$ print $example2
0$ print $example3
0===
This approach is available to both ksh and bash, and would immediately strip out the zeroes or otherwise negate the value to “0” if there were some alpha values. Same applies to a fully zero-filled value. This eliminates the need for any additional function definitions at all…but is this a matter for portability? Sure it restricts the math to integers, but sed would totally skew the numbers if it was used to massage the value.
[Your page doesn’t seem to restrict the discussion to a particular shell that I can see.]
I’m not surprised this isn’t working, Ben: you can’t use floating point / real numbers in a shell script, so even your initial assignment of hrs_total=0.00 is going to fail before you even get into the for loop. It’s possible that you need to do this in awk or, better, Perl, to get it working.
I’m fairly new to this and I do follow the logic. However, I have problems trying to tie it all up together. For instance I have a file ‘benshrs’ with figures I need to add up in columns 21 thru 27. Some are all zeroes. What’s the best way (complete script)that combines a. the cut -c 21-27 b. strip the leading zeroes except when it is all zeroes, c. add up the figures in do … done loop, and lastly print the final sum.
My crude attempt (below) has failed miserably!
hrs_total=0.00
for line_num in `cut -c 21-27 $0`
do
line_hrs= `expr `sed ‘s/0*//’ $line_num` + 0`
(hrs_total=$hrs_total + $line_hrs)
done
hrs_total=`echo “scale=2;$hrs_total/100” | bc`
echo “Hours Total = $hrs_total”
Here’s my usual solution, pretty much equivalent to Alejandro’s use of expr:
newValue=`echo $oldValue | awk ‘{print $1 + 0}’`
or
newValue=`echo $oldValue | awk ‘{printf “%d\n”, $1}’
I figure out a much simpler way of doing it:
newValue=`expr $oldValue + 0`
Note that this only works with “data that are amounts”, as the question says.
Darn, you’re right, Dave! Using the ‘sed’ solution, simply add a test after the conditional:
if [ -z “$valueWithoutZeroes” ] ; then
valueWithoutZeroes=”0″
fi
that should do the trick!
Not to be confused with this Dave Taylor, I’m the Dave Taylor from the game industry, and I’m visiting because my uncle Paul Taylor thought I might have been doing this as I share your penchant for answering things.
It’s a neat site, and I think you’re providing a darned handy service, btw.
I think there’s a small bug with both solutions. You’ve got an issue if one of the lines reads thusly:
0000000000
You’d want to strip that down to just “0”, but both algorithms you use basically delete the line, which could cause some pretty serious issues, depending on what’s eating the output.