linux - Running sed in parallel -
i naively ventured use following command process data file:
cat old.one | parallel --pipe 'sed -r "s/\./\,/g"' > new.one
the goal replace "." ",". resulting file differs obtained sequential treatment:
sed -r "s/\./\,/g" old.one > new.one
maybe parallel work can done somehow differently? here great without semaphores, , combine parts @ end.
solution
thanks lot! here results:
sed: 13.834 s
sed -r "s/./\,/g" old.one > new.one
parallel sed: 12.489 s
cat old.one | parallel -k --pipe 'sed -r "s/./\,/g"' > new.one
tr: 6.480 s
cat old.one | tr "." "," > new.one
parallel tr: 5.848 s
cat new.one | parallel -k --pipe tr "." "," > old.one
if works correctly (-j1):
cat old.one | parallel -j1 --pipe 'sed -r "s/\./\,/g"' > new.one
then should work (-k):
cat old.one | parallel -k --pipe 'sed -r "s/\./\,/g"' > new.one
--pipe slow, if speed of essence, use --pipe-part instead decent block size:
parallel -a old.one -k --block 30m --pipe-part 'sed -r "s/\./\,/g"' > new.one
Comments
Post a Comment