text-processing

reference:
10 Awk Tips, Tricks and Pitfalls
FIND -EXEC VS. FIND | XARGS

charset

[!NOTE|label:references:]
ASCII Table, ISO 1252 Latin-1 Chart & Character Set
ASCII Character Chart with Decimal, Binary and Hexadecimal Conversions
The difference between "binary" and "text" files
control character

HEX

NAME

ABBREVIATION

ESCAPE

CODE

0x00

null

NUL

\0

^@

0x07

bell

BEL

\a

^G

0x08

backspace

\b

^H

0x09

horizontal tab

^I

0x0a

line feed

^J

0x0b

vertical tab

\v

^K

0x0c

form feed

\f

^L

0x0d

carriage return

^M

0x1a

Control-Z

SUB

^Z

0x1b

escape

ESC

\e

^[

list all charset
```
$ iconv --list
```

convert text-file from utf-8 to iso-8859-1

$ iconv --from-code=UTF-8 --to-code=ISO-8859-1 inputfile.txt > outputfile.txt

escape

[!NOTE|label:references:]
Difference between single and double quotes in Bash
How to escape single quotes within single quoted strings

ESCAPING SEQUENCES

COMMENTS

\'

single quote

\"

double quote

\\

backslash

new line

horizontal tab

carriage return

\?

question mark

ASCII Printable characters
String interpolation

single quota & double quotas
[!TIP|label:references:]
- 3.1.2.4 ANSI-C Quoting
- 3.1.2.5 Locale-Specific Translation Prefixing a double-quoted string with a dollar sign ($), such as $"hello, world", will cause the string to be translated according to the current locale
```
$ echo $'aa\'bb'
aa'bb

# ascii code
#            hex  octal  hex   octal
$ echo -e  "\x27  \047  \\x22  \042"
'  '  "  "
$ echo -e "Let\x27s get coding!"
Let's get coding!
$ echo -e "Let\x22s get coding!"
Let"s get coding!
```

encryption

`base64`

$ echo "marslo" | base64 -w0
bWFyc2xvCg==

decryption

$ echo "bWFyc2xvCg==" | base64 --decode
marslo

show

align

[!NOTE|label:see also]
iMarslo: padRight
iMarslo: column
iMarslo: pretty auth can-i
printTable | gdbtek/linux-cookbooks/libraries/util.bash

# right-align
$ printf  _"%10s"_ "foobar"
_    foobar_

# left-align
$ printf  _"%-10s"_ "foobar"
_foobar    _

numfmt

[!NOTE|label:references:]
* iMarslo : math
* numfmt - A number reformatting utility
numfmt(1) — Linux manual page
numfmt - Convert numbers from/to human-readable strings
The Numfmt Command Tutorial With Examples For Beginners

setup

# generic
$ npm install numfmt

# osx
$ brew install coreutils

$ brew list coreutils | grep bin
/usr/local/Cellar/coreutils/9.4/bin/md5sum
/usr/local/Cellar/coreutils/9.4/bin/gsha512sum
/usr/local/Cellar/coreutils/9.4/bin/gusers
/usr/local/Cellar/coreutils/9.4/bin/gprintenv
/usr/local/Cellar/coreutils/9.4/bin/gmknod
/usr/local/Cellar/coreutils/9.4/bin/shuf
/usr/local/Cellar/coreutils/9.4/bin/gdd
/usr/local/Cellar/coreutils/9.4/bin/gtsort
/usr/local/Cellar/coreutils/9.4/bin/grealpath
/usr/local/Cellar/coreutils/9.4/bin/grmdir
/usr/local/Cellar/coreutils/9.4/bin/gfold
/usr/local/Cellar/coreutils/9.4/bin/gnl
/usr/local/Cellar/coreutils/9.4/bin/greadlink
/usr/local/Cellar/coreutils/9.4/bin/gshred
/usr/local/Cellar/coreutils/9.4/bin/gmv
/usr/local/Cellar/coreutils/9.4/bin/runcon
/usr/local/Cellar/coreutils/9.4/bin/gmkdir
/usr/local/Cellar/coreutils/9.4/bin/gkill
/usr/local/Cellar/coreutils/9.4/bin/guniq
/usr/local/Cellar/coreutils/9.4/bin/gpr
/usr/local/Cellar/coreutils/9.4/bin/ptx
/usr/local/Cellar/coreutils/9.4/bin/ghead
/usr/local/Cellar/coreutils/9.4/bin/glink
/usr/local/Cellar/coreutils/9.4/bin/gstat
/usr/local/Cellar/coreutils/9.4/bin/gmktemp
/usr/local/Cellar/coreutils/9.4/bin/gyes
/usr/local/Cellar/coreutils/9.4/bin/gsha1sum
/usr/local/Cellar/coreutils/9.4/bin/b2sum
/usr/local/Cellar/coreutils/9.4/bin/grm
/usr/local/Cellar/coreutils/9.4/bin/gsha256sum
/usr/local/Cellar/coreutils/9.4/bin/gfalse
/usr/local/Cellar/coreutils/9.4/bin/gwho
/usr/local/Cellar/coreutils/9.4/bin/gcut
/usr/local/Cellar/coreutils/9.4/bin/gvdir
/usr/local/Cellar/coreutils/9.4/bin/gdir
/usr/local/Cellar/coreutils/9.4/bin/gchmod
/usr/local/Cellar/coreutils/9.4/bin/gbase32
/usr/local/Cellar/coreutils/9.4/bin/sha224sum
/usr/local/Cellar/coreutils/9.4/bin/ghostid
/usr/local/Cellar/coreutils/9.4/bin/gnohup
/usr/local/Cellar/coreutils/9.4/bin/gtr
/usr/local/Cellar/coreutils/9.4/bin/gdirname
/usr/local/Cellar/coreutils/9.4/bin/gsha384sum
/usr/local/Cellar/coreutils/9.4/bin/gchroot
/usr/local/Cellar/coreutils/9.4/bin/gpaste
/usr/local/Cellar/coreutils/9.4/bin/timeout
/usr/local/Cellar/coreutils/9.4/bin/tac
/usr/local/Cellar/coreutils/9.4/bin/numfmt
/usr/local/Cellar/coreutils/9.4/bin/gid
/usr/local/Cellar/coreutils/9.4/bin/gpinky
/usr/local/Cellar/coreutils/9.4/bin/genv
/usr/local/Cellar/coreutils/9.4/bin/basenc
/usr/local/Cellar/coreutils/9.4/bin/nproc
/usr/local/Cellar/coreutils/9.4/bin/gln
/usr/local/Cellar/coreutils/9.4/bin/gbasename
/usr/local/Cellar/coreutils/9.4/bin/gtruncate
/usr/local/Cellar/coreutils/9.4/bin/stdbuf
/usr/local/Cellar/coreutils/9.4/bin/chcon
/usr/local/Cellar/coreutils/9.4/bin/gcp
/usr/local/Cellar/coreutils/9.4/bin/gls
/usr/local/Cellar/coreutils/9.4/bin/factor
/usr/local/Cellar/coreutils/9.4/bin/gtrue
/usr/local/Cellar/coreutils/9.4/bin/gchown
/usr/local/Cellar/coreutils/9.4/bin/gsync
/usr/local/Cellar/coreutils/9.4/bin/guptime
/usr/local/Cellar/coreutils/9.4/bin/gsum
/usr/local/Cellar/coreutils/9.4/bin/gtac
/usr/local/Cellar/coreutils/9.4/bin/gexpand
/usr/local/Cellar/coreutils/9.4/bin/gruncon
/usr/local/Cellar/coreutils/9.4/bin/gpathchk
/usr/local/Cellar/coreutils/9.4/bin/gnice
/usr/local/Cellar/coreutils/9.4/bin/gecho
/usr/local/Cellar/coreutils/9.4/bin/gdu
/usr/local/Cellar/coreutils/9.4/bin/gb2sum
/usr/local/Cellar/coreutils/9.4/bin/gtouch
/usr/local/Cellar/coreutils/9.4/bin/gmkfifo
/usr/local/Cellar/coreutils/9.4/bin/gdf
/usr/local/Cellar/coreutils/9.4/bin/gjoin
/usr/local/Cellar/coreutils/9.4/bin/gtest
/usr/local/Cellar/coreutils/9.4/bin/gmd5sum
/usr/local/Cellar/coreutils/9.4/bin/gunexpand
/usr/local/Cellar/coreutils/9.4/bin/gsort
/usr/local/Cellar/coreutils/9.4/bin/gshuf
/usr/local/Cellar/coreutils/9.4/bin/gfmt
/usr/local/Cellar/coreutils/9.4/bin/gunlink
/usr/local/Cellar/coreutils/9.4/bin/gcsplit
/usr/local/Cellar/coreutils/9.4/bin/g[
/usr/local/Cellar/coreutils/9.4/bin/gwhoami
/usr/local/Cellar/coreutils/9.4/bin/gsplit
/usr/local/Cellar/coreutils/9.4/bin/gseq
/usr/local/Cellar/coreutils/9.4/bin/sha1sum
/usr/local/Cellar/coreutils/9.4/bin/sha256sum
/usr/local/Cellar/coreutils/9.4/bin/gdircolors
/usr/local/Cellar/coreutils/9.4/bin/ginstall
/usr/local/Cellar/coreutils/9.4/bin/gsha224sum
/usr/local/Cellar/coreutils/9.4/bin/shred
/usr/local/Cellar/coreutils/9.4/bin/sha384sum
/usr/local/Cellar/coreutils/9.4/bin/gcomm
/usr/local/Cellar/coreutils/9.4/bin/gtty
/usr/local/Cellar/coreutils/9.4/bin/gcksum
/usr/local/Cellar/coreutils/9.4/bin/gexpr
/usr/local/Cellar/coreutils/9.4/bin/gbase64
/usr/local/Cellar/coreutils/9.4/bin/gwc
/usr/local/Cellar/coreutils/9.4/bin/gnproc
/usr/local/Cellar/coreutils/9.4/bin/base32
/usr/local/Cellar/coreutils/9.4/bin/gptx
/usr/local/Cellar/coreutils/9.4/bin/gtimeout
/usr/local/Cellar/coreutils/9.4/bin/pinky
/usr/local/Cellar/coreutils/9.4/bin/hostid
/usr/local/Cellar/coreutils/9.4/bin/gpwd
/usr/local/Cellar/coreutils/9.4/bin/gtail
/usr/local/Cellar/coreutils/9.4/bin/gchcon
/usr/local/Cellar/coreutils/9.4/bin/glogname
/usr/local/Cellar/coreutils/9.4/bin/guname
/usr/local/Cellar/coreutils/9.4/bin/gtee
/usr/local/Cellar/coreutils/9.4/bin/gstty
/usr/local/Cellar/coreutils/9.4/bin/gchgrp
/usr/local/Cellar/coreutils/9.4/bin/gcat
/usr/local/Cellar/coreutils/9.4/bin/ggroups
/usr/local/Cellar/coreutils/9.4/bin/gsleep
/usr/local/Cellar/coreutils/9.4/bin/sha512sum
/usr/local/Cellar/coreutils/9.4/bin/gfactor
/usr/local/Cellar/coreutils/9.4/bin/god
/usr/local/Cellar/coreutils/9.4/bin/gprintf
/usr/local/Cellar/coreutils/9.4/bin/gstdbuf
/usr/local/Cellar/coreutils/9.4/bin/gnumfmt
/usr/local/Cellar/coreutils/9.4/bin/gbasenc
/usr/local/Cellar/coreutils/9.4/bin/gdate
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/tee
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/md5sum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/split
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/cat
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/shuf
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/mkfifo
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/pathchk
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/runcon
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/expand
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/tty
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/basename
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/install
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/nice
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/truncate
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/echo
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/du
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/ptx
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/join
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/df
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/pwd
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/test
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/csplit
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sort
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/whoami
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/touch
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/unlink
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/b2sum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sleep
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/fmt
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/stty
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/logname
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/chgrp
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/printenv
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/seq
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/uname
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha224sum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/od
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/date
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/base64
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/realpath
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/readlink
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/dircolors
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/timeout
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/tac
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/numfmt
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/wc
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/basenc
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/comm
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/nproc
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/expr
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/stdbuf
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/cksum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/printf
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/groups
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/chcon
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/factor
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/tail
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/env
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/pr
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/head
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/kill
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/uniq
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/stat
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/link
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/tsort
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/mknod
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/users
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/dd
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/who
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha1sum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/mktemp
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/cut
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha256sum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/dir
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/mkdir
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/nl
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/shred
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/fold
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/rmdir
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha384sum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/mv
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/dirname
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/id
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/base32
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/pinky
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/ln
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/hostid
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/chroot
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/ls
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/true
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/cp
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sync
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/yes
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/unexpand
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/chown
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/chmod
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/uptime
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/rm
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/vdir
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/false
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha512sum
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/[
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/tr
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/paste
/usr/local/Cellar/coreutils/9.4/libexec/gnubin/nohup

usage

$ bc -l <<< 'obase=2;0;0;15;255'
0
0
1111
11111111

$ bc -l <<< 'obase=2;0;0;15;255' | numfmt --format=%08f
00000000
00000000
00001111
11111111

$ bc -l <<< 'obase=2;0;0;15;255' | numfmt --format=%08f | xargs
00000000 00000000 00001111 11111111

$ numfmt --to=si --format "%f bottles of beer on the wall"  99999999
100M bottles of beer on the wall

convert format

$ echo 1G | numfmt --from=si
1000000000
$ echo 1G | numfmt --from=iec
1073741824

$ echo 500G | numfmt --from=si --to=iec
466G

$ numfmt --field=2 --from-unit=1024 --to=iec-i --suffix B < /proc/meminfo  | sed 's/ kB//' | head -n4
MemTotal:          1008GiB
MemFree:           816GiB
MemAvailable:      941GiB
Buffers:          3.1MiB

$ watch -n.1 \
>  'numfmt --header --field=2 --to=iec-i --round=nearest < /proc/interrupts |
>   LC_ALL=en_US numfmt --header --field=3 --group --invalid=ignore --padding=16 |
>   pr -TW$COLUMNS'

$ for method in up down nearest; do
>   echo $method
>   numfmt --to=iec --round=$method 4095 4096 4097
> done | paste - - - -
up      4.0K  4.0K  4.1K
down    3.9K  4.0K  4.0K
nearest 4.0K  4.0K  4.0K

padding

$ du -s * | numfmt --to=si --padding=10
        12 awk.md
        40 character.md
         4 html.md
        16 json.md
         8 markdown.md
         4 regex.md
        16 sed.md

$ du -s * | numfmt --to=si --padding=-10
12         awk.md
40         character.md
4          html.md
16         json.md
8          markdown.md
4          regex.md
16         sed.md

field

$ ls -l
total 100
-rw-r--r-- 1 marslo staff  9720 Sep  7 00:37 awk.md
-rw-r--r-- 1 marslo staff 40500 Sep  7 00:51 character.md
$ ls -l | numfmt --field 5 --to=si
total 100
-rw-r--r-- 1 marslo staff  9.8K Sep  7 00:37 awk.md
-rw-r--r-- 1 marslo staff   41K Sep  7 00:51 character.md

$ df -B1 | head -3
Filesystem                                         1B-blocks          Used     Available Use% Mounted on
devtmpfs                                        540881096704             0  540881096704   0% /dev
tmpfs                                           540899667968        258048  540899409920   1% /dev/shm
$ df -B1 | head -3 | numfmt --header --field 2-4 --to=si
Filesystem                                         1B-blocks          Used     Available Use% Mounted on
devtmpfs                                                541G             0          541G   0% /dev
tmpfs                                                   541G          259K          541G   1% /dev/shm

`q`

[!NOTE|label:references:]
harelba/q
q - Run SQL directly on CSV or TSV files
GET CONTRIBUTOR STATS FROM GIT

$ git log --author="marslo" --format=tformat: --numstat | q -t "select sum(c1), sum(c2) from -"
60650.0   66363.0

combinations

single line to multiple lines

[!TIP]
$ echo 'a b c'
a b c

xargs -n<x>

$ echo 'a b c' | xargs -n1
a
b
c

$ echo {a..c}.{1..2} | xargs -n1 | xargs -I{} echo -{}-
-a.1-
-a.2-
-b.1-
-b.2-
-c.1-
-c.2-

fmt

$ echo 'a b c' | fmt -1
a
b
c

$ echo {a..c}.{1..2} | fmt -1 | xargs -I{} echo -{}-
-a.1-
-a.2-
-b.1-
-b.2-
-c.1-
-c.2-

awk

$ echo 'a b c' | awk '{ OFS=RS; $1=$1 }1'
a
b
c

tr
```
$ echo 'a b c' | tr -s ' ' '\n'
a
b
c
```
printf
```
$ printf '%s\n' a b c
a
b
c
```

execute commands from file

create files

[!TIP]
precondition
$ cat a.txt
a b c

$ echo 'a b c' | xargs -n1 -t touch
touch a
touch b
touch c

$ echo 'a b c' | xargs -n1 -p touch
touch a?...y
touch b?...y
touch c?...y

-t, --verbose
          Print the command line on the standard error output before executing it.
-p, --interactive
          Prompt  the  user  about  whether to run each command line and read a line from the terminal.
          Only run the command line if the response starts with `y' or `Y'.  Implies -t.
-I replace-str
          Replace occurrences of replace-str in the initial-arguments with names read from standard in-
          put.  Also, unquoted blanks do not terminate input items; instead the separator is  the  new-
          line character.  Implies -x and -L 1.

combine every 2 lines

[!NOTE|label:references:]
How to merge every two lines into one from the command line?
Automatic documentation of gitconfig aliases using sed or awk
sample output
$ echo -e "1\na\n2\nb\n3\nc
1
a
2
b
3
c
also using for sed output :
$ git --no-pager log -3 --no-color | sed -nr 's!^commit\s*(.+)$!\1!p; s!^\s*Change-Id:\s*(.*$)!\1!p'
d9a9adfb591bb129d6b1af9532fea0fcf069b176
I5d99e9ccd4edbba608e7da70e575fd6bd091ce42
7aed870eeb203336e5e29b03714905f28ec3e60d
Ie3b4d5fd09a43385a44282a2e6961e220ae6293a
09a652f7c78f416da7a561451ed274f930c27dec
I244d2fec5d25e453fd30d08d1c75c16143b7f7a3

xargs

$ echo -e "1\na\n2\nb\n3\nc" | xargs -n2 -d'\n'
1 a
2 b
3 c

paste

$ echo -e "1\na\n2\nb\n3\nc" | paste -s -d',\n'
1,a
2,b
3,c

$ echo -e "1\na\n2\nb\n3\nc" | paste -d " "  - -
1 a
2 b
3 c

sed

$ echo -e "1\na\n2\nb\n3\nc" | sed 'N;s/\n/ : /'
1 : a
2 : b
3 : c

awk

$ echo -e "1\na\n2\nb\n3\nc" | awk '{ key=$0; getline; print key " : " $0; }'
1 : a
2 : b
3 : c

# or
$ echo -e "1\na\n2\nb\n3\nc" | awk 'ORS=NR%2?FS:RS'
1 a
2 b
3 c

# or
$ echo -e "1\na\n2\nb\n3\nc" | awk 'NR%2{ printf "%s : ",$0;next; }1'
1 : a
2 : b
3 : c

# or
$ echo -e "1\na\n2\nb\n3\nc" | awk '{
                                      if ( NR%2 != 0 ) line=$0; else { printf("%s : %s\n", line, $0); line=""; }
                                    } END {
                                      if ( length(line) ) print line;
                                    }'
1 : a
2 : b
3 : c

while

$ echo -e "1\na\n2\nb\n3\nc" | while read line1; do read line2; echo "$line1 : $line2"; done
1 : a
2 : b
3 : c

combine every 3 lines

paste

# or every 3 lines
$ echo -e "1\na\n2\nb\n3\nc" | paste -d ' '  - - -
1 a 2
b 3 c

awk

$ echo -e "1\na\n2\nb\n3\nc" | awk 'NR%3{ printf "%s : ",$0;next; }1'
1 : a : 2
b : 3 : c

xargs

$ echo {1..9} | fmt -1 | xargs -n3
1 2 3
4 5 6
7 8 9

format output

[!TIP|label:sample data]
$ paste <(sort a.txt) <(sort b.txt) | expand --tabs=10
a         a
b         b
d         c
f         d
          e
$ pr -w 30 -m -t a.txt b.txt
a        a
b        b
d        c
f        d
         e
display two files side by side by pr
display two files side by side by paste
Displaying Files Side by Side in Linux
Print two files in two columns side-by-side
11.4. Side-by-Side diffs: sdiff

merge files line by line
```
$ paste -d ',:' file1 file2 file3
```

echo

[!TIP]
echo -e didn't print newlines and contiguous whitespaces

$ echo -e "a\t\tb"
a      b
$ echo -e $(echo -e "a\t\tb")
a b

print file with ansicolor

$ command cat c.txt
get  get/exec  get/subresource  list  create  create/exec  get/subresource
\e[32;1myes\e[0m  \e[32;1myes\e[0m       \e[32;1myes\e[0m              \e[32;1myes\e[0m   \e[32;1myes\e[0m     no           no

$ echo -ne $(command cat c.txt | sed  's/$/\\n/' | sed 's/ /\\a /g')
get  get/exec  get/subresource  list  create  create/exec  get/subresource
 yes  yes       yes              yes   yes     no           no

$ echo -ne $(command cat c.txt | sed  's/$/\\n/' | sed 's/ /\\033 /g')
get  get/exec  get/subresource  list  create  create/exec  get/subresource
 yes  yes       yes              yes   yes     no           no

diff

[!NOTE|label:references:]
Linux diff command
* GNU Diffutils
* Comparing and Merging Files
13.1 Options to diff
Diff and "--GTYPE-group-format"
diff - output line-numbers
diffstat

show all status

$ diff --side-by-side <(sort a.txt) <(sort b.txt)
a               a
b               b
                > c
d               d
f               | e

show diff only

$ diff --suppress-common-lines --side-by-side <(sort a.txt) <(sort b.txt)
                    > c
f                   | e

show diff with --<GTYPE>-group-format

VARIABLE

APPLICABLE

old

GTYPE, LTYPE

new

GTYPE, LTYPE

unchanged

GTYPE, LTYPE

changed

GTYPE

[!NOTE|label:tips:]
line format:
[G]TYPE : [g]roup : --<GTYPE>-group-format
group format
[L]TYPE : [l]ine : --<LTYPE>-line-format

$ diff --old-group-format="L %<" --new-group-format="R %>" --unchanged-group-format=""  a.txt b.txt
R c
L f
R e

# with line number
$ diff --unchanged-line-format="" --old-line-format="< %dn: %L" --new-line-format="> %dn: %L" <(sort a.txt) <(sort b.txt)
> 3: c
< 4: f
> 5: e

# beging-end
$ diff --old-group-format='\begin{em}
  -> %<\end{em}
  -> ' --new-group-format='\begin{bf}
  -> %>\end{bf}
  -> ' --changed-group-format='\begin{em}
  -> %<\end{em}
  -> \begin{bf}
  -> %>\end{bf}
  -> ' --unchanged-group-format='%=' \
  -> <(sort a.txt) <(sort b.txt)
a
b
\begin{bf}
c
\end{bf}
d
\begin{em}
f
\end{em}
\begin{bf}
e
\end{bf}

$ diff \
  ->    --unchanged-group-format='' \
  ->    --old-group-format='-------- %dn line%(n=1?:s) deleted at %df: %<' \
  ->    --new-group-format='-------- %dN line%(N=1?:s) added after %de: %>' \
  ->    --changed-group-format='-------- %dn line%(n=1?:s) changed at %df: %<-------- to: %>' \
  -> <(sort a.txt) <(sort b.txt)
-------- 1 line added after 2:
c
-------- 1 line changed at 4:
f
-------- to:
e

show common

$ diff --unchanged-line-format="%L" --new-line-format="" --old-line-format="" <(sort a.txt) <(sort b.txt)
a
b
d

create patch

[!NOTE|label:references:]
Is this a good way to create a patch?
$ diff -Naru file_original file_updated > file.patch

$ diff -c <(sort a.txt) <(sort b.txt)
*** /dev/fd/63  2023-09-12 21:51:50.828885643 -0700
--- /dev/fd/62  2023-09-12 21:51:50.829641102 -0700
***************
*** 1,4 ****
  a
  b
  d
! f
--- 1,5 ----
  a
  b
+ c
  d
! e

$ diff -u <(sort a.txt) <(sort b.txt)
--- /dev/fd/63  2023-09-12 21:51:53.561211803 -0700
+++ /dev/fd/62  2023-09-12 21:51:53.561824746 -0700
@@ -1,4 +1,5 @@
 a
 b
+c
 d
-f
+e

$ diff -i <(sort a.txt) <(sort b.txt)
2a3
> c
4c5
< f
---
> e

diff dirs

$ diff <(cd dir1 && find | sort) <(cd dir2 && find | sort)

colordiff side-by-side

$ colordiff -yW"`tput cols`" /path/to/file1 /path/to/file2

comm

diff

$ comm -3 a.txt b.txt
  c
  e
f

$ comm -3 <(sort a.txt) <(sort b.txt) | column -t -s $'\t' --table-columns '==== a.txt ====,==== b.txt ===='
==== a.txt ====  ==== b.txt ====
                 c
                 e
f

common

$ comm -12 <(sort a.txt) <(sort b.txt)
a
b
d

join

[!NOTE|label:references:]
How can I join elements of a Bash array into a delimited string?
How to join() array elements in a bash script

function join_by {
  local d=${1-} f=${2-}
  if shift 2; then
    printf %s "$f" "${@/#/$d}"
  fi
}

via printf

$ foo=( a "b c" d )
$ printf -v joined '%s,' "${foo[@]}"
$ echo "${joined%,}"
a,b c,d

$ printf -v joined '|,%s,|' "${foo[@]}"
$ echo "${joined%,}"
|,a,||,b c,||,d,|

another solution

# join by single char
$ foo=(a "b c" d)
$ bar=$(IFS=, ; echo "${foo[*]}")
$ echo "$bar"
a,b c,d

alignment

[!TIP]
Bash shell script output alignment

expand (POSIX)

pr (POSIX)

rs (BSD)

column (BSD)

[!NOTE|label:references:]
How do I center-align a column in UNIX?
Set alignment of numeric columns when columnating data
mycolumn
Using awk to align columns in text file?

$ ( printf "PERM LINKS OWNER GROUP SIZE MONTH DAY HH:MM/YEAR NAME\n"; ls -l | sed 1d ) | column -t
PERM        LINKS  OWNER   GROUP  SIZE  MONTH  DAY  HH:MM/YEAR  NAME
-rw-r--r--  1      marslo  staff  8     Sep    12   20:10       a.txt
-rw-r--r--  1      marslo  staff  10    Sep    12   19:34       b.txt

$ paste <(echo -e "foo\n\nbarbarbar") <(seq 3) | column -t
foo        1
2
barbarbar  3

$ paste <(echo -e "foo\n\nbarbarbar") <(seq 3) | column -t -s $'\t'
foo        1
           2
barbarbar  3

with header

$ paste <(echo -e "foo\n\nbarbarbar") <(seq 3) | column -t -s $'\t' --table-columns '====LEFT====,====RIGHT===='
====LEFT====  ====RIGHT====
foo           1
              2
barbarbar     3

column with

$ echo -e 'a very long string..........\t112232432\tanotherfield\na smaller string\t123124343\tanotherfield\n' | column -t -s $'\t'
a very long string..........  112232432  anotherfield
a smaller string              123124343  anotherfield

sort

[!NOTE|label:references:]
Unix sort using unknown delimiter (last column)

sort the last column

awk: print( $NF" "$0 ) | sort | cut -f2- -d' '

$ echo -e '5 5 0 0 622 20\n6 3 2 0 439 8\n5 2 3 0 450 8'
5 5 0 0 622 20
6 3 2 0 439 8
5 2 3 0 450 12

$ echo -e '5 5 0 0 622 20\n6 3 2 0 439 8\n5 2 3 0 450 12' |
  awk '{print($NF" "$0)}' |
  sort -k1,1 -n -r -t' ' |
  cut -f2- -d' '
5 5 0 0 622 20
5 2 3 0 450 12
6 3 2 0 439 8

awk: similar with rev for words

$ echo -e '5 5 0 0 622 20\n6 3 2 0 439 8\n5 2 3 0 450 12' |
  awk '{ for (i=NF; i>0; i--) printf("%s ",$i); printf("\n")}' |             # rev
  sort -k1,1 -nr -t' ' |
  awk '{ for (i=NF; i>0; i--) printf("%s ",$i); printf("\n")}'               # rev
5 5 0 0 622 20
5 2 3 0 450 12
6 3 2 0 439 8

get lines

get second-to-last line

[!NOTE|label:references:]
How to read the second-to-last line in a file using Bash?

sed

$ sed -n 'x;$p' <<\INPUT
  a
  b
  c
  d
  INPUT
c

# or
$ echo -e 'a\nb\nc\nd' | sed -n -e '${x;1!p;};h'
c

# tac + sed
$ echo -e 'a\nb\nc\nd' | tac | sed -n '2p'
c

tail & head

$ echo -e 'a\nb\nc\nd' | tail -2 | head -1
c

get next line by the pattern

$ cat a.txt
1a
2b
3c        * (pattern /3c/)
4d        > (wanted)
5e        > (wanted)
6f
7g

awk

$ cat a.txt | awk '$0=="3c"{getline; print; getline; print}'
4d
5e

$ cat a.txt | awk '/3c/{getline; print; getline; print}'
4d
5e

$ cat a.txt | awk '/^3c$/ {s=NR;next} s && NR<=s+2'
4d
5e

$ cat a.txt | awk '{if(a-->0){print;next}} /3c/{a=2}'
4d
5e

or get second column of next line of pattern

$ awk '/company.domain.com$/{getline; print}' ~/.marslo/.netrc
login marslo

$ awk '/company.domain.com$/{getline; print $2}' ~/.marslo/.netrc
marslo

sed

$ cat a.txt | sed -n '/3c/{n;p;n;p}'
4d
5e

$ cat a.txt | sed -n '/3c/{N;p;n;p}'
3c
4d
5e

to get docker registry mirrors

# exclude the pattern
$ docker system info | sed -n '/Registry Mirrors:/{n;p;}'
  https://artifactory.domain.com/

# including pattern
$ docker system info | sed -n '/Registry Mirrors:/{p;n;p;}'
 Registry Mirrors:
  https://artifactory.domain.com/

change next line of pattern

[!NOTE|label:references:]

Find Matching Text and Replace the Next Line

$ cat revenue.txt
total 4000 dollars' revenue
- Quarter 1:
  Revenue: 1200 dollars; Profit: 700 dollars
- Quarter 2:
  Revenue: 1000 dollars; Profit: 650 dollars
- Quarter 3:
  Revenue: 1200 dollars; Profit: 800 dollars
- Quarter 4:
  Revenue: 600 dollars; Profit: -200 dollars
Profit: 1950 dollars

replace dollars to $ right after line of /Quarter [1-4]/

sed

#                        ╭╴ next
$ sed '/Quarter [1-4]:/{ n; s/dollars/$/g }' revenue.txt
total 4000 dollars' revenue
- Quarter 1:
  Revenue: 1200 $; Profit: 700 $
- Quarter 2:
  Revenue: 1000 $; Profit: 650 $
- Quarter 3:
  Revenue: 1200 $; Profit: 800 $
- Quarter 4:
  Revenue: 600 $; Profit: -200 $
Profit: 1950 dollars

awk

#                              next
#                             ╭╴╴╴╴╮
$ awk '/Quarter [1-4]:/{ rl = NR + 1 } NR == rl { gsub( /dollars/,"$") } 1'
total 4000 dollars' revenue
- Quarter 1:
  Revenue: 1200 $; Profit: 700 $
- Quarter 2:
  Revenue: 1000 $; Profit: 650 $
- Quarter 3:
  Revenue: 1200 $; Profit: 800 $
- Quarter 4:
  Revenue: 600 $; Profit: -200 $
Profit: 1950 dollars

replace dollars to $ every 3 lines after /Quarter [1-4]/

#                        ╭╴ next
#                        ╷ ╭╴ next
#                        ╷ ╷ ╭╴ next
$ sed '/Quarter [1-4]:/{ n;n;n; s/dollars/$/g }' revenue.txt
total 4000 dollars' revenue
- Quarter 1:
  Revenue: 1200 dollars; Profit: 700 dollars
- Quarter 2:
  Revenue: 1000 $; Profit: 650 $
- Quarter 3:
  Revenue: 1200 dollars; Profit: 800 dollars
- Quarter 4:
  Revenue: 600 $; Profit: -200 $
Profit: 1950 dollars

get lines between 2 patterns

[!NOTE|label:reference:]
How to print lines between two patterns, inclusive or exclusive (in sed, AWK or Perl)?
Print lines between PAT1 and PAT2
How to select lines between two marker patterns which may occur multiple times with awk/sed
Print lines between (and excluding) two patterns
see also
* imarslo : get first matching patten ( for CERTIFICATE )

[!TIP] sample data:

$ cat a.txt
1a
2b
3c        * (start)
4d
5e
6f
7g
8h        * (end)
9i
10j
11k


### awk
- include pattern
  ```bash
  $ cat a.txt | awk '/3c/,/8h/'
  3c
  4d
  5e
  6f
  7g
  8h

sed

[!NOTE|label:references:]
Print lines in file from the match line until end of file

include all patterns

$ cat a.txt | sed -n '/3c/,/8h/p'
3c
4d
5e
6f
7g
8h

exclude both patterns

$ sed -n '/3c/,/8h/{//!p;}' a.txt
4d
5e
6f
7g

$ sed -n '/3c/,/8h/{/3c/!{/8h/!p}}' a.txt
4d
5e
6f
7g

#                     + delete from line 1 to /3c/
#                     |      + delete /8h/ to end `$`
#                  +-----+ +-----+
$ cat a.txt | sed '1,/3c/d;/8h/,$d'
4d
5e
6f
7g

#                        + not delete since `/3c/` to `/8h`
#                        |      + delete all the others
#                  +---------+ +-+
$ cat a.txt | sed '/3c/,/8h/!d;//d'
4d
5e
6f
7g

exclude single pattern

$ sed -n '/3c/,/8h/{/8h/!p}' a.txt
3c
4d
5e
6f
7g

$ sed -n '/3c/,/8h/{/3c/!p}' a.txt
4d
5e
6f
7g
8h

with empty line

[!NOTE]
use case :
* imarslo : show top summary

$ cat a.txt
1a
2b
3c      * (start)
4d
5e
6f
        * (ending)
7g
8h
9i
10j
11k

$ cat a.txt | sed -n '/3c/,/^$/p'
3c
4d
5e
6f

get line from pattern to the end

[!TIP|label:references:]

sample content:

$ echo -e '1\n2\n\n3\n4'
1
2
            * (start. pattern: `^\s*$`)
3
4

get from first empty line to the end

[!NOTE|label:references:]
Print lines in file from the match line until end of file

including pattern

[!TIP]
solution: to print from pattern to end ,$ == ,$p
for both CRLF and LF

$ echo -e '1\n2\n\n3\n4' | sed -n '/^\s*$/,$p'

3
4

sed

[!TIP]
solution: using line number to end: n,$ -> "n"',$p'
head -n1 : for first matches pattern line number
tail -n1 : for the last matches pattern line number
* iMarslo: sed get line number of pattern

$ command cat -A a | nl
     1  1^M$
     2  ^M$
     3  2^M$
     4  3^M$
     5  4^M$

$ cat a | sed -n "$(sed -n '/^\s*$/ =' a | tail -n1)"' ,$p'

2
3
4

# or
$ echo -e '1\n\n2\n\n3\n4' | sed -n '/^\s*$/h;/^\s*$/!H;$!b;x;p'

3
4

awk

$ echo -e '1\n\n2\n\n3\n4' | awk '/^\s*$/,0'

2

3
4

not including pattern

[!TIP]
solution: to delete / not print from first line to pattern
delete: /d
not print: -n /!p
for both CRLF and LF

# not print
#                              `-n`   + from 1st line to line of pattern `/^\s*$/`
#                               ^  +------+ + not print
$ echo -e '1\n2\n\n3\n4' | sed -n '1,/^\s*$/!p'
3
4

# delete
#                                  + from 1st line to line of pattern `/^\s*$/`
#                                  |     + delete
#                               +------+ |
$ echo -e '1\n2\n\n3\n4' | sed '1,/^\s*$/d'
3
4

#                                      + not delete since `/^\s*$/` to end
#                                      |       + delete the others
#                                 +---------+ +-+
$ echo -e '1\n\n2\n\n3\n4' | sed '/^\s*$/,$!d;//d'
2
3
4

[!TIP]
solution: with matched line number + 1 : "$(( n+1 ))"',$p'
head -n1 : for first matches pattern line number
tail -n1 : for the last matches pattern line number

$ command cat -A a | nl
     1  1^M$
     2  ^M$
     3  2^M$
     4  3^M$
     5  4^M$

$ cat a | sed -n "$(( $(sed -n '/^\s*$/ =' a | head -n1 )+1 ))"' ,$p'
2
3
4

get from last empty line ( `^$` ) to end

[!NOTE|label:references:]
* iMarslo : get CRLF from linux
* iMarslo: check line ending
How to get all lines from a file after the last empty line?
sed only the last match pattern
sed: return last occurrence match until end of file

awk

# awk
## LF
$ echo -e '1\n\n2\n3\n4' | awk -v RS='\n\n' 'END{printf "%s",$0}'
2
3
4
## CRLF
$ echo -e '1\r\n\r\n2\r\n3\r\n4\r' | awk -v RS='\r\n\r\n' 'END{printf "%s",$0}'
2
3
4

# or
$ echo -e '1\n\n2\n\n3\n4' |
  awk '/^\s*$/ { buf = "" } { buf = buf "\n" $0 } END { print buf }' |
  sed 1d

3
4

tac + awk

# tac + awk
## LF
$ echo -e '1\n\n2\n3\n4' | tac | awk '/^$/{exit}1' | tac
2
3
4
## CRLF
$ echo -e '1\r\n\r\n2\r\n3\r\n4\r' | tac | awk '/^\s*\r$/{exit}1'  | tac
2
3
4

sed

[!WARNING]
for LF only, not support CRLF \r

$ echo -e '1\n\n2\n\n3\n4'
1

2

3
4

$ echo -e '1\n\n2\n\n3\n4' | sed -n '/^\s*$/{g;D;}; N; $p;'
3
4

$ echo -e '1\n\n2\n\n3\n4' | sed -n '/^\s*$/{h;b};H;${x;p}'

3
4

# or
$ echo -e '1\n\n2\n\n3\n4' | sed -n '/^\s*$/h;/^\s*$/!H;$!b;x;p'

3
4

# or
$ echo -e '1\n\n2\n\n3\n4' | sed -n 'H; /^\s*$/h; ${g;p;}'

3
4

reverse search empty line

[!NOTE|label:for show TODO]
How do I get sed to print a range of lines until it sees two consecutive blank lines?
#2916 possibility to suppress output if stdin is empty and set header manually

# without reverse search
$ fd -tf --color never |
  xargs -r -I{} bash -c "sed -ne '/TODO:/,/^\s*$/p' {} | bat -l groovy"

# to suppress output if stdin for `bat`
$ while read -r file; do
  _content=$(sed -ne '/TODO:/,/^\s*$/p' "${file}");
  [[ -n "${_content}" ]] && echo "${_content}" | bat -l groovy;
done < <(fd -tf --color never)

return first matching pattern

references:
How to select first occurrence between two patterns including them

[!TIP]

$ cat sample.crt
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
second paragraph
-----END CERTIFICATE-----

sed

$ cat sample.crt | sed '/-END CERTIFICATE-/q'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----

# or `-n /../p`
#                     `-n`                                         `p`
#                      |                                            |
#                      v                                            v
$ cat sample.crt | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p; /-END CERTIFICATE-/q'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----

# or `/../!d`
#                   no `-n`                                     `!d`
#                     |                                           |
#                     v                                           v
$ cat sample.crt | sed '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/!d; /-END CERTIFICATE-/q'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----

awk

$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/{a=1}; a; /-END CERTIFICATE-/{exit}'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----

# or
$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/ {print} /-END CERTIFICATE-/ {exit}'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----

# or
$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/ {print;f=1} f&&/-END CERTIFICATE-/ {exit}'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----

# or
$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/ {f=1} /-END CERTIFICATE-/ {f=0;print;exit} f'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----

return second matching pattern search range

references:
Copying second occurrence pattern to a new file in sed

[!TIP]

$ cat sample.crt
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
second paragraph
-----END CERTIFICATE-----

$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/ && c++, /-END CERTIFICATE-/'
-----BEGIN CERTIFICATE-----
second paragraph
-----END CERTIFICATE-----

return the last matching pattern search range

[!NOTE|label:references:]
sed script to print lines between the last occurence of a pattern and an empty line

$ cat a
pattern
1
2
3

pattern
4
5

pattern    * (start: the last pattern)
6
7
8
9
           * (end)
10
11

sed

$ sed -ne '
  /pattern/{
    $d;n
    :loop
      s/\n$//;tdone
      $bdone;N
    bloop
    :done
    x
  }
  ${x;/./p;}
' a
6
7
8
9

# or
$ sed -e '
  /pattern/,/^$/!ba
  /./!ba
  H;/pattern/{z;x;}
  :a
  $!d;x;s/.//
' a
6
7
8
9

# or GNU sed
$ sed -Ez '
  s/.*pattern\n(([^\n]+\n)+)(\n.*)?/\1/
' a
6
7
8
9

awk

$ awk '/pattern/,/^$/ { arr[NR]=$0; if (/pattern/) line1=NR; if (/^$/) line2=NR}END{ if (line1) for(i=++line1;i<line2;i++) print arr[i]}' a
6
7
8
9

# or
$ awk -v RS='' -F '\n' '$1 ~ /pattern/ { hold = $0 } END { if (hold != "") print hold }' a | sed 1d
6
7
8
9

replace the last matching pattern

# the last `boy` -> `boys`
$ printf "%s\n" boy girl boy girl boy girl | sed -z 's/.*boy/&s/'
boy
girl
boy
girl
boys
girl

`xargs`

references:
xargs
Brainiarc7/xargs-primer-brief.md
running multiple commands with xargs
Xargs Command in Linux
How to Use the Linux xargs Command
xargs - Construct an argument list and run a command
Why does the wc utility generate multiple lines with "total"?
$ xargs --show-limits < /dev/null
# solution for multiple `total` in $ `git ls-files -z | xargs -0 wc -l`:
$ git ls-files -z | wc -l --files0-from=-
xargs split at newlines not spaces
tips:
--delimiter=delim, -d delim
-I{} == -i
-n max-args, --max-args=max-args
-t, --verbose

complex commands with xargs

[!NOTE|label:references:]
using xargs with output piped to awk throws syntax error
How to use xarg (or awk or sed) to run each line of screen output as a command exactly as displayed?

$ echo ip1 ip2 ip3 ... |
       fmt -1 |
       xargs -i printf 'echo -e "\\n..... {} ....."; /sbin/ping -t1 -c1 -W0 {} | sed "/^$/d"\n' |
       xargs -d\\n -n1 bash -c

# so xargs will execute : `echo -e "\n..... {} ....."; /sbin/ping -t1 -c1 -W0 {} | sed "/^$/d"` one by one

or using $@

$ echo ip1 ip2 ip3 ... |
       fmt -1 |
       xargs -n1 bash -c 'echo -e "\n...... $@ ......"; /sbin/ping -t1 -c1 -W0 "$@" | sed '/^$/d'' _

multiple move

$ mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 -I % cp -a % ~/backups

# multiple cp
$ find /path -type f -name '*~' -print0 | xargs -0 sh -c 'if [ $# -gt 0 ]; then cp -a "$@" ~/backup; fi' sh

subset of arguments

$ echo {0..9} | xargs -n 2
0 1
2 3
4 5
6 7
8 9

sort all shell script by line number

[!TIP] Pipe xargs into find

$ find . -name "*.sh" | xargs wc -l | sort -hr

# better solution
$ find . -name "*.sh" -print0 | wc -l --files0-from=- | sort -hr

diff every git commit against its parent

$ git log --format="%H %P" | xargs -L 1 git diff

running multiple commands with xargs

[!TIP] precondition:
$ cat a.txt
a b c
123
###this is a comment

$ myCommandWithDifferentQuotes=$(cat <<'EOF'
  -> echo "command 1: $@"; echo 'will you do the fandango?'; echo "command 2: $@"; echo
  -> EOF
  -> )

$ < a.txt xargs -I @@ bash -c "$myCommandWithDifferentQuotes" -- @@
command 1: a b c
will you do the fandango?
command 2: a b c

command 1: 123
will you do the fandango?
command 2: 123

command 1: ###this is a comment
will you do the fandango?
command 2: ###this is a comment

$ cat a.txt | xargs -I @@ bash -c "$myCommandWithDifferentQuotes" -- @@

$ while read stuff; do
    echo "command 1: $stuff";
    echo 'will you do the fandango?';
    echo "command 2: $stuff";
    echo
  done < a.txt

compress sub-folders

$ find . -maxdepth 1 ! -path . -type d -print0 |
       xargs -0 -I @@ bash -c '{ \
         tar caf "@@.tar.lzop" "@@" \
         && echo Completed compressing directory "@@" ; \
       }'

ping multiple IPs

[!TIP]

 -a file, --arg-file=file
       Read items from file instead of standard input.  If you use this option, stdin remains unchanged  when
       commands are run.  Otherwise, stdin is redirected from /dev/null.

$ cat a.txt
8.8.8.8
1.1.1.1

$ xargs -L1 -a a.txt /sbin/ping -c 1
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=44 time=82.868 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 82.868/82.868/82.868/0.000 ms

PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=63 time=1.016 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.016/1.016/1.016/0.000 ms

$ echo domain-{1..4}.com | fmt -1 | xargs -L1 ping -c1 -t1 -W0

read by char

$ printf 'mark spitz' | while read -r -n1 c; do printf "[%c]" "$c"; done
[m][a][r][k][][s][p][i][t][z]

`find`

[!NOTE|label:reference:]
Everything CLI: FIND -EXEC VS. FIND | XARGS
How to get only names from find command without path
Remove path from find command output

output format

[!TIP|label:man find:]
man find
%P File's name with the name of the command line argument under which it was found removed.
%f File's name with any leading directories removed (only the last element).

# has `./` by default
$ find . -type f -iname "*cfssl*"
./cfssl/cfssl-scan
./cfssl/cfssl-certinfo
./cfssl/cfssl-bundle
./cfssl/cfssl
./cfssl/cfssl-newkey
./cfssl/cfssljson

# remove `./` by `-printf`
$ find . -type f -iname "*cfssl*" -printf '%P\n'
cfssl/cfssl-scan
cfssl/cfssl-certinfo
cfssl/cfssl-bundle
cfssl/cfssl
cfssl/cfssl-newkey
cfssl/cfssljson

# or remove `./` by sed
$ find . -type f -iname "*cfssl*" | sed 's|^./||'
cfssl/cfssl-scan
cfssl/cfssl-certinfo
cfssl/cfssl-bundle
cfssl/cfssl
cfssl/cfssl-newkey
cfssl/cfssljson

# filename only
$ find . -type f -iname "*cfssl*" -printf '%f\n'
cfssl-scan
cfssl-certinfo
cfssl-bundle
cfssl
cfssl-newkey
cfssljson

output file name only

# has `./` by default
$ find . -type f
./cfssl-scan
./cfssl-certinfo
./cfssl-bundle
./cfssl
./cfssl-newkey
./multirootca
./mkbundle
./cfssljson

# to show filename only by `-exec basename`
$ find . -type f -exec basename {} -print \;
cfssl-scan
cfssl-certinfo
cfssl-bundle
cfssl
cfssl-newkey
multirootca
mkbundle
cfssljson

# or
$ find . -type f -execdir basename {} ';'
cfssl-scan
cfssl-certinfo
cfssl-bundle
cfssl
cfssl-newkey
multirootca
mkbundle
cfssljson

cat config file in all `.git` folder

xargs && cat

$ find . -type d -name '.git' -print0 | xargs -0 -I {} cat {}/config

find && -exec

$ find . -type d -name '.git' -exec cat {}/config \;

`exec` and `sed`

change IP address in batch processing

$ find ${JENKINS_HOME}/jobs \
       -type f \
       -name "config.xml" \
       -maxdepth 2 \
       -exec sed -i 's/1.2.3./4.5.6./g' {} \; -print

find and rename

$ find -iname "*.sh" -exec rename "s/.sh$/.shell/" {} \; -print

find and exclude

$ find . -regextype posix-egrep -regex ".*\.(js|vue|s?css|php|html|json)$" -and -not -regex ".*/(node_modules|vendor)/.*"

$ find . -regex-type posix-extended -regex ".*def/incoming.*|.*456/incoming.*" -prune -o -print

`find` && `tar`

[!TIP] more can be found in imarslo: find and tar

backup all config.xml in JENKINS_HOME

$ find ${JENKINS_HOME}/jobs -maxdepth 2 -name config\.xml -type f -print | tar czf ~/config.xml.tar.gz --files-from -

back build history

$ find ${JENKINS_HOME}/jobs -name builds -prune -o -type f -print | tar czf ~/m.tar.gz --files-from -

find by timestamp

[!NOTE|label:references:]
Find files newer than a day and copy
How to find the difference in days between two dates?
$ echo $(( ($(date +%s -d 20210131)-$(date +%s -d 20210101))/86400 )) days
30 days

# or in `%y%m%d` format
$ echo $(( ($(date --date="230301" +%s) - $(date --date="240301" +%s) )/(60*60*24) )) days
-366 days

in a date range

$ find . -type f -newermt "2010-01-01" ! -newermt "2010-06-01"

via `mtime`

[!TIP|label:tricky on -mtime:]
Understanding find with atime, ctime, and mtime
+n: for greater than n
-n: for less than n
n: for exactly n
i.e.:
-mtime -14: less than 14 days, not including 14 days == 13 days ago and less
-mtime +14: more than 14 days, not including 14 days == 15 days ago and more
* iMarslo : how many days from timestamps

# get all files since 2023-10-16
$ diff=$(( ($(date --date "24-02-29" +%s) - $(date --date "231016" +%s) )/(60*60*24) ))
$ find . -type f -daystart -mtime -$((diff+1)) -printf "%T+ | %p\n" | sort | wc -l
33

# copy all files modified since 2023-10-16
$ find . -type f -daystart -mtime -$((diff+1)) -exec cp -a --parents -t /path/to/target "{}" \+

# with timezone
$ diff=$(( ($(date -d "2015-03-11 UTC" +%s) - $(date -d "2015-03-05 UTC" +%s)) / (60*60*24) ))

# find older than x days
$ find . -type f -mtime +7 -exec ls -l {} \;

# delete older than x days: https://www.commandlinefu.com/commands/view/1394/delete-files-older-than..
$ find /dir_name -mtime +5 -exec rm {} \

via `newermt`

[!TIP|label:tips for -newerXY]

What does newermt mean in find command?

-newerXY reference
          Compares the timestamp of the current file with reference.   The
          reference  argument  is  normally the name of a file (and one of
          its timestamps is used for the comparison) but it may also be  a
          string  describing  an  absolute time.  X and Y are placeholders
          for other letters, and these letters select which time belonging
          to how reference is used for the comparison.

          a   The access time of the file reference
          B   The birth time of the file reference
          c   The inode status change time of reference
          m   The modification time of the file reference
          t   reference is interpreted directly as a time

List of files modified between perticular time period
How to find files between two dates using "find"?

find files in data range

$ find ./ -newermt "2016-01-18" ! -newermt '2016-01-19'
$ find . -type f -newermt "2014-10-08 10:17:00" ! -newermt "2014-10-08 10:53:00"

$ find . -type f -newermt '2023-10-16 00:00:00'  | wc -l
33
# or
$ find . -type f -newermt '16 Oct 2023 00:00:00' | wc -l
33
# or with difference timestamp format
$ find . -type f -newermt "@$(date +%s -d '10/16/2023 0:00:00 PDT')" -printf "%T+ | %p\n" | sort | wc -l
33

# copy all files modified since 2023-10-16
$ find . -type f -newermt '2023-10-16 00:00:00' -exec cp -a --parents -t /path/to/target "{}" \+

latest 5 modified files recursively

$ find . -type f -printf '%T@ %TY-%Tm-%Td %TH:%TM:%.2TS %p\n' | sort -nr | head -n 5 | cut -f2- -d" "

inject commands inside find

[!NOTE|label:references:]
Find functions, commands, and builtins [duplicate]

$ find -exec bash -c '
         print_echo() { printf "This is print_echo Function: %s\n" "$@"; };
         print_echo "$@"
       ' find-bash {} +

printf

[!NOTE|label:references:]
How to Use find -printf in Linux?

formats

[!NOTE|label:references:]
printf time formats
find . -printf "%T<format>\n"
printf format

time format

FORMAT

DESCRIPTION

EXAMPLE

%A

last access time

%A+: 2023-02-20+05:19:18.0000000000

%T

last modification time

%T@: 1676899158.0000000000

%t

last modification time in ctime format

Mon Feb 20 05:19:18.0000000000 2023

%C

last status change time

%C+: 2024-11-20+03:30:18.8905999140

%c

last status change time in ctime format

Wed Nov 20 03:30:18.8905999140 2024

%B

birth time

%B@: 1676899158.0000000000

$ touch -d "2023-02-20 05:19:18" sample.txt
$ stat sample.txt
Access: 2023-02-20 05:19:18.000000000 -0800
Modify: 2023-02-20 05:19:18.000000000 -0800
Change: 2024-11-20 03:30:18.890599914 -0800
 Birth: 2023-02-20 05:19:18.000000000 -0800

$ find . -maxdepth 1 -name 'sample.txt' -printf '%%A+: %A+\n%%A@: %A@\n%%T+: %T+\n%%T@: %T@\n%%C+: %C+\n%%C@: %C@\n%%B+: %B+\n%%B@: %B@\n%%t: %t\n%%c: %c\n'
%A+: 2023-02-20+05:19:18.0000000000
%A@: 1676899158.0000000000
%T+: 2023-02-20+05:19:18.0000000000
%T@: 1676899158.0000000000
%C+: 2024-11-20+03:30:18.8905999140
%C@: 1732102218.8905999140
%B+: 2023-02-20+05:19:18.0000000000
%B@: 1676899158.0000000000
%t: Mon Feb 20 05:19:18.0000000000 2023
%c: Wed Nov 20 03:30:18.8905999140 2024

$ find . -maxdepth 1 -name 'sample.txt' -printf 'week%AW %Aj/365 %Ax %Ar\n%%A+: %A+\n'
week08 051/365 02/20/2023 05:19:18 AM
%A+: 2023-02-20+05:19:18.0000000000

TIME FIELD

DESCRIPTION

EXAMPLE

+

date time

2023-02-20+05:19:18.0000000000

@

unix epoch

1676899158.0000000000

H

hour

00..23

k / I

hour in 24-hour / 12-hour format

00..23 / 01..12

M

minute

00..59

S

second

00..60

p

AM/PM

AM / PM

T / X

time in 24-hour format

hh:mm:ss:xxxxxxxxxx

Z

timezone

PST / PDT

$ find . -maxdepth 1 -name 'sample.txt' -printf '%%TT: %TT\n%%TX: %TX\n'
%TT: 05:19:18.0000000000
%TX: 05:19:18.0000000000

DATA FIELD

DESCRIPTION

EXAMPLE

a / A

abbreviated / full weekday

Wed / Wednesday

b(h) / B

abbreviated / full month name

Jan / January

m

month

01..12

d

day of month

01..31

w

day of week

01->Monday; 02->Tuesday

j

day of year

001..366

U / W

week number: Sunday / Monday as first day

00..53

y / Y

last 2-digits-of-year / 4-digits-of-year

00..99 / 1970..

r

time in 12-hour format

hh:mm:ss [A/P]M

F

full date; same as %Y-%m-%d

2023-02-20

D

date; same as %m/%d/%y

02/20/23

x

locale date

02/20/2023

$ find . -maxdepth 1 -name 'sample.txt' -printf '%%TX: %TX\n%%Tx: %Tx\n%%TD: %TD\n%%TF: %TF\n%%Tr: %Tr\n'
%TX: 05:19:18.0000000000
%Tx: 02/20/2023
%TD: 02/20/23
%TF: 2023-02-20
%Tr: 05:19:18 AM

name format
FORMAT
DESCRIPTION
EXAMPLE
%p
file's name
./sample.txt
%P
file's name without starting-point
sample.txt
%f
basename
sample.txt
%h
leading directories of file's name
.
```
$ find . -maxdepth 1 -name '.vimrc' -printf '%%p: %p\n%%P: %P\n%%f: %f\n%%h: %h\n'
%p: ./.vimrc
%P: .vimrc
%f: .vimrc
%h: .
```

permision format

FORMAT

DESCRIPTION

EXAMPLE

%m

file's mode

644

%M

file's mode in human-readable format

-rw-r--r--

$ find . -type f -printf "\n%Td-%Tm-%TY %Tr %p" | head -1
4-10-2023 02:38:42 AM /Users/marslo/.marslo/.marslorc

T : time in 24-hour format : hh:mm:ss.xxxxxxxxxx

$ find . -type f -printf "\n%Td-%Tm-%TY %TT %p" | head -1
14-10-2023 02:38:42.5626405780 /Users/marslo/.marslo/.marslorc

X : locale time : hh:mm:ss.xxxxxxxxxx

c : locale time in ctime format

$ find . -type f -printf "\n%Tc %p" | head -1
Sat Oct 14 02:38:42 2023 /Users/marslo/.marslo/.marslorc

D : date : mm/dd/yy
F : date : yyyy-mm-dd
x : locale date : mm/dd/yy
R : hour and minute in 24 hour format : HH:MM

+ : date and time

$ find . -printf "%T+ | %p\n" | head -1
2023-10-14+02:38:42.5626405780 | /Users/marslo/.marslo/.marslor

Formatting Tips

center-align

$ find . -printf "%15TA | %p\n"
         Monday | /Users/marslo/.marslo/bin/iweather
       Saturday | /Users/marslo/.marslo/.marslorc

left-align

$ find . -printf "%-15TA | %p\n"
Monday          | /Users/marslo/.marslo/bin/iweather
Saturday        | /Users/marslo/.marslo/.marslorc

mixed align

$ find . -type f -printf "%10T+ %-10TA | %m | %p\n" | sort -r | head -2
2023-10-16+20:25:53.5005192040 Monday     | 755 | /Users/marslo/.marslo/bin/iweather
2023-10-14+02:38:42.5626405780 Saturday   | 755 | /Users/marslo/.marslo/.marslorc

# more
find . -type d -printf "%d    %-30p %-10u %-10g %-5m %T+\n" | sort

tips

faster find and replace string

$ sh -c 'S=askapache R=htaccess; find . -mount -type f | xargs -P5 -iFF grep -l -m1 "$S" FF | xargs -P5 -iFF sed -i -e "s%${S}%${R}%g" FF'

find duplicate files

[!TIP|label:rules:] size first, then md5 hash

$ find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

random nubmer between X to 1

$ echo $[RANDOM%X+1]

# or: https://www.commandlinefu.com/commands/view/14051/random-number-between-1-and-x
$ (od -An -t u8 -N8 </dev/urandom; echo X '* 2 64^/1+p') | dc

# or: https://www.commandlinefu.com/commands/view/6297/random-number-between-1-and-x
$ echo "$(od -An -N4 -tu4 /dev/urandom) % 5 + 1" | bc

# between 1 - 256: https://www.commandlinefu.com/commands/view/12702/random-number-between-1-and-256
$ od -An -N1 -tu1 /dev/random

# decimal in 1 - 2d6: https://www.commandlinefu.com/commands/view/6269/random-decimal-in-the-interval-0-n-1-and-2d6-dice-roll
$ awk 'BEGIN { srand(); print rand() }'

trim

trim tailing chars

str='1234567890'

awk + rev

$ echo $str | rev | cut -c4- | rev
1234567

${var:: -x})
```
$ echo ${str:: -3}
1234567
```

remove leading & trailing whitespace

$ str="    aaaa    bbbb      "
$ echo "$str" | sed 's:^ *::; s: *$::'

# i.e.:
$ echo .$(echo "$str" | sed 's:^ *::; s: *$::').
.aaaa bbbb.

remove all spaces
```
$ echo .${str// }.
.aaaabbbb.
```

remove leading space(s)

$ echo .${str##+([[:space:]])}.
.aaaa bbbb .

remove leading space(s)

$ echo .${str%%+([[:space:]])}.
. aaaa bbbb.

function in pip

function trim() { IFS='' read -r str; echo "${str}" | sed -e 's/^[[:blank:]]*//;s/[[:blank:]]*$//'; }
$ echo ..$(echo "   aaa     bbb   " | trim)..
..aaa bbb..

remove empty lines

[!NOTE|label:references:]
Delete empty lines using sed
sed
'/^[[:space:]]*$/d'
'/^\s*$/d'
'/^$/d'
-n '/^\s*$/!p'
grep
.
-v '^$'
-v '^\s*$'
-v '^[[:space:]]*$'
awk
/./
'NF' or 'NF > 0'
'!/^$/'
'length'
'/^[ \t]*$/ {next;} {print}'
'!/^[ \t]*$/'

# original
$ cal | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊
····················␊

# awk 'NF'
$ cal | awk 'NF' | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊

# sed '/^\s*$/d'
$ cal | sed '/^\s*$/d' | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊

remove empty line at the end of file

[!NOTE|label:references:]
Remove an empty line at the end of a file (bash)

$ cat a.txt | sed '${/^[[:space:]]*$/d;}'
a
b

c
d

remove duplicate empty lines

[!NOTE|label:references:]
Removing duplicate blank lines with awk

# original file
$ cat a.txt --style='numbers'
   1 a
   2 b
   3
   4
   5
   6 c
   7
   8 d
   9
  10

$ awk 'NF || p; { p = NF }' p=1 a.txt | bat --style='numbers'
   1 a
   2 b
   3
   4 c
   5
   6 d
   7

# or
$ awk 'NF{c=1} (c++)<3' a.txt | bat --style='numbers'
   1 a
   2 b
   3
   4 c
   5
   6 d
   7

# or
$ awk -v RS= -v ORS='\n\n' '1' a.txt | bat --style='numbers'
   1 a
   2 b
   3
   4 c
   5
   6 d
   7

# or
$ awk '!NF{found++} found>1 && !NF{next} NF{found=""} 1' a.txt | bat --style='numbers'
   1 a
   2 b
   3
   4 c
   5
   6 d
   7

search and replace

[!NOTE|label:reference]
shellcheck SC2001
sample code:
str='aa  bb      cc'

${variable//search/replace}

$ shopt -s extglob
$ echo ${str//+( )/|}
aa|bb|cc

$ echo "${str//+([[:blank:]])/|}"
aa|bb|cc

sed

# DO NOT USE "${str}"
$ echo ${str} | sed 's: :|:g'
aa|bb|cc

$ echo "$str" | sed 's:[ ][ ]*:|:g'
aa|bb|cc

# or
$ echo "$str" | sed 's:\s\s*:|:g'
aa|bb|cc

echo "${string:0:$(( position - 1 ))}${replacement}${string:position}"

or

$ sed 's:\s\s*:|:g' <<< "${str}" aa|bb|cc


- [tr](https://stackoverflow.com/a/50259880/2940319)
```bash
$ echo "$str" | tr -s ' ' '|'
aa|bb|cc

replace with position

$ string=aaaaa
$ replacement=b
$ position=3
$ echo "${string:0:$(( position - 1 ))}${replacement}${string:position}"
aabaa

$ echo "${string:0:position-1}${replacement}${string:position}"
aabaa

check line ending

[!NOTE|label:references:]
* iMarslo : get CRLF from linux
check ascii via terminal
$ man ascii
$ cat /usr/share/misc/ascii
How Hexdump works
ASCII Table
ASCII/Binary of 0x0a
Newline

CHARACTER ENCODING

ABBREVIATION

HEX

DEC

ESCAPE SEQUENCE

UNIX or Unix-like

ASCII

MS-DOS

ASCII

CR LF

0D 0A

13 10

Commodore 8-bit machines

ASCII

QNX pre-POSIX

ASCII

\036

Acorn BBC and RISC OS

ASCII

LF CR

0A 0D

10 13

Atari 8-bit machines

ATASCII

155

IBM mainframe systems

EBCDIC

\025

ZX80 and ZX81

non-ASCII

NEWLINE

118

od -c

$ echo 'abc' | od -c
0000000   a   b   c  \n
0000004

$ echo -n 'abc' | od -c
0000000   a   b   c
0000003

hexdump -c

$ echo 'abc' | hexdump -c
0000000   a   b   c  \n
0000004
$ echo -n 'abc' | hexdump -c
0000000   a   b   c
0000003

hexdump -C

$ echo 'abc' | hexdump -C
00000000  61 62 63 0a                                       |abc.|
00000004
#                  ^
#                  |
#              0x0a: LF
$ echo -n 'abc' | hexdump -C
00000000  61 62 63                                          |abc|
00000003

$ cat a.txt | hexdump -C
#                    0x0a
#                     v
00000000  61 61 61 61 0a                                    |aaaa.|
00000005

$ unix2dos a.txt
unix2dos: converting file a.txt to DOS format...
$ cat a.txt | hexdump -C
#                   0x0d 0x0a
#                     v   v
00000000  61 61 61 61 0d 0a                                 |aaaa..|
00000006

$ cat a.txt | hexdump -c
0000000   a   a   a   a  \r  \n
0000006
$ file a.txt
a.txt: ASCII text, with CRLF line terminators

vim

$ vim a.txt
:%!hexdump -C

# or
$ vim -c '%!xxd' a.txt

remove the ending '\n'

[!NOTE|label:references:]
Why should text files end with a newline?
original file
$ cat foo.txt
abc
efg

$ cat -A foo.txt
abc$
efg$

$ cat foo.txt | od -c
0000000   a   b   c  \n   e   f   g  \n
0000010

truncate

$ truncate -s -1 foo.txt
$ od -c foo.txt
0000000   a   b   c  \n   e   f   g
0000007

sed

$ sed -z s/.$// foo.txt | od -c
0000000   a   b   c  \n   e   f   g
0000007

$ sed -z s/\\n$// foo.txt | od -c
0000000   a   b   c  \n   e   f   g
0000007

$ sed -z 's/\n$//' foo.txt | od -c
0000000   a   b   c  \n   e   f   g
0000007

printf

$ printf %s "$(< foo.txt)" | od -c
0000000   a   b   c  \n   e   f   g
0000007

head

$ head -c -1 foo.txt | od -c
0000000   a   b   c  \n   e   f   g
0000007

vim

$ od -c foo.txt
0000000   a   b   c  \n   e   f   g  \n
0000010

$ vim -c "set binary noeol" -c "wq" foo.txt
$ od -c foo.txt
0000000   a   b   c  \n   e   f   g
0000007

# or : https://stackoverflow.com/a/39627416/2940319
$ vim -c "set noendofline nofixendofline" -c "wq" foo.txt
$ od -c foo.txt
0000000   a   b   c  \n   e   f   g
0000007

add '\n' to line-ending

[!TIP]
for ssh private key issue:
$ ssh -vT sample.host
...
Load key "~/.ssh/id_ed25519": error in libcrypto
references:
How to add a newline to the end of a file?
Add new line character at the end of file in bash

check last char in the file

# unqualified key
$ tail -c1 ~/.ssh/id_ed25519
$ tail -c1 ~/.ssh/id_ed25519 | /usr/bin/xxd -u -p
2D

# qualified key
$ tail -c1 ~/.ssh/id_ed25519

$ tail -c1 ~/.ssh/id_ed25519 | /usr/bin/xxd -u -p
0A
$ tail -c1 ~/.ssh/id_ed25519 | hexdump -v -e '/1 "%02X"'
0A

add new line

$ [ -n "$(tail -c1 file)" ] && echo >> ~/.ssh/id_ed25519
# or
$ [ -z "$(tail -c1 file)" ] || printf '\n' >>file

# or: https://stackoverflow.com/a/35279563/2940319
$ echo \ >> file.txt

# or: https://stackoverflow.com/a/65136212/2940319
$ sed -i -z 's/$/\n/g' file.txt

# performance for various solutions
$ [ -n "$(tail -c1 file)" ] && printf '\n' >>file  0.013 sec
$ vi -ecwq file                                    2.544 sec
$ paste file 1<> file                             31.943 sec
$ ed -s file <<< w                             1m  4.422 sec
$ sed -i -e '$a\' file                         3m 20.931 sec

add new line ending without modifying the file

$ echo -n "$(cat ~/.ssh/id_ed25519)"$'\n' | tail -c1 | /usr/bin/xxd -u -p
0A

# or
$ cat file.txt | sed -e '$a\'

# or: https://stackoverflow.com/a/65136212/2940319
$ cat a.org.txt | sed -z 's/$/\n/g'

# or
$ echo '' >> file                               # fix the last line line-ending
$ sed '${/^[[:space:]]*$/d;}' -i file           # remove the empty lines at end of file

fold

check the params valid

available params should be contained by 'iwfabcem'

# case insensitive
param=$( tr '[:upper:]' '[:lower:]' <<< "$1" )

for _p in $(echo "${param}" | fold -w1); do
  [[ ! 'iwfabcem' =~ ${_p} ]] && exits='yes' && break
done

insert

insert new line

insert right after the second match string

DCR DCR DCR

DCR DCR check DCR

$ echo -e "DCR\nDCR\nDCR" | awk 'BEGIN {t=0}; { print }; /DCR/ { t++; if ( t==2) { print "check" } }'

insert after the matched string

with gsub

$ cat project.config |
  awk '{
    gsub( /.+access .(refs\/for\/|\^)refs\/heads\/main.+$/,
          "&\n\tpush = block group Registered Users\n\tsubmit = block group Registered Users\n\tpushMerge = block group Registered Users",
          $0 )
  }1'

ORIGINAL

AFTER INSERT

$ cat project.config

$ cat project.config | awk '{
      gsub( /.+access .(refs\/for\/|\^)refs\/heads\/main.+$/,
            "&\n\t---a---\n\t---b---",
            $0 )
      }1'

[access "^refs/heads/main"]
        label-Code-Review = -2..+2 group user/John Doe (jdoe)
        label-Verified = -1..+1 group user/John Doe (jdoe)
        push = group user/John Doe (jdoe)
        submit = group user/John Doe (jdoe)
[access "refs/for/refs/heads/main"]
        push = group user/John Doe (jdoe)
        addPatchSet = group user/John Doe (jdoe)
        abandon = group user/John Doe (jdoe)
        deleteOwnChanges = group user/John Doe (jdoe)
        editAssignee = group user/John Doe (jdoe)
        editTopicName = group user/John Doe (jdoe)
        label-Code-Review = -2..+2 group user/John Doe (jdoe)
        label-Verified = -1..+1 group user/John Doe (jdoe)
        owner = group user/John Doe (jdoe)
        read = group user/John Doe (jdoe)
        removeReviewer = group user/John Doe (jdoe)
        submit = group user/John Doe (jdoe)
        create = group user/John Doe (jdoe)
        pushMerge = group user/John Doe (jdoe)
[access "^refs/heads/user/jdoe/.*"]
        label-Code-Review = -2..+2 group user/John Doe (jdoe)
        label-Verified = -1..+1 group user/John Doe (jdoe)
        push = group user/John Doe (jdoe)
        submit = group user/John Doe (jdoe

[access "^refs/heads/main"]
        ---a---
        ---b---
        label-Code-Review = -2..+2 group user/John Doe (jdoe)
        label-Verified = -1..+1 group user/John Doe (jdoe)
        push = group user/John Doe (jdoe)
        submit = group user/John Doe (jdoe)
[access "refs/for/refs/heads/main"]
        ---a---
        ---b---
        push = group user/John Doe (jdoe)
        addPatchSet = group user/John Doe (jdoe)
        abandon = group user/John Doe (jdoe)
        deleteOwnChanges = group user/John Doe (jdoe)
        editAssignee = group user/John Doe (jdoe)
        editTopicName = group user/John Doe (jdoe)
        label-Code-Review = -2..+2 group user/John Doe (jdoe)
        label-Verified = -1..+1 group user/John Doe (jdoe)
        owner = group user/John Doe (jdoe)
        read = group user/John Doe (jdoe)
        removeReviewer = group user/John Doe (jdoe)
        submit = group user/John Doe (jdoe)
        create = group user/John Doe (jdoe)
        pushMerge = group user/John Doe (jdoe)
[access "^refs/heads/user/jdoe/.*"]
        label-Code-Review = -2..+2 group user/John Doe (jdoe)
        label-Verified = -1..+1 group user/John Doe (jdoe)
        push = group user/John Doe (jdoe)
        submit = group user/John Doe (jdoe)

with variable

[!NOTE|label:references:]
How to insert a line after a match using sed (or awk) in a formatted JSON file?

#                                       + first line without extra space
#                                       |                  + the others new line requires 4 spaces
#                               +--------------+   +----------------+
$ cat sample.json | awk -v new='"gender": "male",\n    "state": "CA",' '{print} sub(/"name":.*John Doe.*$
/,""){print $0 new}'
{
  "id": "1234567",
  "properties": {
    "name": "John Doe",
    "gender": "male",
    "state": "CA",
    "age": 30
  }
}

# or
$ cat sample.json |
      awk -v insertVal="femal" '/"name":.+John.+Doe.*$/{$0=$0"\n    \"gender\": \""insertVal"\","} {print}'
{
  "id": "1234567",
  "properties": {
    "name": "John Doe",
    "gender": "femal",
    "age": 30
  }
}

ORIGINAL

AFTER INSERT

INSERT TWO LINES

$ cat sample.json

$ cat sample.json | 
          awk -v new='"gender": "male",' '{print} sub(/"name":.*John Doe.*$/,""){print $0 new}'

$ cat sample.json | 
          awk -v new='"gender": "male",\n    "state": "CA",' '{print} sub(/"name":.*John Doe.*$/,""){print $0 new}'

{
  "id": "1234567",
  "properties": {
    "name": "John Doe",
    "age": 30
  }
}

{
  "id": "1234567",
  "properties": {
    "name": "John Doe",
    "gender": "male",
    "age": 30
  }
}

{
  "id": "1234567",
  "properties": {
    "name": "John Doe",
    "gender": "male",
    "state": "CA",
    "age": 30
  }
}

with sed

$ cat sample.json | sed -e '/John Doe/{p;s/name.*"/gender": "male"/' -e '}'
{
  "id": "1234567",
  "properties": {
    "name": "John Doe",
    "gender": "male",                   # new added
    "age": 30
  }
}

insert new line base on pattern

[!NOTE|label:references:]
awk script for pattern match and line break

awk

$ echo "9089.00 ----- kl jkjjljk lkkk; (909099)  9097.00 ----- HGJJHHJ jcxkjlkjvhvlk jhdkjksdfkhfskd 898.00 ----- HHHH" |
  awk '{ gsub("[0-9][0-9]*[.]00", RS "&");print }'

9089.00 ----- kl jkjjljk lkkk; (909099)
9097.00 ----- HGJJHHJ jcxkjlkjvhvlk jhdkjksdfkhfskd
898.00 ----- HHHH

sed

$ echo "9089.00 ----- kl jkjjljk lkkk; (909099)  9097.00 ----- HGJJHHJ jcxkjlkjvhvlk jhdkjksdfkhfskd 898.00 ----- HHHH" |
  sed "s/\([0-9]*\.00\)/\n\1/g"

9089.00 ----- kl jkjjljk lkkk; (909099)
9097.00 ----- HGJJHHJ jcxkjlkjvhvlk jhdkjksdfkhfskd
898.00 ----- HHHH

write a file without indent space

$ sed -e 's:^\s*::' > ~/file-without-indent-space.txt < <(echo "items.find ({
      \"repo\": \"repo-name\",
      \"type\" : \"folder\" ,
      \"depth\" : \"1\",
      \"created\" : { \"\$before\" : \"4mo\" }
    })
")

$ cat ~/file-without-indent-space.txt
items.find ({
"repo": "repo-name",
"type" : "folder" ,
"depth" : "1",
"created" : { "$before" : "4mo" }
})

$ sed -e 's:^\s*::' > find.aql <<-'EOF'
                    items.find ({
                      "repo": "${product}-${stg}-local",
                      "type" : "folder" ,
                      "depth" : "1",
                      "created" : { "${opt}": "4mo" }
                    })
EOF

$ sed -e 's:^\s*::' <<-'EOF' items.find ({ "repo": "${product}-${stg}-local", "type" : "folder" , "depth" : "1", "created" : { "${opt}": "4mo" } }) EOF items.find ({ "repo": "${product}-${stg}-local", "type" : "folder" , "depth" : "1", "created" : { "${opt}": "4mo" } })

cat

`<< -` and `<<`

Here Documents:

This type of redirection instructs the shell to read input from the current source until a line containing only delimiter (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input for a command.
The format of here-documents is:
      <<[-]word
              here-document
      delimiter
No parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. In the latter case, the character sequence \ is ignored, and \ must be used to quote the characters , $, and `.

cat with specific character

$ man tab
...
      -T, --show-tabs

          display TAB characters as ^I

other references:

$ cat -A sample.sh LANG=C tr a-z A-Z <<- END_TEXT$ Here doc with <<$ A single space character (i.e. 0x20 ) is at the beginning of this line$ ^IThis line begins with a single TAB character i.e 0x09 as does the next line$ ^IEND_TEXT$ $ echo The intended end was before this line$

$ bash sample.sh HERE DOC WITH <<- A SINGLE SPACE CHARACTER (I.E. 0X20 ) IS AT THE BEGINNING OF THIS LINE THIS LINE BEGINS WITH A SINGLE TAB CHARACTER I.E 0X09 AS DOES THE NEXT LINE The intended end was before this line

$ cat -A sample.sh LANG=C tr a-z A-Z << END_TEXT$ Here doc with <<$ A single space character (i.e. 0x20 ) is at the beginning of this line$ ^IThis line begins with a single TAB character i.e 0x09 as does the next line$ ^IEND_TEXT$ $ echo The intended end was before this line$

$ bash sample.sh sample.sh: line 7: warning: here-document at line 1 delimited by end-of-file (wanted `END_TEXT') HERE DOC WITH << A SINGLE SPACE CHARACTER (I.E. 0X20 ) IS AT THE BEGINNING OF THIS LINE THIS LINE BEGINS WITH A SINGLE TAB CHARACTER I.E 0X09 AS DOES THE NEXT LINE END_TEXT

ECHO THE INTENDED END WAS BEFORE THIS LINE

Previoushavefun Nextawk

Last updated 4 months ago

Was this helpful?

charset

escape

encryption

base64

show

align

numfmt

combinations

single line to multiple lines

execute commands from file

combine every 2 lines

combine every 3 lines

format output

echo

diff

comm

join

alignment

sort

sort the last column

get lines

get second-to-last line

get next line by the pattern

change next line of pattern

get lines between 2 patterns

sed

with empty line

get line from pattern to the end

get from first empty line to the end

get from last empty line ( ^$ ) to end

reverse search empty line

return first matching pattern

sed

awk

return second matching pattern search range

return the last matching pattern search range

sed

awk

replace the last matching pattern

xargs

complex commands with xargs

sort all shell script by line number

diff every git commit against its parent

compress sub-folders

ping multiple IPs

find

output format

output file name only

cat config file in all .git folder

exec and sed

find and rename

find && tar

find by timestamp

via mtime

via newermt

inject commands inside find

printf

formats

tips

trim

trim tailing chars

remove empty lines

remove empty line at the end of file

remove duplicate empty lines

or

check line ending

remove the ending '\n'

add '\n' to line-ending

fold

check the params valid

insert

insert new line

insert right after the second match string

insert after the matched string

insert new line base on pattern

write a file without indent space

cat

<< - and <<

`base64`

get from last empty line ( `^$` ) to end

`xargs`

`find`

cat config file in all `.git` folder

`exec` and `sed`

`find` && `tar`

via `mtime`

via `newermt`

`<< -` and `<<`