text-processing
reference:
charset
[!NOTE|label:references:]
HEX
NAME
ABBREVIATION
ESCAPE
CODE
0x00
null
NUL
\0
^@
0x07
bell
BEL
\a
^G
0x08
backspace
BS
\b
^H
0x09
horizontal tab
HT
^I
0x0a
line feed
LF
^J
0x0b
vertical tab
VT
\v
^K
0x0c
form feed
FF
\f
^L
0x0d
carriage return
CR
^M
0x1a
Control-Z
SUB
-
^Z
0x1b
escape
ESC
\e
^[
list all charset
$ iconv --listconvert text-file from utf-8 to iso-8859-1
$ iconv --from-code=UTF-8 --to-code=ISO-8859-1 inputfile.txt > outputfile.txt
echo ascii in bash
# single echo
# ' is necessary
# v
$ printf "%d" "'${var}"
# more example
$ printf "%s\n" {a..z} |
while read -r c; do printf "%s = %d\n" "$c" "'$c"; done
# or format output
$ printf "%s\n" {a..z} {A..Z} |
while read -r c; do printf "%s = %-3d\n" "$c" "'$c"; done |
awk 'ORS=(NR%13?" ":"\n")'
a = 97 b = 98 c = 99 d = 100 e = 101 f = 102 g = 103 h = 104 i = 105 j = 106 k = 107 l = 108 m = 109
n = 110 o = 111 p = 112 q = 113 r = 114 s = 115 t = 116 u = 117 v = 118 w = 119 x = 120 y = 121 z = 122
A = 65 B = 66 C = 67 D = 68 E = 69 F = 70 G = 71 H = 72 I = 73 J = 74 K = 75 L = 76 M = 77
N = 78 O = 79 P = 80 Q = 81 R = 82 S = 83 T = 84 U = 85 V = 86 W = 87 X = 88 Y = 89 Z = 90escape
[!NOTE|label:references:]
ESCAPING SEQUENCES
COMMENTS
\'
single quote
\"
double quote
\\
backslash
new line
horizontal tab
carriage return
\?
question mark
single quota & double quotas
[!TIP|label:references:]
3.1.2.5 Locale-Specific Translation Prefixing a double-quoted string with a dollar sign (
$), such as$"hello, world", will cause the string to be translated according to the current locale
$ echo $'aa\'bb' aa'bb # ascii code # hex octal hex octal $ echo -e "\x27 \047 \\x22 \042" ' ' " " $ echo -e "Let\x27s get coding!" Let's get coding! $ echo -e "Let\x22s get coding!" Let"s get coding!
control character
$ read -rep "$(printf "\033[37;3m%s\033[0m" 'input: ')" input
# SOH STX SOH STX
# +--+ +--+ +--+ +--+
$ read -rep "$(printf "\001\033[37;3m\002%s\001\033[0m\002" 'input: ')" input
CHAR/SEQUENCE
ASCII
USAGE/MEANING
EXAMPLE
TYPICAL SCENARIO
\001
SOH (Start Of Heading)
(0x01)
start of non-printing section
(== \[ in PS1)
\001\033[31m\002
Wrap ANSI color codes to avoid cursor position miscalculation
\002
STX (Start Of Text)
(0x02)
end of non-printing section
(== \] in PS1)
\001\033[0m\002
Wrap ANSI color codes to avoid cursor position miscalculation
\e
\033
ESC (Escape, 0x1B)
(0x1B)
begins an ANSI escape sequence
\e[31m (set text red)
Control text color, cursor position, terminal behavior
\a
BEL (BELL)
(0x07)
triggers terminal bell sound
echo -e "\a"
Alert user with sound
CR (Carriage Return)
(0x0D)
returns cursor to beginning of line
printf "Progress: 50%%\r"
Overwrite text on same line ( i.e. spinner/progress bar )
LF (line feed)
(0x0A)
newline
echo -e "Line1\nLine2"
Standard line break
TAB (Horizontal Tab)
(0x09)
horizontal tab
echo -e "Name:\tJohn"
Column-style output, Align text
\b
BS (Delete)
(0x08)
deletes one character backward
echo -e "abc\b\bxyz" (result: axyz)
Inline deletions
\xHH
Hex Byte
represents a character by its hex value
\x1B == \e
\x41 → A
Injecting arbitrary bytes
\uXXXX
Unicode Char
4-digit Unicode character (Bash extension, requires terminal support)
\u2714 → ✔️
Emojis or symbols in terminal
\nnn
Octal Char
3-digit octal character
\101 → A
Injecting arbitrary bytes
\033[XXm
SGR (Select Graphic Rendition)
text color/style control (i.e.: 31m=red，1m=bold，0m=reset)
\033[32;1mSuccess!\033[0m
Highlight key information
\033[K
-
erase to end of line
echo -e "Loading...\033[K"
Clear the content at the end of each line dynamically
\033[nA
-
cursor up n lines
\033[2A
Multi-line Output Control
\033[nB
-
cursor down n lines
\033[3B
Multi-line Output Control
\033[nC
-
cursor forward n columns
\033[10C
Align complex output
\033[nD
-
cursor backward n columns
\033[5D
Rollback Edit
encryption
base64
base64$ echo "marslo" | base64 -w0
bWFyc2xvCg==decryption
$ echo "bWFyc2xvCg==" | base64 --decode marslo
show
align
[!NOTE|label:see also]
# right-align
$ printf _"%10s"_ "foobar"
_ foobar_
# left-align
$ printf _"%-10s"_ "foobar"
_foobar _numfmt
[!NOTE|label:references:]
setup
# generic $ npm install numfmt # osx $ brew install coreutils$ brew list coreutils | grep bin /usr/local/Cellar/coreutils/9.4/bin/md5sum /usr/local/Cellar/coreutils/9.4/bin/gsha512sum /usr/local/Cellar/coreutils/9.4/bin/gusers /usr/local/Cellar/coreutils/9.4/bin/gprintenv /usr/local/Cellar/coreutils/9.4/bin/gmknod /usr/local/Cellar/coreutils/9.4/bin/shuf /usr/local/Cellar/coreutils/9.4/bin/gdd /usr/local/Cellar/coreutils/9.4/bin/gtsort /usr/local/Cellar/coreutils/9.4/bin/grealpath /usr/local/Cellar/coreutils/9.4/bin/grmdir /usr/local/Cellar/coreutils/9.4/bin/gfold /usr/local/Cellar/coreutils/9.4/bin/gnl /usr/local/Cellar/coreutils/9.4/bin/greadlink /usr/local/Cellar/coreutils/9.4/bin/gshred /usr/local/Cellar/coreutils/9.4/bin/gmv /usr/local/Cellar/coreutils/9.4/bin/runcon /usr/local/Cellar/coreutils/9.4/bin/gmkdir /usr/local/Cellar/coreutils/9.4/bin/gkill /usr/local/Cellar/coreutils/9.4/bin/guniq /usr/local/Cellar/coreutils/9.4/bin/gpr /usr/local/Cellar/coreutils/9.4/bin/ptx /usr/local/Cellar/coreutils/9.4/bin/ghead /usr/local/Cellar/coreutils/9.4/bin/glink /usr/local/Cellar/coreutils/9.4/bin/gstat /usr/local/Cellar/coreutils/9.4/bin/gmktemp /usr/local/Cellar/coreutils/9.4/bin/gyes /usr/local/Cellar/coreutils/9.4/bin/gsha1sum /usr/local/Cellar/coreutils/9.4/bin/b2sum /usr/local/Cellar/coreutils/9.4/bin/grm /usr/local/Cellar/coreutils/9.4/bin/gsha256sum /usr/local/Cellar/coreutils/9.4/bin/gfalse /usr/local/Cellar/coreutils/9.4/bin/gwho /usr/local/Cellar/coreutils/9.4/bin/gcut /usr/local/Cellar/coreutils/9.4/bin/gvdir /usr/local/Cellar/coreutils/9.4/bin/gdir /usr/local/Cellar/coreutils/9.4/bin/gchmod /usr/local/Cellar/coreutils/9.4/bin/gbase32 /usr/local/Cellar/coreutils/9.4/bin/sha224sum /usr/local/Cellar/coreutils/9.4/bin/ghostid /usr/local/Cellar/coreutils/9.4/bin/gnohup /usr/local/Cellar/coreutils/9.4/bin/gtr /usr/local/Cellar/coreutils/9.4/bin/gdirname /usr/local/Cellar/coreutils/9.4/bin/gsha384sum /usr/local/Cellar/coreutils/9.4/bin/gchroot /usr/local/Cellar/coreutils/9.4/bin/gpaste /usr/local/Cellar/coreutils/9.4/bin/timeout /usr/local/Cellar/coreutils/9.4/bin/tac /usr/local/Cellar/coreutils/9.4/bin/numfmt /usr/local/Cellar/coreutils/9.4/bin/gid /usr/local/Cellar/coreutils/9.4/bin/gpinky /usr/local/Cellar/coreutils/9.4/bin/genv /usr/local/Cellar/coreutils/9.4/bin/basenc /usr/local/Cellar/coreutils/9.4/bin/nproc /usr/local/Cellar/coreutils/9.4/bin/gln /usr/local/Cellar/coreutils/9.4/bin/gbasename /usr/local/Cellar/coreutils/9.4/bin/gtruncate /usr/local/Cellar/coreutils/9.4/bin/stdbuf /usr/local/Cellar/coreutils/9.4/bin/chcon /usr/local/Cellar/coreutils/9.4/bin/gcp /usr/local/Cellar/coreutils/9.4/bin/gls /usr/local/Cellar/coreutils/9.4/bin/factor /usr/local/Cellar/coreutils/9.4/bin/gtrue /usr/local/Cellar/coreutils/9.4/bin/gchown /usr/local/Cellar/coreutils/9.4/bin/gsync /usr/local/Cellar/coreutils/9.4/bin/guptime /usr/local/Cellar/coreutils/9.4/bin/gsum /usr/local/Cellar/coreutils/9.4/bin/gtac /usr/local/Cellar/coreutils/9.4/bin/gexpand /usr/local/Cellar/coreutils/9.4/bin/gruncon /usr/local/Cellar/coreutils/9.4/bin/gpathchk /usr/local/Cellar/coreutils/9.4/bin/gnice /usr/local/Cellar/coreutils/9.4/bin/gecho /usr/local/Cellar/coreutils/9.4/bin/gdu /usr/local/Cellar/coreutils/9.4/bin/gb2sum /usr/local/Cellar/coreutils/9.4/bin/gtouch /usr/local/Cellar/coreutils/9.4/bin/gmkfifo /usr/local/Cellar/coreutils/9.4/bin/gdf /usr/local/Cellar/coreutils/9.4/bin/gjoin /usr/local/Cellar/coreutils/9.4/bin/gtest /usr/local/Cellar/coreutils/9.4/bin/gmd5sum /usr/local/Cellar/coreutils/9.4/bin/gunexpand /usr/local/Cellar/coreutils/9.4/bin/gsort /usr/local/Cellar/coreutils/9.4/bin/gshuf /usr/local/Cellar/coreutils/9.4/bin/gfmt /usr/local/Cellar/coreutils/9.4/bin/gunlink /usr/local/Cellar/coreutils/9.4/bin/gcsplit /usr/local/Cellar/coreutils/9.4/bin/g[ /usr/local/Cellar/coreutils/9.4/bin/gwhoami /usr/local/Cellar/coreutils/9.4/bin/gsplit /usr/local/Cellar/coreutils/9.4/bin/gseq /usr/local/Cellar/coreutils/9.4/bin/sha1sum /usr/local/Cellar/coreutils/9.4/bin/sha256sum /usr/local/Cellar/coreutils/9.4/bin/gdircolors /usr/local/Cellar/coreutils/9.4/bin/ginstall /usr/local/Cellar/coreutils/9.4/bin/gsha224sum /usr/local/Cellar/coreutils/9.4/bin/shred /usr/local/Cellar/coreutils/9.4/bin/sha384sum /usr/local/Cellar/coreutils/9.4/bin/gcomm /usr/local/Cellar/coreutils/9.4/bin/gtty /usr/local/Cellar/coreutils/9.4/bin/gcksum /usr/local/Cellar/coreutils/9.4/bin/gexpr /usr/local/Cellar/coreutils/9.4/bin/gbase64 /usr/local/Cellar/coreutils/9.4/bin/gwc /usr/local/Cellar/coreutils/9.4/bin/gnproc /usr/local/Cellar/coreutils/9.4/bin/base32 /usr/local/Cellar/coreutils/9.4/bin/gptx /usr/local/Cellar/coreutils/9.4/bin/gtimeout /usr/local/Cellar/coreutils/9.4/bin/pinky /usr/local/Cellar/coreutils/9.4/bin/hostid /usr/local/Cellar/coreutils/9.4/bin/gpwd /usr/local/Cellar/coreutils/9.4/bin/gtail /usr/local/Cellar/coreutils/9.4/bin/gchcon /usr/local/Cellar/coreutils/9.4/bin/glogname /usr/local/Cellar/coreutils/9.4/bin/guname /usr/local/Cellar/coreutils/9.4/bin/gtee /usr/local/Cellar/coreutils/9.4/bin/gstty /usr/local/Cellar/coreutils/9.4/bin/gchgrp /usr/local/Cellar/coreutils/9.4/bin/gcat /usr/local/Cellar/coreutils/9.4/bin/ggroups /usr/local/Cellar/coreutils/9.4/bin/gsleep /usr/local/Cellar/coreutils/9.4/bin/sha512sum /usr/local/Cellar/coreutils/9.4/bin/gfactor /usr/local/Cellar/coreutils/9.4/bin/god /usr/local/Cellar/coreutils/9.4/bin/gprintf /usr/local/Cellar/coreutils/9.4/bin/gstdbuf /usr/local/Cellar/coreutils/9.4/bin/gnumfmt /usr/local/Cellar/coreutils/9.4/bin/gbasenc /usr/local/Cellar/coreutils/9.4/bin/gdate /usr/local/Cellar/coreutils/9.4/libexec/gnubin/tee /usr/local/Cellar/coreutils/9.4/libexec/gnubin/md5sum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/split /usr/local/Cellar/coreutils/9.4/libexec/gnubin/cat /usr/local/Cellar/coreutils/9.4/libexec/gnubin/shuf /usr/local/Cellar/coreutils/9.4/libexec/gnubin/mkfifo /usr/local/Cellar/coreutils/9.4/libexec/gnubin/pathchk /usr/local/Cellar/coreutils/9.4/libexec/gnubin/runcon /usr/local/Cellar/coreutils/9.4/libexec/gnubin/expand /usr/local/Cellar/coreutils/9.4/libexec/gnubin/tty /usr/local/Cellar/coreutils/9.4/libexec/gnubin/basename /usr/local/Cellar/coreutils/9.4/libexec/gnubin/install /usr/local/Cellar/coreutils/9.4/libexec/gnubin/nice /usr/local/Cellar/coreutils/9.4/libexec/gnubin/truncate /usr/local/Cellar/coreutils/9.4/libexec/gnubin/echo /usr/local/Cellar/coreutils/9.4/libexec/gnubin/du /usr/local/Cellar/coreutils/9.4/libexec/gnubin/ptx /usr/local/Cellar/coreutils/9.4/libexec/gnubin/join /usr/local/Cellar/coreutils/9.4/libexec/gnubin/df /usr/local/Cellar/coreutils/9.4/libexec/gnubin/pwd /usr/local/Cellar/coreutils/9.4/libexec/gnubin/test /usr/local/Cellar/coreutils/9.4/libexec/gnubin/csplit /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sort /usr/local/Cellar/coreutils/9.4/libexec/gnubin/whoami /usr/local/Cellar/coreutils/9.4/libexec/gnubin/touch /usr/local/Cellar/coreutils/9.4/libexec/gnubin/unlink /usr/local/Cellar/coreutils/9.4/libexec/gnubin/b2sum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sleep /usr/local/Cellar/coreutils/9.4/libexec/gnubin/fmt /usr/local/Cellar/coreutils/9.4/libexec/gnubin/stty /usr/local/Cellar/coreutils/9.4/libexec/gnubin/logname /usr/local/Cellar/coreutils/9.4/libexec/gnubin/chgrp /usr/local/Cellar/coreutils/9.4/libexec/gnubin/printenv /usr/local/Cellar/coreutils/9.4/libexec/gnubin/seq /usr/local/Cellar/coreutils/9.4/libexec/gnubin/uname /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha224sum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/od /usr/local/Cellar/coreutils/9.4/libexec/gnubin/date /usr/local/Cellar/coreutils/9.4/libexec/gnubin/base64 /usr/local/Cellar/coreutils/9.4/libexec/gnubin/realpath /usr/local/Cellar/coreutils/9.4/libexec/gnubin/readlink /usr/local/Cellar/coreutils/9.4/libexec/gnubin/dircolors /usr/local/Cellar/coreutils/9.4/libexec/gnubin/timeout /usr/local/Cellar/coreutils/9.4/libexec/gnubin/tac /usr/local/Cellar/coreutils/9.4/libexec/gnubin/numfmt /usr/local/Cellar/coreutils/9.4/libexec/gnubin/wc /usr/local/Cellar/coreutils/9.4/libexec/gnubin/basenc /usr/local/Cellar/coreutils/9.4/libexec/gnubin/comm /usr/local/Cellar/coreutils/9.4/libexec/gnubin/nproc /usr/local/Cellar/coreutils/9.4/libexec/gnubin/expr /usr/local/Cellar/coreutils/9.4/libexec/gnubin/stdbuf /usr/local/Cellar/coreutils/9.4/libexec/gnubin/cksum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/printf /usr/local/Cellar/coreutils/9.4/libexec/gnubin/groups /usr/local/Cellar/coreutils/9.4/libexec/gnubin/chcon /usr/local/Cellar/coreutils/9.4/libexec/gnubin/factor /usr/local/Cellar/coreutils/9.4/libexec/gnubin/tail /usr/local/Cellar/coreutils/9.4/libexec/gnubin/env /usr/local/Cellar/coreutils/9.4/libexec/gnubin/pr /usr/local/Cellar/coreutils/9.4/libexec/gnubin/head /usr/local/Cellar/coreutils/9.4/libexec/gnubin/kill /usr/local/Cellar/coreutils/9.4/libexec/gnubin/uniq /usr/local/Cellar/coreutils/9.4/libexec/gnubin/stat /usr/local/Cellar/coreutils/9.4/libexec/gnubin/link /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/tsort /usr/local/Cellar/coreutils/9.4/libexec/gnubin/mknod /usr/local/Cellar/coreutils/9.4/libexec/gnubin/users /usr/local/Cellar/coreutils/9.4/libexec/gnubin/dd /usr/local/Cellar/coreutils/9.4/libexec/gnubin/who /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha1sum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/mktemp /usr/local/Cellar/coreutils/9.4/libexec/gnubin/cut /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha256sum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/dir /usr/local/Cellar/coreutils/9.4/libexec/gnubin/mkdir /usr/local/Cellar/coreutils/9.4/libexec/gnubin/nl /usr/local/Cellar/coreutils/9.4/libexec/gnubin/shred /usr/local/Cellar/coreutils/9.4/libexec/gnubin/fold /usr/local/Cellar/coreutils/9.4/libexec/gnubin/rmdir /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha384sum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/mv /usr/local/Cellar/coreutils/9.4/libexec/gnubin/dirname /usr/local/Cellar/coreutils/9.4/libexec/gnubin/id /usr/local/Cellar/coreutils/9.4/libexec/gnubin/base32 /usr/local/Cellar/coreutils/9.4/libexec/gnubin/pinky /usr/local/Cellar/coreutils/9.4/libexec/gnubin/ln /usr/local/Cellar/coreutils/9.4/libexec/gnubin/hostid /usr/local/Cellar/coreutils/9.4/libexec/gnubin/chroot /usr/local/Cellar/coreutils/9.4/libexec/gnubin/ls /usr/local/Cellar/coreutils/9.4/libexec/gnubin/true /usr/local/Cellar/coreutils/9.4/libexec/gnubin/cp /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sync /usr/local/Cellar/coreutils/9.4/libexec/gnubin/yes /usr/local/Cellar/coreutils/9.4/libexec/gnubin/unexpand /usr/local/Cellar/coreutils/9.4/libexec/gnubin/chown /usr/local/Cellar/coreutils/9.4/libexec/gnubin/chmod /usr/local/Cellar/coreutils/9.4/libexec/gnubin/uptime /usr/local/Cellar/coreutils/9.4/libexec/gnubin/rm /usr/local/Cellar/coreutils/9.4/libexec/gnubin/vdir /usr/local/Cellar/coreutils/9.4/libexec/gnubin/false /usr/local/Cellar/coreutils/9.4/libexec/gnubin/sha512sum /usr/local/Cellar/coreutils/9.4/libexec/gnubin/[ /usr/local/Cellar/coreutils/9.4/libexec/gnubin/tr /usr/local/Cellar/coreutils/9.4/libexec/gnubin/paste /usr/local/Cellar/coreutils/9.4/libexec/gnubin/nohupusage
$ bc -l <<< 'obase=2;0;0;15;255' 0 0 1111 11111111 $ bc -l <<< 'obase=2;0;0;15;255' | numfmt --format=%08f 00000000 00000000 00001111 11111111 $ bc -l <<< 'obase=2;0;0;15;255' | numfmt --format=%08f | xargs 00000000 00000000 00001111 11111111 $ numfmt --to=si --format "%f bottles of beer on the wall" 99999999 100M bottles of beer on the wallconvert format
$ echo 1G | numfmt --from=si 1000000000 $ echo 1G | numfmt --from=iec 1073741824 $ echo 500G | numfmt --from=si --to=iec 466G $ numfmt --field=2 --from-unit=1024 --to=iec-i --suffix B < /proc/meminfo | sed 's/ kB//' | head -n4 MemTotal: 1008GiB MemFree: 816GiB MemAvailable: 941GiB Buffers: 3.1MiB $ watch -n.1 \ > 'numfmt --header --field=2 --to=iec-i --round=nearest < /proc/interrupts | > LC_ALL=en_US numfmt --header --field=3 --group --invalid=ignore --padding=16 | > pr -TW$COLUMNS' $ for method in up down nearest; do > echo $method > numfmt --to=iec --round=$method 4095 4096 4097 > done | paste - - - - up 4.0K 4.0K 4.1K down 3.9K 4.0K 4.0K nearest 4.0K 4.0K 4.0Kpadding
$ du -s * | numfmt --to=si --padding=10 12 awk.md 40 character.md 4 html.md 16 json.md 8 markdown.md 4 regex.md 16 sed.md $ du -s * | numfmt --to=si --padding=-10 12 awk.md 40 character.md 4 html.md 16 json.md 8 markdown.md 4 regex.md 16 sed.mdfield
$ ls -l total 100 -rw-r--r-- 1 marslo staff 9720 Sep 7 00:37 awk.md -rw-r--r-- 1 marslo staff 40500 Sep 7 00:51 character.md $ ls -l | numfmt --field 5 --to=si total 100 -rw-r--r-- 1 marslo staff 9.8K Sep 7 00:37 awk.md -rw-r--r-- 1 marslo staff 41K Sep 7 00:51 character.md $ df -B1 | head -3 Filesystem 1B-blocks Used Available Use% Mounted on devtmpfs 540881096704 0 540881096704 0% /dev tmpfs 540899667968 258048 540899409920 1% /dev/shm $ df -B1 | head -3 | numfmt --header --field 2-4 --to=si Filesystem 1B-blocks Used Available Use% Mounted on devtmpfs 541G 0 541G 0% /dev tmpfs 541G 259K 541G 1% /dev/shm
[!NOTE|label:references:]
$ git log --author="marslo" --format=tformat: --numstat | q -t "select sum(c1), sum(c2) from -"
60650.0 66363.0combinations
single line to multiple lines
[!TIP]
$ echo 'a b c' a b c
xargs -n<x>$ echo 'a b c' | xargs -n1 a b c $ echo {a..c}.{1..2} | xargs -n1 | xargs -I{} echo -{}- -a.1- -a.2- -b.1- -b.2- -c.1- -c.2-$ echo 'a b c' | fmt -1 a b c $ echo {a..c}.{1..2} | fmt -1 | xargs -I{} echo -{}- -a.1- -a.2- -b.1- -b.2- -c.1- -c.2-$ echo 'a b c' | awk '{ OFS=RS; $1=$1 }1' a b ctr$ echo 'a b c' | tr -s ' ' '\n' a b c$ printf '%s\n' a b c a b c
execute commands from file
[!TIP]
precondition
$ cat a.txt a b c
$ echo 'a b c' | xargs -n1 -t touch touch a touch b touch c $ echo 'a b c' | xargs -n1 -p touch touch a?...y touch b?...y touch c?...y-t, --verbose Print the command line on the standard error output before executing it. -p, --interactive Prompt the user about whether to run each command line and read a line from the terminal. Only run the command line if the response starts with `y' or `Y'. Implies -t. -I replace-str Replace occurrences of replace-str in the initial-arguments with names read from standard in- put. Also, unquoted blanks do not terminate input items; instead the separator is the new- line character. Implies -x and -L 1.
combine every 2 lines
[!NOTE|label:references:]
sample output
$ echo -e "1\na\n2\nb\n3\nc 1 a 2 b 3 calso using for sed output :
$ git --no-pager log -3 --no-color | sed -nr 's!^commit\s*(.+)$!\1!p; s!^\s*Change-Id:\s*(.*$)!\1!p' d9a9adfb591bb129d6b1af9532fea0fcf069b176 I5d99e9ccd4edbba608e7da70e575fd6bd091ce42 7aed870eeb203336e5e29b03714905f28ec3e60d Ie3b4d5fd09a43385a44282a2e6961e220ae6293a 09a652f7c78f416da7a561451ed274f930c27dec I244d2fec5d25e453fd30d08d1c75c16143b7f7a3
xargs
$ echo -e "1\na\n2\nb\n3\nc" | xargs -n2 -d'\n' 1 a 2 b 3 cpaste
$ echo -e "1\na\n2\nb\n3\nc" | paste -s -d',\n' 1,a 2,b 3,c $ echo -e "1\na\n2\nb\n3\nc" | paste -d " " - - 1 a 2 b 3 csed
$ echo -e "1\na\n2\nb\n3\nc" | sed 'N;s/\n/ : /' 1 : a 2 : b 3 : cawk
$ echo -e "1\na\n2\nb\n3\nc" | awk '{ key=$0; getline; print key " : " $0; }' 1 : a 2 : b 3 : c # or $ echo -e "1\na\n2\nb\n3\nc" | awk 'ORS=NR%2?FS:RS' 1 a 2 b 3 c # or $ echo -e "1\na\n2\nb\n3\nc" | awk 'NR%2{ printf "%s : ",$0;next; }1' 1 : a 2 : b 3 : c # or $ echo -e "1\na\n2\nb\n3\nc" | awk '{ if ( NR%2 != 0 ) line=$0; else { printf("%s : %s\n", line, $0); line=""; } } END { if ( length(line) ) print line; }' 1 : a 2 : b 3 : cwhile
$ echo -e "1\na\n2\nb\n3\nc" | while read line1; do read line2; echo "$line1 : $line2"; done 1 : a 2 : b 3 : c
combine every 3 lines
paste
# or every 3 lines $ echo -e "1\na\n2\nb\n3\nc" | paste -d ' ' - - - 1 a 2 b 3 cawk
$ echo -e "1\na\n2\nb\n3\nc" | awk 'NR%3{ printf "%s : ",$0;next; }1' 1 : a : 2 b : 3 : cxargs
$ echo {1..9} | fmt -1 | xargs -n3 1 2 3 4 5 6 7 8 9
format output
[!TIP|label:sample data]
$ paste <(sort a.txt) <(sort b.txt) | expand --tabs=10 a a b b d c f d e $ pr -w 30 -m -t a.txt b.txt a a b b d c f d e
$ paste -d ',:' file1 file2 file3
echo
[!TIP]
$ echo -e "a\t\tb"
a b
$ echo -e $(echo -e "a\t\tb")
a b
print file with ansicolor
$ command cat c.txt get get/exec get/subresource list create create/exec get/subresource \e[32;1myes\e[0m \e[32;1myes\e[0m \e[32;1myes\e[0m \e[32;1myes\e[0m \e[32;1myes\e[0m no no $ echo -ne $(command cat c.txt | sed 's/$/\\n/' | sed 's/ /\\a /g') get get/exec get/subresource list create create/exec get/subresource yes yes yes yes yes no no $ echo -ne $(command cat c.txt | sed 's/$/\\n/' | sed 's/ /\\033 /g') get get/exec get/subresource list create create/exec get/subresource yes yes yes yes yes no no
echo -ne file with ansicolor
diff
[!NOTE|label:references:]
show all status
$ diff --side-by-side <(sort a.txt) <(sort b.txt) a a b b > c d d f | eshow diff only
$ diff --suppress-common-lines --side-by-side <(sort a.txt) <(sort b.txt) > c f | eshow diff with
--<GTYPE>-group-formatVARIABLEAPPLICABLEoldGTYPE,LTYPEnewGTYPE,LTYPEunchangedGTYPE,LTYPEchangedGTYPE[!NOTE|label:tips:]
[G]TYPE:[g]roup:--<GTYPE>-group-format
[L]TYPE:[l]ine:--<LTYPE>-line-format
$ diff --old-group-format="L %<" --new-group-format="R %>" --unchanged-group-format="" a.txt b.txt R c L f R e # with line number $ diff --unchanged-line-format="" --old-line-format="< %dn: %L" --new-line-format="> %dn: %L" <(sort a.txt) <(sort b.txt) > 3: c < 4: f > 5: e # beging-end $ diff --old-group-format='\begin{em} %<\end{em} ' --new-group-format='\begin{bf} %>\end{bf} ' --changed-group-format='\begin{em} %<\end{em} \begin{bf} %>\end{bf} ' --unchanged-group-format='%=' \ <(sort a.txt) <(sort b.txt) a b \begin{bf} c \end{bf} d \begin{em} f \end{em} \begin{bf} e \end{bf} $ diff \ -> --unchanged-group-format='' \ -> --old-group-format='-------- %dn line%(n=1?:s) deleted at %df: %<' \ -> --new-group-format='-------- %dN line%(N=1?:s) added after %de: %>' \ -> --changed-group-format='-------- %dn line%(n=1?:s) changed at %df: %<-------- to: %>' \ -> <(sort a.txt) <(sort b.txt) -------- 1 line added after 2: c -------- 1 line changed at 4: f -------- to: e
show common
$ diff --unchanged-line-format="%L" --new-line-format="" --old-line-format="" <(sort a.txt) <(sort b.txt) a b dcreate patch
[!NOTE|label:references:]
Is this a good way to create a patch?
$ diff -Naru file_original file_updated > file.patch
$ diff -c <(sort a.txt) <(sort b.txt) *** /dev/fd/63 2023-09-12 21:51:50.828885643 -0700 --- /dev/fd/62 2023-09-12 21:51:50.829641102 -0700 *************** *** 1,4 **** a b d ! f --- 1,5 ---- a b + c d ! e $ diff -u <(sort a.txt) <(sort b.txt) --- /dev/fd/63 2023-09-12 21:51:53.561211803 -0700 +++ /dev/fd/62 2023-09-12 21:51:53.561824746 -0700 @@ -1,4 +1,5 @@ a b +c d -f +e $ diff -i <(sort a.txt) <(sort b.txt) 2a3 > c 4c5 < f --- > e$ diff <(cd dir1 && find | sort) <(cd dir2 && find | sort)$ colordiff -yW"`tput cols`" /path/to/file1 /path/to/file2
comm
diff
$ comm -3 a.txt b.txt c e f $ comm -3 <(sort a.txt) <(sort b.txt) | column -t -s $'\t' --table-columns '==== a.txt ====,==== b.txt ====' ==== a.txt ==== ==== b.txt ==== c e fcommon
$ comm -12 <(sort a.txt) <(sort b.txt) a b d
join
[!NOTE|label:references:]
function join_by {
local d=${1-} f=${2-}
if shift 2; then
printf %s "$f" "${@/#/$d}"
fi
}$ foo=( a "b c" d ) $ printf -v joined '%s,' "${foo[@]}" $ echo "${joined%,}" a,b c,d $ printf -v joined '|,%s,|' "${foo[@]}" $ echo "${joined%,}" |,a,||,b c,||,d,|# join by single char $ foo=(a "b c" d) $ bar=$(IFS=, ; echo "${foo[*]}") $ echo "$bar" a,b c,d
alignment
[!TIP]
column (BSD)
[!NOTE|label:references:]
$ ( printf "PERM LINKS OWNER GROUP SIZE MONTH DAY HH:MM/YEAR NAME\n"; ls -l | sed 1d ) | column -t
PERM LINKS OWNER GROUP SIZE MONTH DAY HH:MM/YEAR NAME
-rw-r--r-- 1 marslo staff 8 Sep 12 20:10 a.txt
-rw-r--r-- 1 marslo staff 10 Sep 12 19:34 b.txt
$ paste <(echo -e "foo\n\nbarbarbar") <(seq 3) | column -t
foo 1
2
barbarbar 3
$ paste <(echo -e "foo\n\nbarbarbar") <(seq 3) | column -t -s $'\t'
foo 1
2
barbarbar 3with header
$ paste <(echo -e "foo\n\nbarbarbar") <(seq 3) | column -t -s $'\t' --table-columns '====LEFT====,====RIGHT====' ====LEFT==== ====RIGHT==== foo 1 2 barbarbar 3$ echo -e 'a very long string..........\t112232432\tanotherfield\na smaller string\t123124343\tanotherfield\n' | column -t -s $'\t' a very long string.......... 112232432 anotherfield a smaller string 123124343 anotherfield
sort
[!NOTE|label:references:]
sort the last column
awk:
print( $NF" "$0 ) | sort | cut -f2- -d' '$ echo -e '5 5 0 0 622 20\n6 3 2 0 439 8\n5 2 3 0 450 8' 5 5 0 0 622 20 6 3 2 0 439 8 5 2 3 0 450 12 $ echo -e '5 5 0 0 622 20\n6 3 2 0 439 8\n5 2 3 0 450 12' | awk '{print($NF" "$0)}' | sort -k1,1 -n -r -t' ' | cut -f2- -d' ' 5 5 0 0 622 20 5 2 3 0 450 12 6 3 2 0 439 8awk: similar with rev for words
$ echo -e '5 5 0 0 622 20\n6 3 2 0 439 8\n5 2 3 0 450 12' | awk '{ for (i=NF; i>0; i--) printf("%s ",$i); printf("\n")}' | # rev sort -k1,1 -nr -t' ' | awk '{ for (i=NF; i>0; i--) printf("%s ",$i); printf("\n")}' # rev 5 5 0 0 622 20 5 2 3 0 450 12 6 3 2 0 439 8
get lines
get second-to-last line
[!NOTE|label:references:]
sed
$ sed -n 'x;$p' <<\INPUT a b c d INPUT c # or $ echo -e 'a\nb\nc\nd' | sed -n -e '${x;1!p;};h' c # tac + sed $ echo -e 'a\nb\nc\nd' | tac | sed -n '2p' ctail & head
$ echo -e 'a\nb\nc\nd' | tail -2 | head -1 c
get next line by the pattern
$ cat a.txt
1a
2b
3c * (pattern /3c/)
4d > (wanted)
5e > (wanted)
6f
7gawk
$ cat a.txt | awk '$0=="3c"{getline; print; getline; print}' 4d 5e $ cat a.txt | awk '/3c/{getline; print; getline; print}' 4d 5eor
$ cat a.txt | awk '/^3c$/ {s=NR;next} s && NR<=s+2' 4d 5eor
$ cat a.txt | awk '{if(a-->0){print;next}} /3c/{a=2}' 4d 5eor get second column of next line of pattern
$ awk '/company.domain.com$/{getline; print}' ~/.marslo/.netrc login marslo $ awk '/company.domain.com$/{getline; print $2}' ~/.marslo/.netrc marslo
sed
$ cat a.txt | sed -n '/3c/{n;p;n;p}' 4d 5e $ cat a.txt | sed -n '/3c/{N;p;n;p}' 3c 4d 5eto get docker registry mirrors
# exclude the pattern $ docker system info | sed -n '/Registry Mirrors:/{n;p;}' https://artifactory.domain.com/ # including pattern $ docker system info | sed -n '/Registry Mirrors:/{p;n;p;}' Registry Mirrors: https://artifactory.domain.com/
change next line of pattern
[!NOTE|label:references:]
Find Matching Text and Replace the Next Line
$ cat revenue.txt total 4000 dollars' revenue - Quarter 1: Revenue: 1200 dollars; Profit: 700 dollars - Quarter 2: Revenue: 1000 dollars; Profit: 650 dollars - Quarter 3: Revenue: 1200 dollars; Profit: 800 dollars - Quarter 4: Revenue: 600 dollars; Profit: -200 dollars Profit: 1950 dollars
replace
dollarsto$right after line of/Quarter [1-4]/sed
# ╭╴ next $ sed '/Quarter [1-4]:/{ n; s/dollars/$/g }' revenue.txt total 4000 dollars' revenue - Quarter 1: Revenue: 1200 $; Profit: 700 $ - Quarter 2: Revenue: 1000 $; Profit: 650 $ - Quarter 3: Revenue: 1200 $; Profit: 800 $ - Quarter 4: Revenue: 600 $; Profit: -200 $ Profit: 1950 dollarsawk
# next # ╭╴╴╴╴╮ $ awk '/Quarter [1-4]:/{ rl = NR + 1 } NR == rl { gsub( /dollars/,"$") } 1' total 4000 dollars' revenue - Quarter 1: Revenue: 1200 $; Profit: 700 $ - Quarter 2: Revenue: 1000 $; Profit: 650 $ - Quarter 3: Revenue: 1200 $; Profit: 800 $ - Quarter 4: Revenue: 600 $; Profit: -200 $ Profit: 1950 dollars
replace
dollarsto$every 3 lines after/Quarter [1-4]/# ╭╴ next # ╷ ╭╴ next # ╷ ╷ ╭╴ next $ sed '/Quarter [1-4]:/{ n;n;n; s/dollars/$/g }' revenue.txt total 4000 dollars' revenue - Quarter 1: Revenue: 1200 dollars; Profit: 700 dollars - Quarter 2: Revenue: 1000 $; Profit: 650 $ - Quarter 3: Revenue: 1200 dollars; Profit: 800 dollars - Quarter 4: Revenue: 600 $; Profit: -200 $ Profit: 1950 dollars
get lines between 2 patterns
[!NOTE|label:reference:]
[!TIP] sample data:
$ cat a.txt 1a 2b 3c * (start) 4d 5e 6f 7g 8h * (end) 9i 10j 11k
### awk
- include pattern
```bash
$ cat a.txt | awk '/3c/,/8h/'
3c
4d
5e
6f
7g
8hsed
[!NOTE|label:references:]
include all patterns
$ cat a.txt | sed -n '/3c/,/8h/p' 3c 4d 5e 6f 7g 8hexclude both patterns
$ sed -n '/3c/,/8h/{//!p;}' a.txt 4d 5e 6f 7g $ sed -n '/3c/,/8h/{/3c/!{/8h/!p}}' a.txt 4d 5e 6f 7g # + delete from line 1 to /3c/ # | + delete /8h/ to end `$` # +-----+ +-----+ $ cat a.txt | sed '1,/3c/d;/8h/,$d' 4d 5e 6f 7g # + not delete since `/3c/` to `/8h` # | + delete all the others # +---------+ +-+ $ cat a.txt | sed '/3c/,/8h/!d;//d' 4d 5e 6f 7gexclude single pattern
$ sed -n '/3c/,/8h/{/8h/!p}' a.txt 3c 4d 5e 6f 7g $ sed -n '/3c/,/8h/{/3c/!p}' a.txt 4d 5e 6f 7g 8h
with empty line
[!NOTE]
use case :
$ cat a.txt
1a
2b
3c * (start)
4d
5e
6f
* (ending)
7g
8h
9i
10j
11k
$ cat a.txt | sed -n '/3c/,/^$/p'
3c
4d
5e
6fget line from pattern to the end
[!TIP|label:references:]
sample content:
$ echo -e '1\n2\n\n3\n4' 1 2 * (start. pattern: `^\s*$`) 3 4
get from first empty line to the end
[!NOTE|label:references:]
including pattern
[!TIP]
solution: to print from pattern to end
,$==,$pfor both CRLF and LF
$ echo -e '1\n2\n\n3\n4' | sed -n '/^\s*$/,$p' 3 4sed
[!TIP]
solution: using line number to end:
n,$->"n"',$p'head -n1: for first matches pattern line numbertail -n1: for the last matches pattern line number
$ command cat -A a | nl 1 1^M$ 2 ^M$ 3 2^M$ 4 3^M$ 5 4^M$ $ cat a | sed -n "$(sed -n '/^\s*$/ =' a | tail -n1)"' ,$p' 2 3 4 # or $ echo -e '1\n\n2\n\n3\n4' | sed -n '/^\s*$/h;/^\s*$/!H;$!b;x;p' 3 4awk
$ echo -e '1\n\n2\n\n3\n4' | awk '/^\s*$/,0' 2 3 4
not including pattern
[!TIP]
solution: to delete / not print from first line to pattern
delete:
/dnot print: -n
/!p
for both CRLF and LF
# not print # `-n` + from 1st line to line of pattern `/^\s*$/` # ^ +------+ + not print $ echo -e '1\n2\n\n3\n4' | sed -n '1,/^\s*$/!p' 3 4 # delete # + from 1st line to line of pattern `/^\s*$/` # | + delete # +------+ | $ echo -e '1\n2\n\n3\n4' | sed '1,/^\s*$/d' 3 4 # + not delete since `/^\s*$/` to end # | + delete the others # +---------+ +-+ $ echo -e '1\n\n2\n\n3\n4' | sed '/^\s*$/,$!d;//d' 2 3 4[!TIP]
solution: with matched line number + 1 :
"$(( n+1 ))"',$p'head -n1: for first matches pattern line numbertail -n1: for the last matches pattern line number
$ command cat -A a | nl 1 1^M$ 2 ^M$ 3 2^M$ 4 3^M$ 5 4^M$ $ cat a | sed -n "$(( $(sed -n '/^\s*$/ =' a | head -n1 )+1 ))"' ,$p' 2 3 4
get from last empty line ( ^$ ) to end
^$ ) to end[!NOTE|label:references:]
awk
# awk ## LF $ echo -e '1\n\n2\n3\n4' | awk -v RS='\n\n' 'END{printf "%s",$0}' 2 3 4 ## CRLF $ echo -e '1\r\n\r\n2\r\n3\r\n4\r' | awk -v RS='\r\n\r\n' 'END{printf "%s",$0}' 2 3 4 # or $ echo -e '1\n\n2\n\n3\n4' | awk '/^\s*$/ { buf = "" } { buf = buf "\n" $0 } END { print buf }' | sed 1d 3 4tac + awk
# tac + awk ## LF $ echo -e '1\n\n2\n3\n4' | tac | awk '/^$/{exit}1' | tac 2 3 4 ## CRLF $ echo -e '1\r\n\r\n2\r\n3\r\n4\r' | tac | awk '/^\s*\r$/{exit}1' | tac 2 3 4sed
[!WARNING]
for LF only, not support CRLF
\r
$ echo -e '1\n\n2\n\n3\n4' 1 2 3 4 $ echo -e '1\n\n2\n\n3\n4' | sed -n '/^\s*$/{g;D;}; N; $p;' 3 4$ echo -e '1\n\n2\n\n3\n4' | sed -n '/^\s*$/{h;b};H;${x;p}' 3 4 # or $ echo -e '1\n\n2\n\n3\n4' | sed -n '/^\s*$/h;/^\s*$/!H;$!b;x;p' 3 4 # or $ echo -e '1\n\n2\n\n3\n4' | sed -n 'H; /^\s*$/h; ${g;p;}' 3 4
reverse search empty line
[!NOTE|label:for show TODO]
# without reverse search
$ fd -tf --color never |
xargs -r -I{} bash -c "sed -ne '/TODO:/,/^\s*$/p' {} | bat -l groovy"
# to suppress output if stdin for `bat`
$ while read -r file; do
_content=$(sed -ne '/TODO:/,/^\s*$/p' "${file}");
[[ -n "${_content}" ]] && echo "${_content}" | bat -l groovy;
done < <(fd -tf --color never)return first matching pattern
{% hint style='tip' %}
references:
[!TIP]
$ cat sample.crt -----BEGIN CERTIFICATE----- first paragraph -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- second paragraph -----END CERTIFICATE-----
sed
$ cat sample.crt | sed '/-END CERTIFICATE-/q'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----
# or `-n /../p`
# `-n` `p`
# | |
# v v
$ cat sample.crt | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p; /-END CERTIFICATE-/q'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----
# or `/../!d`
# no `-n` `!d`
# | |
# v v
$ cat sample.crt | sed '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/!d; /-END CERTIFICATE-/q'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----awk
$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/{a=1}; a; /-END CERTIFICATE-/{exit}'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----
# or
$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/ {print} /-END CERTIFICATE-/ {exit}'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----
# or
$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/ {print;f=1} f&&/-END CERTIFICATE-/ {exit}'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----
# or
$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/ {f=1} /-END CERTIFICATE-/ {f=0;print;exit} f'
-----BEGIN CERTIFICATE-----
first paragraph
-----END CERTIFICATE-----return second matching pattern search range
{% hint style='tip' %}
references:
Copying second occurrence pattern to a new file in sed {% endhint %}
[!TIP]
$ cat sample.crt -----BEGIN CERTIFICATE----- first paragraph -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- second paragraph -----END CERTIFICATE-----
$ cat sample.crt | awk '/-BEGIN CERTIFICATE-/ && c++, /-END CERTIFICATE-/'
-----BEGIN CERTIFICATE-----
second paragraph
-----END CERTIFICATE-----return the last matching pattern search range
[!NOTE|label:references:]
$ cat a
pattern
1
2
3
pattern
4
5
pattern * (start: the last pattern)
6
7
8
9
* (end)
10
11sed
$ sed -ne '
/pattern/{
$d;n
:loop
s/\n$//;tdone
$bdone;N
bloop
:done
x
}
${x;/./p;}
' a
6
7
8
9
# or
$ sed -e '
/pattern/,/^$/!ba
/./!ba
H;/pattern/{z;x;}
:a
$!d;x;s/.//
' a
6
7
8
9
# or GNU sed
$ sed -Ez '
s/.*pattern\n(([^\n]+\n)+)(\n.*)?/\1/
' a
6
7
8
9awk
$ awk '/pattern/,/^$/ { arr[NR]=$0; if (/pattern/) line1=NR; if (/^$/) line2=NR}END{ if (line1) for(i=++line1;i<line2;i++) print arr[i]}' a
6
7
8
9
# or
$ awk -v RS='' -F '\n' '$1 ~ /pattern/ { hold = $0 } END { if (hold != "") print hold }' a | sed 1d
6
7
8
9replace the last matching pattern
# the last `boy` -> `boys`
$ printf "%s\n" boy girl boy girl boy girl | sed -z 's/.*boy/&s/'
boy
girl
boy
girl
boys
girlxargs
xargs{% hint style='tip' %}
references:
Why does the wc utility generate multiple lines with "total"?
$ xargs --show-limits < /dev/null # solution for multiple `total` in $ `git ls-files -z | xargs -0 wc -l`: $ git ls-files -z | wc -l --files0-from=-tips:
--delimiter=delim,-d delim
-I{}==-i
-n max-args,--max-args=max-args
-t,--verbose{% endhint %}
complex commands with xargs
[!NOTE|label:references:]
$ echo ip1 ip2 ip3 ... |
fmt -1 |
xargs -i printf 'echo -e "\\n..... {} ....."; /sbin/ping -t1 -c1 -W0 {} | sed "/^$/d"\n' |
xargs -d\\n -n1 bash -c
# so xargs will execute : `echo -e "\n..... {} ....."; /sbin/ping -t1 -c1 -W0 {} | sed "/^$/d"` one by oneor using
$@$ echo ip1 ip2 ip3 ... | fmt -1 | xargs -n1 bash -c 'echo -e "\n...... $@ ......"; /sbin/ping -t1 -c1 -W0 "$@" | sed '/^$/d'' _
$ mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 -I % cp -a % ~/backups# multiple cp
$ find /path -type f -name '*~' -print0 | xargs -0 sh -c 'if [ $# -gt 0 ]; then cp -a "$@" ~/backup; fi' sh$ echo {0..9} | xargs -n 2
0 1
2 3
4 5
6 7
8 9sort all shell script by line number
[!TIP] Pipe
xargsintofind
$ find . -name "*.sh" | xargs wc -l | sort -hr
# better solution
$ find . -name "*.sh" -print0 | wc -l --files0-from=- | sort -hrdiff every git commit against its parent
$ git log --format="%H %P" | xargs -L 1 git diff[!TIP] precondition:
$ cat a.txt a b c 123 ###this is a comment
$ myCommandWithDifferentQuotes=$(cat <<'EOF'
-> echo "command 1: $@"; echo 'will you do the fandango?'; echo "command 2: $@"; echo
-> EOF
-> )
$ < a.txt xargs -I @@ bash -c "$myCommandWithDifferentQuotes" -- @@
command 1: a b c
will you do the fandango?
command 2: a b c
command 1: 123
will you do the fandango?
command 2: 123
command 1: ###this is a comment
will you do the fandango?
command 2: ###this is a commentor
$ cat a.txt | xargs -I @@ bash -c "$myCommandWithDifferentQuotes" -- @@$ while read stuff; do echo "command 1: $stuff"; echo 'will you do the fandango?'; echo "command 2: $stuff"; echo done < a.txt
compress sub-folders
$ find . -maxdepth 1 ! -path . -type d -print0 |
xargs -0 -I @@ bash -c '{ \
tar caf "@@.tar.lzop" "@@" \
&& echo Completed compressing directory "@@" ; \
}'ping multiple IPs
[!TIP]
-a file, --arg-file=file Read items from file instead of standard input. If you use this option, stdin remains unchanged when commands are run. Otherwise, stdin is redirected from /dev/null.
$ cat a.txt
8.8.8.8
1.1.1.1
$ xargs -L1 -a a.txt /sbin/ping -c 1
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=44 time=82.868 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 82.868/82.868/82.868/0.000 ms
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=63 time=1.016 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.016/1.016/1.016/0.000 msor
$ echo domain-{1..4}.com | fmt -1 | xargs -L1 ping -c1 -t1 -W0
$ printf 'mark spitz' | while read -r -n1 c; do printf "[%c]" "$c"; done
[m][a][r][k][][s][p][i][t][z]
# or
$ while read -r -n1 c; do printf "[%c]" "$c"; done <<< "mark spitz"
[m][a][r][k][][s][p][i][t][z][]find
find[!NOTE|label:reference:]
output format
[!TIP|label:man find:]
%P File's name with the name of the command line argument under which it was found removed.
%f File's name with any leading directories removed (only the last element).
# has `./` by default
$ find . -type f -iname "*cfssl*"
./cfssl/cfssl-scan
./cfssl/cfssl-certinfo
./cfssl/cfssl-bundle
./cfssl/cfssl
./cfssl/cfssl-newkey
./cfssl/cfssljson
# remove `./` by `-printf`
$ find . -type f -iname "*cfssl*" -printf '%P\n'
cfssl/cfssl-scan
cfssl/cfssl-certinfo
cfssl/cfssl-bundle
cfssl/cfssl
cfssl/cfssl-newkey
cfssl/cfssljson
# or remove `./` by sed
$ find . -type f -iname "*cfssl*" | sed 's|^./||'
cfssl/cfssl-scan
cfssl/cfssl-certinfo
cfssl/cfssl-bundle
cfssl/cfssl
cfssl/cfssl-newkey
cfssl/cfssljson
# filename only
$ find . -type f -iname "*cfssl*" -printf '%f\n'
cfssl-scan
cfssl-certinfo
cfssl-bundle
cfssl
cfssl-newkey
cfssljsonoutput file name only
# has `./` by default
$ find . -type f
./cfssl-scan
./cfssl-certinfo
./cfssl-bundle
./cfssl
./cfssl-newkey
./multirootca
./mkbundle
./cfssljson
# to show filename only by `-exec basename`
$ find . -type f -exec basename {} -print \;
cfssl-scan
cfssl-certinfo
cfssl-bundle
cfssl
cfssl-newkey
multirootca
mkbundle
cfssljson
# or
$ find . -type f -execdir basename {} ';'
cfssl-scan
cfssl-certinfo
cfssl-bundle
cfssl
cfssl-newkey
multirootca
mkbundle
cfssljsoncat config file in all .git folder
.git folderxargs&&cat$ find . -type d -name '.git' -print0 | xargs -0 -I {} cat {}/configfind&&-exec$ find . -type d -name '.git' -exec cat {}/config \;
exec and sed
exec and sedchange IP address in batch processing
$ find ${JENKINS_HOME}/jobs \ -type f \ -name "config.xml" \ -maxdepth 2 \ -exec sed -i 's/1.2.3./4.5.6./g' {} \; -print
find and rename
$ find -iname "*.sh" -exec rename "s/.sh$/.shell/" {} \; -print$ find . -regextype posix-egrep -regex ".*\.(js|vue|s?css|php|html|json)$" -and -not -regex ".*/(node_modules|vendor)/.*"$ find . -regex-type posix-extended -regex ".*def/incoming.*|.*456/incoming.*" -prune -o -print
find && tar
find && tar[!TIP] more can be found in imarslo: find and tar
backup all
config.xmlin JENKINS_HOME$ find ${JENKINS_HOME}/jobs -maxdepth 2 -name config\.xml -type f -print | tar czf ~/config.xml.tar.gz --files-from -back build history
$ find ${JENKINS_HOME}/jobs -name builds -prune -o -type f -print | tar czf ~/m.tar.gz --files-from -
find by timestamp
[!NOTE|label:references:]
How to find the difference in days between two dates?
$ echo $(( ($(date +%s -d 20210131)-$(date +%s -d 20210101))/86400 )) days 30 days # or in `%y%m%d` format $ echo $(( ($(date --date="230301" +%s) - $(date --date="240301" +%s) )/(60*60*24) )) days -366 days
$ find . -type f -newermt "2010-01-01" ! -newermt "2010-06-01"via mtime
mtime[!TIP|label:tricky on
-mtime:]
Understanding find with atime, ctime, and mtime
+n: for greater than n
-n: for less than n
n: for exactly ni.e.:
-mtime -14: less than 14 days, not including 14 days == 13 days ago and less
-mtime +14: more than 14 days, not including 14 days == 15 days ago and more
# get all files since 2023-10-16
$ diff=$(( ($(date --date "24-02-29" +%s) - $(date --date "231016" +%s) )/(60*60*24) ))
$ find . -type f -daystart -mtime -$((diff+1)) -printf "%T+ | %p\n" | sort | wc -l
33
# copy all files modified since 2023-10-16
$ find . -type f -daystart -mtime -$((diff+1)) -exec cp -a --parents -t /path/to/target "{}" \+
# with timezone
$ diff=$(( ($(date -d "2015-03-11 UTC" +%s) - $(date -d "2015-03-05 UTC" +%s)) / (60*60*24) ))
# find older than x days
$ find . -type f -mtime +7 -exec ls -l {} \;
# delete older than x days: https://www.commandlinefu.com/commands/view/1394/delete-files-older-than..
$ find /dir_name -mtime +5 -exec rm {} \via newermt
newermt[!TIP|label:tips for
-newerXY]
What does newermt mean in find command?
-newerXY reference Compares the timestamp of the current file with reference. The reference argument is normally the name of a file (and one of its timestamps is used for the comparison) but it may also be a string describing an absolute time. X and Y are placeholders for other letters, and these letters select which time belonging to how reference is used for the comparison. a The access time of the file reference B The birth time of the file reference c The inode status change time of reference m The modification time of the file reference t reference is interpreted directly as a time$ find ./ -newermt "2016-01-18" ! -newermt '2016-01-19' $ find . -type f -newermt "2014-10-08 10:17:00" ! -newermt "2014-10-08 10:53:00"
$ find . -type f -newermt '2023-10-16 00:00:00' | wc -l
33
# or
$ find . -type f -newermt '16 Oct 2023 00:00:00' | wc -l
33
# or with difference timestamp format
$ find . -type f -newermt "@$(date +%s -d '10/16/2023 0:00:00 PDT')" -printf "%T+ | %p\n" | sort | wc -l
33
# copy all files modified since 2023-10-16
$ find . -type f -newermt '2023-10-16 00:00:00' -exec cp -a --parents -t /path/to/target "{}" \+$ find . -type f -printf '%T@ %TY-%Tm-%Td %TH:%TM:%.2TS %p\n' | sort -nr | head -n 5 | cut -f2- -d" "inject commands inside find
[!NOTE|label:references:]
$ find -exec bash -c '
print_echo() { printf "This is print_echo Function: %s\n" "$@"; };
print_echo "$@"
' find-bash {} +printf
[!NOTE|label:references:]
formats
[!NOTE|label:references:]
find . -printf "%T<format>\n"
time format
FORMATDESCRIPTIONEXAMPLE%Alast access time
%A+:2023-02-20+05:19:18.0000000000%Tlast modification time
%T@:1676899158.0000000000%tlast modification time in ctime format
Mon Feb 20 05:19:18.0000000000 2023%Clast status change time
%C+:2024-11-20+03:30:18.8905999140%clast status change time in ctime format
Wed Nov 20 03:30:18.8905999140 2024%Bbirth time
%B@:1676899158.0000000000$ touch -d "2023-02-20 05:19:18" sample.txt $ stat sample.txt Access: 2023-02-20 05:19:18.000000000 -0800 Modify: 2023-02-20 05:19:18.000000000 -0800 Change: 2024-11-20 03:30:18.890599914 -0800 Birth: 2023-02-20 05:19:18.000000000 -0800 $ find . -maxdepth 1 -name 'sample.txt' -printf '%%A+: %A+\n%%A@: %A@\n%%T+: %T+\n%%T@: %T@\n%%C+: %C+\n%%C@: %C@\n%%B+: %B+\n%%B@: %B@\n%%t: %t\n%%c: %c\n' %A+: 2023-02-20+05:19:18.0000000000 %A@: 1676899158.0000000000 %T+: 2023-02-20+05:19:18.0000000000 %T@: 1676899158.0000000000 %C+: 2024-11-20+03:30:18.8905999140 %C@: 1732102218.8905999140 %B+: 2023-02-20+05:19:18.0000000000 %B@: 1676899158.0000000000 %t: Mon Feb 20 05:19:18.0000000000 2023 %c: Wed Nov 20 03:30:18.8905999140 2024 $ find . -maxdepth 1 -name 'sample.txt' -printf 'week%AW %Aj/365 %Ax %Ar\n%%A+: %A+\n' week08 051/365 02/20/2023 05:19:18 AM %A+: 2023-02-20+05:19:18.0000000000TIME FIELDDESCRIPTIONEXAMPLE+date time
2023-02-20+05:19:18.0000000000@unix epoch
1676899158.0000000000Hhour
00..23k/Ihour in 24-hour / 12-hour format
00..23/01..12Mminute
00..59Ssecond
00..60pAM/PM
AM/PMT/Xtime in 24-hour format
hh:mm:ss:xxxxxxxxxxZtimezone
PST/PDT$ find . -maxdepth 1 -name 'sample.txt' -printf '%%TT: %TT\n%%TX: %TX\n' %TT: 05:19:18.0000000000 %TX: 05:19:18.0000000000DATA FIELDDESCRIPTIONEXAMPLEa/Aabbreviated / full weekday
Wed/Wednesdayb(h) /Babbreviated / full month name
Jan/Januarymmonth
01..12dday of month
01..31wday of week
01->Monday;02->Tuesdayjday of year
001..366U/Wweek number: Sunday / Monday as first day
00..53y/Ylast 2-digits-of-year / 4-digits-of-year
00..99/1970..rtime in 12-hour format
hh:mm:ss [A/P]MFfull date; same as
%Y-%m-%d2023-02-20Ddate; same as
%m/%d/%y02/20/23xlocale date
02/20/2023$ find . -maxdepth 1 -name 'sample.txt' -printf '%%TX: %TX\n%%Tx: %Tx\n%%TD: %TD\n%%TF: %TF\n%%Tr: %Tr\n' %TX: 05:19:18.0000000000 %Tx: 02/20/2023 %TD: 02/20/23 %TF: 2023-02-20 %Tr: 05:19:18 AMname format
FORMATDESCRIPTIONEXAMPLE%pfile's name
./sample.txt%Pfile's name without starting-point
sample.txt%fbasename
sample.txt%hleading directories of file's name
.$ find . -maxdepth 1 -name '.vimrc' -printf '%%p: %p\n%%P: %P\n%%f: %f\n%%h: %h\n' %p: ./.vimrc %P: .vimrc %f: .vimrc %h: .permision format
FORMATDESCRIPTIONEXAMPLE%mfile's mode
644%Mfile's mode in human-readable format
-rw-r--r--$ find . -type f -printf "\n%Td-%Tm-%TY %Tr %p" | head -1 4-10-2023 02:38:42 AM /Users/marslo/.marslo/.marslorcT: time in 24-hour format :hh:mm:ss.xxxxxxxxxx$ find . -type f -printf "\n%Td-%Tm-%TY %TT %p" | head -1 14-10-2023 02:38:42.5626405780 /Users/marslo/.marslo/.marslorcX: locale time :hh:mm:ss.xxxxxxxxxxc: locale time in ctime format$ find . -type f -printf "\n%Tc %p" | head -1 Sat Oct 14 02:38:42 2023 /Users/marslo/.marslo/.marslorcD: date :mm/dd/yyF: date :yyyy-mm-ddx: locale date :mm/dd/yyR: hour and minute in 24 hour format :HH:MM+: date and time$ find . -printf "%T+ | %p\n" | head -1 2023-10-14+02:38:42.5626405780 | /Users/marslo/.marslo/.marslorcenter-align
$ find . -printf "%15TA | %p\n" Monday | /Users/marslo/.marslo/bin/iweather Saturday | /Users/marslo/.marslo/.marslorcleft-align
$ find . -printf "%-15TA | %p\n" Monday | /Users/marslo/.marslo/bin/iweather Saturday | /Users/marslo/.marslo/.marslorcmixed align
$ find . -type f -printf "%10T+ %-10TA | %m | %p\n" | sort -r | head -2 2023-10-16+20:25:53.5005192040 Monday | 755 | /Users/marslo/.marslo/bin/iweather 2023-10-14+02:38:42.5626405780 Saturday | 755 | /Users/marslo/.marslo/.marslorc # more find . -type d -printf "%d %-30p %-10u %-10g %-5m %T+\n" | sort
tips
$ sh -c 'S=askapache R=htaccess; find . -mount -type f | xargs -P5 -iFF grep -l -m1 "$S" FF | xargs -P5 -iFF sed -i -e "s%${S}%${R}%g" FF'[!TIP|label:rules:] size first, then md5 hash
$ find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate$ echo $[RANDOM%X+1]
# or: https://www.commandlinefu.com/commands/view/14051/random-number-between-1-and-x
$ (od -An -t u8 -N8 </dev/urandom; echo X '* 2 64^/1+p') | dc
# or: https://www.commandlinefu.com/commands/view/6297/random-number-between-1-and-x
$ echo "$(od -An -N4 -tu4 /dev/urandom) % 5 + 1" | bc
# between 1 - 256: https://www.commandlinefu.com/commands/view/12702/random-number-between-1-and-256
$ od -An -N1 -tu1 /dev/random
# decimal in 1 - 2d6: https://www.commandlinefu.com/commands/view/6269/random-decimal-in-the-interval-0-n-1-and-2d6-dice-roll
$ awk 'BEGIN { srand(); print rand() }'trim
trim tailing chars
str='1234567890'
awk+rev$ echo $str | rev | cut -c4- | rev 1234567${var:: -x})$ echo ${str:: -3} 1234567
$ str=" aaaa bbbb "
$ echo "$str" | sed 's:^ *::; s: *$::'
# i.e.:
$ echo .$(echo "$str" | sed 's:^ *::; s: *$::').
.aaaa bbbb.remove all spaces
$ echo .${str// }. .aaaabbbb.$ echo .${str##+([[:space:]])}. .aaaa bbbb .$ echo .${str%%+([[:space:]])}. . aaaa bbbb.function in pip
function trim() { IFS='' read -r str; echo "${str}" | sed -e 's/^[[:blank:]]*//;s/[[:blank:]]*$//'; } $ echo ..$(echo " aaa bbb " | trim).. ..aaa bbb..
remove empty lines
[!NOTE|label:references:]
# original
$ cal | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊
····················␊
# awk 'NF'
$ cal | awk 'NF' | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊
# sed '/^\s*$/d'
$ cal | sed '/^\s*$/d' | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊remove empty line at the end of file
[!NOTE|label:references:]
$ cat a.txt | sed '${/^[[:space:]]*$/d;}'
a
b
c
dremove duplicate empty lines
[!NOTE|label:references:]
# original file
$ cat a.txt --style='numbers'
1 a
2 b
3
4
5
6 c
7
8 d
9
10
$ awk 'NF || p; { p = NF }' p=1 a.txt | bat --style='numbers'
1 a
2 b
3
4 c
5
6 d
7
# or
$ awk 'NF{c=1} (c++)<3' a.txt | bat --style='numbers'
1 a
2 b
3
4 c
5
6 d
7
# or
$ awk -v RS= -v ORS='\n\n' '1' a.txt | bat --style='numbers'
1 a
2 b
3
4 c
5
6 d
7
# or
$ awk '!NF{found++} found>1 && !NF{next} NF{found=""} 1' a.txt | bat --style='numbers'
1 a
2 b
3
4 c
5
6 d
7[!NOTE|label:reference]
sample code:
str='aa bb cc'
${variable//search/replace}$ shopt -s extglob $ echo ${str//+( )/|} aa|bb|cc$ echo "${str//+([[:blank:]])/|}" aa|bb|ccsed
# DO NOT USE "${str}" $ echo ${str} | sed 's: :|:g' aa|bb|cc$ echo "$str" | sed 's:[ ][ ]*:|:g' aa|bb|cc # or $ echo "$str" | sed 's:\s\s*:|:g' aa|bb|cc
echo "${string:0:$(( position - 1 ))}${replacement}${string:position}"
or
$ sed 's:\s\s*:|:g' <<< "${str}" aa|bb|cc
- [tr](https://stackoverflow.com/a/50259880/2940319)
```bash
$ echo "$str" | tr -s ' ' '|'
aa|bb|cc$ string=aaaaa
$ replacement=b
$ position=3
$ echo "${string:0:$(( position - 1 ))}${replacement}${string:position}"
aabaaor
$ echo "${string:0:position-1}${replacement}${string:position}" aabaa
check line ending
[!NOTE|label:references:]
check ascii via terminal
$ man ascii
$ cat /usr/share/misc/ascii
$ printf "%d" "'${char}", i.e.:$ printf "%d" "'a" => 97
OS
CHARACTER ENCODING
ABBREVIATION
HEX
DEC
ESCAPE SEQUENCE
UNIX or Unix-like
ASCII
LF
0A
10
MS-DOS
ASCII
CR LF
0D 0A
13 10
\r
Commodore 8-bit machines
ASCII
CR
0D
13
QNX pre-POSIX
ASCII
RS
1E
30
\036
Acorn BBC and RISC OS
ASCII
LF CR
0A 0D
10 13
\n
Atari 8-bit machines
ATASCII
-
9B
155
-
IBM mainframe systems
EBCDIC
NL
15
21
\025
ZX80 and ZX81
non-ASCII
NEWLINE
76
118
-
od -c$ echo 'abc' | od -c 0000000 a b c \n 0000004 $ echo -n 'abc' | od -c 0000000 a b c 0000003hexdump -c$ echo 'abc' | hexdump -c 0000000 a b c \n 0000004 $ echo -n 'abc' | hexdump -c 0000000 a b c 0000003hexdump -C$ echo 'abc' | hexdump -C 00000000 61 62 63 0a |abc.| 00000004 # ^ # | # 0x0a: LF $ echo -n 'abc' | hexdump -C 00000000 61 62 63 |abc| 00000003 $ cat a.txt | hexdump -C # 0x0a # v 00000000 61 61 61 61 0a |aaaa.| 00000005 $ unix2dos a.txt unix2dos: converting file a.txt to DOS format... $ cat a.txt | hexdump -C # 0x0d 0x0a # v v 00000000 61 61 61 61 0d 0a |aaaa..| 00000006 $ cat a.txt | hexdump -c 0000000 a a a a \r \n 0000006 $ file a.txt a.txt: ASCII text, with CRLF line terminatorsvim
$ vim a.txt :%!hexdump -C # or $ vim -c '%!xxd' a.txt
remove the ending '\n'
[!NOTE|label:references:]
original file
$ cat foo.txt abc efg $ cat -A foo.txt abc$ efg$ $ cat foo.txt | od -c 0000000 a b c \n e f g \n 0000010
$ truncate -s -1 foo.txt $ od -c foo.txt 0000000 a b c \n e f g 0000007$ sed -z s/.$// foo.txt | od -c 0000000 a b c \n e f g 0000007 $ sed -z s/\\n$// foo.txt | od -c 0000000 a b c \n e f g 0000007 $ sed -z 's/\n$//' foo.txt | od -c 0000000 a b c \n e f g 0000007$ printf %s "$(< foo.txt)" | od -c 0000000 a b c \n e f g 0000007$ head -c -1 foo.txt | od -c 0000000 a b c \n e f g 0000007$ od -c foo.txt 0000000 a b c \n e f g \n 0000010 $ vim -c "set binary noeol" -c "wq" foo.txt $ od -c foo.txt 0000000 a b c \n e f g 0000007 # or : https://stackoverflow.com/a/39627416/2940319 $ vim -c "set noendofline nofixendofline" -c "wq" foo.txt $ od -c foo.txt 0000000 a b c \n e f g 0000007
add '\n' to line-ending
[!TIP]
for ssh private key issue:
$ ssh -vT sample.host ... Load key "~/.ssh/id_ed25519": error in libcrypto
check last char in the file
# unqualified key $ tail -c1 ~/.ssh/id_ed25519 $ tail -c1 ~/.ssh/id_ed25519 | /usr/bin/xxd -u -p 2D # qualified key $ tail -c1 ~/.ssh/id_ed25519 $ tail -c1 ~/.ssh/id_ed25519 | /usr/bin/xxd -u -p 0A $ tail -c1 ~/.ssh/id_ed25519 | hexdump -v -e '/1 "%02X"' 0Aadd new line
$ [ -n "$(tail -c1 file)" ] && echo >> ~/.ssh/id_ed25519 # or $ [ -z "$(tail -c1 file)" ] || printf '\n' >>file # or: https://stackoverflow.com/a/35279563/2940319 $ echo \ >> file.txt # or: https://stackoverflow.com/a/65136212/2940319 $ sed -i -z 's/$/\n/g' file.txt # performance for various solutions $ [ -n "$(tail -c1 file)" ] && printf '\n' >>file 0.013 sec $ vi -ecwq file 2.544 sec $ paste file 1<> file 31.943 sec $ ed -s file <<< w 1m 4.422 sec $ sed -i -e '$a\' file 3m 20.931 secadd new line ending without modifying the file
$ echo -n "$(cat ~/.ssh/id_ed25519)"$'\n' | tail -c1 | /usr/bin/xxd -u -p 0A # or $ cat file.txt | sed -e '$a\' # or: https://stackoverflow.com/a/65136212/2940319 $ cat a.org.txt | sed -z 's/$/\n/g' # or $ echo '' >> file # fix the last line line-ending $ sed '${/^[[:space:]]*$/d;}' -i file # remove the empty lines at end of file
fold
check the params valid
{% hint style='tip' %}
available params should be contained by 'iwfabcem' {% endhint %}
# case insensitive
param=$( tr '[:upper:]' '[:lower:]' <<< "$1" )
for _p in $(echo "${param}" | fold -w1); do
[[ ! 'iwfabcem' =~ ${_p} ]] && exits='yes' && break
doneinsert
insert new line
insert right after the second match string
{% codetabs name="original", type="bash" -%} DCR DCR DCR {%- language name="expected", type="bash" -%} DCR DCR check DCR {%- endcodetabs %}
$ echo -e "DCR\nDCR\nDCR" | awk 'BEGIN {t=0}; { print }; /DCR/ { t++; if ( t==2) { print "check" } }'insert after the matched string
with
gsub$ cat project.config | awk '{ gsub( /.+access .(refs\/for\/|\^)refs\/heads\/main.+$/, "&\n\tpush = block group Registered Users\n\tsubmit = block group Registered Users\n\tpushMerge = block group Registered Users", $0 ) }1'ORIGINALAFTER INSERT$ cat project.config$ cat project.config | awk '{ gsub( /.+access .(refs\/for\/|\^)refs\/heads\/main.+$/, "&\n\t---a---\n\t---b---", $0 ) }1'[access "^refs/heads/main"] label-Code-Review = -2..+2 group user/John Doe (jdoe) label-Verified = -1..+1 group user/John Doe (jdoe) push = group user/John Doe (jdoe) submit = group user/John Doe (jdoe) [access "refs/for/refs/heads/main"] push = group user/John Doe (jdoe) addPatchSet = group user/John Doe (jdoe) abandon = group user/John Doe (jdoe) deleteOwnChanges = group user/John Doe (jdoe) editAssignee = group user/John Doe (jdoe) editTopicName = group user/John Doe (jdoe) label-Code-Review = -2..+2 group user/John Doe (jdoe) label-Verified = -1..+1 group user/John Doe (jdoe) owner = group user/John Doe (jdoe) read = group user/John Doe (jdoe) removeReviewer = group user/John Doe (jdoe) submit = group user/John Doe (jdoe) create = group user/John Doe (jdoe) pushMerge = group user/John Doe (jdoe) [access "^refs/heads/user/jdoe/.*"] label-Code-Review = -2..+2 group user/John Doe (jdoe) label-Verified = -1..+1 group user/John Doe (jdoe) push = group user/John Doe (jdoe) submit = group user/John Doe (jdoe[access "^refs/heads/main"] ---a--- ---b--- label-Code-Review = -2..+2 group user/John Doe (jdoe) label-Verified = -1..+1 group user/John Doe (jdoe) push = group user/John Doe (jdoe) submit = group user/John Doe (jdoe) [access "refs/for/refs/heads/main"] ---a--- ---b--- push = group user/John Doe (jdoe) addPatchSet = group user/John Doe (jdoe) abandon = group user/John Doe (jdoe) deleteOwnChanges = group user/John Doe (jdoe) editAssignee = group user/John Doe (jdoe) editTopicName = group user/John Doe (jdoe) label-Code-Review = -2..+2 group user/John Doe (jdoe) label-Verified = -1..+1 group user/John Doe (jdoe) owner = group user/John Doe (jdoe) read = group user/John Doe (jdoe) removeReviewer = group user/John Doe (jdoe) submit = group user/John Doe (jdoe) create = group user/John Doe (jdoe) pushMerge = group user/John Doe (jdoe) [access "^refs/heads/user/jdoe/.*"] label-Code-Review = -2..+2 group user/John Doe (jdoe) label-Verified = -1..+1 group user/John Doe (jdoe) push = group user/John Doe (jdoe) submit = group user/John Doe (jdoe)with variable
[!NOTE|label:references:]
# + first line without extra space # | + the others new line requires 4 spaces # +--------------+ +----------------+ $ cat sample.json | awk -v new='"gender": "male",\n "state": "CA",' '{print} sub(/"name":.*John Doe.*$ /,""){print $0 new}' { "id": "1234567", "properties": { "name": "John Doe", "gender": "male", "state": "CA", "age": 30 } } # or $ cat sample.json | awk -v insertVal="femal" '/"name":.+John.+Doe.*$/{$0=$0"\n \"gender\": \""insertVal"\","} {print}' { "id": "1234567", "properties": { "name": "John Doe", "gender": "femal", "age": 30 } }ORIGINALAFTER INSERTINSERT TWO LINES$ cat sample.json$ cat sample.json | awk -v new='"gender": "male",' '{print} sub(/"name":.*John Doe.*$/,""){print $0 new}'$ cat sample.json | awk -v new='"gender": "male",\n "state": "CA",' '{print} sub(/"name":.*John Doe.*$/,""){print $0 new}'{ "id": "1234567", "properties": { "name": "John Doe", "age": 30 } }{ "id": "1234567", "properties": { "name": "John Doe", "gender": "male", "age": 30 } }{ "id": "1234567", "properties": { "name": "John Doe", "gender": "male", "state": "CA", "age": 30 } }with sed
$ cat sample.json | sed -e '/John Doe/{p;s/name.*"/gender": "male"/' -e '}' { "id": "1234567", "properties": { "name": "John Doe", "gender": "male", # new added "age": 30 } }
insert new line base on pattern
[!NOTE|label:references:]
awk
$ echo "9089.00 ----- kl jkjjljk lkkk; (909099) 9097.00 ----- HGJJHHJ jcxkjlkjvhvlk jhdkjksdfkhfskd 898.00 ----- HHHH" | awk '{ gsub("[0-9][0-9]*[.]00", RS "&");print }' 9089.00 ----- kl jkjjljk lkkk; (909099) 9097.00 ----- HGJJHHJ jcxkjlkjvhvlk jhdkjksdfkhfskd 898.00 ----- HHHHsed
$ echo "9089.00 ----- kl jkjjljk lkkk; (909099) 9097.00 ----- HGJJHHJ jcxkjlkjvhvlk jhdkjksdfkhfskd 898.00 ----- HHHH" | sed "s/\([0-9]*\.00\)/\n\1/g" 9089.00 ----- kl jkjjljk lkkk; (909099) 9097.00 ----- HGJJHHJ jcxkjlkjvhvlk jhdkjksdfkhfskd 898.00 ----- HHHH
write a file without indent space
$ sed -e 's:^\s*::' > ~/file-without-indent-space.txt < <(echo "items.find ({
\"repo\": \"repo-name\",
\"type\" : \"folder\" ,
\"depth\" : \"1\",
\"created\" : { \"\$before\" : \"4mo\" }
})
")
$ cat ~/file-without-indent-space.txt
items.find ({
"repo": "repo-name",
"type" : "folder" ,
"depth" : "1",
"created" : { "$before" : "4mo" }
})or
$ sed -e 's:^\s*::' > find.aql <<-'EOF' items.find ({ "repo": "${product}-${stg}-local", "type" : "folder" , "depth" : "1", "created" : { "${opt}": "4mo" } }) EOF
{% codetabs name="example", type="bash" -%} $ sed -e 's:^\s*::' <<-'EOF' items.find ({ "repo": "${product}-${stg}-local", "type" : "folder" , "depth" : "1", "created" : { "${opt}": "4mo" } }) EOF items.find ({ "repo": "${product}-${stg}-local", "type" : "folder" , "depth" : "1", "created" : { "${opt}": "4mo" } }) {%- endcodetabs %}
cat
<< - and <<
<< - and <<This type of redirection instructs the shell to read input from the current source until a line containing only delimiter (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input for a command.
The format of here-documents is:
bash
<<[-]word here-document delimiterNo parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. In the latter case, the character sequence \ is ignored, and \ must be used to quote the characters \, $, and `.
cat with specific character
$ man tab ... -T, --show-tabsdisplay TAB characters as ^I
$ cat -A sample.sh LANG=C tr a-z A-Z <<- END_TEXT$ Here doc with <<$ A single space character (i.e. 0x20 ) is at the beginning of this line$ ^IThis line begins with a single TAB character i.e 0x09 as does the next line$ ^IEND_TEXT$ $ echo The intended end was before this line$
$ bash sample.sh HERE DOC WITH <<- A SINGLE SPACE CHARACTER (I.E. 0X20 ) IS AT THE BEGINNING OF THIS LINE THIS LINE BEGINS WITH A SINGLE TAB CHARACTER I.E 0X09 AS DOES THE NEXT LINE The intended end was before this line
$ cat -A sample.sh LANG=C tr a-z A-Z << END_TEXT$ Here doc with <<$ A single space character (i.e. 0x20 ) is at the beginning of this line$ ^IThis line begins with a single TAB character i.e 0x09 as does the next line$ ^IEND_TEXT$ $ echo The intended end was before this line$
$ bash sample.sh sample.sh: line 7: warning: here-document at line 1 delimited by end-of-file (wanted `END_TEXT') HERE DOC WITH << A SINGLE SPACE CHARACTER (I.E. 0X20 ) IS AT THE BEGINNING OF THIS LINE THIS LINE BEGINS WITH A SINGLE TAB CHARACTER I.E 0X09 AS DOES THE NEXT LINE END_TEXT
ECHO THE INTENDED END WAS BEFORE THIS LINE
Last updated
Was this helpful?