sed

• execute multiple sed commands

  • example : show only root and nobody in /etc/passwd

• range

  • specific line

  • until empty line

  • n~m range

  • pattern matches range

• print

  • print all lines

  • range print

  • print matched pattern

  • print line number of matched pattern

• delete

  • delete all

  • range delete

  • conditional delete

• substitute

  • substitute-flags

  • multiple replaces

• get matched pattern

  • &

  • substitution grouping

• cheatsheet

  • get first matching patten ( for CERTIFICATE )

  • remove both '#' and empty lines

  • remove tailing spaces

  • show top summary

  • transform literal string in regex

  • escape

• tricky

[!NOTE] references:

• sed, a stream editor

• character classes

CHARACTER	REGULAR EXPRESSION	EXPLANATION
[[:alnum:]]	[A-Za-z0-9]	Alphanumeric characters
[[:alpha:]]	[A-Za-z]	Alphabetic characters
[[:blank:]]	[ \t]	Space or tab characters only
[[:cntrl:]]	[\x00-\x1F\x7F]	Control characters
[[:digit:]]	[0-9]	Numeric characters
[[:graph:]]	[!-~]	Printable and visible characters
[[:lower:]]	[a-z]	Lower-case alphabetic characters
[[:print:]]	[ -~]	Printable (non-Control) characters
[[:punct:]]	[!-/:-@[-{-~]`	Punctuation characters
[[:space:]]	[ \t\v\f\n\r]	All whitespace chars
[[:upper:]]	[A-Z]	Upper-case alphabetic characters
[[:xdigit:]]	[0-9a-fA-F]	Hexadecimal digit characters

execute multiple sed commands

[!TIP]

-e command
       Append the editing commands specified by the command argument to the list of commands.

references:

• 50 sed Command Examples

example : show only root and nobody in /etc/passwd

• -e :

  $ sed -n -e '/^root/p' -e '/^nobody/p' /etc/passwd
  nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
  root:*:0:0:System Administrator:/var/root:/bin/sh

• ; :

  $ sed -n -e '/^root/p;/^nobody/p' /etc/passwd
  nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
  root:*:0:0:System Administrator:/var/root:/bin/sh

• '{}' :

  $ sed -n '{
      /^root/p
      /^nobody/p
    }' /etc/passwd
  nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
  root:*:0:0:System Administrator:/var/root:/bin/sh

range

specific line

• 2nd line : N <opt>

until empty line

references:

• Remove everything before a blank line using sed

• Remove empty line before a pattern using sed

• Grep starting from a fixed text, until the first blank line

• Delete unknown number of lines from * until blank line

• remove only the first blank line sed

• Use sed to insert text before two blank lines

• htop output to human readable file

• $HOME/.toprc

$ top -bn1 | sed -n '0,/^\s*$/p'
top - 03:41:45 up 258 days, 19:05,  1 user,  load average: 2.33, 0.92, 0.95
Tasks: 856 total,   2 running, 447 sleeping,   0 stopped,  36 zombie
%Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52802012+total, 11152644+free, 24536944 used, 39195673+buff/cache
KiB Swap:        0 total,        0 free,        0 used. 49137280+avail Mem

• or

  $ top -bn1 | sed -e '/^$/Q'
  top - 03:45:55 up 258 days, 19:09,  1 user,  load average: 0.17, 0.51, 0.77
  Tasks: 857 total,   2 running, 448 sleeping,   0 stopped,  36 zombie
  %Cpu(s):  0.1 us,  0.4 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  KiB Mem : 52802012+total, 11151089+free, 24546520 used, 39196272+buff/cache
  KiB Swap:        0 total,        0 free,        0 used. 49136291+avail Mem

n~m range

• n~m lines : n,m <opt>

• n to end lines : n,$ <opt>

• m lines starting with n : n, +m <opt>

• start n skip m via ~ :

  pattern	matches	comments
  1~2	1,3,5,7,...	start frmo 1, print every 2 lines
  2~2	2,4,6,8,...	start from 2, print every 2 lines
  1~3	1,4,7,10,...	start from 1, print every 3 lines
  2~3	2,5,8,11,...	start from 2, print every 3 lines

pattern matches range

• between pattern_1 to pattern_2 : /pattern_1/,/pattern_2/ <opt>

• first line to pattern_2 : 0,/pattern_2/ <opt>

from pattern to first empty line

$ top -bn1 | sed -n '/^top.*/,/^\s*$/p'
top - 03:49:02 up 258 days, 19:13,  1 user,  load average: 0.43, 0.41, 0.68
Tasks: 853 total,   1 running, 448 sleeping,   0 stopped,  36 zombie
%Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52802012+total, 11150942+free, 24543500 used, 39196720+buff/cache
KiB Swap:        0 total,        0 free,        0 used. 49136582+avail Mem

print

print all lines

• print every line twice

  $ sed 'p' employee.txt
  101,John Doe,CEO
  101,John Doe,CEO
  102,Jason Smith,IT Manager
  102,Jason Smith,IT Manager
  103,Raj Reddy,Sysadmin
  103,Raj Reddy,Sysadmin
  104,Anand Ram,Developer
  104,Anand Ram,Developer
  105,Jane Miller,Sales Manager
  105,Jane Miller,Sales Manager

• print all lines : $ sed -n 'p' employee.txt

range print

• print the 2nd line : $ sed -n '2 p' employee.txt

• n,m range

  • print 1~4 lines : $ sed -n '1,4 p' employee.txt

  • print all lines since the 2nd line: $ sed -n '2,$ p' employee.txt

• ~ to skip lines

  • print only odd numbered lines : sed -n '1~2 p' employee.txt

• + ( n, +m ) : sed -n 'n,+m p' employee.txt

print matched pattern

• find pattern to the end :

  $ sed -n '/Raj/,$ p' employee.txt
  103,Raj Reddy,Sysadmin
  104,Anand Ram,Developer
  105,Jane Miller,Sales Manager

• find pattern and line after the matches line :

  $ sed -n '/Raj/, +1 p' employee.txt
  103,Raj Reddy,Sysadmin
  104,Anand Ram,Developer

• find pattern to 4th line :

  $ sed -n '/Raj/,4 p' employee.txt
  103,Raj Reddy,Sysadmin
  104,Anand Ram,Developer

• find pattern until find another pattern ( Jason to Anand ) :

  $ sed -n '/Jason/,/Anand/p' employee.txt
  102,Jason Smith,IT Manager
  103,Raj Reddy,Sysadmin
  104,Anand Ram,Developer

print line number of matched pattern

[!NOTE|label:references:]

• sed replace last line matching pattern

• Perform sed operations on given line numbers

# original
$ printf "%s\r\n\r\n%s\r\n%s\r\n\r\n%s\r\n" 1 2 3 4 | command cat -A | nl
     1  1^M$
     2  ^M$
     3  2^M$
     4  3^M$
     5  ^M$
     6  4^M$

# result
$ printf "%s\r\n\r\n%s\r\n%s\r\n\r\n%s\r\n" 1 2 3 4 | sed -n '/^\s*\r$/ ='
2
5
# or
$ sed -n '/^\s*\r$/ =' < <(printf "%s\r\n\r\n%s\r\n%s\r\n\r\n%s\r\n" 1 2 3 4)
2
5

# first matched line number
$ printf "%s\r\n\r\n%s\r\n%s\r\n%s\r\n" 1 2 3 4 | sed -n '/^\s*\r$/ =' | head -n1
2

# last matched line number
$ printf "%s\r\n\r\n%s\r\n%s\r\n\r\n%s\r\n" 1 2 3 4 | sed -n '/^\s*\r$/ =' | tail -n1
5

• or

  # via sed
  $ sed -n '10,20{=;p}'
  ## or ##
  $ nl file.txt | sed -n '10,20p'
  ## or ##
  $ sed -n '10,20{=;p}' file.txt | sed '{N; s/\n/ /}'
  
  # via awk
  $ awk 'FNR==10,FNR==20 {print FNR ":" $0}' file.txt
  ## or ##
  $ awk 'FNR >= 10 {print FNR ":" $0}; FNR == 20 {exit}' file.txt
  ## or ##
  $ awk -v OFS=: 'FNR >= 10 {print FILENAME, FNR, $0}; FNR == 20 {nextfile}' ./*.txt

delete

delete all

$ sed 'd' employee.txt

range delete

• delete the 2nd line : $ sed '2 d' /path/to/file

• delete between 1 and 4 lines : $ sed '1,4 d' /path/to/file

conditional delete

• delete all empty lines: $ sed '/^$/ d' /path/to/file

• delete all comment lines : $ sed '/^#/ d' /path/to/file

substitute

substitute-flags

flag	comments
i	ignore case flag
g	global flag
1,2,...	number flag
p	print flag
w	write flag
e	execute flag

multiple replaces

$ sed '{
    s/Developer/IT Manager/
    s/Manager/Director/
  }' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Director
103,Raj Reddy,Sysadmin
104,Anand Ram,IT Director
105,Jane Miller,Sales Director

get matched pattern

&

When & is used in the replacement-string, it replaces it with whatever text matched the original-string or the regular-expression.

$ sed 's/^[0-9][0-9][0-9]/<&>/g' employee.txt
<101>,John Doe,CEO
<102>,Jason Smith,IT Manager
<103>,Raj Reddy,Sysadmin
<104>,Anand Ram,Developer
<105>,Jane Miller,Sales Manager

substitution grouping

$ sed 's/^\([0-9][0-9][0-9]\).*/<\1>/g' employee.txt
<101>
<102>
<103>
<104>
<105>

or via -r

$ sed -r 's/^([0-9][0-9][0-9]).*/<\1>/g' employee.txt
<101>
<102>
<103>
<104>
<105>

# or
$ sed -nr 's/^([0-9][0-9][0-9])(.*)/<\1>\2/gp' employee.txt
<101>,John Doe,CEO
<102>,Jason Smith,IT Manager
<103>,Raj Reddy,Sysadmin
<104>,Anand Ram,Developer
<105>,Jane Miller,Sales Manager

cheatsheet

reference:

• Sed cheatsheet

• ssstonebraker/sed cheatsheet

• sed Cheat Sheet

• Erik's Cheat Sheet : sed

• cheatsheet_sed.pdf

more samples:

• * imarslo : get lines between 2 matched patterns

get first matching patten ( for CERTIFICATE )

[!TIP]

• sample.crt

  $ cat sample.crt
  -----BEGIN CERTIFICATE-----
  first paragraph
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  second paragraph
  -----END CERTIFICATE-----

• regular pattern

  $ cat sample.crt | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
  -----BEGIN CERTIFICATE-----
  first paragraph
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  second paragraph
  -----END CERTIFICATE-----
  
  # or for short
  $ cat sample.crt | sed -ne '/-BEGIN/,/-END/p'
  -----BEGIN CERTIFICATE-----
  first paragraph
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  second paragraph
  -----END CERTIFICATE-----

• get first

  [!TIP]

  • How to print the text between the first occurence of a pair of strings? [duplicate]

  • How to select first occurrence between two patterns including them

  more :

  • * imarslo : get second matching pattern

  # or `-n /../p`
  #                     `-n`                                         `p`
  #                       |                                           |
  #                       v                                           v
  $ cat sample.crt | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p; /-END CERTIFICATE-/q'
  -----BEGIN CERTIFICATE-----
  first paragraph
  -----END CERTIFICATE-----
  
  # or `/../!d`
  #                   no `-n`                                     `!d`
  #                     |                                           |
  #                     v                                           v
  $ cat sample.crt | sed '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/!d; /-END CERTIFICATE-/q'
  -----BEGIN CERTIFICATE-----
  first paragraph
  -----END CERTIFICATE-----
  
  # or for short
  $ cat sample.crt | sed '/-END CERTIFICATE-/q'
  -----BEGIN CERTIFICATE-----
  first paragraph
  -----END CERTIFICATE-----
  
  # or
  $ cat sample.crt | sed '/-END/q'
  -----BEGIN CERTIFICATE-----
  first paragraph
  -----END CERTIFICATE-----

remove both '#' and empty lines

[!NOTE|label:references:]

• * iMarslo: remove empty lines

$ .. | sed -r '/^(#.*)$/d' | sed -r '/^\s*$/d'

# or
$ .. | sed -r '/^(#.*)$/d;/^\s*$/d'

# or
$ .. | sed -r '/(^#.*)|(^\s*)$/d'

• example

  $ ldapsearch CN=marslo DN | sed -r '/^(#.*)$/d;/^\s*$/d'
  dn: CN=marslo,OU=Workers,DC=company,DC=com

remove tailing spaces

[!TIP|label:available patterns]

• 's/[ \t]*$//'

• 's/[[:blank:]]*$//'

• 's/[[:space:]]*$//'

$ cal | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊
····················␊

# remove empty line
$ cal | sed -r '/^(#.*)$/d;/^\s*$/d' | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊

# remove trailing spaces
$ cal | sed 's/[[:space:]]*$//' | cat -pp -A
····January·2024␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31␊
␊

show top summary

[!NOTE] see sed until empty line

• contains empty line

  $ top -bn1 | sed -n '0,/^\s*$/p'
  top - 03:41:45 up 258 days, 19:05,  1 user,  load average: 2.33, 0.92, 0.95
  Tasks: 856 total,   2 running, 447 sleeping,   0 stopped,  36 zombie
  %Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  KiB Mem : 52802012+total, 11152644+free, 24536944 used, 39195673+buff/cache
  KiB Swap:        0 total,        0 free,        0 used. 49137280+avail Mem
  

• without empty line

  [!TIP|label:references:]

  • Exiting with "q" and "d" - SED Tutorial

  manual:

  • The "q" command prints the current line again in less the -n flag was used on the command line and exits the script completely

  q[exit-code]
    (quit) Exit sed without processing any more commands or input.
  
  Q[exit-code]
    (quit) This command is the same as q, but will not print the contents of pattern space. Like q, it provides the ability to return an exit code to the caller.

  $ top -bn1 | sed -e '/^$/Q'
  top - 03:45:55 up 258 days, 19:09,  1 user,  load average: 0.17, 0.51, 0.77
  Tasks: 857 total,   2 running, 448 sleeping,   0 stopped,  36 zombie
  %Cpu(s):  0.1 us,  0.4 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  KiB Mem : 52802012+total, 11151089+free, 24546520 used, 39196272+buff/cache
  KiB Swap:        0 total,        0 free,        0 used. 49136291+avail Mem

transform literal string in regex

$ echo 'Hello (world) $100 + 20?' | sed -r 's/[.[\*^$(){}+?|]/\\&/g'
Hello \(world\) \$100 \+ 20\?

escape

[!NOTE|label:references:]

• * How to escape single quotes within single quoted strings

• [\ to \](https://stackoverflow.com/a/67170003/2940319)

  $ echo "\(\)" | sed 's/\\/\\\\\\/g'
  \\\(\\\)

  • more

    alias rxvt='urxvt -fg'\''#111111'\'' -bg '\''#111111'\''
    #          │         │┊┊|       │┊┊│     │┊┊│       │┊┊│
    #          └─STRING──┘┊┊└─STRIN─┘┊┊└─STR─┘┊┊└─STRIN─┘┊┊│
    #                     ┊┊         ┊┊       ┊┊         ┊┊│
    #                     ┊┊         ┊┊       ┊┊         ┊┊│
    #                     └┴─────────┴┴───┰───┴┴─────────┴┘│
    #                         All escaped single quotes    │
    #                                                      │
    #                                                      ?
    
    alias rc='sed '"'"':a;N;$!ba;s/\n/, /g'"'"
    alias rc='sed '\'':a;N;$!ba;s/\n/, /g'\'

tricky

• add ' or " to strings

  [!NOTE|label:references:]

  • * iMarslo: sed quota

  • How can I keep quotes in Bash arguments?

  • How can I preserve quotes in printing a Bash script's arguments?

  $ GIT_OPT="-a -b --c=1 2 3 -d=4 5 6"
  $ GIT_OPT=$(echo "${GIT_OPT}" |
              sed -r 's/\s+-/\n-/g' |
              sed -r "s/^([^=]+)=(.+)$/\1='\2'/g" |
              sed -e 'N;s/\n/ /'
             )
  $ echo $GIT_OPT
  -a -b --c='1 2 3' -d='4 5 6'
  
  $ GIT_OPT="--c=1 2 3 --d=4 5 6"
  $ GIT_OPT=$(echo "${GIT_OPT}" |
              sed -r 's/\s+--/\n--/g' |
              sed -r "s/^([^=]+)=(.+)$/\1='\2'/g" |
              sed -e 'N;s/\n/ /'
             )
  $ echo $GIT_OPT
  --c='1 2 3' --d='4 5 6'
  
  $ GIT_OPT="--c=1 2 3 --d=4 5 6"
  $ GIT_OPT=$(echo "${GIT_OPT}" |
              sed -r 's/\s+--/\n--/g' |
              sed -r "s/^([^=]+)=(.+)$/\1=\"\2\"/g" |
              sed -e 'N;s/\n/ /'
             )
  $ echo $GIT_OPT
  --c="1 2 3" --d="4 5 6"