sed

[!NOTE] references:

CHARACTERREGULAR EXPRESSIONEXPLANATION

[[:alnum:]]

[A-Za-z0-9]

Alphanumeric characters

[[:alpha:]]

[A-Za-z]

Alphabetic characters

[[:blank:]]

[ \t]

Space or tab characters only

[[:cntrl:]]

[\x00-\x1F\x7F]

Control characters

[[:digit:]]

[0-9]

Numeric characters

[[:graph:]]

[!-~]

Printable and visible characters

[[:lower:]]

[a-z]

Lower-case alphabetic characters

[[:print:]]

[ -~]

Printable (non-Control) characters

[[:punct:]]

[!-/:-@[-{-~]`

Punctuation characters

[[:space:]]

[ \t\v\f\n\r]

All whitespace chars

[[:upper:]]

[A-Z]

Upper-case alphabetic characters

[[:xdigit:]]

[0-9a-fA-F]

Hexadecimal digit characters

execute multiple sed commands

[!TIP]

-e command
       Append the editing commands specified by the command argument to the list of commands.

references:

example : show only root and nobody in /etc/passwd

  • -e :

    $ sed -n -e '/^root/p' -e '/^nobody/p' /etc/passwd
    nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
    root:*:0:0:System Administrator:/var/root:/bin/sh
  • ; :

    $ sed -n -e '/^root/p;/^nobody/p' /etc/passwd
    nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
    root:*:0:0:System Administrator:/var/root:/bin/sh
  • '{}' :

    $ sed -n '{
        /^root/p
        /^nobody/p
      }' /etc/passwd
    nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
    root:*:0:0:System Administrator:/var/root:/bin/sh

range

specific line

  • 2nd line : N <opt>

until empty line

$ top -bn1 | sed -n '0,/^\s*$/p'
top - 03:41:45 up 258 days, 19:05,  1 user,  load average: 2.33, 0.92, 0.95
Tasks: 856 total,   2 running, 447 sleeping,   0 stopped,  36 zombie
%Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52802012+total, 11152644+free, 24536944 used, 39195673+buff/cache
KiB Swap:        0 total,        0 free,        0 used. 49137280+avail Mem
  • or

    $ top -bn1 | sed -e '/^$/Q'
    top - 03:45:55 up 258 days, 19:09,  1 user,  load average: 0.17, 0.51, 0.77
    Tasks: 857 total,   2 running, 448 sleeping,   0 stopped,  36 zombie
    %Cpu(s):  0.1 us,  0.4 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 52802012+total, 11151089+free, 24546520 used, 39196272+buff/cache
    KiB Swap:        0 total,        0 free,        0 used. 49136291+avail Mem

n~m range

  • n~m lines : n,m <opt>

  • n to end lines : n,$ <opt>

  • m lines starting with n : n, +m <opt>

  • start n skip m via ~ :

    patternmatchescomments

    1~2

    1,3,5,7,...

    start frmo 1, print every 2 lines

    2~2

    2,4,6,8,...

    start from 2, print every 2 lines

    1~3

    1,4,7,10,...

    start from 1, print every 3 lines

    2~3

    2,5,8,11,...

    start from 2, print every 3 lines

pattern matches range

  • between pattern_1 to pattern_2 : /pattern_1/,/pattern_2/ <opt>

  • first line to pattern_2 : 0,/pattern_2/ <opt>

from pattern to first empty line

$ top -bn1 | sed -n '/^top.*/,/^\s*$/p'
top - 03:49:02 up 258 days, 19:13,  1 user,  load average: 0.43, 0.41, 0.68
Tasks: 853 total,   1 running, 448 sleeping,   0 stopped,  36 zombie
%Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52802012+total, 11150942+free, 24543500 used, 39196720+buff/cache
KiB Swap:        0 total,        0 free,        0 used. 49136582+avail Mem

print

  • print every line twice

    $ sed 'p' employee.txt
    101,John Doe,CEO
    101,John Doe,CEO
    102,Jason Smith,IT Manager
    102,Jason Smith,IT Manager
    103,Raj Reddy,Sysadmin
    103,Raj Reddy,Sysadmin
    104,Anand Ram,Developer
    104,Anand Ram,Developer
    105,Jane Miller,Sales Manager
    105,Jane Miller,Sales Manager
  • print all lines : $ sed -n 'p' employee.txt

range print

  • print the 2nd line : $ sed -n '2 p' employee.txt

  • n,m range

    • print 1~4 lines : $ sed -n '1,4 p' employee.txt

    • print all lines since the 2nd line: $ sed -n '2,$ p' employee.txt

  • ~ to skip lines

    • print only odd numbered lines : sed -n '1~2 p' employee.txt

  • + ( n, +m ) : sed -n 'n,+m p' employee.txt

  • find pattern to the end :

    $ sed -n '/Raj/,$ p' employee.txt
    103,Raj Reddy,Sysadmin
    104,Anand Ram,Developer
    105,Jane Miller,Sales Manager
  • find pattern and line after the matches line :

    $ sed -n '/Raj/, +1 p' employee.txt
    103,Raj Reddy,Sysadmin
    104,Anand Ram,Developer
  • find pattern to 4th line :

    $ sed -n '/Raj/,4 p' employee.txt
    103,Raj Reddy,Sysadmin
    104,Anand Ram,Developer
  • find pattern until find another pattern ( Jason to Anand ) :

    $ sed -n '/Jason/,/Anand/p' employee.txt
    102,Jason Smith,IT Manager
    103,Raj Reddy,Sysadmin
    104,Anand Ram,Developer

[!NOTE|label:references:]

# original
$ printf "%s\r\n\r\n%s\r\n%s\r\n\r\n%s\r\n" 1 2 3 4 | command cat -A | nl
     1  1^M$
     2  ^M$
     3  2^M$
     4  3^M$
     5  ^M$
     6  4^M$

# result
$ printf "%s\r\n\r\n%s\r\n%s\r\n\r\n%s\r\n" 1 2 3 4 | sed -n '/^\s*\r$/ ='
2
5
# or
$ sed -n '/^\s*\r$/ =' < <(printf "%s\r\n\r\n%s\r\n%s\r\n\r\n%s\r\n" 1 2 3 4)
2
5

# first matched line number
$ printf "%s\r\n\r\n%s\r\n%s\r\n%s\r\n" 1 2 3 4 | sed -n '/^\s*\r$/ =' | head -n1
2

# last matched line number
$ printf "%s\r\n\r\n%s\r\n%s\r\n\r\n%s\r\n" 1 2 3 4 | sed -n '/^\s*\r$/ =' | tail -n1
5
  • or

    # via sed
    $ sed -n '10,20{=;p}'
    ## or ##
    $ nl file.txt | sed -n '10,20p'
    ## or ##
    $ sed -n '10,20{=;p}' file.txt | sed '{N; s/\n/ /}'
    
    # via awk
    $ awk 'FNR==10,FNR==20 {print FNR ":" $0}' file.txt
    ## or ##
    $ awk 'FNR >= 10 {print FNR ":" $0}; FNR == 20 {exit}' file.txt
    ## or ##
    $ awk -v OFS=: 'FNR >= 10 {print FILENAME, FNR, $0}; FNR == 20 {nextfile}' ./*.txt

delete

delete all

$ sed 'd' employee.txt

range delete

  • delete the 2nd line : $ sed '2 d' /path/to/file

  • delete between 1 and 4 lines : $ sed '1,4 d' /path/to/file

conditional delete

  • delete all empty lines: $ sed '/^$/ d' /path/to/file

  • delete all comment lines : $ sed '/^#/ d' /path/to/file

substitute

substitute-flags

flagcomments

i

ignore case flag

g

global flag

1,2,...

number flag

p

print flag

w

write flag

e

execute flag

multiple replaces

$ sed '{
    s/Developer/IT Manager/
    s/Manager/Director/
  }' employee.txt
101,John Doe,CEO
102,Jason Smith,IT Director
103,Raj Reddy,Sysadmin
104,Anand Ram,IT Director
105,Jane Miller,Sales Director

get matched pattern

&

When & is used in the replacement-string, it replaces it with whatever text matched the original-string or the regular-expression.

$ sed 's/^[0-9][0-9][0-9]/<&>/g' employee.txt
<101>,John Doe,CEO
<102>,Jason Smith,IT Manager
<103>,Raj Reddy,Sysadmin
<104>,Anand Ram,Developer
<105>,Jane Miller,Sales Manager

substitution grouping

$ sed 's/^\([0-9][0-9][0-9]\).*/<\1>/g' employee.txt
<101>
<102>
<103>
<104>
<105>

or via -r

$ sed -r 's/^([0-9][0-9][0-9]).*/<\1>/g' employee.txt
<101>
<102>
<103>
<104>
<105>

# or
$ sed -nr 's/^([0-9][0-9][0-9])(.*)/<\1>\2/gp' employee.txt
<101>,John Doe,CEO
<102>,Jason Smith,IT Manager
<103>,Raj Reddy,Sysadmin
<104>,Anand Ram,Developer
<105>,Jane Miller,Sales Manager

cheatsheet

get first matching patten ( for CERTIFICATE )

[!TIP]

  • sample.crt

    $ cat sample.crt
    -----BEGIN CERTIFICATE-----
    first paragraph
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    second paragraph
    -----END CERTIFICATE-----
  • regular pattern

    $ cat sample.crt | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
    -----BEGIN CERTIFICATE-----
    first paragraph
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    second paragraph
    -----END CERTIFICATE-----
    
    # or for short
    $ cat sample.crt | sed -ne '/-BEGIN/,/-END/p'
    -----BEGIN CERTIFICATE-----
    first paragraph
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    second paragraph
    -----END CERTIFICATE-----
  • get first

    [!TIP]

    more :

    # or `-n /../p`
    #                     `-n`                                         `p`
    #                       |                                           |
    #                       v                                           v
    $ cat sample.crt | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p; /-END CERTIFICATE-/q'
    -----BEGIN CERTIFICATE-----
    first paragraph
    -----END CERTIFICATE-----
    
    # or `/../!d`
    #                   no `-n`                                     `!d`
    #                     |                                           |
    #                     v                                           v
    $ cat sample.crt | sed '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/!d; /-END CERTIFICATE-/q'
    -----BEGIN CERTIFICATE-----
    first paragraph
    -----END CERTIFICATE-----
    
    # or for short
    $ cat sample.crt | sed '/-END CERTIFICATE-/q'
    -----BEGIN CERTIFICATE-----
    first paragraph
    -----END CERTIFICATE-----
    
    # or
    $ cat sample.crt | sed '/-END/q'
    -----BEGIN CERTIFICATE-----
    first paragraph
    -----END CERTIFICATE-----

remove both '#' and empty lines

[!NOTE|label:references:]

$ .. | sed -r '/^(#.*)$/d' | sed -r '/^\s*$/d'

# or
$ .. | sed -r '/^(#.*)$/d;/^\s*$/d'

# or
$ .. | sed -r '/(^#.*)|(^\s*)$/d'
  • example

    $ ldapsearch CN=marslo DN | sed -r '/^(#.*)$/d;/^\s*$/d'
    dn: CN=marslo,OU=Workers,DC=company,DC=com

remove tailing spaces

[!TIP|label:available patterns]

  • 's/[ \t]*$//'

  • 's/[[:blank:]]*$//'

  • 's/[[:space:]]*$//'

$ cal | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊
····················␊

# remove empty line
$ cal | sed -r '/^(#.*)$/d;/^\s*$/d' | cat -pp -A
····January·2024····␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31·········␊

# remove trailing spaces
$ cal | sed 's/[[:space:]]*$//' | cat -pp -A
····January·2024␊
Su·Mo·Tu·We·Th·Fr·Sa␊
····1··2··3··4··5··6␊
·7··8··9·10·11·12·13␊
14·15·16·17·18·19·20␊
21·22·23·24·25·26·27␊
28·29·30·31␊

show top summary

[!NOTE] see sed until empty line

  • contains empty line

    $ top -bn1 | sed -n '0,/^\s*$/p'
    top - 03:41:45 up 258 days, 19:05,  1 user,  load average: 2.33, 0.92, 0.95
    Tasks: 856 total,   2 running, 447 sleeping,   0 stopped,  36 zombie
    %Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 52802012+total, 11152644+free, 24536944 used, 39195673+buff/cache
    KiB Swap:        0 total,        0 free,        0 used. 49137280+avail Mem
    
  • without empty line

    [!TIP] references:

    manual:

    • The "q" command prints the current line again in less the -n flag was used on the command line and exits the script completely

    q[exit-code]
      (quit) Exit sed without processing any more commands or input.
    
    Q[exit-code]
      (quit) This command is the same as q, but will not print the contents of pattern space. Like q, it provides the ability to return an exit code to the caller.
    $ top -bn1 | sed -e '/^$/Q'
    top - 03:45:55 up 258 days, 19:09,  1 user,  load average: 0.17, 0.51, 0.77
    Tasks: 857 total,   2 running, 448 sleeping,   0 stopped,  36 zombie
    %Cpu(s):  0.1 us,  0.4 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 52802012+total, 11151089+free, 24546520 used, 39196272+buff/cache
    KiB Swap:        0 total,        0 free,        0 used. 49136291+avail Mem

escape

[!NOTE|label:references:]

  • [\ to \](https://stackoverflow.com/a/67170003/2940319)

    $ echo "\(\)" | sed 's/\\/\\\\\\/g'
    \\\(\\\)
    • more

      alias rxvt='urxvt -fg'\''#111111'\'' -bg '\''#111111'\''
      #          │         │┊┊|       │┊┊│     │┊┊│       │┊┊│
      #          └─STRING──┘┊┊└─STRIN─┘┊┊└─STR─┘┊┊└─STRIN─┘┊┊│
      #                     ┊┊         ┊┊       ┊┊         ┊┊│
      #                     ┊┊         ┊┊       ┊┊         ┊┊│
      #                     └┴─────────┴┴───┰───┴┴─────────┴┘│
      #                         All escaped single quotes    │
      #                                                      │
      #                                                      ?
      
      alias rc='sed '"'"':a;N;$!ba;s/\n/, /g'"'"
      alias rc='sed '\'':a;N;$!ba;s/\n/, /g'\'

tricky

  • add ' or " to strings

    [!NOTE|label:references:]

    $ GIT_OPT="-a -b --c=1 2 3 -d=4 5 6"
    $ GIT_OPT=$(echo "${GIT_OPT}" |
                sed -r 's/\s+-/\n-/g' |
                sed -r "s/^([^=]+)=(.+)$/\1='\2'/g" |
                sed -e 'N;s/\n/ /'
               )
    $ echo $GIT_OPT
    -a -b --c='1 2 3' -d='4 5 6'
    
    $ GIT_OPT="--c=1 2 3 --d=4 5 6"
    $ GIT_OPT=$(echo "${GIT_OPT}" |
                sed -r 's/\s+--/\n--/g' |
                sed -r "s/^([^=]+)=(.+)$/\1='\2'/g" |
                sed -e 'N;s/\n/ /'
               )
    $ echo $GIT_OPT
    --c='1 2 3' --d='4 5 6'
    
    $ GIT_OPT="--c=1 2 3 --d=4 5 6"
    $ GIT_OPT=$(echo "${GIT_OPT}" |
                sed -r 's/\s+--/\n--/g' |
                sed -r "s/^([^=]+)=(.+)$/\1=\"\2\"/g" |
                sed -e 'N;s/\n/ /'
               )
    $ echo $GIT_OPT
    --c="1 2 3" --d="4 5 6"

Last updated