cmd

• compress

  • zip

  • tar

• extract

  • unzip

  • tar

• network

  • ip address

  • check port

• file

  • check file text or binary

• system

• clock/time/date

• download

• backup

• others

compress

zip

[!TIP|label:references:]

• how to zip files (with pattern) from another zip archive without extracting in linux

$ zip -r file.zip /path/to/folder

# with password
$ zip -r -P password file.zip /path/to/folder

# zip files via pipe ( with xargs )
$ find . -name "*.txt" | xargs zip file.zip

# repackage - zip specific files from another zip package
#               pattern  --copy
#              +--------+  ++
$ zip org.zip '**/bin/*.o' -U -O new.zip

# append new files into existing zip
$ zip -u file.zip file.txt
# or
$ zip -ur file.zip !(file.zip)

• create zip from stdin

  [!NOTE|label:references:]

  • * how to build Info-Zip for macos
  • * How do you specify filenames within a zip when creating it on the command line from a pipe?
  • stdin into zip command, how do I specify a file name?
  • Problems with zipnote writing to files

  $ echo 'foo bar' | zip > file.zip
  $ printf "@ -\n@=filename.txt\n" | zipnote -w file.zip

• delete after compress

  [!NOTE|label:references:]

  • -m, --move : delete files after adding to archive

  $ zip -m file.zip file.txt
  # or
  $ zip -mqj file.zip file.txt

tar

[!NOTE|label:references:] options:

SUFFIX	EXTRACT	COMPRESS
tar	-xf	-cf
tgz	-xzf	-czf
tar.gz	-xzf	-czf
tar.bz2	-xjf -xjSf	-cjf -cjSf
tar.xz	-xJf	-cJf
tar.Z	-xZf	-cZf
tar.lz	-x --lzip	-c --lzip

$ tar -czf file.tar.gz /path/to/folder
# or
$ tar cvjSf file.tar.bz2 *

• delete files after compress

  $ tar -cvjf archive.tar.bz2 --remove-files archive/

• remove files from tar

  $ tar vf file.tar --delete file.txt
  # or
  $ tar --delete -f file.tar file.txt
  
  # -- example --
  $ tar tf file.tar
  a.txt
  b.txt
  c.txt
  d.txt
  $ tar vf file.tar --delete a.txt
  $ tar tf file.tar
  b.txt
  c.txt
  d.txt

extract

unzip

$ unzip file.zip

# to specific folder
$ unzip file.zip -d /path/to/extract/folder

# with password
$ unzip file.zip -P password -d /path/to/extract/folder

# extract all zip files in a folder
$ find . -name "*.zip" -exec unzip {} \;

# extract specific `file.txt` in a zip file
$ unzip -p file.zip file.txt
# or
$ unzip -j file.zip -d /path/to/extract/folder file.txt

# unzip files via pipe ( with xargs )
$ unzip -l file.zip | grep -v Archive | awk '{print $4}' | xargs -I {} unzip -j file.zip {}

• get shasum for individual file in zip

  # without file name ( single file )
  $ unzip -p file.zip filename.txt | sha256sum
  1f2ec52b774368781bed1d1fb140a92e0eb6348090619c9291f9a5a3c8e8d151  -
  # or with wildcards
  $ unzip -p file.zip '*.txt' | sha256sum
  1f2ec52b774368781bed1d1fb140a92e0eb6348090619c9291f9a5a3c8e8d151  -
  
  # with file name ( multiple files )
  #                                          using `|` instead of `/` to avoid conflict with file path
  $ unzip -l file.zip |                                                   | |  |
          grep -E '.*.txt' --color=never |                                | |  |
          awk '{print $NF}' |                                             v v  v
          xargs -I '{}' bash -c "unzip -p file.zip {} | sha256sum | sed 's|-|{}|g'"
  1f2ec52b774368781bed1d1fb140a92e0eb6348090619c9291f9a5a3c8e8d151  filename.txt
  1f2ec52b774368781bed1d1fb140a92e0eb6348090619c9291f9a5a3c8e8d151  text.txt

tar

[!NOTE|label:references:] options:

SUFFIX	EXTRACT	COMPRESS
tar	-xf	-cf
tgz	-xzf	-czf
tar.gz	-xzf	-czf
tar.bz2	-xjf -xjSf	-cjf -cjSf
tar.xz	-xJf	-cJf
tar.Z	-xZf	-cZf
tar.lz	-x --lzip	-c --lzip

• references:

  • Deleting empty files in tar.gz file in bash

• check folder structure

  $ tar tf name.tar | awk -F'/' '{print $1}' | uniq

• strip path

  [!NOTE|label:references:]

  • How do I extract files without folder structure using tar

  # -- using `--strip-*` --
  $ tar xzf name.tar.gz -C /srv/www --strip-components 2
  # old version
  $ tar xzf name.tar.gz -C /srv/www --strip-path 2
  
  # -- using `--transform` --
  $ tar xzf name.tgz --transform='s/.*\///'
  # or
  $ tar -zxf name.tar.gz --absolute-names --no-anchored img*.gif --transform='s:^folder[1-3]/::'
  # or
  $ tar xf archpackage.tar --transform="s,usr/bin,myapp,;s,usr/share/doc/myapp,myapp," usr/bin/myapp  usr/share/doc/myapp/myapp-example-config.toml

• check shasum

  # shasum without file name
  $ tar -O -xf name.tar --wildcards ./usr/bin/*.tgz | sha256sum
  
  # shasum with file name
  $ tar tf name.tar |
        grep -E './usr/bin/.*.tgz' --color=never |
        xargs -I '{}' bash -c "tar -O -xf name.tar --wildcards {} | sha256sum | sed 's|-|{}|g'"

network

ip address

• get subnet IP address

  $ ip addr show eno1 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'
  192.168.1.105
  fe80::e5ca:1027:b572:9998

• get public IP address

  $ curl -4 icanhazip.com
  182.150.46.248

check port

$ sudo lsof -i:1111
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ss-server 903 nobody    8u  IPv4  20522      0t0  UDP *:1111
obfs-serv 909   root    7u  IPv4  20649      0t0  TCP *:1111 (LISTEN)

$ sudo netstatus -tunpla | grep 1111
tcp        0      0 0.0.0.0:1111            0.0.0.0:*               LISTEN      909/obfs-server
udp        0      0 0.0.0.0:1111            0.0.0.0:*                           903/ss-server

• list all ports

  $ lsof -i
  # or stop by ctrl-c
  $ lsof -i -r
  
  # or: https://www.commandlinefu.com/commands/view/2545/list-programs-with-open-ports-and-connections
  $ netstat -ntauple

• show apps that use internet connection

  $ lsof -P -i -n
  $ lsof -P -i -n | cut -f 1 -d " " | uniq | tail -n +2
  
  # or : https://www.commandlinefu.com/commands/view/3546/show-apps-that-use-internet-connection-at-the-moment.-multi-language
  $ ss -p
  $ ss -p | cut -f2 -sd\"

file

• create a bunch of dummy text files

  $ base64 /dev/urandom | head -c 33554432 | split -b 8192 -da 4 - dummy.

• modify timestamp interactively

  $ touch -d $(zenity --calendar --date-format=%F) filename

• change/modify timestamp

  $ touch --date "2010-01-05" /tmp/filename

• run a command whenever a file is touched

  $ ontouchdo(){ while :; do a=$(stat -c%Y "$1"); [ "$b" != "$a" ] && b="$a" && sh -c "$2"; sleep 1; done }

• efficiently print a line deep in a huge log file

  $ sed '1000000!d;q' huge.log
  # or
  $ sed '999995,1000005!d' < my_massive_file

• transmit a file like a http server

  $ while ( nc -l 80 < /file.htm > : ) ; do : ; done &

• create date-based tgz of current dir, runs in the background, very very cool

  $ alias tarred='( ( D=`builtin pwd`; F=$(date +$HOME/`sed "s,[/ ],#,g" <<< ${D/${HOME}/}`#-%F.tgz); tar --ignore-failed-read --transform "s,^${D%/*},`date +${D%/*}.%F`,S" -czPf "$F" "$D" &>/dev/null ) & )'
  
  # or
  $ alias tarred='( ( D=`builtin pwd`; F=$(date +$HOME/`sed "s,[/ ],#,g" <<< ${D/${HOME}/}`#-%F.tgz); S=$SECONDS; tar --ignore-failed-read --transform "s,^${D%/*},`date +${D%/*}.%F`,S" -czPf "$"F "$D" && logger -s "Tarred $D to $F in $(($SECONDS-$S)) seconds" ) & )'

• backup and remove files with access time older than 5 days

  $ tar -zcvpf backup_`date +"%Y%m%d_%H%M%S"`.tar.gz `find <target> -atime +5` 2> /dev/null | xargs rm -fr ;

• chmod with reference file

  $ chmod --reference file1 file2
  
  # or: https://www.commandlinefu.com/commands/view/5234/makes-the-permissions-of-file2-the-same-as-file1
  # tips: man setfacl
  $ getfacl file1 | setfacl --set-file=- file2

• share file via http 80

  $ nc -v -l 80 < file.ext

check file text or binary

[!NOTE|label:references:]

• How to check if a file is binary?

  • file
  • grep
  • perl
• grep: (standard input): binary file matches

$ find . -type f -print0 | perl -0nE 'say if -f and -s _ and -T _'

# verify
$ find . -type f -print0 | perl -0nE 'say if -f and -s _ and -T _' | grep -a -E '\.db$'

system

• execute commnd at a specific time

  $ echo "ls -l" | at midnight
  # or
  $ at midnight <<< "ls -l"
  # or
  $ echo "ls -l" | at 10:00
  
  # set alarm to wake up: https://www.commandlinefu.com/commands/view/2502/set-an-alarm-to-wake-up-2
  $ echo "aplay path/to/song" | at [time]
  
  # timer with sound alarm: https://www.commandlinefu.com/commands/view/8341/timer-with-sound-alarm
  $ say(){ mplayer -user-agent Mozilla "http://translate.google.com/translate_tts?tl=en&q=$(echo $* | sed 's#\ #\+#g')" > /dev/null 2>&1 ; };
  $ sleep 3s && say "wake up, you bastard"
  
  # alarm with fade-in, for graceful awakening: https://www.commandlinefu.com/commands/view/9794/set-up-alarm-with-fade-in-for-graceful-awakening
  $ at 8:30 <<<'mpc volume 20; mpc play; for i in `seq 1 16`; do sleep 2; mpc volume +5; done'

• sort disk usage

  $ du | sort -gr

• 32bit or 64bit

  $ getconf LONG_BIT
  64

• list most offen used command

  $ history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head

• eavesdrop on your system

  $ diff <(lsof -p 1234) <(sleep 10; lsof -p 1234)
  # or
  $ tcpdump -i eth0 -s 0 -v -w /tmp/traffic.pcap

• synchronize date and time with a server over ssh

  $ date --set="$(ssh user@server date)"
  # or
  $ ssh -t user@host 'sudo date --set "$(ssh user@host date)"'

• last reboot time

  $ who -b

• current runlevel

  $ who -r

• backup files incremental with rsync to a ntfs-partition

  $ rsync -rtvu --modify-window=1 --progress /media/SOURCE/ /media/TARGET/

• make sudo forget password instantly

  $ sudo -K

• faster rsync

  $ rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>

• when was your OS installed?

  $ ls -lct /etc | tail -1 | awk '{print $6, $7}'

• check for login failures and summarize

  [!NOTE|label:references:]

  • * iMarslo: displays the attempted via SSH

  $ zgrep "Failed password" /var/log/auth.log* | awk '{print $9}' | sort | uniq -c | sort -nr | less

• find a count of how many times invalid users have attempted to access your system

  $ gunzip -c /var/log/auth.log.*.gz | cat - /var/log/auth.log /var/log/auth.log.0 | grep "Invalid user" | awk '{print $8;}' | sort | uniq -c | less

clock/time/date

[!TIP|lavel:see also:]

• * iMarslo: date

• get how many days left this years

  $ echo "There are $(($(date +%j -d"Dec 31, $(date +%Y)")-$(date +%j))) left in year $(date +%Y)."
  There are 323 left in year 2014.

• get week number

  $ date +"%V"
  08

• display a cool clock on your terminal

  $ watch -t -n1 "date +%T|figlet"

• put a console clock in top right corner

  $ while sleep 1;do tput sc;tput cup 0 $(($(tput cols)-29));date;tput rc;done &

• remind yourself to leave in 15 minutes

  $ leave +15

• binary clock

  $ watch -n 1 'echo "obase=2;`date +%s`" | bc'

• countdown clock

  $ MIN=1 && for i in $(seq $(($MIN*60)) -1 1); do echo -n "$i, "; sleep 1; done; echo -e "\n\nBOOOM! Time to start."

• countdown clock for new year

  $ while V=$((`date +%s -d"2025-01-01"`-`date +%s`));do if [ $V == 0 ];then figlet 'Happy New Year!';break;else figlet $V;sleep 1;clear;fi;done

download

• parallel file downloading with wget

  $ wget -nv http://en.wikipedia.org/wiki/Linux -O- | egrep -o "http://[^[:space:]]*.jpg" | xargs -P 10 -r -n 1 wget -nv

• log download speed

  $ echo $(date +%s) > start-time; URL=http://www.google.com; while true; do echo $(curl -L --w %{speed_download} -o/dev/null -s $URL) >> bps; sleep 10; done &
  
  # show as graph view
  $ gnuplot -persist <(echo "plot 'bps' with lines")

• download every font from dafont.com to current folder

  $ d="www.dafont.com/alpha.php?"; for c in {a..z}; do l=`curl -s "${d}lettre=${c}"|sed -n 's/.*ge=\([0-9]\{2\}\).*/\1/p'`; for((p=1;p<=l;p++));do for u in `curl -s "${d}page=${p}&lettre=${c}"|egrep -o "http\S*.com/dl/\?f=\w*"`; do aria2c "${u}"; done; done; done

• download free e-books

  $ wget -erobots=off --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" -H -r -l2 --max-redirect=1 -w 5 --random-wait -PmyBooksFolder -nd --no-parent -A.pdf http://URL

backup

• database backup

  $ backup() { for i in "$@"; do cp -va $i $i.$(date +%Y%m%d-%H%M%S); done }

• backup mysql

  $ for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump $I | gzip > "$I.sql.gz"; done

• backup multiple files

  $ cp -bfS.bak filename filename

others

• advanced ls

  $ find $PWD -maxdepth 1 -printf '%.5m %10M %#9u:%-9g %#5U:%-5G [%AD | %TD | %CD] [%Y] %p\n'
  
  # or
  $ find $PWD -maxdepth 1 -printf '%.5m %10M %#9u:%-9g %#5U:%-5G [%AD | %TD | %CD] [%Y] %p\n' | sort -rgbS 50%

• DOS tree

  $ find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g':wa
  .
  |____a_b
  |____b_a

• the most frequent used words of a text file

  $ cat WAR_AND_PEACE_By_LeoTolstoi.txt | tr -cs "[:alnum:]" "\n"| tr "[:lower:]" "[:upper:]" | awk '{h[$1]++}END{for (i in h){print h[i]" "i}}'|sort -nr | cat -n | head -n 30

• count how many times a string appears in a (source code) tree

  $ grep -or string path/ | wc -l

• matrix

  $ echo -e "\e[32m"; while :; do for i in {1..16}; do r=`echo -e "\x$(echo $(($(($RANDOM % 26)) + $(echo $([[ $(($RANDOM % 3)) == 1 ]] && echo -n 48 || echo -n $([[ $(($RANDOM % 3)) == 2 ]] && echo -n 97 || echo -n 65))))) 16 o p | dc)"`; if [[ $(($RANDOM % 5)) == 1 ]]; then if [[ $(($RANDOM % 4)) == 1 ]]; then v+="\e[1m $r   "; else v+="\e[2m $r   "; fi; else v+="     "; fi; done; echo -e "$v"; v=""; done
  
  # https://www.commandlinefu.com/commands/view/2531/matrix-style
  $ tr -c "[:digit:]" " " < /dev/urandom | dd cbs=$COLUMNS conv=unblock | GREP_COLOR="1;32" grep --color "[^ ]"
  
  # https://www.commandlinefu.com/commands/view/3652/matrix-style
  $ echo -e "\e[32m"; while :; do for i in {1..16}; do r="$(($RANDOM % 2))"; if [[ $(($RANDOM % 5)) == 1 ]]; then if [[ $(($RANDOM % 4)) == 1 ]]; then v+="\e[1m $r "; else v+="\e[2m $r "; fi; else v+=" "; fi; done; echo -e "$v"; v=""; done
  
  # https://www.commandlinefu.com/commands/view/4352/another-matrix-style-implementation
  $ COL=$(( $(tput cols) / 2 )); clear; tput setaf 2; while :; do tput cup $((RANDOM%COL)) $((RANDOM%COL)); printf "%$((RANDOM%COL))s" $((RANDOM%2)); done
  
  # https://www.commandlinefu.com/commands/view/2542/matrix-style
  $ LC_ALL=C tr -c "[:digit:]" " " < /dev/urandom | dd cbs=$COLUMNS conv=unblock | GREP_COLOR="1;32" grep --color "[^ ]"
  
  # https://www.commandlinefu.com/commands/view/4384/another-matrix-style-implementation
  $ echo -ne "\e[32m" ; while true ; do echo -ne "\e[$(($RANDOM % 2 + 1))m" ; tr -c "[:print:]" " " < /dev/urandom | dd count=1 bs=50 2> /dev/null ; done
  
  # https://www.commandlinefu.com/commands/view/2615/matrix-style
  $ echo -e "\e[31m"; while $t; do for i in `seq 1 30`;do r="$[($RANDOM % 2)]";h="$[($RANDOM % 4)]";if [ $h -eq 1 ]; then v="\e[1m $r";else v="\e[2m $r";fi;v2="$v2 $v";done;echo -e $v2;v2="";done;

• notify send in Gnome

  $ notify-send ["<title>"] "<body>"

• notify via inotifywait

  $ inotifywait -mrq -e CREATE --format %w%f /path/to/dir | while read FILE; do chmod g=u "$FILE"; done

• change xterm window title

  $ echo "^[]0;My_Title_Goes _Here^G"

• retry previous cmd until succeed

  $ until !!; do :; done
  
  # or: https://www.commandlinefu.com/commands/view/12238/retry-the-previous-command-until-it-exits-successfully
  $ until !!; do done
  
  # or: https://www.commandlinefu.com/commands/view/12236/retry-the-previous-command-until-it-exits-successfully
  $ !!; while [ $? -ne 0 ]; do !!; done

• retry until succeed

  $ util <command>; do echo "retyring"; sleep 1; done

• check unread Gmail from the command line

  $ curl -u username:password --silent "https://mail.google.com/mail/feed/atom" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | sed -n "s/<title>\(.*\)<\/title.*name>\(.*\)<\/name>.*/\2 - \1/p"
  
  # from mac:
  $ curl -u username:password --silent "https://mail.google.com/mail/feed/atom" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | perl -pe 's/^<title>(.*)<\/title>.*<name>(.*)<\/name>.*$/$2 - $1/'
  
  # or: https://www.commandlinefu.com/commands/view/9490/check-your-unread-gmail-from-the-command-line
  $ curl -u username --silent "https://mail.google.com/mail/feed/atom" | awk 'BEGIN{FS="\n";RS="(</entry>\n)?<entry>"}NR!=1{print "\033[1;31m"$9"\033[0;32m ("$10")\033[0m:\t\033[1;33m"$2"\033[0m"}' | sed -e 's,<[^>]*>,,g' | column -t -s $'\t'

• search cmdful via curl

  $ cmdfu(){ curl "http://www.commandlinefu.com/commands/matching/$@/$(echo -n $@ | openssl base64)/plaintext"; }
  
  # or: https://www.commandlinefu.com/commands/view/10233/search-commandlinefu.com-from-the-command-line-using-the-api
  # usage: cmdfu search 'command'
  $ cmdfu(){ curl "http://www.commandlinefu.com/commands/matching/$(echo "$@" | sed 's/ /-/g')/$(echo -n $@ | base64)/plaintext" --silent | vim -R - }
  
  # or: https://www.commandlinefu.com/commands/view/9654/search-commandlinefu.com-from-the-command-line-using-the-api
  $ cmdfu(){ curl "http://www.commandlinefu.com/commands/matching/$@/$(echo -n $@ | openssl base64)/plaintext" --silent | sed "s/\(^#.*\)/\x1b[32m\1\x1b[0m/g" | less -R }

• backup cmdfu commands

  # usage: clfavs username password num_favourite_commands /path/to/file
  $ clfavs(){ URL="http://www.commandlinefu.com"; wget -O - --save-cookies c --post-data "username=$1&password=$2&submit=Let+me+in" $URL/users/signin; for i in `seq 0 25 $3`; do wget -O - --load-cookies c $URL/commands/favourites/plaintext/$i >>$4; done; rm -f c; }
  # or
  $ clfavs(){ URL="http://www.commandlinefu.com"; wget -O - --save-cookies c --post-data "openid=$1&submit=Let+me+in" $URL/users/openid; for i in `seq 0 25 $3`; do wget -O - --load-cookies c $URL/commands/favourites/plaintext/$i >>$4; done; rm -f c; }