ssh

• ssh key

  • generate ssh key

  • get servers public key

  • add ssh key into agent

  • get public key from private key

  • verify public and private key pair

  • generate new passphrase

  • get fingerprinter

  • with openssl

  • keys performance

• ssh

  • force use password

  • allows ssh-rsa

  • force bypass ~/.ssh/config

  • ssh and tar

    • copy multiple files to remote server

    • tar all and extra in remote

    • copy local file content into remote

  • with proxy

    • using command directly

  • start app remotely

    • launch firefox

  • create a persistent connection to a machine

• ssh certificate

  • SSH User Certificates

    • ca

    • sign and generate cert key

    • update for existing key

    • login via specific cert

  • SSH Host Certificates

• ssh tunnel

  • two servers

    • -L

    • -R

  • three serves

    • scenario 1

    • scenario 2

• config

  • ssh config

  • sshd_config

    • banner and motd

    • disable login password

    • disallow group to use password

    • disallowing user to use tcp forwarding

    • displaying a special banner for users not in the staff group

    • allowing root login from host rootallowed.example.com

    • allowing anyone to use gatewayports from the local net

    • avoid ssh timeout by sending a keepalive message

• debug

  • debug git

  • debug ssh

    • check sshd log

• tips

  • play video via ssh

  • displays failed login via SSH

  • compare files via ssh

  • disconnect

  • execute shell commands via ssh

    • ctrl-c to break while loop

    • execute local script via ssh

  • how ssh works

reference:

• * How SSH Really Works
• SSH port forwarding - Example, command, server config
• 透过代理连接SSH
• SSH ProxyCommand example: Going through one host to reach another server
• OpenSSH/Cookbook/Proxies and Jump Hosts
• Tunneling SSH over PageKite
• Transparent Multi-hop SSH
• SSH via HTTP proxy in OSX
• ssh key
• 6 ssh authentication methods to secure connection (sshd_config)
• 8 ways to prevent brute force SSH attacks in Linux (CentOS/RHEL 7)
• * SSH 是如何工作的 | How does SSH work

ssh key

[!TIP|label:resolve ssh permission issue:]

$ sudo su devops -c "restorecon -Rf ~/.ssh"
# or
$ /usr/sbin/restorecon .ssh/ .ssh/authorized_keys

generate ssh key

$ keyname='marslo@china'

# rsa4096
$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/${keyname} -C "${keyname}" -P "" -q

# ed25519
$ ssh-keygen -t ed25519 -C "${keyname}" -f ~/.ssh/${keyname} -P '' -q
$ ssh-keygen -t ed25519 -o -a 100 -C "${keyname}" -f ~/.ssh/${keyname} -P '' -q

get servers public key

$ ssh-keyscan -H www.sample.com

# or
$ ssh-keyscan -p 29418 -t rsa www.sample.com
$ ssh-keyscan -p 29418 -t rsa www.sample.com >> ~/.ssh/known_hosts

• upload the local ~/.ssh/know_hosts

  [!NOTE|label:references:]

  • We updated our RSA SSH host key

  $ ssh-keygen -R github.com
  # Host github.com found: line 63
  /home/marslo/.ssh/known_hosts updated.
  Original contents retained as /home/marslo/.ssh/known_hosts.old
  
  # or
  $ curl -sL https://api.github.com/meta | jq -r '.ssh_keys | .[]'  | sed -e 's/^/github.com /' >> ~/.ssh/know_hosts
  
  # more details
  $ curl -sL https://api.github.com/meta | jq -r '.ssh_keys | .[]'
  ssh-ed25519 AAAA***9GKJl
  ecdsa-sha2-nistp256 AAAA***ockg=
  ssh-rsa AAAA***wsjk=
  $ curl -sL https://api.github.com/meta | jq -r '.ssh_keys | .[]'  | sed -e 's/^/github.com /'
  github.com ssh-ed25519 AAAA***9GKJl
  github.com ecdsa-sha2-nistp256 AAAA***ockg=
  github.com ssh-rsa AAAA***wsjk=

add ssh key into agent

$ ssh-add ~/.ssh/${keyname}

# e.g.:
$ ssh-add ~/.ssh/id_ed25519
Identity added: /Users/marslo/.ssh/id_ed25519 (marslo@devops)

$ ssh-agent -s
SSH_AUTH_SOCK=/var/folders/s3/mg_f3cv54nn7y758j_t46zt40000gn/T//ssh-MgCrKA3ZS06N/agent.25376; export SSH_AUTH_SOCK;
SSH_AGENT_PID=25377; export SSH_AGENT_PID;
echo Agent pid 25377;

get public key from private key

$ ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub

verify public and private key pair

$ diff [-qs] <( ssh-keygen -y -e -f /path/to/private ) \
             <( ssh-keygen -y -e -f /path/to/public.pub )

generate new passphrase

$ ssh-keygen -p -f /path/to/private

get fingerprinter

• sha256:

  $ ssh-keygen -l -f ~/.ssh/id_rsa
  
  # or
  $ ssh-keygen -l -f ~/.ssh/id_rsa.pub

• md5:

  $ ssh-keygen -E md5 -l -f ~/.ssh/id_rsa
  
  # or
  $ ssh-keygen -E md5 -l -f ~/.ssh/id_rsa.pub

with openssl

$ openssl pkey -in ~/.ssh/ec2/primary.pem -pubout -outform DER | openssl md5 -c

keys performance

$ openssl speed rsa1024 rsa2048 dsa1024 dsa2048
Doing 1024 bit private rsa's for 10s: 91211 1024 bit private RSA's in 9.97s
Doing 1024 bit public rsa's for 10s: 1161461 1024 bit public RSA's in 9.93s
Doing 2048 bit private rsa's for 10s: 12305 2048 bit private RSA's in 9.94s
Doing 2048 bit public rsa's for 10s: 403455 2048 bit public RSA's in 9.98s
Doing 1024 bit sign dsa's for 10s: 84873 1024 bit DSA signs in 10.00s
Doing 1024 bit verify dsa's for 10s: 109544 1024 bit DSA verify in 9.99s
Doing 2048 bit sign dsa's for 10s: 30010 2048 bit DSA signs in 9.99s
Doing 2048 bit verify dsa's for 10s: 33202 2048 bit DSA verify in 9.98s
OpenSSL 1.0.2t  10 Sep 2019
built on: reproducible build, date unspecified
options:bn(64,64) rc4(ptr,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang -I. -I.. -I../include  -fPIC -fno-common -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -arch x86_64 -O3 -DL_ENDIAN -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
                  sign    verify    sign/s verify/s
rsa 1024 bits 0.000109s 0.000009s   9148.5 116964.9
rsa 2048 bits 0.000808s 0.000025s   1237.9  40426.4
                  sign    verify    sign/s verify/s
dsa 1024 bits 0.000118s 0.000091s   8487.3  10965.4
dsa 2048 bits 0.000333s 0.000301s   3004.0   3326.9

ssh

force use password

$ ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no [e-mail]

allows ssh-rsa

[!TIP|label:references:]

• openssh v8.8

$ ssh -o HostkeyAlgorithms=+ssh-rsa -p <port> -l <user> <host>

• for git commands

  [!NOTE|label:references:]

  • since git v2.33.1

  $ GIT_SSH_COMMAND="ssh -o HostkeyAlgorithms=+ssh-rsa" git clone
  # since v2.41.0
  $ GIT_SSH_COMMAND="ssh -o HostkeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa" git clone
  
  # or
  $ git config [--global] core.sshCommand 'ssh -o HostKeyAlgorithms=+ssh-rsa'
  # since v2.41.0
  $ git config [--global] core.sshCommand 'ssh -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa'

force bypass ~/.ssh/config

[!NOTE|label:references:]

• How to tell ssh client to ignore ssh config file
• $ man ssh

       -F configfile
               Specifies an alternative per-user configuration file.  If a configuration file is given on
               the command line, the system-wide configuration file (/etc/ssh/ssh_config) will be ignored.
               The default for the per-user configuration file is ~/.ssh/config.  If set to “none”, no
               configuration files will be read.

$ ssh -F none [e-mail]

# or
$ ssh -vvT -F /dev/null -i ~/.ssh/id_ed25519 [e-mail]

ssh and tar

[!TIP|label:references:]

• * iMarslo: bash sugar - find and tar for JENKINS_HOME

copy multiple files to remote server

$ tar cvzf - -T list_of_filenames | ssh hostname tar xzf -

# or
$ tar cvf - /path/*.jpg | ssh [e-mail] "tar xvf -"

tar all and extra in remote

# ssh -C
#    no `-z`      `-C`
#       |           |
#       v           v
$ tar cf - . | ssh -C hostname "cd ~/.marslo/test/; tar xvf -"
Warning: Permanently added '10.69.78.40' (ECDSA) to the list of known hosts.
./
./temp/
./a/
./a/a.txt
./c/
./b/

# tar z-flag
#     `-z`       no `-C`
#       |          |
#       v          v
$ tar cfz - . | ssh hostname "cd ~/.marslo/test/; tar xvzf -"

copy local file content into remote

[!NOTE|label:references:]

• How to copy a file without using scp inside an ssh session?

$ ssh user@remote 'cat > /path/to/remote/file' < /path/to/local/file

# or
$ cat /path/to/local/file | ssh user@remote 'cat > /path/to/remote/file'

# to retrieve the file
$ ssh user@remote 'cat /path/to/remote/file' > pbcopy

with proxy

using command directly

• Linux:

  $ ssh -o 'ProxyCommand nc -x proxy.url.com proxy-port %h %p' -vT [e-mail]
  
  # or
  $ ssh -o 'ProxyCommand corkscrew proxy.url.com proxy-port %h %p' -vT [e-mail]

• windows:

  • git for windows

    $ ssh -o 'ProxyCommand connect -H http://proxy.url.com:proxy-port %h %p' [e-mail]

  • cygwin

    $ apt-cyg install corkscrew
    $ ssh -o 'ProxyCommand corkscrew proxy.url.com proxy-port %h %p' [e-mail]
    
    # or
    $ apt-cyg install nc
    $ ssh -o 'ProxyCommand nc -X connect -x proxy.url.com:proxy-port %h %p' -vT [e-mail]

start app remotely

$ ssh -f [e-mail] DISPLAY=:0.0 smplayer movie.avi

launch firefox

$ ssh -fY user@REMOTESERVER firefox -no-remote

create a persistent connection to a machine

$ ssh -MNf <user>@<host>

ssh certificate

[!NOTE|label:references:]

• * OpenSSH/Cookbook/Certificate-based Authentication
• * Configure ssh certificate based authentication
• 14.3.3. Creating SSH CA Certificate Signing Keys

  • Files Associated with SSH Host Certificates

    • Certificate Authority (ca) - a private key generated for signing other keys

    • Certificate public key - the public component of the certificate authority

    • Host Public Key - the actual key that the SSH daemon uses to identify itself to the clients

    • Host Certificate - the signature made for the Host Public Key using the Certificate Authority

• How to Generate and Configure SSH Certificate-Based Authentication
• If you’re not using SSH certificates you’re doing SSH wrong
• How to configure and setup SSH certificates for SSH authentication
• How To Configure SSH Key-Based Authentication on a Linux Server
• Tightening SSH access using short-lived SSH certificates
• How to setup SSH certificates for SSH authentication

SSH User Certificates

ca

remote $ ssh-keygen -t ed25519 -b 4096 -f devops@ca -C 'devops@ca' -P '' -q
remote $ sudo cp devops@ca* /etc/ssh/
remote $ echo 'TrustedUserCAKeys /etc/ssh/[e-mail]' | sudo tee -a /etc/ssh/sshd_config

remote $ sudo grep TrustedUserCAKeys /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/[e-mail]

sign and generate cert key

# download devops@ca
#                                 key id  principals
#                                    v        v
local $ ssh-keygen -s devops@ca -I marslo -n marslo -V +52w ~/.ssh/marslo.pub
$ ls ~/.ssh/marslo*
/Users/marslo/.ssh/marslo  /Users/marslo/.ssh/marslo-cert.pub  /Users/marslo/.ssh/marslo.pub

• verify

  $ ssh-keygen -Lf ~/.ssh/marslo-cert.pub
  /Users/marslo/.ssh/marslo-cert.pub:
          Type: [e-mail] user certificate
          Public key: ED25519-CERT SHA256:JfJnCDxjWhLwW3BcBX3XycBr3dT3JtHTwD1M4H3828E
          Signing CA: ED25519 SHA256:5dNlpIIjX/pdoNSpmtfcGQijSrl3W87TByA9KeCe2M0 (using ssh-ed25519)
          Key ID: "marslo"
          Serial: 0
          Valid: from 2023-08-17T17:41:00 to 2024-08-15T17:42:52
          Principals:
                  marslo
          Critical Options: (none)
          Extensions:
                  permit-X11-forwarding
                  permit-agent-forwarding
                  permit-port-forwarding
                  permit-pty
                  permit-user-rc
  
  $ ssh -i ~/.ssh/marslo marslo@remote "cat /home/marslo/.ssh/authorized_keys; du -hs /home/marslo/.ssh/authorized_keys; hostname"
  0 /home/marslo/.ssh/authorized_keys
  remote

update for existing key

#                                       serial
#                                         v
$ ssh-keygen -s devops@ca -I 'edcba' -z '0002' -n marslo marslo.pub
Signed user key marslo-cert.pub: id "edcba" serial 2 for marslo valid forever

# verify
$ ssh-keygen -f marslo-cert.pub -L
marslo-cert.pub:
        Type: [e-mail] user certificate
        Public key: ED25519-CERT SHA256:JfJnCDxjWhLwW3BcBX3XycBr3dT3JtHTwD1M4H3828E
        Signing CA: ED25519 SHA256:5dNlpIIjX/pdoNSpmtfcGQijSrl3W87TByA9KeCe2M0 (using ssh-ed25519)
        Key ID: "edcba"
        Serial: 2
        Valid: forever
        Principals:
                marslo
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

login via specific cert

[!NOTE|label:references:]

• OpenSSH/Cookbook/Certificate-based Authentication

  $ ssh -o CertificateFile=server01.ed25519-cert.pub -i server01.ed25519 \
          [e-mail]

$ ssh [e-mail]
[e-mail]'s password:

$ ssh -o CertificateFile=marslo-cert.pub [e-mail] "du -hs ~/.ssh/authorized_keys"
0 /home/marslo/.ssh/authorized_keys

# via config
$ cat ~/.ssh/confg
Host example sample.server.com
     Hostname sample.server.com
     User marslo
     IdentitiesOnly yes
     IdentityFile /home/marslo/.ssh/marslo
     CertificateFile /home/marslo/.ssh/marslo-cert.pub

SSH Host Certificates

ssh tunnel

references:

• ssh端口转发(跳板机)实战详解
• SSH 通过跳板机直接访问内网机器

[!TIP|label:references:]

• key point:

  • -L : <--

  • -R : -->

• basic command line

  $ ssh -Nf -L <localhost/localip>:<local_port>:<target>:<target_port> <to-be-mapping/localhost>

• usage:

  • 1 -> [2 ->] 3 : ssh host2:port2:host3:port3 host1:port1

  • if ignore host2. default using local.host

• references:

  • SSH connection through host in the middle

    $ ssh -t reachable_host ssh unreachable_host

two servers

-L

in jumper

[!TIP]

• purpose:

  local:22 --> jumper:6666

# -L : <--
$ ssh [e-mail]
$ ssh -Nf -L [jumper.server:]6666:my.server:22 [e-mail]

# verify
$ sudo netstat -an | grep 6666
tcp 0 0 jumper.ip:6666 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:6666 0.0.0.0:* LISTEN
tcp 0 0 ::1:6666 :::* LISTEN

-R

[!TIP]

• status:

                 current status
                      |
                      v
         mymac:22 <--------- jumper:22
  
   purpose:  |                  ^
             + ---------------> +

# login to jumper first via GUI
                         destination
                             |
                             v
<jumper> $ ssh -Nf -R 6666:<mymac>:22 marslo@<mymac>
<jumper> $ ps auxfww | grep ssh | grep -v grep
marslo    2281  0.0  0.0  11716  4064 ?        Ss   19:55   0:00      \_ ssh -Nf -R 6666:<my.pc.ip>:22 marslo@<my.pc.ip>
<jumper> $ sudo netstat -anp | grep ssh
tcp        0      0 <jumper>:46176    <mymac>:22        ESTABLISHED 2281/ssh

# verify in <mymac>
<mymac> $ netstat -an | grep 6666
tcp4       0      0  127.0.0.1.6666         *.*                    LISTEN
tcp6       0      0  ::1.6666               *.*                    LISTEN

<mymac> $ ssh -p 7777 localhost
## or
<mymac> $ cat ~/.ssh/config
Host  jumper
      Hostname              localhost
      Port                  7777
<mymac> $ ssh jumper

three serves

scenario 1

in jumper

> - purpose: > ```bash > local:6666 <--- jumper:6666 <--- remote:6666 > ```

# -R : -->
$ ssh user@jumper
$ ssh -Nf -R [jumper:]6666:local:6666 root@remote

# verify
## in remote
$ ssh root@remote
$ scp -P 6666 root@localhost:/remote/path/file /local/path
## in local
$ scp -P 6666 /local/path/file root@localhost:/remote/path/file

$ ssh user@jumper
<jumper> $ ps awwx | grep ssh | grep 6666
17549 ? Ss 0:00 ssh -Nf -L 6666:remote:22 root@remote
18740 ? Ss 3:50 ssh -Nf -R 6666:local:6666 root@remote

scenario 2

[!TIP]

• details :

                      current status
                      --------------
                     |              |
  
            mymac  <----  jumper  ----> destination
  
              |                             ^
  wants:      v                             |
               -------------->--------------
                     visit directly

# login to jumper
<jumper> $ ssh -Nf -R7777:<destination>:22 marslo@<mymac>
<jumper> $ ps auxfww | grep ssh | grep -v grep
marslo      41  0.0  0.0  11716  5832 ?        Ss   21:51   0:00      \_ ssh -Nf -R7777:<destination>:22 marslo@<my.mac.ip>

# verify in mymac
<mymac> $ ssh -p 7777 localhost       # localhost in <mymac> == <destination>
<mymac> $ netstat -an | grep 7777
tcp6       0      0  ::1.7777               ::1.65371              ESTABLISHED
tcp6       0      0  ::1.65371              ::1.7777               ESTABLISHED
tcp4       0      0  127.0.0.1.7777         *.*                    LISTEN
tcp6       0      0  ::1.7777               *.*                    LISTEN

config

ssh config

$ cat ~/.ssh/config
HOST  *
      LogLevel                   ERROR
      HostkeyAlgorithms          +ssh-rsa
      # PubkeyAcceptedAlgorithms +ssh-rsa              # since git v2.41.0
      GSSAPIAuthentication       no
      StrictHostKeyChecking      no
      UserKnownHostsFile         /dev/null
      IdentityFile               ~/.ssh/id_ed25519
      IdentityFile               ~/.ssh/id_rsa         # keep the older key if necessary

Include config.d/*

Host  github.com
      User                  [e-mail]
      Hostname              ssh.github.com
      IdentityFile          /C/Marslo/my@key
      IdentitiesOnly        yes
      Port                  443
      # ProxyCommand        nc -X connect -x proxy.com:8080 %h %p
      # ProxyCommand        /usr/bin/nc -X 5 -x 127.0.0.1:1087 %h %p
      # ProxyCommand        /usr/local/bin/corkscrew 127.0.0.1 1087 %h %p

sshd_config

references:

• Three locks for your SSH door
• 保护 SSH 的三把锁
• 7 Default OpenSSH Security Options You Should Change in /etc/ssh/sshd_config
• sshd_config (4)

[!TIP]

• disable Root Login : PermitRootLogin

• allow only specific users or groups : AllowUsers, AllowGroups

• deny specific users or groups : DenyUsers, DenyGroups

• change sshd port number : Port

• change login grace time : LoginGraceTime

• Restrict the Interface (IP Address) to Login : ListenAddress

• disconnect ssh when no activity : ClientAliveInterval

banner and motd

[!TIP] files:

• /etc/pam.d/sshd : session optional pam_motd.so : /usr/lib64/security/pam_motd.so

• /etc/motd

• /etc/ssh/sshd_config : Banner /path/to/banner

• force disable all banner or motd

  $ ssh user@host
  $ touch ~/.hushlogin

disable login password

$ cat /etc/ssh/sshd_config
ChallengeResponseAuthentication no
PasswordAuthentication no
UsePAM no

• scripts:

  TIMESTAMPE=$(date +"%Y%m%d%H%M%S")
  SSHDFILE="/etc/ssh/sshd_config"
  sudo cp "${SSHDFILE}{,.org.${TIMESTAMPE}}"
  
  sudo bash -c '/bin/sed -i -e "s:^\(UsePAM.*$\):# \1:" ${SSHDFILE}'
  sudo bash -c '/bin/sed -i -e "s:^\(PermitRootLogin.*$\):# \1:" ${SSHDFILE}'
  sudo bash -c '/bin/sed -i -e "s:^\(ChallengeResponseAuthentication.*$\):# \1:" ${SSHDFILE}'
  sudo bash -c '/bin/sed -i -e "s:^\(PasswordAuthentication.*$\):# \1:" ${SSHDFILE}'
  
  if ! grep 'Add my marslo' ${SSHDFILE} > /dev/null 2>&1; then
    sudo bash -c "cat >> ${SSHDFILE}" << EOF
  
      # Add my marslo
      PermitRootLogin no
      UsePAM no
      ChallengeResponseAuthentication no
      PasswordAuthentication no
      PrintMotd yes
      Banner /etc/ssh/server.banner
    EOF
  fi

disallow group to use password

[!TIP|label:references:]

• Directive 'UsePAM' is not allowed within a Match block

• Directive 'ChallengeResponseAuthentication' is not allowed within a Match block

• Directive 'PrintMotd' is not allowed within a Match block

• Directive 'LoginGraceTime' is not allowed within a Match block

if ! grep 'Add my marslo' ${SSHDFILE} > /dev/null 2>&1; then
  sudo bash -c "cat >> ${SSHDFILE}" << EOF

    # Add my marslo
    Match Group devops
                PasswordAuthentication no
                Banner /etc/ssh/server.banner
                MaxAuthTries 3
                ClientAliveInterval 600       # 60*10 secs
  EOF
fi

disallowing user to use tcp forwarding

Match User testuser
           AllowTcpForwarding no

displaying a special banner for users not in the staff group

Match Group *,!staff
  Banner /etc/banner.text

allowing root login from host rootallowed.example.com

Match Host rootallowed.example.com
  PermitRootLogin yes

allowing anyone to use gatewayports from the local net

Match Address 192.168.0.0/24
  GatewayPorts yes

avoid ssh timeout by sending a keepalive message

$ echo 'ServerAliveInterval 60' >> /etc/ssh/ssh_config

# https://www.commandlinefu.com/commands/view/28/command-to-keep-an-ssh-connection-open
$ watch -n 30 uptime

debug

debug git

• GIT_SSH_COMMAND

  $ GIT_SSH_COMMAND="ssh -vvT" git clone [e-mail]:Marslo/myblog.git
  Cloning into 'myblog'...
  OpenSSH_7.9p1, LibreSSL 2.7.3
  debug1: Reading configuration data /Users/marslo/.ssh/config
  debug1: /Users/marslo/.ssh/config line 1: Applying options for *
  debug1: /Users/marslo/.ssh/config line 13: Applying options for github.com
  debug1: Reading configuration data /etc/ssh/ssh_config
  debug1: /etc/ssh/ssh_config line 48: Applying options for *
  debug1: Connecting to github.com port 22.
  ^C

$ GIT_TRACE=1 git st
00:30:44.772137 git.c:703               trace: exec: git-st
00:30:44.772540 run-command.c:663       trace: run_command: git-st
00:30:44.772894 git.c:384               trace: alias expansion: st => status
00:30:44.772903 git.c:764               trace: exec: git status
00:30:44.772907 run-command.c:663       trace: run_command: git status
00:30:44.777379 git.c:440               trace: built-in: git status
On branch master
Your branch is up to date with 'origin/master'.

00:30:44.782714 run-command.c:663       trace: run_command: GIT_INDEX_FILE=.git/index git submodule summary --cached --for-status --summary-limit -1 HEAD
00:30:44.787490 git.c:703               trace: exec: git-submodule summary --cached --for-status --summary-limit -1 HEAD
00:30:44.788038 run-command.c:663       trace: run_command: git-submodule summary --cached --for-status --summary-limit -1 HEAD
00:30:44.838222 git.c:440               trace: built-in: git rev-parse --git-dir
00:30:44.845054 git.c:440               trace: built-in: git rev-parse --git-path objects
00:30:44.852811 git.c:440               trace: built-in: git rev-parse -q --git-dir
00:30:44.870362 git.c:440               trace: built-in: git rev-parse --show-prefix
00:30:44.878755 git.c:440               trace: built-in: git rev-parse --show-toplevel
00:30:44.893984 git.c:440               trace: built-in: git rev-parse -q --verify --default HEAD HEAD
00:30:44.899709 git.c:440               trace: built-in: git rev-parse --show-toplevel
00:30:44.905200 git.c:440               trace: built-in: git rev-parse --sq --prefix  --
00:30:44.911762 git.c:440               trace: built-in: git diff-index --cached --ignore-submodules=dirty --raw 52c94664ffc09cde2308c6bf9824ca0355ff5ff7 --
00:30:44.917374 run-command.c:663       trace: run_command: GIT_INDEX_FILE=.git/index git submodule summary --files --for-status --summary-limit -1
00:30:44.922165 git.c:703               trace: exec: git-submodule summary --files --for-status --summary-limit -1
00:30:44.922568 run-command.c:663       trace: run_command: git-submodule summary --files --for-status --summary-limit -1
00:30:44.965375 git.c:440               trace: built-in: git rev-parse --git-dir
00:30:44.972784 git.c:440               trace: built-in: git rev-parse --git-path objects
00:30:44.979117 git.c:440               trace: built-in: git rev-parse -q --git-dir
00:30:44.991077 git.c:440               trace: built-in: git rev-parse --show-prefix
00:30:44.997718 git.c:440               trace: built-in: git rev-parse --show-toplevel
00:30:45.012365 git.c:440               trace: built-in: git rev-parse -q --verify --default HEAD
00:30:45.018759 git.c:440               trace: built-in: git rev-parse --show-toplevel
00:30:45.024687 git.c:440               trace: built-in: git rev-parse --sq --prefix  --
00:30:45.031664 git.c:440               trace: built-in: git diff-files --ignore-submodules=dirty --raw --
nothing to commit, working tree clean

debug ssh

references:

• OpenSSH Test Mode
• OpenSSH Tip: Check Syntax Errors before Restarting SSHD Server
• How to check SSH server's configuration validity

• debug mode

  $ sudo /usr/sbin/sshd -d [-d] [-d]

• test mode

  # -T : extended test mode
  $ sudo /usr/sbin/sshd -T [-f /path/to/sshd_config]
  
  # -t : test mode
  $ sudo /usr/sbin/sshd -t [-f /path/to/sshd_config]

check sshd log

[!NOTE|label:references:]

• How to check sshd log?

# -- check `/var/log/auth.log` --
$ grep 'sshd' /var/log/auth.log
# or
$ tail -f -n 500 /var/log/auth.log | grep 'sshd'

# -- check with journalctl --
## -t, --identifier
$ journalctl -t sshd

## -u, --unit
$ journalctl -u ssh

## -o json-pretty: with json format
$ journalctl -t sshd -o json-pretty

## -b0: since last boot
$ journalctl -t sshd -b0

## reserve order
$ journalctl -t sshd -b0 -r

tips

play video via ssh

$ ssh USER@HOST cat REMOTE_FILE.mp4 | tee LOCAL_FILE.mp4 | mplayer -

displays failed login via SSH

[!NOTE|label:references:]

• * iMarslo: check local failed login

$ awk '/sshd/ && /Failed/ {gsub(/invalid user/,""); printf "%-12s %-16s %s-%s-%s\n", $9, $11, $1, $2, $3}' /var/log/auth.log

compare files via ssh

$ ssh user@host cat /path/to/remotefile | diff /path/to/localfile -

disconnect

[!TIP] Enter + ~ + . + Enter

references:

• Linux / UNIX kill unresponsive hung SSH session
• Kill an unresponsive ssh session without closing the terminal
• How to kill or disconnect hung ssh session in Linux

$ <Enter>
$ ~?
Supported escape sequences:
 ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~C   - open a command line
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

execute shell commands via ssh

[!NOTE|label:references:]

• * iMarslo: knrun
• Bash while read loop breaking early

  $ ssh -n username@hosts "cmd"

• Source a file after ssh login drops to prompt | docker run -it over ssh: the input device is not a TTY

  $ ssh -t username@hosts "cmd"

ctrl-c to break while loop

trap exit SIGINT SIGTERM; while read -r _node; do
  ssh -n username@${_node} "cmd"
done <<< "${hostname}"

execute local script via ssh

[!NOTE|label:references:]

• How to use SSH to run a local shell script on a remote machine?

trap exit SIGINT SIGTERM; while read -r _node; do
  ssh -q username@${_node} 'bash -s' < '/path/to/scipt.sh'
done <<< "${hostname}"

# with eval
trap exit SIGINT SIGTERM; while read -r _node; do
  declare sshCmd="ssh -q username@${_node} 'bash -s' < \"/path/to/scipt.sh\""
  eval "${sshCmd}"
done <<< "${hostname}"

# or: https://www.commandlinefu.com/commands/view/5772/run-complex-remote-shell-cmds-over-ssh-without-escaping-quotes
$ ssh host -l user $(<cmd.txt)

how ssh works

how ssh works