troubleshooting

• troubleshooting

  • tools

  • thread dump

• other tools

[!TIP|label:official recommended]

• Prepare Jenkins for Support

• CloudBees Jenkins JVM troubleshooting

  • * Supported Java 8 arguments

  • * Supported Java 11 arguments

  • * collectPerformanceData Script

• Diagnosing Errors

• Tuning Jenkins GC For Responsiveness and Stability with Large Instances

• ** Java Heap settings Best Practice

  • Minimum and maximum heap sizes

• How to Troubleshoot and Address Jenkins Startup Performances

troubleshooting

[!NOTE|label:references:]

• How to Troubleshoot and Address Jenkins Startup Performances

• Required Data: Jenkins Hang Issue On Linux

• collectPerformanceData Script

  • collectPerformanceData.sh

• Understanding Thread Dumps

  • fastthread.io

• gceasy.io

• What causes high CPU usage and how can I reduce it

• Unrecognized VM Option

  • UseGCLogFileRotation

  • GCLogFileSize=100m

  • PrintGCDateStamps

  • PrintGCCause

  • PrintTenuringDistribution

  • PrintReferenceGC

  • PrintAdaptiveSizePolicy

tools

[!NOTE|label:tools]

• iostat

• nfsiostat
• nfsstat
• vmstat - understanding Thread DumpUnderstanding Thread Dumpss
• top

  • top -H
• netstat
• jstack
• stap

• prepare

  $ apt update
  $ apt install sudo vim netstat net-tools sysstat nfs-common
  $ sudo systemctl start sysstat
  $ sudo systemctl enable sysstat
  $ cat /etc/cron.d/sysstat

  [!NOTE|label:tips for sar]

  • if you wanted to check your memory usage instead, you could use the -r argument rather than -u

    $ sar -r 2 30

• collectPerformanceData.sh

  $ curl -sO https://s3.amazonaws.com/cloudbees-jenkins-scripts/e206a5-linux/collectPerformanceData.sh
  $ chmod +x collectPerformanceData.sh
  
  $ sudo -u $JENKINS_USER sh collectPerformanceData.sh $JENKINS_PID 300 5
  $ or
  <jenkins> $ bash collectPerformanceData.sh $JENKINS_PID 300 5
  [INFO] Collected a threadDump for PID 8.
  [INFO] A new collection will start in 5 seconds.
  [INFO] Taking top data collection.
  [INFO] Taking TopdashH data collection.
  [INFO] Taking vmstat data collection.
  [INFO] Taking netstat collection.
  [INFO] Taking iostat data collection.
  [INFO] Taking nfsiostat data collection.
  [INFO] Taking nfsstat data collection.

  [!NOTE]

  • 300: "Length to run the script in seconds"

  • 5: "Intervals to execute commands in seconds"

thread dump

• generated via monitor plugin

  

  generate heap dump
• generated via jmap

  [!NOTE|label:references:]

  • List All the Classes Loaded in the JVM

  $ pid=$(ps auxfww | grep devops-jenkins | awk '{print $2}')
  $ jmap -dump:format=b,file=/opt/tmp/heapdump.bin ${pid}

• analysis via

  • fastthread.io
  • MAT: Eclipse Memory Analyzer

  

  Eclipse Memory Analyzer

  • VisualVM

    • i.e.: visualvm.exe --jdkhome "C:\Software\Java\jdk1.6.0" --userdir "C:\Temp\visualvm_userdir"

  

  VisualVM

other tools

• Decimal to Hexadecimal Converter
• GC Log Analyzer