• 1 System
  • 1.1 Terminal
    • 1.1.1 Use watch command with a piped chain of commands
    • 1.1.2 Change the width of the remote console
  • 1.2 SSH and SFTP
    • 1.2.1 Show progress
    • 1.2.2 VNC connects to Server
    • 1.2.3 Run command using SSH
    • 1.2.4 SSH Allow only Key
    • 1.2.5 Calculate RSA Key Fingerprint
    • 1.2.6 SFTP Batch Mode
  • 1.3 File and Folder
    • 1.3.1 Find files that match one of multiple patterns
    • 1.3.2 Create multi-part 7z
    • 1.3.3 Append file in the column-wise way
    • 1.3.4 List all files but exclude directories
    • 1.3.5 List files not matching
    • 1.3.6 Insert zero to filename
    • 1.3.7 Transpose a file
    • 1.3.8 Change the permission of all sub-folders of a folder
    • 1.3.9 Directory must have x permission
    • 1.3.10 Replace String in a Directory
    • 1.3.11 Encoding Problem in Uncompressed Zip file
    • 1.3.12 Get only files created after a date
  • 1.4 Administration
    • 1.4.1 Memory information shown in free
    • 1.4.2 Remove packages with dependencies in CentOS
    • 1.4.3 List files installed from a RPM
    • 1.4.4 Remove systemd services
    • 1.4.5 Find information for a running process
    • 1.4.6 Get the start time of a process
    • 1.4.7 Force Stopping Multi Threads Process
    • 1.4.8 Find and kill zombie process
    • 1.4.9 Increase the ps column width for column “user”
    • 1.4.10 Show CPU Usage
    • 1.4.11 Show which CPU is used by PID
  • 1.5 Network
    • 1.5.1 Find out which process eats up the bandwidth
• 2 Image, Video and Audeio Manipulations
  • 2.1 Video
    • 2.1.1 Merge two clips and place them next to each other
    • 2.1.2 Concatenate videos
  • 2.2 Conversion
    • 2.2.1 Convert series of images to video
    • 2.2.2 Convert MP4 to GIF
  • 2.3 Image
    • 2.3.1 Annotate image
• 3 PDF Manipulations
  • 3.0.1 Remove Margin Whitespace in a PDF Figure
  • 3.0.2 Get Box Info of PDF
  • 3.0.3 Crop a PDF Figure
  • 3.0.4 Concatenate PDF Figures Horizontally
• 4 HPC
  • 4.1 slurm
    • 4.1.1 Find where the job is submitted

1 System

1.1 Terminal

1.1.1 Use watch command with a piped chain of commands

watch 'command | othertool | yet-another-tool'

1.1.2 Change the width of the remote console

stty rows 50 cols 132

Reference: How to change the width of remote serial console? [duplicate]

The above solution may cause messed text in Vim. Maybe a better solution is resizable serial console window?.

1.2 SSH and SFTP

1.2.1 Show progress

progress
   Toggle display of progress meter.

1.2.2 VNC connects to Server

1. On Linux server, start the VNC server

vncserver -fp catalogue:/etc/X11/fontpath.d

-fp catalogue:/etc/X11/fontpath.d is required. No idea of the reason.

It will prompts the password.

2. On Windows, use ssh to forward port

ssh -p 10001 [email protected]_ip -L 7000:master:5908

master is the host name, 5908 is the port which the VNC server listens to. 7000 is the port number on the client side which ssh forwards to 5908.

2. On Windows, start VNC viewer to connect to localhost. The remote URl is localhost:7000.

1.2.3 Run command using SSH

ssh [email protected] date
ssh [email protected] bash my.sh

1.2.4 SSH Allow only Key

• Edit /etc/ssh/sshd_config to set PasswordAuthentication no.
• Restart systemctl restart sshd.service

1.2.5 Calculate RSA Key Fingerprint

ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub

1.2.6 SFTP Batch Mode

sftp -oBatchMode=no -b YOUR_COMMAND_FILE_PATH [email protected]

If it requires the password, use sshpass to pass the password.

sshpass -p YOUR_PASSWORD sftp -oBatchMode=no -b YOUR_COMMAND_FILE_PATH [email protected]

1.3 File and Folder

1.3.1 Find files that match one of multiple patterns

Use -o, which means “or”:

find Documents \( -name "*.py" -o -name "*.html" \)

1.3.2 Create multi-part 7z

7z -v100m a my_zip.7z my_folder/

1.3.3 Append file in the column-wise way

$ paste f1 f2 | tr -d '\t'
123456789abcdefghij
9876543211111111111
0000000003333333333

1.3.4 List all files but exclude directories

find . -type f

1.3.5 List files not matching

ls | grep -v '\.jar$'

1.3.6 Insert zero to filename

fbase=PressureContour_SliceCore_REF_G1R2P1It
  for f_fullname in ${fbase}[0-9]*; do
  fname=${f_fullname%.*}
  mv "$f_fullname" $(printf "${fbase}%05d.png" "${fname#${fbase}}")
done

1.3.7 Transpose a file

Run the following file as transpose.sh file_name

awk '
{ 
    for (i=1; i<=NF; i++)  {
        a[NR,i] = $i
    }
}
NF>p { p = NF }
END {    
    for(j=1; j<=p; j++) {
        str=a[1,j]
        for(i=2; i<=NR; i++){
            str=str" "a[i,j];
        }
        print str
    }
}' $1

1.3.8 Change the permission of all sub-folders of a folder

To change all the directories to 755 (drwxr-xr-x):

find /opt/lampp/htdocs -type d -exec chmod 755 {} \;

To change all the files to 644 (-rw-r--r--):

find /opt/lampp/htdocs -type f -exec chmod 644 {} \;

1.3.9 Directory must have x permission

To view the files in a directory, the directory must have a x permission.

1.3.10 Replace String in a Directory

grep -rwnil abc | while read f; do sed -i 's/abc/ref/gI' $f; done

1.3.11 Encoding Problem in Uncompressed Zip file

Use unzip -O cp936 file.zip

1.3.12 Get only files created after a date

For example, to find all files in the current directory that have been modified since yesterday (24 hours ago) use:

find . -maxdepth 1 -mtime -1

Note that to find files modified before 24 hours ago, you have to use -mtime +1 instead of -mtime -1.

1.4 Administration

1.4.1 Memory information shown in free

❯ free -m
              total        used        free      shared  buff/cache   available
Mem:         191891       22150        1698        4171      168042      164146
Swap:             0           0           0

available includes free and a part of buff/cache.

If free is close to 0, do not worry. If the following things happen, you must check what is going on.

• available memory is close to zero
• swap used increases or fluctuates
• dmesg | grep oom-killer shows the OutOfMemory-killer at work

1.4.2 Remove packages with dependencies in CentOS

yum autoremove [package_name]

1.4.3 List files installed from a RPM

repoquery --installed -l httpd

1.4.4 Remove systemd services

systemctl stop [servicename]
systemctl disable [servicename]
rm /etc/systemd/system/[servicename]
rm /etc/systemd/system/[servicename] # and symlinks that might be related
rm /usr/lib/systemd/system/[servicename] 
rm /usr/lib/systemd/system/[servicename] # and symlinks that might be related
systemctl daemon-reload
systemctl reset-failed

1.4.5 Find information for a running process

You can get this information from /proc filesystem, it stores information about running processes.

cat /proc/<pid>/environ
cd /proc/<pid>/cwd; pwd -P
cat /proc/<pid>/cmdline

Another method is

lsof -p $PID

1.4.6 Get the start time of a process

The parameter of lstart of ps does this job.

ps -eo pid,lstart,cmd

1.4.7 Force Stopping Multi Threads Process

ps aux | grep nfs_opt | awk '{print $2}' | while read i; do kill -9 $i; done

1.4.8 Find and kill zombie process

The stat of the zombie process is Z.

ps aux | grep 'Z'

You have to kill the parent process of the zombie process.

$ pstree -p -s ${zombie_pid}
init(1)---cnid_metad(1311)---cnid_dbd(5145)
$ kill 5145 # Usually kill the 1st one

1.4.9 Increase the ps column width for column “user”

The following specfies the width of the column user to be 16. The key is to use the option o or -o.

alias ps_mod='ps ax o user:16,pid,pcpu,pmem,vsz,rss,stat,start_time,time,cmd'

1.4.10 Show CPU Usage

RED='\033[0;31m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
echo -e "${RED}The CPU usage(2nd column) and memory usage(3rd column) of the currently login users are as follows,${NC}"

own=$(id -nu)
cpus=$(lscpu | grep "^CPU(s):" | awk '{print $2}')
normal_users=$(cut -d: -f1,3 /etc/passwd | egrep ':[0-9]{4}$' | cut -d: -f1)
# Assume that the length of username is less than 20
proc_users=$(ps ax o user:20 | awk '{print $1}' | sort -u)
proc_normal_users=''
for pu in $proc_users; do
  for nu in $normal_users; do
    if [[ $pu == $nu ]]; then
      proc_normal_users=$proc_normal_users$pu" "
    fi
  done
done

# for user in $(who | awk '{print $1}' | sort -u)
for user in $proc_normal_users
do
  # print other user's CPU usage in parallel but skip own one because
  # spawning many processes will increase our CPU usage significantly
  if [ "$user" = "$own" ]; then continue; fi
  (top -b -n 1 -u "$user" | awk -v user=$user -v CPUS=$cpus 'NR>7 { sum += $9; } END { print user, sum, sum/CPUS; }') &
  # don't spawn too many processes in parallel
  sleep 0.05
done
wait

# print own CPU usage after all spawned processes completed
top -b -n 1 -u "$own" | awk -v user=$own -v CPUS=$cpus 'NR>7 { sum += $9; } END { print user, sum, sum/CPUS; }'

echo -e "${BLUE}In the 2nd column, 100 means 1 CPU is being used.${NC}"
echo -e "${RED}If someone is using this node, try other nodes please!${NC}"

1.4.11 Show which CPU is used by PID

PSR number shows the processor number the process is assigned to.

Thus use ps -aF or ps -q 7810 -o psr.

1.5 Network

1.5.1 Find out which process eats up the bandwidth

sudo nethogs wlan0

2 Image, Video and Audeio Manipulations

2.1 Video

2.1.1 Merge two clips and place them next to each other

ffmpeg -i left.mp4 -i right.mp4 -filter_complex hstack output.mp4

To stack more than 2 videos, you can simply specify hstack=inputs=3.

2.1.2 Concatenate videos

ffmpeg -f concat -safe 0 -i <(for f in ./*.ogv; do echo "file '$PWD/$f'"; done) -c copy output.ogv

Reference: ffmpeg concat demuxer

2.2 Conversion

2.2.1 Convert series of images to video

1. Use convert

convert -delay 25 -loop 0 -quality 100 Pressure*.png Pressure.mp4

Note: The video file by convert can be played by video player but not by Chrome.

2. Use ffmpeg

By pipe

cat PressureContour_SliceCore_REF_G1R2P1It0*.png | ~/Softwares/ffmpeg/ffmpeg -f image2pipe -framerate 4 -i - -c:v libx264 -pix_fmt yuv420p out.mp4

Another method is sequential reading. However, it requires that the index of the image should be starting from 1. If not, you need to specify the -start_number 50. But the second index number should be 51. That is bad design.

The above ffmpeg method produce a video file which is played perfectly in Chrome but in apps like PotPlayer it shows wrongs frames at the end of the video file. For example, ffmpeg says this video is 9.23s. PotPlayer says this video is 10s and shows wrong frames after 9.23s.

Reference: ffmpeg wiki: Create a video slideshow from images

Another problem is that h264 codec requires even dimensions.

ffmpeg -r 24 -i frame_%05d.jpg -vcodec libx264 -y -an video.mp4 -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2"

Reference: FFMPEG (libx264) “height not divisible by 2”

2.2.2 Convert MP4 to GIF

Use ffmpeg to convert. Use convert from imagemagick to reduce size.

ffmpeg -i input.mp4 -pix_fmt rgb24 input_raw.gif
convert -loop 0 -layers Optimize input_raw.gif input.gif

2.3 Image

2.3.1 Annotate image

Try using -gravity North (this will move your text to the top of the image) and then adding an offset (-annotate +0+100) to move down your text:

convert temp.jpg -gravity North -pointsize 30 -annotate +0+100 'Love you mom' temp1.jpg

3 PDF Manipulations

3.0.1 Remove Margin Whitespace in a PDF Figure

Use pdfcrop

3.0.2 Get Box Info of PDF

Use pdfinfo

pdfinfo -box input_file.pdf

3.0.3 Crop a PDF Figure

Use Ghostscript. Shell script is as follows,

#!/bin/bash
input_fname=$1
left=$2
bottom=$3
right=$4
top=$5
if [ "$top" == "" ]; then
  echo "Usage: ./crop_pdf.sh input_file.pdf left bottom right top"
else
  extension="${input_fname##*.}"
  filename="${input_fname%.*}"
  output_fname="${filename}_crop.pdf"
  # Use pdfinfo -box input_file.pdf to get the box info of this pdf.
  gs -sDEVICE=pdfwrite -sOutputFile=$output_fname -dBATCH -dNOPAUSE -c "[ /CropBox [ $left $bottom $right $top ] /PAGES pdfmark" -f $input_fname
fi

3.0.4 Concatenate PDF Figures Horizontally

Use pdfjam which is installed by tlmgr in the TexLive distro. The usage is as follows,

pdfjam input_location_discontinuous_crop.pdf input_location_merged_crop.pdf --nup 2x1 --outfile input_location.pdf

4 HPC

4.1 slurm

4.1.1 Find where the job is submitted

$ scontrol show job 1665191
    JobId=1665191 Name=tasktest
    ...
    Shared=OK Contiguous=0 Licenses=(null) Network=(null)
    Command=/lustre/work/.../slurm_test/task.submit
    WorkDir=/lustre/work/.../slurm_test