Command toolbox

Linux - Disable RH subscription manager

vi /etc/yum/pluginconf.d/subscription-manager.conf
enabled=0

Linux - Delete logs older than

find /aa/bb -name '*.log' -type f -mtime +30 -exec rm {} +

Linux - Export amount of lines of log to tar.gz

cat <log_file_name> | wc -l
sed -n '777850,805947p' <log_file_name> | gzip > <output_file_name>

Linux - Pack bunch of logs in their own tar.gz

find . -type f \( -name "server.log.2021-02*" ! -name "*tar.gz" \) -exec tar czvf {}.tar.gz {} --remove-files \;

Linux - VirtualHere with legacy usb printer.

This is server-client solution for sharing USB devices over network.
If you have old usb printer connected to USB in your linx server you know that cups will not provide high quality printing.
So, my HP DeskJet 920c neded encapsulation of usb. Unfortunately freely available usbip is not a solution since on client side Iā€™m using Windows and usbip drivers for windows are not signed. Hard way you can use it but it involves mapping device and starting windows in disabled drivers signing check.

It is a pain.

Looking for solutions for it there are only commercially available solutions.
VirtuHere trial allows you use only one device on server side in trial mode with no time restrictions.
Taking under consideration my infrequent usage of printer - it is just fine.
Bellow installation covering Linux server and windows client.

Get server for your operating system from:

https://www.virtualhere.com/usb_server_software

Publish package as you want on your server.

Run first only to generate config.ini file

Move config.ini file to your favorite destination, mine is /etc/virtualhere/config.ini
Because trial gives you opportunity to work only with one USB device chose one:

  lsusb
  Bus 002 Device 002: ID 0bc2:231a Seagate RSS LLC Expansion Portable
  Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
  Bus 001 Device 003: ID 03f0:1504 HP, Inc DeskJet 920c
  Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Mine ID is 03f0, so update config with:

  AllowedDevices=03f0
  ServerName=my_server_name
  Hostname=my_server_name

Create service:

  systemctl edit --full --force virtualhere.service

  [Unit]
  Description=VirtualHere Client
  After=network.target
  [Service]
  Type=forking
  ExecStart=/usr/sbin/vhusbdx86_64 -b -c /etc/virtualhere/config.ini
  [Install]
  WantedBy=multi-user.target
  systemctl enable virtualhere.service

  systemctl start virtualhere.service

Enable access over firewall to port where your service is present.

Now get client from virtualhere page:

https://www.virtualhere.com/usb_client_software

Linux - Scale columns for ps dump

ps -eo user:20,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,group:50,args > /tmp/ps_202107010942.txt

Docker - volume to containers assigment

docker ps -aq | xargs docker inspect -f '{{ .Name }}{{ printf "\n" }}{{ range .Mounts }}{{ printf "\n\t" }}{{ .Type }} {{ if eq .Type "bind" }}{{ .Source }}{{ end }}{{ .Name }} => {{ .Destination }}{{ end }}{{ printf "\n" }}'

Docker - overlay to container assigment

docker inspect $(docker ps -qa) |  jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"'

Linux - Simple serving your http status

This is python 3.9 code that serves simple HTTP server with custom response.
It is useful for testing purposes, for example to check if your firewall is working properly.

#!/bin/python3.9
import http.server

PORT = 8000

class MyHandler(http.server.BaseHTTPRequestHandler):
    def send_response(self, code, message=None):
        """Send the response header and log the response code.

        In contrast to base class, do not send two standard headers
        with the server software version and the current date.
        """
        self.log_request(code)
        if message is None:
            if code in self.responses:
                message = self.responses[code][0]
            else:
                message = ''
        if self.request_version != 'HTTP/0.9':
            self.wfile.write("%s %d %s\r\n" %
                             (self.protocol_version, code, message))

    def do_GET(self):
        self.send_response(500)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        self.wfile.write("Internal Server Error\n")


httpd = http.server.HTTPServer(("", PORT), MyHandler)
print(("serving at port"), PORT)
httpd.serve_forever()

Podman - Keep nft rules after nftables reload/restart

This solution is for situation when nftables are used as main firewall tool, without firewalld.
If you have own rules for nftables in named tables call to systemctl restart nftables or systemctl reload nftables, will kill your podman network.
Override of systemd nftables.nftables is needed as follows:

systemctl edit nftables

[Service]
ExecReload=
ExecReload=/sbin/nft 'flush table inet firewall; include "/etc/sysconfig/nftables.conf";'
ExecStartPost=/usr/bin/podman network reload --all

ExecStartPost simple calls command after restart. ExecReload is modification of original command, instead flush ruleset we are flushing only our table keeping podman rules intact.

vi - Remove CRLF (^M) with vim

Go to binary mode withi your vi/vim

vi -b <your.file.encoded.in.windows>

Replace using
^M sign is created by combination of Ctrl+V+M

%s/^M$//

Linux - Find CVE number for package specified

rpm -q --changelog {package-name} | grep CVE-NUMBER

Linux - Password generator

tr -dc [[:alnum:]^%#@!=+] </dev/urandom | head -c 12

Nginx - htpasword generator

As you see - you need to have file already in place. +
----
printf "username:$(openssl passwd -apr1)" >> /path/to/htpasswd
----

Linux - Get Hostname & IP linux

$(hostname -s)
# OL8
$(ip route get 1 | awk '{print $(NF-2);exit}')
# other
ip route get 1 | sed -n 's/^.*src \([0-9.]*\) .*$/\1/p'
# maybe
ip route get 1 | awk '{print $NF;exit}'

Docker - safelly remove dangling overlays

docker volume ls -qf dangling=true | xargs -r docker volume rm

Grep - recursively search for text, file + date

This command will search recursively in all files in current directory for <something> and print modification date of file and file name.
It is useful to find out when file was modified last time and what is the content of file

grep -rIH <something> * | awk -F: '{"stat -c %y " $1 | getline r; print r": "$0 }'

Linux - Get directory sizes in tree without tree to easy browse in spreadsheet

This command will create a pseudo tree of directories with their sizes in MB.
It will sort directories by size and replace directory names with -- to make it easier to read in spreadsheet.

du -mx | tac | sed 's_[^/[:cntrl:]]*/_--_g;s/-/|/' > /tmp/pseudotree.txt

Podman - complaining about overlays

If you are using podman and you get Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

/etc/containers/storage.conf

[storage]
driver = "overlay"
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"

Linux - Reset USB devices

If you have problems with USB devices, you can reset them by unbinding and binding them again.

for i in /sys/bus/pci/drivers/[uoex]hci_hcd/*:*; do
  [ -e "$i" ] || continue
  echo "${i##*/}" > "${i%/*}/unbind"
  echo "${i##*/}" > "${i%/*}/bind"
done

Linux - Resize oracle linux volume

This porcedure make your oracle linux 9 partition bigger - simply remove crash report parttion that eats up 15 GB.
Of course - do your own backup :-)

pvresize /dev/sda3
lvextend -l +100%FREE -r /dev/mapper/ocivolume-root
or
lvextend -l +100%FREE /dev/mapper/ocivolume-root
xfs_growfs /
# disable now performance metrics collector
systemctl disable --now pmcd pmlogger pmie
# Size before
df -h

#rescan volumes

echo "1" | sudo tee /sys/class/block/sda/device/rescan

# Backup and unmount
tar -czpf oled.tar.gz /var/oled
umount /var/oled

# Remove
lvremove /dev/mapper/ocivolume-oled

lvcreate -L 1G -n oled ocivolume
mkfs.xfs  /dev/mapper/ocivolume-oled

# Remount and restore data
mount /dev/mapper/ocivolume-oled  /var/oled/
tar -xzf oled.tar.gz -C /

# Size after
df -h

# extend root
lvextend -l +100%FREE -r /dev/mapper/ocivolume-root

# sometimes -r does not work , remove it and then resize fs manually with:
xfs_growfs ...

Linux - Find where your alias come from

You can use bash tracing to find out where your alias is defined.
This will print all aliases defined in your bash session, including those defined in /etc/bashrc and ~/.bashrc.
You can also use grep to filter the output to show only aliases defined in /home or /etc.

PS4='+$BASH_SOURCE> ' BASH_XTRACEFD=7 bash -xl 7> /tmp/mylog
grep "alias " /tmp/mylog | grep -e /home -e /etc

Linux - Bash functions

There are some additional functions to help in bash.

Linux - SAR - magic sed replacer

This is a magic tool that will help you replace a block of text from one to another.
Remember that it will ignore preceding spaces and tabs in search, but the block replacement will be exact

sar() {
# sar will replace any text block to any other
# text block
# for separation of from - to , use ----
# search will ignore line tabs
# sar <from> <to>
sed -i -f <(sed -nf <(echo '
:a;/^----$/!{;s,[/\.*[],\\&,g;s/$/\\/;s/\s*/\\s*/;
H;x;1s,^.,s/,;1! s/^/N\;/;x;n;ba;};x;s,\\$,/,;x;
:b;n;s,[/\],\\&,g;$!s/$/\\/;H;$!bb;g;s,/\n,/\\\n,;
s,$,/\;P\;D,p') <(cat $1) ) $2
}

This is example usage:

sar <(echo'
</Context>
----
            <CookieProcessor sameSiteCookies="none" />
        </Context>

') /opt/server/conf/context.xml

Above will replace any line contaning </Context> to block

           <CookieProcessor sameSiteCookies="none" />
        </Context>

In file /opt/server/conf/context.xml

Linux - rsync - nicer look

This is a function that will help you to use rsync in a more readable way.

rsyncc() {
# this is rsync function that will print tree of folders an readable
# file status
# rsyncc <zrodlo> <cel>
rsync -avvr --human-readable --out-format="%t %i %-8''b %f  > $_%n" $1 $2
}

Linux - LookBusy

LookBusy is a tool that can be used to generate CPU load on a system.
It can be useful for testing purposes, such as checking system performance or stress testing.
Or keep your system busy when you need it :-)

  • Compile

curl -L http://www.devin.com/lookbusy/download/lookbusy-1.4.tar.gz -o lookbusy-1.4.tar.gz
tar -xzvf lookbusy-1.4.tar.gz
cd lookbusy-1.4/
./configure
make
make install

  • Add service

systemctl edit --full --force lookbusy.service

  • Adapt service settings

[Unit]
Description=lookbusy service

[Service]
Type=simple
ExecStart=/usr/local/bin/lookbusy -c 5 -m 2048MB
Restart=always
RestartSec=10
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

  • Start service

systemctl enable --now lookbusy.service

Linux - Change user ID

Original user GID:2000, ID: 1005, username foo.

usermod -u 2005 foo
groupmod -g 3000 foo

find / -group 2000 -exec chgrp -h foo {} \;
find / -user 1005 -exec chown -h foo {} \;

Check:

ls -l /home/foo/
id -u foo
id -g foo
# search for 'foo' in the passswd file #
grep foo /etc/passwd # search for 'foo' in the group file #
grep foo /etc/group # use the find command to locate files owned by ' foo'#
find / -user foo -ls
find / -group sales -ls

Linux - EPEL enable ol9

sudo su -c "dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm"
sudo /usr/bin/crb enable

Linux - Install Python modules offline

On machine WITH access to internet create folder C:\piprepo go to it (in this example it is windows machine)
Also add platform identifiers to ensure that your target platform is covered

mkdir C:\piprepo
cd C:\piprepo
python -m pip download requests -d"C:\piprepo" --platform=any --only-binary=:all:

Now you have *.whl files in folder, create requirements.txt. Execute in folder:

Windows:
(for %i in (*.whl) do @echo %~nxi) > requirements.txt
Linux:
ls *.whl > requirements.txt

Now put entire directory on machine that has no internet acces and execute following in folder location:

python -m pip install -r requirements.txt --find-links="..location of\piprepo" --no-index

Now your packages are installed.

Borg - remove file/files from backup

borg recreate /media/usb/bulbobackup/bulbo.borg --progress --dry-run --verbose --list --filter=x --exclude="re:data/photoprism_data/jarek/files/Documents/rsa/(id.*|ora.*)"
borg compact --progress --verbose /media/usb/bulbobackup/bulbo.borg

filter=x shows files excluded from backup.
exclude="re:data/photoprism_data/jarek/files/Documents/rsa/(id.|ora.)" adding re: you have regular expression, not adding anything you are using filesystem patterns.
Progress and verbose are not working in dry-run mode.

QNAP - autostart ARM based - UPS reload after powerdown

Enable autostart System ā†’ Hardware ā†’ General

/etc/init.d/init_disk.sh mount_flash_config
vi /tmp/nasconfig_tmp/autorun.sh

(sleep 300; echo 0 > /sys/bus/usb/devices/usb1/1-2/authorized; sleep 2; echo 1 > /sys/bus/usb/devices/usb1/1-2/authorized) &

/etc/init.d/init_disk.sh umount_flash_config

HERE is original qnap documentation for autostart

5862759311cce4ddb52de03213208869
Figure 1. Localizatio n of autostart

QNAP - script for drives health monitoring

First you need set MAIL_FROM, MAIL_TO and MAIL_SUBJECT variables in the script.
You can also change the colors used for terminal output by modifying the GREEN, YELLOW, RED, and NC variables.
You can also change the path to the smartctl command if it is not in your PATH.
You need also configured msmtp to send emails.
You can use this script to monitor SMART status of drives in your QNAP NAS. It will check each drive and send an email alert if any issues are detected.
Make sure to have smartmontools and msmtp installed on your QNAP NAS.
You can save this script as smart_monitor.sh and make it executable with chmod +x smart_monitor.sh.
Then you can run it manually or set up autostart.

#!/bin/bash

# Mail configuration
MAIL_TO="admin@example.com"
MAIL_FROM="monitor@example.com"
MAIL_SUBJECT="šŸ›‘ SMART alert from $(hostname)"
MAIL_BODY="/tmp/smart_alert_body.txt"
> "$MAIL_BODY"

# Colors for terminal
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
RED='\033[0;31m'
NC='\033[0m'

echo "=== SMART STATUS CHECK ==="

problem_found=0

for dev in /dev/sg*; do
    model=$(smartctl -i -d sat "$dev" 2>/dev/null | grep "Device Model" | cut -d: -f2- | xargs)
    serial=$(smartctl -i -d sat "$dev" 2>/dev/null | grep "Serial Number" | cut -d: -f2- | xargs)
    health=$(smartctl -H -d sat "$dev" 2>/dev/null | grep "SMART overall-health" | awk -F: '{print $2}' | xargs)
    realloc=$(smartctl -A -d sat "$dev" 2>/dev/null | awk '$1 == 5 {print $10}')
    pend=$(smartctl -A -d sat "$dev" 2>/dev/null | awk '$1 == 197 {print $10}')
    uncor=$(smartctl -A -d sat "$dev" 2>/dev/null | awk '$1 == 198 {print $10}')

    status="${GREEN}OK${NC}"
    plain_status="OK"

    if [[ "$health" != "PASSED" ]] || [[ "$realloc" -gt 0 ]] || [[ "$pend" -gt 0 ]] || [[ "$uncor" -gt 0 ]]; then
        status="${YELLOW}WARNING${NC}"
        plain_status="WARNING"
        problem_found=1
    fi

    if [[ "$realloc" -gt 10 ]] || [[ "$pend" -gt 0 ]] || [[ "$uncor" -gt 0 ]] || [[ "$health" == *"FAILED"* ]]; then
        status="${RED}ERROR${NC}"
        plain_status="ERROR"
        problem_found=1
    fi

    echo -e "\nšŸ“¦ Disk: $dev ($model / SN: $serial)"
    echo -e "šŸ“Š SMART Status: $health"
    echo -e "šŸ” Reallocated: $realloc | ā³ Pending: $pend | āŒ Offline Uncorrectable: $uncor"
    echo -e "šŸ§­ Assessment: $status"

    if [[ "$plain_status" != "OK" ]]; then
        {
            echo "Disk: $dev ($model / SN: $serial)"
            echo "SMART Status: $health"
            echo "Reallocated: $realloc"
            echo "Pending:     $pend"
            echo "Uncorrectable: $uncor"
            echo "Assessment: $plain_status"
            echo "---------------------------------------"
        } >> "$MAIL_BODY"
    fi
done

echo -e "\nDisk scan completed."

# Send mail if problems are detected
if [[ $problem_found -eq 1 ]]; then
    {
        echo "From: SMART Monitor <$MAIL_FROM>"
        echo "To: Admin <$MAIL_TO>"
        echo "Subject: $MAIL_SUBJECT"
        echo "Content-Type: text/plain; charset=UTF-8"
        echo
        cat "$MAIL_BODY"
    } | msmtp --debug --from=default -t

    echo -e "${RED}Alert sent via msmtp to $MAIL_TO${NC}"
else
    echo -e "${GREEN}No SMART issues ā€“ message not sent.${NC}"
fi

rm -f "$MAIL_BODY"

Linux - Count patern in files

grep -oIRH "abc" /var/log | cut -d: -f1 | sort | uniq -c | sort -nr

Linux - Disabling flooding logs from user

https://wiki.archlinux.org/index.php/systemd/User

loginctl enable-linger username

Exif - Modifiy dates of media files (jpg) in current folder

find -name '*.jpg' | while read PIC; do
    DATE=$(exiftool -p '$DateTimeOriginal' $PIC |
    sed 's/[: ]//g')
    touch -t $(echo $DATE | sed 's/\(..$\)/\.\1/') $PIC
done

Exif - Modifiy dates for mp4 files

mp4 files in my case has no tag DateTimeOriginal

find -name '*.mp4' | while read PIC; do
    DATE=$(exiftool -p '$CreateDate' $PIC |
    sed 's/[: ]//g')
    touch -t $(echo $DATE | sed 's/\(..$\)/\.\1/') $PIC
done
24 Jul 25 15:37 CEST
Edit this page