operations

Change the frontpage notice on usegalaxy.eu

Fork the repo, change the file and create a PR: https://github.com/galaxyproject/galaxy-hub/blob/master/content/bare/eu/usegalaxy/notices.md

decode Galaxy id

user@sn06:~$ . /opt/galaxy/venv/bin/activate
(venv) user@sn06:~$ cd /opt/galaxy
(venv) user@sn06:/opt/galaxy$ python server/scripts/secret_decoder_ring.py decode ec81bbe85ee13506
746380

or using gxadmin

user@sn06:~$ . /opt/galaxy/venv/bin/activate
(venv) user@sn06:~$ GALAXY_ROOT=/opt/galaxy/server GALAXY_CONFIG_FILE=/opt/galaxy/config/galaxy.yml gxadmin galaxy decode ec81bbe85ee13506
746380

non responsive yum

rm -f /var/lib/rpm/__*
rpm --rebuilddb -v -v
yum clean all

visit all compute nodes and execute one command

pdsh -g cloud 'singularity --version | colordiff'

GPUs

Check if tensorflow is compiled with GPU support and a GPU is available.

import tensorflow as tf;
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True));
print(tf.test.is_built_with_cuda()); print( tf.test.is_gpu_available())

Check the utilization of the GPU on the host system:

> nvidia-smi 

Sun Aug 25 22:41:01 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   40C    P0    28W /  70W |      0MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:06.0 Off |                    0 |
| N/A   42C    P0    28W /  70W |      0MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            Off  | 00000000:00:07.0 Off |                    0 |
| N/A   42C    P0    27W /  70W |      0MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            Off  | 00000000:00:08.0 Off |                    0 |
| N/A   43C    P0    26W /  70W |      0MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Grafana

Installing or updating Grafana plugins can be done via the grafana-cli tools. Please note that the plugin folder is mounted from a volume and resides in /vdb/grafana/plugins/

ssh stats.galaxyproject.eu
sudo grafana-cli --pluginsDir /vdb/grafana/plugins/ plugins update grafana-worldmap-panel
sudo /etc/init.d/grafana-server restart

How to find out any password from Jenkins

find out the credential name from the “Bindings” tab in the project’s configuration.

find the encrypted value:

root@build:~$ grep -A1 vault-pass-usegalaxy-star /opt/jenkins/jenkins/jobs/usegalaxy-eu/config.xml
           <description>vault-pass-usegalaxy-star</description>
           <secret>{supersecretstringhere}</secret>

decrypt

go to jenkins → manage jenkins → script console https://build.galaxyproject.eu/script

google “jenkins decrypt secret” because you can never remember
println(hudson.util.Secret.fromString(“{supersecretstringhere}”).getPlainText())

done!

How to roll-back Jenkins

Find jenkins.war, in our case its at /usr/share/java. Rename the jenkins.war file:

/usr/share/java$ mv jenkins.war jenkins.war_2.375

Get older Jenkins version and restart.

/usr/share/java$ wget https://updates.jenkins.io/download/war/2.374/jenkins.war
/usr/share/java$ systemctl restart jenkins

or Instead of rollback try the newest release with sudo yum update jenkins

How Condor honors TIaaS priorities

All the details in this https://github.com/usegalaxy-eu/issues/issues/277

PostgreSQL configuarions

PGTune is a PG configurator assistent
configuration files are under /var/lib/pgsql/13/data/

How to install pdsh with genders support on Centos8

https://gist.github.com/gmauro/cc97ff1287282469ce98c2b8035100f2

debug ‘D’ state in processe

Get all processes in D state:

ps axl awk ‘$10 ~ /D/’

Looking at file handlers of a thread yields to:

root@sn06:~$ ll /proc/215503/task/296960/fd/** 
lr-x------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/0 -> /dev/null
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/1 -> 'socket:[3487300049]'
lr-x------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/10 -> /data/jwd01/main/048/946/48946081
lr-x------ 1 galaxy galaxy 64 Aug  2 17:11 /proc/215503/task/296960/fd/11 -> /data/jwd01/main/048/946/48946885
lr-x------ 1 galaxy galaxy 64 Aug  2 17:11 /proc/215503/task/296960/fd/12 -> /data/jwd01/main/048/950/48950933
lr-x------ 1 galaxy galaxy 64 Aug  2 17:11 /proc/215503/task/296960/fd/13 -> /data/jwd01/main/048/950/48950933
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/2 -> 'socket:[3487300049]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/3 -> 'socket:[3487336618]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/4 -> 'socket:[3487342602]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/41 -> 'socket:[3487388502]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/42 -> 'socket:[3487426468]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/43 -> 'socket:[3487471967]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/44 -> 'socket:[3487492268]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/45 -> 'socket:[3501137915]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/47 -> 'socket:[3487842897]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/5 -> 'socket:[3487337597]'
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/53 -> 'socket:[3518012875]'
l-wx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/6 -> /opt/galaxy/server/compliance.log
lrwx------ 1 galaxy galaxy 64 Aug  2 17:09 /proc/215503/task/296960/fd/7 -> 'socket:[3526229330]'
lr-x------ 1 galaxy galaxy 64 Aug  2 17:11 /proc/215503/task/296960/fd/8 -> /data/jwd01/main/048/946/48946885/metadata
lr-x------ 1 galaxy galaxy 64 Aug  2 17:11 /proc/215503/task/296960/fd/9 -> /data/jwd01/main/048/946/48946081/outputs

Which looks ok, Galaxy is cleaning up jobs.