This Blog is to share our knowledge and expertise on Linux System Administration and VMware Administration

Tuesday, December 26, 2017

Configuring the systemd journal for persistence in RHEL7

Tuesday, December 26, 2017 0

Configuring the systemd journal for persistence in RHEL7

By default, the journal doesn't store log files on disk, only in memory or the /run/log/journal directory. This is sufficient for the recent log history (with the journal) but not for long-term log retention should you decide to go with journal only and not with any other syslog solution.

Configuring journald to keep more logs than memory allows is fairly simple, as follows:
Open /etc/systemd/journald.conf with your favorite text editor with root permissions by executing the following command:
vim /etc/systemd/journald.conf

Ensure that the line containing Storage is either remarked or set to auto or persistent and save it, as follows:

[root@nsk ~]# vim /etc/systemd/journald.conf
#  This file is part of systemd.
...
..
[Journal]
Storage=auto

If you select auto, the journal directory needs to be manually created. The following command would be useful for this:

[root@nsk ~]# mkdir -p /var/log/journal
[root@nsk ~]#

Now, restart the journal service by executing the following command:

[root@nsk ~]# systemctl restart systemd-journald

There are many other options that can be set for the journal daemon.
By default, all the data stored by journald is compressed, but you could disable this using Compress=no.
It is recommended to limit the size of the journal files by either specifying a maximum retention age (MaxRetentionSec), a global maximum size usage (SystemMaxUse), or a maximum size usage per file (SystemMaxFileSize).


Monday, December 25, 2017

Screen command in Linux

Monday, December 25, 2017 0

Screen command in Linux


NAME
       screen - screen manager with VT100/ANSI terminal emulation

Screen  is  a full-screen window manager that multiplexes a physical terminal between several processes. When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can  use the  program  as  you  normally  would.  Then,  at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill existing windows, view a list of windows, turn output logging on and off, copy-and-paste text between windows, view  the  scrollback history,  switch  between  windows in whatever manner you wish, etc. All windows run their programs completely independent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the user’s  terminal.   When  a program  terminates,  screen  (per default) kills the window that contained it.  If this window was in the foreground, the display switches to the previous window; if none are left, screen exits.

#screen -h 2000 -t SERVER_UPDATE -S SERVER_UPDATE

Here
  -h  num - Specifies the history scrollback buffer to be num lines high.
  -t  name - sets the title for the default shell or specified program. 
  -S  sessionname  -  When  creating  a  new  session,  this  option can be used to specify a meaningful name for the session. This name identifies the session for
            "screen -list" and "screen -r" actions. It substitutes the default [tty.host] suffix.

Prints a list of screen session
#screen -ls
There is a screen on:
        27137.pts-1.testserver       (Attached)
1 Socket in /var/run/screen/S-root.

Detach the screen session
#screen -D 27137.pts-1.testserver
[27137.pts-1.testserver power detached.]

Attach the screen session
#screen -r 27137.pts-1.testserver

Please check man screen page for more options.

Sunday, December 24, 2017

Configuring logrotate for yum in RHEL7

Sunday, December 24, 2017 0

Configuring logrotate for yum in RHEL7


Every time you use yum to install and/or update packages, it logs to /var/log/yum.log. 

I do recommend keeping your complete yum history as it doesn't grow a lot, unless you reinstall packages a lot.

For a rich interface to your yum history, I suggest you use yum history.

By default, your yum log file is rotated yearly, and even then, it only rotates if the size of your log file exceeds 30 KB, and your logs are only kept for 4 years. 

virtual servers have the potential to stay "alive" beyond these 3-4 years.

How to do it…
Modify /etc/logrotate.d/yum to the following:

/var/log/yum.log {
    missingok
    notifempty
    size 30k
    rotate 100
    yearly
    create 0600 root root
}

How it works…
This configuration will only rotate the yum log when it exceeds 30 KB in size on a yearly basis, and it will keep 100 rotated logs, which is basically log files for 100 years!

Friday, December 22, 2017

Install KVM in RHEL7

Friday, December 22, 2017 0

Install KVM in RHEL7

By default, a RHEL 7 system doesn't come with a KVM or libvirt preinstalled. This can be installed in three ways:

Through the graphical setup during the system's setupVia a kickstart installationThrough a manual installation from the command line


To install a KVM, you will require at least 6 GB of free disk space, 2 GB of RAM, and an additional core or thread per guest.

Check whether your CPU supports a virtualization flag (such as SVM or VMX). Some hardware vendors disable this in the BIOS, so you may want to check your BIOS as well. Run the following command:

# grep -E 'svm|vmx' /proc/cpuinfo
flags    : ... svm ...
Check whether the hardware virtualization modules (such as kvm_intel and kvm) are loaded in the kernel using the following command:

# lsmod | grep kvm
kvm_intel             155648  0
kvm                      495616  1 kvm_intel

Manual installation
This way of installing a KVM is generally done once the base system is installed by some other means. 

Install the software needed to provide an environment to host virtualized guests with the following command:
# yum -y install qemu-kvm qemu-img libvirt

The installation of these packages will include quite a lot of dependencies.

Install additional utilities required to configure libvirt and install virtual machines by running this command:
# yum -y install virt-install libvirt-python python-virthost libvirt-client

By default, the libvirt daemon is marked to autostart on each boot. Check whether it is enabled by executing the following command:
# systemctl status libvirtd

libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: inactive
   Docs: man:libvirtd(8)
    http://libvirt.org

If for some reason this is not the case, mark it for autostart by executing the following:
# systemctl enable libvirtd
To manually stop/start/restart the libvirt daemon, this is what you'll need to execute:
# systemctl stop libvirtd
# systemctl start libvirtd
# systemctl restart libvirtd

Kickstart installation
Installing a KVM during kickstart offers you an easy way to automate the installation of KVM instances. 

Add the following package groups to your kickstarted file in the %packages section:
@virtualization-hypervisor
@virtualization-client
@virtualization-platform
@virtualization-tools
Start the installation of your host with this kickstart file.

Graphical setup during the system's setup
This is probably the least common way of installing a KVM. The only time I used this was during the course of writing this recipe. Here's how you can do this:

Boot from the RHEL 7 Installation media.
Complete all steps besides the Software selection step.
Go to Software Selection to complete the KVM software selection.
Select the Virtualization host radio button in Base Environment, and check the Virtualization Platform checkbox in Add-Ons for Selected Environment:
Finalize the installation.
On the Installation Summary screen, complete any other steps and click on Begin Installation.

Wednesday, December 20, 2017

ssh_exchange_identification: Connection closed by remote host - Password less authentication setup

Wednesday, December 20, 2017 0

ssh_exchange_identification: Connection closed by remote host - Password less authentication setup 


Situation:
While setup passwordless authentication from testserver2 (192.181.166.55) to testserver1 (192.181.130.55)  server, getting  error "ssh_exchange_identification: Connection closed by remote host"

[kanachim@testserver2 .ssh]$ ssh -vv testserver1
OpenSSH_5.3p1-hpn13v7, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.181.166.55 [192.181.166.55] port 22.
debug1: Connection established.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /home/kanachim/.ssh/id_rsa type 1
debug1: identity file /home/kanachim/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host

Solution:
Check /etc/hosts.deny on server testserver1

root@host testserver1# grep sshd /etc/hosts.deny 
# DenyHosts: Mon Dec 18 22:10:38 2017 | sshd: 192.181.166.55
sshd: 192.181.166.55

Remove the sshd entry from hosts.deny on testserver1

Login to testserver2 Switch to user kanachim & create new key

-bash-3.2$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/kanachim/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/kanachim/.ssh/id_rsa.
Your public key has been saved in/home/kanachim/.ssh/id_rsa.pub.
The key fingerprint is:
37:6a:69:c1:a4:01:4b:c5:34:be:32:42:90:0b:20:a0 kanachim@192.181.166.55
The key's randomart image is:
+--[ RSA 2048]----+
|B.  o++          |
|=. . +..         |
|E.. . o .        |
|..     *         |
|  . o o S o      |
|   . o   = .     |
|        =        |
|       o         |
|                 |
+-----------------+
-bash-3.2$
-bash-3.2$ ssh-copy-id kanachim@192.181.130.55
Password:
Now try logging into the machine, with "ssh 'kanachim@192.181.130.55'", and check in:
  .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

Now login from testserver2 to testserver1

-bash-3.2$ ssh 192.181.130.55
Last login: Thu Feb 27 00:30:38 2014 from 10.217.230.145
-bash-3.2$ hostname
testserver1

Tuesday, December 19, 2017

Recover RPM DB from a corrupted RPM database in RHEL 7

Tuesday, December 19, 2017 0

Rebuilding corrupted rpm database in RHEL7

Situation:
Although everything is done to ensure that your RPM databases are intact, your RPM database may become corrupt and unuseable. This happens mainly if the filesystem on which the rpm db resides is suddenly inaccessible (full, read-only, reboot, or so on).

Solution:
1.Start by creating a backup of your corrupt rpm db, as follows:
[root@nsk ~]# tar zcvf rpm-db.tar.gz /var/lib/rpm/*
tar: Removing leading `/' from member names
/var/lib/rpm/Basenames
/var/lib/rpm/Conflictname
/var/lib/rpm/__db.001
/var/lib/rpm/__db.002
/var/lib/rpm/__db.003
/var/lib/rpm/Dirnames
/var/lib/rpm/Group
/var/lib/rpm/Installtid
/var/lib/rpm/Name
/var/lib/rpm/Obsoletename
/var/lib/rpm/Packages
/var/lib/rpm/Providename
/var/lib/rpm/Requirename
/var/lib/rpm/Sha1header
/var/lib/rpm/Sigmd5
/var/lib/rpm/Triggername

2.Remove stale lock files if they exist through the following command:
[root@nsk ~]# rm -f /var/lib/rpm/__db*

3.Now, verify the integrity of the Packages database via the following:
[root@nsk ~]# /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages; echo $?
BDB5105 Verification of /var/lib/rpm/Packages succeeded.
0

If it prints 0, proceed to next step.

4. Rename the Packages file (don't delete it, we'll need it!), as follows:
[root@nsk ~]# mv /var/lib/rpm/Packages  /var/lib/rpm/Packages.org

5. Now, dump the Packages db from the original Packages db by executing the following command:
[root@nsk ~]# cd /var/lib/rpm/
 [root@nsk rpm]# /usr/lib/rpm/rpmdb_dump Packages.org | /usr/lib/rpm/rpmdb_load Packages
rpmdb_load: BDB1540 configured environment flags incompatible with existing environment

6.Verify the integrity of the newly created Packages database. Run the following:
[root@nsk rpm]#  /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages; echo $?
BDB5105 Verification of /var/lib/rpm/Packages succeeded.
0

If the exit code is not 0, you will need to restore the database from backup.

7. Rebuild the rpm indexes, as follows:
[root@nsk ~]# rpm -vv --rebuilddb
[root@nsk rpm]# rpm -vv --rebuilddb
D: rebuilding database /var/lib/rpm into /var/lib/rpmrebuilddb.1312
D: opening  db environment /var/lib/rpm private:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db environment /var/lib/rpmrebuilddb.1312 private:0x401
D: opening  db index       /var/lib/rpmrebuilddb.1312/Packages (none) mode=0x42
D: opening  db index       /var/lib/rpmrebuilddb.1312/Packages 0x1 mode=0x42
D: disabling fsync on database
....
...
D: adding "5f7fd424d0773a4202731bff4901d449699b0929" to Sha1header index.
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm
D: closed   db index       /var/lib/rpmrebuilddb.1312/Sha1header
D: closed   db index       /var/lib/rpmrebuilddb.1312/Sigmd5
D: closed   db index       /var/lib/rpmrebuilddb.1312/Installtid
D: closed   db index       /var/lib/rpmrebuilddb.1312/Dirnames
D: closed   db index       /var/lib/rpmrebuilddb.1312/Triggername
D: closed   db index       /var/lib/rpmrebuilddb.1312/Obsoletename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Conflictname
D: closed   db index       /var/lib/rpmrebuilddb.1312/Providename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Requirename
D: closed   db index       /var/lib/rpmrebuilddb.1312/Group
D: closed   db index       /var/lib/rpmrebuilddb.1312/Basenames
D: closed   db index       /var/lib/rpmrebuilddb.1312/Name
D: closed   db index       /var/lib/rpmrebuilddb.1312/Packages
D: closed   db environment /var/lib/rpmrebuilddb.1312

8. Use the following command to check the rpm db with yum for any other issues (this may take a long time):
[root@nsk rpm]# yum check
Loaded plugins: fastestmirror
....
...

9. Restore the SELinux context of the rpm database through the following command:
[root@nsk rpm]# restorecon -R -v /var/lib/rpm

Monday, December 18, 2017

How to reboot the Xen Virtual Machine when was at hung state.

Monday, December 18, 2017 0

How to reboot the Xen Virtual Machine when was at hung state.

If console is not working from the host(Dom0), the console was opened in order to use sysrq magic packages
#xm console xenvm006

From other terminal on the node, run the below  comamnd one by one.
#xm sysrq xenvm006  h
#xm sysrq xenvm006  m    #should show the amount of memory been used
#xm sysrq xenvm006  t      #should show the current tasks

the output is  "console is opened and was not good"  also   "vm was with out resources"
Check the xentop, if  a high cpu usage is shown like 200%

Then reboot the virtual machine.

Open one more terminal & run the below command.

#xm destroy  xenvm006

if the virtual machine is running under cluster,  first disable the vm from cluster to avoid failover.

#clusvcadm -d vm:xenvm006

Then destroy the virtual machine.

Because as the vm is hung state, clusvcadm -d wont do a clean shutdown, so we need to destroy it manually.

#xm create xenvm006  (will start the vm)
#clusvcadm -e vm:xenvm006   (with cluster)

Sunday, December 17, 2017

Understanding the LVM Configuration Files in Linux

Sunday, December 17, 2017 0

Understanding the LVM Configuration Files in Linux

The following files are part of LVM configuration:

/etc/lvm/lvm.conf
    Central configuration file read by the tools.

etc/lvm/lvm_hosttag.conf
    For each host tag, an extra configuration file is read if it exists: lvm_hosttag.conf. If that file defines new tags,
    then further configuration files will be appended to the list of tiles to read in.
In addition to the LVM configuration files, a system running LVM includes the following files that affect LVM system setup:

/etc/lvm/cache
    Device name filter cache file (configurable).

/etc/lvm/backup/
    Directory for automatic volume group metadata backups (configurable).

/etc/lvm/archive/
    Directory for automatic volume group metadata archives (configurable with regard to directory path and archive history depth).

/var/lock/lvm/
    In single-host configuration, lock files to prevent parallel tool runs from corrupting the metadata; in a cluster, cluster-wide DLM is used.