LINUX - Shell script to detect file change

Below is the simple shell script which can be used to detect the file change in a particular directory. It can detect file modified, file touched, new file created, file deleted / moved, files mode change (chmod) / files owner change (chown). It runs continuously after a sleep of specified interval and compare the list of files against the list of files prior to sleep.

Please see the self explanatory shell script below.

Shell script - function to split a string based on a delimiter

#!/bin/ksh

#function to split a string based on a delimiter
function split {

        string=$1
        delimiter=$2

echo "split string '"$string"' with delimiter '"$delimiter"'"
        for word in $(echo $string | tr "$delimiter" "\n")
        do
                echo $word
        done
}

#call the split function with arguments string and delimiter
split "transient;private;protected;public;synchronized;native" ";"

split "float|short|char|double" "|"

UNIX/LINUX - How to encrypt/decrypt a text file

crypt command can be used to encrypt/decrypt a particular text file in UNIX/LINUX. It is a simple encryption algorithm which works on a secret-key basis. It encrypts or decrypts a stream of data from standard input, and writes the result to standard output. Encryption and decryption is a symmetric process.

Encryption
This will create an encrypted file enTextFile.txt  from a text file textFile.txt. You can use either of the commands both does one and the same task.

crypt ENCRYPTION_KEY < textFile.txt > enTextFile.txt

OR

cat textFile.txt | crypt > enTextFile.txt

Decryption
This will decrypt an encrypted file enTextFile.txt and store the output in textFile.txt

crypt ENCRYPTION_KEY < enTextFile.txt (to show data on console)
crypt ENCRYPTION_KEY < enTextFile.txt > textFile.txt

OR

cat enTextFile.txt | crypt > textFile.txt

Linux - Copy/paste block of codes from Windows editor to Vi editor

Below is the command used to Copy/paste a block of code(s) from Windows editor to Vi editor. Basically to replace complete text from a particular file.

First copy the block of code(s) from windows editor then go to Vi editor and use the following command sequentially as given below

<ESC>
d
Shift+g
i
Shift+Insert
<ESC>

Above command says delete (d) the codes form specific line number (0 in this case) upto end of the file i.e. last line (note - capital G, hence Shift+g). Then go to insert mode (i) and paste the copied lines.

Linux - Block of comments in shell script

There is no block comment on shell script however there are few options that you can try out -

Method 1

#!/bin/bash
echo "This is a statement"
: << 'COMMENTS'
echo "The code that you want be commented out goes here"
echo "This echo statement will not be called"
COMMENTS
echo "This statement is outside the comment block"

At the beginning of the block you wish to comment out put -
: << 'COMMENTS'

The '' around the COMMENTS delimiter is important, otherwise things inside the block will be parsed and executed. I have used 'COMMENTS' as it should be a string of characters which is not used in you code, modify it in your code if needed.

And at the end of the block put -
COMMENTS

Method 2

#!/bin/bash
echo "This is a statement"
if [ 1 -eq 0 ]; then
echo "The code that you want be commented out goes here"
echo "This echo statement will not be called"
fi
echo "This statement is outside the comment block"

Added a false condition in the IF block so that it won't get executed.

Method 3

Using vi editor you can easily comment/uncomment from line n to m

Comment

<ESC>
:10,100s/^/#/

Above command says from line 10 to 100 substitute line start (^) with a # sign

Uncomment

<ESC>
:10,100s/^#//

Above command says from line 10 to 100 substitute line start (^) followed by # with noting //

Linux : Base64 Encode & Decode with OpenSSL

Base64 is a group of similar binary-to-text encoding schemes that represent binary data in an ASCII string format by translating it into a radix-64 representation. The term Base64 originates from a specific MIME content transfer encoding.

Base64 encoding schemes are commonly used when there is a need to encode binary data that needs to be stored and transferred over media that is designed to deal with textual data. This is to ensure that the data remains intact without modification during transport. Base64 is commonly used in a number of applications including email via MIME, and storing complex data in XML.

In Linux you can achieve Base64 Encoding and Decoding using pre-installed OpenSSL package.

Encode

Below command will encode any text contents within the quotes and display the encoded contents on a new line.

echo 'text content' | openssl base64

You can also encode multiple lines using below command

echo 'text content' | openssl base64 && echo 'another text content' | openssl base64

Decode

Similarly you can decode using '-d' flag as shown below

echo 'dGV4dCBjb250ZW50Cg==' | openssl base64 -d


Using the OpenSSL package, you can also encode or decode a specific file, as shown below:

encode file
openssl base64 -in 'file.txt' -out 'encodedFile.txt'

decode file
openssl base64 -d -in 'encodedFile.txt' -out 'file.txt'

Unix/Linux - Shell script to find and replace a text from the list of files

Below is the automated shell script which will find a text and replace it with the given string. It will also keep a backup of existing file. It can also be used for global search and replace string from a text file. Please see the self-explanatory shell script below -

replace.sh

#!/bin/bash

#find string
searchString="text1"

#replace with
replaceWith="text2"

echo "searching "$searchString" ..."
grep -l $searchString * >foundIn.txt
chmod 777 foundIn.txt

echo "replacing "$replaceWith" ..."
count=0
while read x
do
  if [ $x != replace.sh ]
  then
          (cat  $x) >$x"_bkp"
          sed "s/${searchString}/${replaceWith}/g" $x"_bkp" >  $x
          count=$(($count+1))
          echo "Found in... "$x
  fi
done <foundIn.txt

echo $count "occurences have been replaced."

Linux/Unix - Shell script to read a text file

Method 1 - By iterating each line using while loop

#!/bin/bash

#Name of the text file
txtFile=myFile.txt

echo "Reading text file "$txtFile" ..."

while read x
do
echo $x
done <$txtFile


Method 2 - By using cat (concatenate and display files) command

#!/bin/bash

#Name of the text file
txtFile=myFile.txt

echo "Reading text file "$txtFile" ..."

cat $txtFile

keytool error: java.io.FileNotFoundException: cacerts (Permission denied)

I got the error when trying to install a certificate into my keystore

Command -
keytool -import -alias aliasName -file fileName.cer -keystore cacerts
keytool -list -keystore cacerts

Solutions

Windows -
This could happen if you are not running the command prompt in administrator mode. Login with the user who has administrator privilege.

For Windows 7 and above - Go to run, type cmd and hit Ctrl+Shift+enter. This will open the command prompt in administrator mode.

For others windows -  Go to start -> all programs -> accessories -> right click command prompt and say run as administrator.

Linux -
First check the write permissions on the keystore. Also if needed login with root user and try the command.

How to use scp command without prompting for password

You may need to write a shell script in which you want to copy a file from one server to another. OfCouse you can easily achieve it with SCP command however when you automate the shell script using crontab it will get stuck because of password prompt. To avoid this you want SCP command to work without prompting for password.

Secure Copy (SCP) allows files to be copied to, from, or between different hosts

Let’s assume you want to copy a file from host server (HOST) to destination server (DEST).

1. On HOST server run below command

$ ssh-keygen -t rsa

First it will prompt for "Enter file in which to save the key" just keep it blank by pressing enter. Then it will prompt for passphrase, again keep it blank by pressing enter. It will generate a public key (id_rsa.pub) and a private key (id_rsa) in the folder <your_home_dir>/.ssh

Output -

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/.ssh/id_rsa.
Your public key has been saved in /home/.ssh/id_rsa.pub.
The key fingerprint is:
1a:7b:3b:d7:5c:f0:6d:f1:01:75:0f:b1:71:6a:15:be user@HOST

2. Share the public key (id_rsa.pub) with DEST server

Copy the file id_rsa.pub from HOST to <your_home_dir>/.ssh folder of DEST. You can use scp or ftp anyway you are comfortable with.

3. Copy the contents of id_rsa.pub to <your_home_dir>/.ssh/authorized_keys2 in DEST server

Below command will append the content of id_rsa.pub to authorized_keys2. Note - you may even copy to authorized_keys whichever file exits on <your_home_dir>/.ssh in DEST server

$ cat id_rsa.pub >> authorized_keys2

Now if you use SCP command to transfer a file from HOST to DEST it won’t prompt for password. I hope it helped you.

Linux - Commands for Grep or Search recursively

Below are the commands for -

Grep for a particular text recursively

find ./ -name "*" | xargs grep "text" {} 2>/dev/null

Search for a particular file recursively

find ./ -name "*.properties" 2>/dev/null

UNIX / LINUX - HELP diff Command

NAME
     diff - compare two files

SYNOPSIS
     diff [-bitw] [-c | -e | -f | -h | -n | -u]  file1 file2

     diff [-bitw] [-C number | -U number]  file1 file2

     diff [-bitw] [-D string] file1 file2

     diff [-bitw] [-c | -e | -f | -h | -n | -u]  [-l]  [-r]  [-s] [-S name] directory1 directory2

DESCRIPTION
     The diff utility will compare  the  contents  of  file1  and file2  and write to standard output a list of changes necessary to convert  file1  into  file2.  This  list  should  be minimal.   Except  in rare circumstances, diff finds a smallest sufficient set of file differences. No output  will  be produced if the files are identical.

     The normal output contains lines of these forms:

     n1 a n3,n4
     n1,n2 d n3
     n1,n2 c n3,n4

     where n1  and  n2  represent  lines  file1  and  n3  and  n4 represent lines in file2 These lines resemble ed(1) commands to convert file1 to file2. By exchanging a for d and reading backward, file2 can be converted to file1. As in ed, identical pairs, where n1=n2 or n3=n4, are abbreviated as a single number.

     Following each of these lines come all the  lines  that  are affected  in  the  first  file  flagged by `<', then all the lines that are affected in the second file flagged by `>'.

OPTIONS
     The following options are supported:

     -b              Ignores trailing blanks  (spaces  and  tabs) and   treats  other  strings  of  blanks  as equivalent.

     -i              Ignores the case of  letters.  For  example, 'A' will compare equal to 'a'.

     -t              Expands characters  in  output  lines. Normal or -c output adds character(s) to the front of each line that may adversely affect the indentation of the original source lines and  make  the  output  lines  difficult  to interpret.  This  option  will  preserve the original source's indentation.

     -w              Ignores all blanks ( and   characters)  and  treats  all  other  strings of blanks   as   equivalent.    For    example, 'if ( a == b )'   will   compare   equal  to 'if(a==b)'.

     The following options are mutually exclusive:

     -c              Produces a listing of differences with three lines  of  context. With this option, output format is modified slightly. That is, output begins  with  identification  of  the  files involved and their creation dates, then each change  is  separated by a line with a dozen *'s. The lines removed from file1 are marked with  '-'.  The  lines  added  to  file2 are marked '+'. Lines that are changed from  one file  to  the other are marked in both files with '!'.

     -C number       Produces a listing of differences  identical to  that produced by -c with number lines of context.

     -D string       Creates a merged version of file1 and  file2 with  C  preprocessor  controls  included so that a compilation  of  the  result  without defining  string  is equivalent to compiling file1,  while  defining  string  will  yield file2.

     -e              Produces a script of only a, c, and  d  commands for the editor ed, which will recreate file2 from file1. In connection with the  -e option, the following shell program may help maintain multiple versions of a  file.  Only an  ancestral  file  ($1)  and  a  chain  of version-to-version  ed  scripts  ($2,$3,...) made  by  diff  need  be on hand. A ``latest version'' appears on the standard output. (shift; cat $*; echo '1,$p') | ed - $1

     -f              Produces a similar script, not  useful  with ed, in the opposite order.

     -h              Does a fast, half-hearted job. It works only when  changed  stretches  are short and well separated, but does work on files of  unlimited length. Options -c, -C, -D, -e, -f, and -n are unavailable with -h.  diff  does  not descend into directories with this option.

     -n              Produces a script similar to -e, but in  the opposite  order  and with a count of changed lines on each insert or delete command.

     -u              Produces a listing of differences with three lines  of  context. The output is similar to that of the -c option, except that the  context is "unified". Removed and changed lines in file1 are marked by  a  '-'  while  lines added  or  changed  in file2 are marked by a '+'. Both versions of changed  lines  appear in  the  output,  while  added, removed, and context lines appear only once. The identification  of  file1  and  file2 is different, with "---" and  "+++"  being  printed  where "***"  and  "---"  would  appear with the -c option. Each change is separated by  a  line of the form @@ -n1,n2 +n3,n4 @@

     -U number       Produces a listing of differences  identical to  that produced by -u with number lines of context.

     The following options are used for comparing directories:

     -l              Produces output in long format.  Before  the diff,  each text file is piped through pr(1) to  paginate  it.   Other  differences   are remembered  and  summarized  after  all text file differences are reported.

     -r              Applies diff recursively to common subdirectories encountered.

     -s              Reports  files  that  are  identical.  These identical  files would not otherwise be mentioned.

     -S name         Starts  a  directory  diff  in  the  middle, beginning with the file name.

OPERANDS
     The following operands are supported:

     file1           A path name of a file  or  directory  to  be
     file2           compared. If either file1 or file2 is -, the standard input will be used in its place.

     directory1      A path name of a directory to be compared.
     directory2

     If only one of file1 and file2 is a directory, diff will  be applied  to the non-directory file and the file contained in the directory file with a filename that is the same  as  the last component of the non-directory file.

USAGE
     See largefile(5) for the description of the behavior of diff  when encountering files greater than or equal to 2 Gbyte ( 2     **31 bytes).

EXAMPLES
     Example 1: Typical output of the diff command

     In the following command, dir1 is a directory  containing  a directory  named  x, dir2 is a directory containing a directory named x, dir1/x and dir2/x  both  contain  files  named date.out, and dir2/x contains a file named y:

     example% diff -r dir1 dir2
     Common subdirectories: dir1/x and dir2/x

     Only in dir2/x: y

     diff -r dir1/x/date.out dir2/x/date.out

     1c1

     < Mon Jul  2 13:12:16 PDT 1990

     ---

     > Tue Jun 19 21:41:39 PDT 1990

ENVIRONMENT VARIABLES
     See environ(5) for descriptions of the following environment variables  that  affect the execution of diff: LANG, LC_ALL, LC_CTYPE, LC_MESSAGES, LC_TIME, and NLSPATH.

     TZ       Determines the locale for  affecting  the  timezone used  for  calculating file timestamps written with the -C and -c options.

EXIT STATUS
     The following exit values are returned:

     0        No differences were found.

     1        Differences were found.

     >1       An error occurred.

FILES
     /tmp/d?????             temporary file used for comparison

     /usr/lib/diffh          executable file for -h option

ATTRIBUTES
     See attributes(5) for descriptions of the  following  attributes:

     ____________________________________________________________
    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    |_____________________________|_____________________________|
    | Availability                | SUNWesu                     |
    |_____________________________|_____________________________|
    | CSI                         | Enabled                     |
    |_____________________________|_____________________________|
    | Interface Stability         | Standard                    |
    |_____________________________|_____________________________|

SEE ALSO
     bdiff(1), cmp(1), comm(1), dircmp(1), ed(1),  pr(1),  sdiff(1), attributes(5), environ(5), largefile(5), standards(5)

NOTES
     Editing scripts produced under the  -e  or  -f  options  are naive  about  creating  lines  consisting of a single period (.).

     Missing NEWLINE at end of file indicates that the last  line of the file in question did not have a NEWLINE. If the lines are different, they will be flagged and output, although the output will seem to indicate they are the same.    

COMP and FC : Windows equivalent of UNIX / LINUX diff command

COMP and FC commands can be used as Windows equivalent of UNIX / LINUX diff command.

COMP

Compares the contents of two files or sets of files.

SYNTAX :

COMP [data1] [data2] [/D] [/A] [/L] [/N=number] [/C] [/OFF[LINE]]

  data1      Specifies location and name(s) of first file(s) to compare.

  data2      Specifies location and name(s) of second files to compare.

  /D         Displays differences in decimal format.

  /A         Displays differences in ASCII characters.

  /L         Displays line numbers for differences.

  /N=number  Compares only the first specified number of lines in each file.

  /C         Disregards case of ASCII letters when comparing files.

  /OFF[LINE] Do not skip files with offline attribute set.

To compare sets of files, use wildcards in data1 and data2 parameters.

FC

Compares two files or sets of files and displays the differences between them

SYNTAX:

FC [/A] [/C] [/L] [/LBn] [/N] [/OFF[LINE]] [/T] [/U] [/W] [/nnnn]

   [drive1:][path1]filename1 [drive2:][path2]filename2

FC /B [drive1:][path1]filename1 [drive2:][path2]filename2

  /A         Displays only first and last lines for each set of differences.

  /B         Performs a binary comparison.

  /C         Disregards the case of letters.

  /L         Compares files as ASCII text.

  /LBn       Sets the maximum consecutive mismatches to the specified

             number of lines.

  /N         Displays the line numbers on an ASCII comparison.

  /OFF[LINE] Do not skip files with offline attribute set.

  /T         Does not expand tabs to spaces.

  /U         Compare files as UNICODE text files.

  /W         Compresses white space (tabs and spaces) for comparison.

  /nnnn      Specifies the number of consecutive lines that must match

             after a mismatch.

  [drive1:][path1]filename1

             Specifies the first file or set of files to compare.

  [drive2:][path2]filename2

             Specifies the second file or set of files to compare.

How to send mail using MAILX command in UNIX / LINUX

Simple MAILX command

Below is the syntax to send simple mail using mailx command in UNIX / Linux.
You can use it as command line or used it in your shell script.


$ mailx -s "Mail Subject" test@xyz.com
type body of the mail
...
..
EOT (Ctrl+d)
$

Send attachment using MAILX command

In some cases you may have to send attachment in the mail. Lets see the syntax of sending an attachment using mailx command.

Below is the syntax to attach a file while sending a mail using mailx command in LINUX / UNIX


( cat /root/MailBody.txt
uuencode /root/file_name.txt file_name.txt 
) | mailx -s "Mail Subject" test@xyz.com


Here file_name.txt is the attachment to be attached in the mail
AND MailBody.txt is the text body of the mail.

UNIX/Linux : Provide jar files in classpath at run time

You may face an issue to provide .jar files in classpath at run time.

Lets say you have a Java file Test.java which requires foo.jar and bar.jar in classpath.

In Windows below mentioned command works well.

javac -cp foo.jar;bar.jar Test.java
java -cp foo.jar;bar.jar Test

Note the semi-colon(;) as jar file separator in windows.

Where as in UNIX/Linux it don't work. To run the above program in UNIX/Linux you need to use below mentioned command

javac -cp foo.jar:bar.jar Test.java
java -cp foo.jar:bar.jar Test

Note the colon(:) as as jar file separator in UNIX/Linux.

Linux : chmod: WARNING: chmod: could not getcwd OR chown: getcwd: Permission denied

When I tried to do chmod for a bulk of files at once using below mentioned chmod command, I used to get below error.
Below command is used for changing permission for a bulk of files recursively in a single shot.

Command : chmod -R 777 *

Error :

chmod: WARNING: chmod: could not getcwd /home/test

Also when i tried to do chown for a bulk of files at once using below mentioned chown command, I used to get error as shown below.
Below command is used to change owner name for a bulk of files recursively in a single shot.

Command : chown -R UserName:GroupName *

Error :

chown: getcwd: Permission denied

However when I do chmod or chown for one particular folder, it runs without error.

Command : chmod -R 777 FolderName
Command : chown -R UserName:GroupName FolderName

Reason :
On some Unix variants, getcwd() will return FALSE if any one of the parent directories does not have the readable or search mode set, even if the current directory does.

Solution :
Type "groups" command and see which other groups you can access.
Go to the parent folder and change the group name.
And then try the above commands, It worked for me and hope it may even work for you.

If not lets discuss it here.

Linux server copy command

In Linux / UNIX if you want to copy a file or a directory recursively from one server to other server you can use SCP (server copy) command


Synatax :

scp <filename> userName@server:/home/path/

OR

scp -r <foldername> userName@server:/home/path/

It will promt for a password, after entering correct password it will copy files or a directory recursively from specified source server to specified destination server.


Options available :

• -p : Preserves the modification and access times, as well as the permissions of the source-file in the destination-file
• -q : Do not display the progress bar
• -r : Recursive, so it copies the contents of the source-file (directory in this case) recursively
• -v : Displays debugging messages