Linux Commands for Big Data Professionals
ls Command:
ls command is used to list down files.
a) List Files using ls with no option
[jmarun@jmarun BD]# ls
a.txt b.txt
[jmarun@jmarun BD]#
b) List Files With option –l
[jmarun@localhost ~]$ ls -l
-rw-rw-r–. 1 jmarun jmarun 0 Feb 11 18:51 jmarun.txt
-rwxr-xr-x. 1 jmarun jmarun 0 Feb 12 18:22 diva.txt
[jmarun@localhost ~]$
c) View Hidden Files
List all files including hidden file starting with ‘.‘.
[jmarun@jmarun ~]$ ls -al
drwx——. 27 jmarun jmarun 4096 Feb 21 14:22 .
drwxr-xr-x. 3 root root 4096 Feb 10 13:52 ..
-rw-rw-r–. 1 jmarun jmarun 0 Feb 11 18:51 jmarun.txt
-rwxr-xr-x. 1 jmarun jmarun 0 Feb 12 18:22 diva.txt
[jmarun@localhost ~]$
d) List Files with Human Readable Format with option -lh
With combination of -lh option, shows sizes in human readable format.
[jmarun@localhost ~]$ ls -lh
-rw-rw-r–. 1 jmarun jmarun 0 Feb 12 18:25 d1.txt
-rw-rw-r–. 1 jmarun jmarun 0 Feb 12 18:25 d2.txt
[jmarun@localhost ~]$
e) List Files and Directories with ‘/’ Character at the end
Using -F option with ls command, will add the ‘/’ Character at the end each directory.
[jmarun@localhost ~]$ ls -F
d1.txt Desktop/ diva.txt* Downloads/ Pictures/ Templates/ xyz/
d2.txt jmarun.txt Documents/ Music/ Public/ Videos/
[jmarun@localhost ~]$
f) List Files in Reverse Order
The following command with ls -r option display files and directories in reverse order.
[jmarun@localhost ~]$ ls -r
xyz Templates Pictures Downloads diva.txt Desktop d1.txt Videos Public jmarun.txt d2.txt
[jmarun@localhost ~]$
g) Recursively list Sub-Directories
ls -R option will list very long listing directory trees. See an example of output of the command.
[jmarun@localhost ~]$ ls -R
d1.txt Desktop diva.txt Downloads Pictures Templates xyz
h) Reverse Output Order
With combination of -ltr will shows latest modification file or directory date as last.
[jmarun@localhost ~]$ ls -ltr
drwxr-xr-x. 2 jmarun jmarun 4096 Feb 10 18:55 Templates
drwxr-xr-x. 2 jmarun jmarun 4096 Feb 10 18:55 Downloads
drwxr-xr-x. 2 jmarun jmarun 4096 Feb 10 18:55 Videos
drwxr-xr-x. 2 jmarun jmarun 4096 Feb 10 18:55 Public
i) Sort Files by File Size
With combination of -lS displays file size in order, will display big in size first.
[jmarun@localhost ~]$ ls -ls
0 -rw-rw-r–. 1 jmarun jmarun 0 Feb 12 18:25 d1.txt
0 -rw-rw-r–. 1 jmarun jmarun 0 Feb 12 18:25 d2.txt
4 drwxr-xr-x. 4 jmarun jmarun 4096 Feb 11 18:03 Desktop
0 -rw-rw-r–. 1 jmarun jmarun 0 Feb 11 18:51 jmarun.
j) Display Inode number of File or Directory
We can see some number printed before file / directory name. With -i options list file /directory with inode number
[jmarun@localhost ~]$ ls -i
272870 d1.txt 272869 diva.txt 397175 Pictures 397282 xyz
272871 d2.txt 397173 Documents 397172 Public
k) Shows version of ls command
Check version of ls command.
[jmarun@localhost ~]$ ls –version
ls (GNU coreutils) 8.4
Copyright (C) 2010 Free Software Foundation, Inc
l) Show Help Page
List help page of ls command with their option.
[jmarun@localhost ~]$ ls –help
Usage: ls [OPTION]… [FILE]…
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor –sort.
m) List Directory Information
With ls -l command list files under directory /tmp. Wherein with -ld parameters displays information of /tmp directory.
[jmarun@localhost ~]$ ls -l /
dr-xr-xr-x. 2 root root 4096 Feb 10 19:34 bin
dr-xr-xr-x. 5 root root 1024 Feb 10 13:52 boot
n) Display UID and GID of Files
To display UID and GID of files and directories. use option -n with ls command.
[jmarun@localhost ~]$ ls -n
-rw-rw-r–. 1 500 500 0 Feb 12 18:25 d1.txt
-rw-rw-r–. 1 500 500 0 Feb 12 18:25 d2.txt
Copying with the cp Command
a) How do I copy files?
cp filename1 filename2 -> copies a file.
b) How do I copy recursively? cp –r dir1 dir2
c) To see copy progress pass –v option to cp command. cp –v –r dir1 dir2
d) How do I confirm file overwriting? cp –i dir1 dir2
e) Preserve the file permission and other attributes.
cp –p file1 file2
Deleting Files with the ‘rm’ Command
The rm command deletes the files.This command has several options,but should be used cautiously.
a) The rm command will delete one or several files from the command line.
rm file1
rm file1 file2 file3
b) One of the safer ways to use rm is through the -i or interactive option, where you’ll be asked if you want to delete the file.
rm -i file1
c) you can also force file deletion by using -f option
rm -f file1
d) when we combine -f and -r, the recursive option , you can delete directories and all files or directories found.
rm -rf <directoryname>
Creating Directories with the ‘mkdir’ Command
The mkdir command can create one or several directories with a single command line.
a) Creating directories
mkdir <directoryName>
b) Creating multiple directories
mkdir <directoryName1> <directoryname2>
c) Creating Child under directories
mkdir temp/child
d) To build a hierarchy of directories with mkdir, you must use the -p,or parent option, for example
mkdir -p temp5/parent/child
Removing Directories with the ‘rmdir’ Command
a) The rmdir command is used to remove directories. To remove a directory, all you have to do is type
rmdir <DirectoryName>
b) Removing directories and sub directories as well
rm -rf
Renaming Files with the ‘mv’ command.
The mv command, called a rename command but know to many as a move command
a) Move the data file1 to file2
mv <file1> <file2>
b) The mv command can work silently, or as with rm, you can use the -i (interactive) option
mv -i <file1> <file2>
Creating Hard and Symbolic Links with the ‘ln’ Command
The ln command creates both types of links.If you use the ln command to create a hard link, you specify a second file on the command line you can use to reference the original file,for Ex
# ln file1 file2
#ln -s file1 file 2
ps command: The ps command will show information about current system process.
Ps-> The user’s currently running processes.
Ps –f -> Full listing of the user’s currently running process.
ps –ef -> Full listing of all process, except kernel process.
Ps –A -> All process, including kernel process.
Ps auxw -> wide listing sorted by percentage of CPU usage.
Last: The last command shows the history of who has logged in to the system since the wtmp file was created.
Who: The who command gives this output for each logged-in user: username,tty .login time and where the user logged in from.
W: The W command is really an extended who
Checking your installation Files
rpm –qa | grep ^x
you should receive the output similar to the following..
xorg-x11-drv-apm-1.2.2-1.1.el6.x86_64
xorg-x11-drv-penmount-1.4.0-5.el6.x86_64
xorg-x11-drv-ast-0.89.9-1.1.el6.x86_64
xorg-x11-drv-aiptek-1.3.0-2.el6.x86_64
Installing the X Files
rpm –ivh <filename>
Moving to different directories with the cd command.
cd /usr/bin
cd ..
cd ../..
cd or cd –
Knowing where you are with the pwd command.
pwd
pwd –help
/bin/pwd –help
Searching directories for matching files with the find command
Syntax: find where-to-look criteria what-to-do
Find /usr –name spell –print
Listing and Combining Files with the cat Command
The cat (Concatenate file) commands are used to send the contents of files to your screen.
a) cat a.txt
[abc@master]$ cat a.txt
Hello Hello Hello
Hello hello hello
[abc@master]$
b) The cat command also has a number of options. If you’d like to see your file with line numbers, you can use the n-option
#cat –n a.txt
1 Hello Hello Hello
2 Hello hello hello
c) You can also use cat to look at several files at once.
# cat –n test*
1 Hello Hello Hello
2 Hello hello hello
[root@localhost ~]# cat a.txt
aksjlkdj
[root@localhost ~]#
[root@localhost ~]# cat b.txt
alksjdjkjaskjkaj
;lajslkdjja
lkashdjfajs
[root@localhost ~]#
d) As you can see, cat has also include a second file in its output.
[root@localhost ~]# cat a.txt b.txt
aksjlkdj
alksjdjkjaskjkaj
;lajslkdjja
lkashdjfajs
[root@localhost ~]#
cat > c.txt
e) To see the numbers using cat
# cat -n jmarun.txt
100 jmarun 20000
200 diiia 10000
399 ksjjkj 30999[
Reading the files with the ‘more’ command
[root@localhost ~]# more c.txt
kajslkj
lkasjdkfjjs
klakjshkdjfhkahs
[root@localhost ~]#
Browsing Files with the ‘less’ command.
less c.txt
Reading the Beginning or End of Files with the head and tail Commands.
head -5 /usr/man/man.txt
head -5 –q /usr/man/man.txt
tail -12 /var/log/message/a.txt
The more command is one of a family of Linux commands called pagers.
Creating Files with the ‘touch’ Command
The touch command is easy to use, and generally, there are two reasons to use it .The first reason is to create a file, and the second is to update a file’s modification date.
a) To create a file with touch,use
# touch newfile
#ls –l newfile
-rw-r–r– 1 jmarun jmarun 0 Feb 21 14:12 newfile
b) To Change time stamp
# touch –t 1225110099 newfile2
Trap: when the program is terminated before it would normal end, we can catch an exit signal.
0- Normal termination, end of script.
1-SIGHUP -> hang up, line disconnected
2-singint-> terminal interrupts, usually ctrl+c
3- SIGQUIT -> Quit Key, Child process to die before terminating.
9-SIGKILL->kill -9 commands can’t trap this type of exit status.
15-SIGTERM àkill command’s default action.
19-SIGSTOP->stop, usually ctrl+z
17 -> dintfunc
Ex: Kill –9 <ps Id>
df -> Report how much free disk space is available for each mount you have.
df –a –all -> include dummy file system.
df –B 100, –block-size=SIZE -> use SIZE –byte blocks
df –h ->human readable àprint sizes in human readable format.
df –I -> list inode information instead of block usage.
Df –k -> like –block –Size =1 k.
df –T ->Print file System Type.
du -> disk usage
du ->tells you how much disk space a file occupies.
du –a ->display the space that each file is taking up.
Du –h -> which can make the output easier to read by displaying it in KB /M/G.
Du –sh -> The -s (for suppress or summarize) option tells du to report only the total disk space occupied by a directory tree and to suppress individual reports for its subdirectories
top àdisplays top CPU process.
The top program provides a dynamic real-time view of a running system.
Uname
Free -> displays information about free and used memory on the system.
-b,-k,-m,-g show output in bytes, KB, MB, or GB
-l show detailed low and high memory statistics
-o use old format (no -/+buffers/cache line)
-t display total for RAM + swap
-s update every [delay] seconds
-c update [count] times
-V display version information and exit
awk-> The awk command is powerful method for processing or analyzing text files,in particulat data files that are organized by lines (rows) and cloumns.
How to add user:
Useradd div –to add user Id
Passwd div –to set password
Set Account disable date:
useradd -e {yyyy-mm-dd} {username}
useradd -e 2008-12-31 jerry
Set default password expiry:
useradd -f {days} {username}
useradd -e 2009-12-31 -f 30 jerry
How Can I see the entire users list on Linux Server?
cat /etc/passwd
Vim /etc/passwd
cat /etc/passwd | grep home | cut -d’:’ -f1
How to convert you as root, create new user and setting password.
[jmarun@localhost Desktop]$ su root
Password: <give your user passwd>
[root@localhost Desktop]# useradd diva1
[root@localhost Desktop]# passwd diva1
Changing password for user diva1.
New password:
BAD PASSWORD: it is too short
BAD PASSWORD: is too simple
Retype new password:
passwd: all authentication tokens updated successfully.
[root@localhost Desktop]#
How to add user in sudo list.
Go to vi /etc/sudoers
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
jmarun ALL=(ALL) ALL
Man ->
Find files with the whereis command.
Whereis find
You can also use whereis to find only the binary version of the program with
Whereis –b find
If whereis cann’t find your request , you’ll get an empty return string, for example
Whreis foo
It will search in entire system.
Not limiting searches to known directories such as /usr/man, /usr/bin, or /usr/sbin can speed up the task of finding files.
Although whereis is faster than using find to locate programs or manual pages.
Locate is faster than whereis
Locating files with locate command.
Finding a file using locate is much faster than the find command because locate will go directly to the database file,find any matching filenames, and print its results.
Locate *.ps
Locate resides in /var/lib
Moving different directories with cd command.
cd or
cd ../.. or
cd or cd –
Knowing where you are wit pwd command.
Go to /user/local and type pwd to know your directory.
ps -ef | grep aneel
pkill -f
Link while copying the data?
cp –i test 1 test2
Getting command Summaries with whatis and apropos
Specifying other directories with ls
# ls /usr/bin
Listing Directories with the dir and vdir commands
#dir :-> this command works like the default ls command,listing the files in sorted columns
Vdir -> The vdir command works like the ls –l option, and presents a long format listing by default,
Graphic Directory listings with the tree command
# tree /var/lib
# tree –d /usr/local/netscape/
Search Inside Files with the ‘grep’ Command
grep command will help to search any words in file
Ex: cat hive.log | grep loaded [Here we are searching loaded files in log]
Compressing files with the ‘gzip’ command
# gzip file.tar
Compressing Files with the ‘compress’ Command
#compress file
To uncompress a file,use
# uncompress file.Z
Running Programs in the Background
# nohup ./jmarun.sh &
Checking the Connection
Using the ‘ipconfig’ Command, This Command will help to identify the IP address of your system
# ifconfig
[jmarun@localhost ~]$ ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:7F:24:89
inet addr:192.168.64.130 Bcast:192.168.64.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe7f:2489/64 Scope:Link
Using the ‘netstat’ Command
The netstat command is the definitive command for checking your network activity,connections,routing tables,and other network messages and statistics.
# netstat
Using the ping Command
# ping <hostname>.com
Find Hostname of the System
#hostname
How to replace One word with another word in vi:
:%s/old-text/new-text/g
Datapedia @JM_Arun 