Okay, here's a clear and complete summary of the "String Manipulation in Shell Scripting, Sed, Awk and Networking Commands" lecture notes, designed for easy revision and ensuring all commands/concepts are included. SEng3208- Lab-Part-III Batch-2024 String Manipulation in Shell Scripting, Sed, Awk and Networking Commands Basics of Pure Bash String Manipulation Assigning Content to a Variable and Printing: $ followed by variable name prints its content (parameter expansion). Shell doesn't strictly type variables (can store strings, integers, reals). Syntax: VariableName='value' VariableName="value" VariableName=value (no quotes needed if no spaces/special chars) Printing: echo $VariableName echo ${VariableName} (good practice, avoids ambiguity) echo "$VariableName" (preserves spaces within the value) Example (strfile.sh): str1='welcome' str2="to" str3=StringForStrings echo $str1 # Output: welcome echo ${str2} # Output: to echo "$str3" # Output: StringForStrings Common String Operations Overview: Concatenation Substring extraction Length determination Comparison Trimming whitespace Printing String Length: # symbol used with parameter expansion: ${#variablename} Example (strfile1.sh): Str=StringForStrings echo ${#Str} # Output: 16 (Corrected from '13' in notes) IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Interactive Example (Find length): echo "Enter a string:" read string length=${#string} echo "The length of the string is: $length" # Output for "hello": The length of the string is: 5 IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Practice Box Example: name="Shell Scripting Expert" echo "String length: ${#name}" # Output: String length: 22 echo "First 5 characters: ${name:0:5}" # Output: First 5 characters: Shell greeting="Welcome, $name" echo "$greeting" # Output: Welcome, Shell Scripting Expert IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Example-2 Substring Extraction: Syntax: ${string:offset:length} (offset is 0-based) Interactive Example: echo "Enter a string:" read string substring=${string:2:3} # Extract 3 chars starting from index 2 echo "The substring is: $substring" # Output for "hello": The substring is: llo IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Example-3 Reverse a String (Without rev command): echo "Enter a string:" read string reversed="" len=${#string} for (( i=$len-1; i>=0; i-- )); do reversed="$reversed${string:$i:1}" # Append char by char from end done echo "The reversed string is: $reversed" IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Example-4 String Concatenation: Syntax: Var=${var1}${var2}${var3} var=$var1$var2$var3 Var="$var1""$var2""$var3" (or Var="$var1$var2$var3") With Separator (e.g., **): Var=${var1}**${var2}**${var3} var="$var1"**"$var2"**"$var3" With Space: var="${var1} ${var2} ${var3}" (using quotes is best) echo "${var1} ${var2} ${var3}" Note: Avoid var=$var1 $var2 $var3 without quotes, as $var2 and $var3 might be treated as commands. Interactive Concatenation Example: echo "Enter first string:" read str1 echo "Enter second string:" read str2 result="$str1$str2" echo "The concatenated string is: $result" # Output for "hello" and "world": helloworld IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Concatenation using Arrays: Create array: arr=("value1" value2 $value3) Print all elements: echo ${arr[@]} Print array length (number of elements): echo ${#arr[@]} Print specific element: echo ${arr[index]} (e.g., echo ${arr[0]}) Note: echo ${arr} is same as echo ${arr[0]}. Example-5 Check if a String Contains a Substring: echo "Enter the main string:" read main_string echo "Enter the substring to search for:" read substring if [[ "$main_string" == *"$substring"* ]]; then # Wildcard matching echo "The substring '$substring' is found in '$main_string'." else echo "The substring '$substring' is not found in '$main_string'." fi # Output for "hello world", "world": The substring 'world' is found in 'hello world'. IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Direct Substring Example (from notes): NAME=Baeldung echo ${NAME:6} # Output: ng (chars from index 6 to end) echo ${NAME:0:4} # Output: Bael (4 chars from index 0) IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Example-6 Convert String to Uppercase: echo "Enter a string:" read string uppercase=$(echo "$string" | tr '[:lower:]' '[:upper:]') echo "The string in uppercase is: $uppercase" # Output for "linux": LINUX IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Bash specific (v4+): uppercase=${string^^} Example-7 Replace a Substring in a String: Syntax: ${string//old_substring/new_substring} (replaces all occurrences) Interactive Example: echo "Enter the main string:" read string echo "Enter the substring to replace:" read old echo "Enter the new substring:" read new result=${string//$old/$new} echo "The new string is: $result" # Output for "hello world", "world", "linux": hello linux IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Pattern Matching (Bash built-in): *: matches any number of characters +: matches one or more characters (requires extglob option: shopt -s extglob) [abc]: matches only given characters Example: if [[ "file.jpg" == *.jpg ]]; then echo "is jpg"; fi (Output: is jpg) String Operators Examples (Comparisons): Equal (= or == inside [[...]]): str1="StringForStrings"; str2="strings"; if [ "$str1" = "$str2" ]; then # Always quote variables in [ ] echo "Both string are same"; else echo "Both string are not same"; # Output fi IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Not Equal (!=): str1="StringForStrings"; str2="strings"; if [ "$str1" != "$str2" ]; then echo "Both string are not same"; # Output else echo "Both string are same"; fi IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Less than (\< escaped for [ ], or < in [[...]]): str1="StringsForStrings"; str2="Strings"; if [[ "$str1" < "$str2" ]]; then # Lexicographical comparison echo "$str1 is less than $str2"; else echo "$str1 is not less than $str2"; # Output fi IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Greater than (\> escaped for [ ], or > in [[...]]): str1="StringsForStrings"; str2="Strings"; # Original example "Geeks" changed to "Strings" for consistency if [[ "$str1" > "$str2" ]]; then echo "$str1 is greater than $str2"; # Output else echo "$str1 is less than $str2"; fi IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Check string length greater than 0 (-n): (String is not empty) str="GeeksforGeeks"; if [ -n "$str" ]; then # Quote $str echo "String is not empty"; # Output else echo "String is empty"; fi IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Check string length equal to 0 (-z): (String is empty) str=""; if [ -z "$str" ]; then # Quote $str echo "String is empty"; # Output else echo "String is not empty"; fi IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Advanced String Operations / Complex String Manipulation Techniques: String Transformation Methods (Bash v4+): Uppercase: ${text^^} Lowercase: ${text,,} Example: text="hello world" uppercase=${text^^} echo "Uppercase: $uppercase" # Output: HELLO WORLD text="HELLO WORLD" lowercase=${text,,} echo "Lowercase: $lowercase" # Output: hello world IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END AWK – Lab Tasks-solutions Print lines and line number from fileinput: STEP1: Create fileinput: $ cat > fileinput welcome to vits hello ^D (Ctrl+D to save and exit) IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END STEP2: Create cmds.awk script: $ cat > cmds.awk {print NR, $0 } # NR: record number (line number), $0: whole line ^D IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END STEP3: Execute: $ awk -f cmds.awk fileinput # Output: # 1 welcome to # 2 vits # 3 hello IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Print first and second field if third field >= 50 (Input separator ':', Output separator ','): STEP1: Create file1: $ cat > file1 sachin:10:100 rahul:11:95 rohit:12:89 ^D IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END STEP2: Execute: $ awk -F':' '$3>=50 {print $1","$2}' file1 # Output: # sachin,10 # rahul,11 # rohit,12 IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END (Note: The prompt in notes was $awk -F':' '$3>=50 {print $1”,”$2}' file1. Corrected ”,” to "," for awk string literal.) Process marks.txt (CSV: studentid,name,Tel,Eng,Math,Sci,Soc) and generate results: Result: studentid, studentname, Total Marks, Pass/Fail Pass criteria: Telugu & English >= 30, Other subjects >= 40. STEP1: Create marks.txt: 1001,name1,99,69,85,56,75 1002,name2,89,69,65,56,55 1003,name3,50,50,50,55,55 1004,name4,69,29,85,56,75 1005,name5,99,69,85,56,11 IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END STEP2: Create marks.awk script: { total=$3+$4+$5+$6+$7 if($3>=30 && $4>=30 && $5>=40 && $6>=40 && $7>=40) print $1,$2,total, "Pass"; # Output fields separated by OFS (default space) else print $1,$2,total, "fail"; } IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Awk IGNORE_WHEN_COPYING_END STEP3: Execute: $ awk -F"," -f marks.awk marks.txt # Example Output for 1001: 1001 name1 384 Pass # Example Output for 1004: 1004 name4 314 fail (due to English < 30) IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END (Note: Awk script can be made more robust by setting OFS="," if comma-separated output is strictly required.) Print fields 1 and 4 from a CSV file (passed as argument) and average of 4th field data at the end. STEP1: Create data file: 12,13,14,15,one 22,23,24,25,two 34,23,45,23,three 44,55,66,77,four IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END STEP2: Execute (Inline awk script): $ awk -F',' '{ sum_f4 += $4; count_f4++; print $1, $4 } END { if (count_f4 > 0) print "Average of 4th field:", sum_f4/count_f4 }' data # The notes show: $awk -F',' '{print $1,$2,$3,$4,($1+$2+$3+$4)/4}' data # This is different; it prints fields 1-4 and their average per line. # The question asks for fields 1 & 4, then overall average of field 4. # My corrected command addresses the question. # If the intention was field-wise average, the notes' command is okay. # Output for my corrected command: # 12 15 # 22 25 # 34 23 # 44 77 # Average of 4th field: 35 IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Demonstrate user-defined functions and system command in awk. STEP1: Create data file (can reuse from task 4). STEP2: Create user.awk script: # user.awk { if($3 > 0) # Assuming field 3 is what needs to be displayed display($3) # Call user-defined function # Example of system command (not in original notes for this example but implied by question) # system("date") # This would execute the 'date' command } function display(item_to_print) { print item_to_print } IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Awk IGNORE_WHEN_COPYING_END STEP3: Execute: $ awk -F',' -f user.awk data # Output (will print the 3rd field of each line from 'data'): # 14 # 24 # 45 # 66 IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Count lines in a file that do not contain vowels. STEP1: Create input file: this is one 213 BCDEFG This is last line IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END STEP2: Create vowels.awk script: BEGIN{count=0} !/[aeiouAEIOU]/ {count++; print} # Print lines without vowels, increment count END{print "Number of lines="count} IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Awk IGNORE_WHEN_COPYING_END (Note: Added AEIOU for case-insensitivity, as typical. Notes only had aeiou.) STEP3: Execute: $ awk -f vowels.awk input # Output: # 213 # BCDEFG # Number of lines=2 IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END Find number of characters, words, and lines in a file (file7). STEP1: Create file7: This is a file YEs NO 1234 IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END STEP2: Create lines.awk script: BEGIN{words=0; characters=0} { characters+=length($0); # Add length of current line (including spaces) words+=NF; # NF: Number of Fields (words) in current line } END{print "lines=",NR," words=",words," Characters=",characters} IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Awk IGNORE_WHEN_COPYING_END STEP3: Execute: $ awk -f lines.awk file7 # Output: # lines= 3 words= 7 Characters= 29 (approx, includes newlines if length counts them, or not if $0 strips them) # More accurately for characters excluding newlines, but length($0) typically includes them if read that way. # `wc file7` gives: 3 lines, 7 words, 30 characters (including final newline char). Awk length($0) might not count trailing newline. IGNORE_WHEN_COPYING_START content_copy download Use code with caution. Bash IGNORE_WHEN_COPYING_END sed: Tasks Create Sample File (filename.txt): Open the door Oxygen is important BEGIN this is the start END We need to BEGIN but no END 1234 Hello 5678 Another BEGIN test END Line without special content IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END Questions & Solutions: a) Print line numbers of lines beginning with "O": Command: sed -n '/^O/=' filename.txt Explanation: -n: Suppress automatic printing. ^O: Regex for lines starting with 'O'. =: Print line number of matched line. Output: 1 2 IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END b) Delete digits in the given input file: Command: sed 's/[0-9]*//g' filename.txt Explanation: s/old/new/g: Substitute command (g for global on a line). [0-9]*: Regex for zero or more digits. //: Replace with nothing (delete). Output: Open the door Oxygen is important BEGIN this is the start END We need to BEGIN but no END Hello Another BEGIN test END Line without special content IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END c) Delete lines that contain both BEGIN and END: Command: sed '/BEGIN/ { /END/d }' filename.txt Explanation: /BEGIN/: Address lines containing "BEGIN". { /END/d }: If that line also contains "END", then delete (d) it. Output: Open the door Oxygen is important We need to BEGIN but no END 1234 Hello 5678 Another BEGIN test END <-- Note: The example output in notes omits this, but it should be deleted. My output corrects this based on logic. If "Another BEGIN test END" is to be kept, the logic in the problem is slightly different from the sed cmd. The provided sed command will delete any line with BEGIN that also has END. Line without special content IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END Correction based on typical Sed behavior: The Another BEGIN test END line will also be deleted. Revisiting the provided output: The notes' output for (c) keeps "Another BEGIN test END". This means the sed might be intended to only delete lines where BEGIN appears before END literally in that order for the first BEGIN. However, the sed command provided is more general. For the purpose of this summary, I'll stick to what the sed command does. If the output is the definitive truth, the sed command would need adjustment. Assuming the provided sed command's logic is primary: Output (as per sed command logic): Open the door Oxygen is important We need to BEGIN but no END 1234 Hello 5678 Line without special content IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END d) Delete lines that contain BEGIN but not END: Command: sed '/BEGIN/ { /END/!d }' filename.txt Explanation: /BEGIN/: Address lines containing "BEGIN". { /END/!d }: If that line also does not contain "END" (!), then delete (d) it. Output: Open the door Oxygen is important BEGIN this is the start END 1234 Hello 5678 Another BEGIN test END Line without special content IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END e) Delete the first character in each line: Command: sed 's/^.//' filename.txt Explanation: ^.: Regex for any single character at the start of the line. //: Replace with nothing. Example Output (showing first few lines): pen the door xygen is important EGIN this is the start END IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END f) Delete the last character in each line: Command: sed 's/.$//' filename.txt Explanation: .$: Regex for any single character at the end of the line. //: Replace with nothing. Example Output (showing first few lines): Open the doo Oxygen is importan BEGIN this is the start EN IGNORE_WHEN_COPYING_START content_copy download Use code with caution. IGNORE_WHEN_COPYING_END LINUX NETWORKING COMMANDS (Ubuntu) ifconfig (Interface Configuration) - Deprecated, use ip Displays network interfaces and IP addresses. Syntax: ifconfig Example: ifconfig (lists all active interfaces, IPs, netmasks, broadcast). ip (IP Address Management) - Modern alternative Syntax: ip addr (or ip a) Example: ip addr show (detailed info for all network interfaces). ping (Packet Internet Groper) Sends ICMP echo requests to test connectivity. Syntax: ping destination Example: ping google.com (Ctrl+C to stop). Limit packets: ping -c 4 google.com traceroute (Trace Route) Traces packet path to a destination. Syntax: traceroute destination Example: traceroute google.com Install if needed: sudo apt install traceroute netstat (Network Statistics) - Older, ss is preferred Displays connections, routing tables, interface stats. Syntax: netstat [options] Example: netstat -an (all active connections and listening ports). netstat -tuln (TCP/UDP listening ports, numeric). ss (Socket Statistics) - Modern, faster alternative to netstat Syntax: ss [options] Example: ss -tuln (all TCP/UDP listening ports and active connections, numeric). dig (DNS Lookup / Domain Information Groper) Performs DNS lookups, shows detailed info. More detailed than nslookup. Syntax: dig domain_name Example: dig google.com nslookup (DNS Query / Name Server Lookup) Queries DNS server for domain/IP mapping. Syntax: nslookup domain_name_or_IP Example: nslookup google.com curl (Client URL) Transfers data from or to a server (HTTP, FTP, etc.). Retrieves web pages. Syntax: curl [options] URL Example: curl http://example.com wget (Web Get) Downloads files from the web. Syntax: wget [options] URL Example: wget http://example.com/file.zip hostname Displays or sets the system's hostname. Syntax: hostname (to display) Example: hostname Set hostname: sudo hostname new_hostname (temporary) Set hostname permanently (systemd): sudo hostnamectl set-hostname newhostname route - Deprecated, use ip route Shows/modifies IP routing table. Syntax: route [options] Example: route -n (displays current routing table with numerical IPs). Modern equivalent: ip route show or ip r nmap (Network Mapper) Scans networks/hosts for open ports and services. Syntax: nmap [options] target Example: nmap 192.168.1.1 Install if needed: sudo apt install nmap nmcli (NetworkManager Command Line Interface) Manages NetworkManager connections, interfaces, devices. Syntax: nmcli [object] [command] Example: nmcli device status ethtool (Ethernet Tool) Displays/modifies network interface driver and hardware settings. Syntax: ethtool interface_name Example: sudo ethtool eth0 iwconfig (Wireless Interface Configuration) - Older, iw is newer Displays/configures wireless network interfaces. Syntax: iwconfig [interface] Example: iwconfig wlan0 systemctl (Manage System Services) Controls systemd services, including network services. Syntax: sudo systemctl [action] service_name Example: sudo systemctl restart NetworkManager or sudo systemctl restart networking ufw (Uncomplicated Firewall) Manages firewall rules. Syntax: sudo ufw [command] Examples: sudo ufw enable (Enable firewall) sudo ufw allow 80 (Allow port 80/HTTP) sudo ufw status (Check firewall status) host (DNS Lookup) (Alternative to nslookup/dig for simple lookups) Syntax: host [hostname or IP] Example: host google.com ifup / ifdown (Bring Interface Up or Down) - Older, NetworkManager/ip preferred Enables/disables network interfaces (if configured in /etc/network/interfaces). Syntax: sudo ifup [interface], sudo ifdown [interface] Example: sudo ifdown eth0, sudo ifup eth0 tcpdump (Network Packet Analyzer) Captures and analyzes network packets. Syntax: sudo tcpdump [options] Example: sudo tcpdump -i eth0 (Captures traffic on eth0). Install if needed: sudo apt install tcpdump This summary covers all the commands and concepts mentioned in the Lab-Part-III notes.