Bash Commands

Extract parent domain from list of domains

awk '{l=split($1,a,"."); print (a[l-1]=="com" || a[l-1]=="co"?a[l-2] OFS:X) a[l-1] OFS a[l]}' OFS="." file |sort -u

File:
sport.yahoo.com
finance.yahoo.com
yahoo.com
bbc.co.uk

Awk Aggregation

### file.tsv content
abc    2
def    5
acb    6
abc    8
def    1


cat file.tsv | awk -F'\t' '{if(agg[$1]=="") agg[$1]=$2; else agg[$1]+=$2;}END{for(x in agg){print x"\t"agg[x]}}'

## For string aggregation

String concat is done in this way agg[$1]=agg[$1]","$2

Remove last character

Case 1: Remove irrespective of what is the last character

echo "lkj" | rev | cut -c 2- | rev

output : lk

Case 2: Remove if last character is /

echo "https://google.com/" | awk '{lCPgUrl=substr($1,length($1),1);if(lCPgUrl=="/") a=substr($1,1,length($1)-1); else a=$1; print a}'

Output: https://google.com

Variable Contains String

if [[ $VARIABLE = *"string1"* ]]; then
  ### do something
fi

Sort

### file.tsv content
abc    2
def    5
acb    6
abc    8
def    1

## Sort by alphabet
cat file.tsv | sort -k1 -t$'\t' 

## Sort by number
cat file.tsv | sort -k2 -t$'\t' -nr

-n is for number
-r is for reverse (decreasing order)

Join

### file1.tsv
apple    2qty
orange    10qty
banana    5qty
kiwi    3qty

### file2.tsv
apple    $10
orange    $8
banana    $5
pineappple    $4

join -1 1 -2 1 -t$'\t' -o1.1,1.2,2.2 <(cat file1.tsv|sort -k1 -t$'\t') <(cat file2.tsv|sort -k1 -t$'\t')

Output:
apple    2qty    $10
banana    5qty    $5
orange    10qty    $8

### Inorder to include all records from file1

join -1 1 -2 1 -t$'\t' -e"-" -a1 -o1.1,1.2,2.2 <(cat file1.tsv|sort -k1 -t$'\t') <(cat file2.tsv|sort -k1 -t$'\t')

Output:
apple    2qty    $10
banana    5qty    $5
kiwi    3qty    -
orange    10qty    $8

### Inorder to include all records from file2

join -1 1 -2 1 -t$'\t' -e"-" -a2 -o2.1,1.2,2.2 <(cat file1.tsv|sort -k1 -t$'\t') <(cat file2.tsv|sort -k1 -t$'\t')

Output:
apple    2qty    $10
banana    5qty    $5
orange    10qty    $8
pineapple    -    $4