Gain insights into data processing using Bash, sed, awk, and RegEx. Explore sorting, searching, replacing, and optimizing data tasks interactively with animated videos, projects, quizzes, and tasks.

tarballBash.tar.gz

code-job

spa-job

code-job-bash

2500+ students have taken this innovative project-based data learning course (includes video lectures). It demonstrates the use of Bash shell (Bash, sed and awk including RegEx) in processing textual data. It can help to learn to sort, search, match, replace, clean and optimize various aspects of data with Bash Shell. The target audience (students, researchers, scientists, journalists, data miners, developers) didn't have to go through any tough learning curve. This course also should have helped RedHat, SuSE and Ubuntu Linux learners and Data Science enthusiasts.  Regularly updated, new projects to come!

- Learn Bash commands interactively
- Projects with own stories and conclusive decisions
- Animated video lectures (for visual learners)
- Demonstrations
- Quizzes
- Learning tasks

Bloom's taxonomy (remember, understand, apply, analyze, evaluate and create) in developing your Linux skills. Learn Scientific Programming! scientificprogramming.io

Learn Data Science with Bash Shell

**Given a text, what are the most frequent words?**


Finding the most frequent words for a given text (e.g., Knight_of_the_Burning_Pestle) is easy, we can build a function `toptokens()`, which is nothing but the `topcrimes()` function developed in our previous project. Let's watch the following video lecture first:



For example, if we want to grab the most frequent words in the Romeo and Juliet play, we can execute the following:

toptokens() { cat $1 | \
csvcut -c "tokens",$2 | \
sort -nr -t "," -k 2 | \
head -n 20 | \
awk -F',' '{print $1 "," $2}' ; }

toptokens plays_and_poems_stat.csv "Romeo_and_Juliet___play___Shakespeare" |  csvlook

**Given an author, what are the most frequent words?**

This is slightly complicated! becuase we again need to perform several steps:

+ For the given author, trim out the plays/ poems names, indclding text types (i.e., `plays | poems`)
+ Combine all the columns, i.e., sum horizontally the frequencies of words for all the texts of that author
+ Sort the words, based on the accumulated frequencies on all works by that author.

Don't be scared! we will take you there.

**Step 1.** Trim out the plays/ poems names, for a given author:

Let's consider that the author in question is Shakespeare. The following `awk` based regular expression will trim out all the bit before the name of the author. If you look closely, you will see that inside the `sed` regex, it's actually finding the pattern of plays `OR` (`|`) poems names that end with the string "Shakespeare" and then replacing inplace (due to the `-i -r`) the whole matched pattern e.g., `Romeo_and_Juliet___play___Shakespeare` with the string `Shakespeare`:

Course Introduction

Project 1: Analyzing the 'US News' University Ranking Data

Project 2: Facebook Data Mining

Project 3: Australian Cities Crime Statistics

Project 4: Shakespearean-era plays and poems data mining

Bash Tutorials

REGEX Tutorials

AWK Tutorials

SED, GREP and Find Tutorials

Beyond the Text Files! Enter into the Big Data Landscape - Concepts

Conclusion

Finding the most frequent words by Shakespeare