Gain insights into enhancing Python data processing with PySpark. Delve into reading, transforming, aggregating data, and creating user-defined functions, boosting efficiency with Apache Spark.

frompandastopyspark.tar.gz

PandasAndPyspark

jupyter

spa-jupyter

spa-snapshot

spa-copy

spa-copy-copy

spa-copy-copy-5klyxp

spa-snapshot-copy

spa-snapshot-copy-copy

spa-copy-copy-5klyxp-copy

Pandas is a popular Python library used to manipulate data, but it has certain limitations in its ability to process large datasets. The Apache Spark analytics library offers significant performance improvements.

This course will help improve your Python-based data processing by leveraging Apache Spark’s multithreading capabilities through the PySpark library. You’ll start by reading data into a PySpark DataFrame before performing basic input/output functions, such as renaming attributes, selecting, and writing data. You’ll move onto transformation functions like aggregation, statistical analysis, and joins before creating custom, user-defined functions. At each step, you’ll get a quick Pandas review before being walked through leveraging the more robust PySpark library to unlock Apache Spark.

By the end of this course, you’ll be able to quickly and reliably process large amounts of data, even stored across multiple files, using PySpark.

From Pandas to PySpark DataFrame



This dataset is an updated version of the “Amazon review dataset" released in 2014. As in the previous version, this dataset includes reviews (ratings, text, helpfulness votes), product metadata (descriptions, category information, price, brand, and image features), and links (also viewed/also bought graphs). In addition, this version provides the following features:



* More reviews:
  * The total number of reviews is 233.1 million (142.8 million in 2014).
* Newer reviews:
  * Current data includes reviews in the range May 1996 - Oct 2018.
* Metadata:
  * They have added transaction metadata for each review shown on the review page. Such information includes the following:
    * Product information, for example color (white or black), size (large or small), package type (hardcover or electronics), and so on.
    * Product images that are taken after the user received the product.
  * Added more detailed metadata of the product landing page. Such detailed information includes:
    * Bullet-point descriptions under the product title.
    * Technical details table (attribute-value pairs).
    * Similar products table.

* More categories:
    * Includes 5 new product categories.

# Dataset sizes
- **Raw review data** (34gb) - This includes all the 233.1 million reviews.

- **Ratings only** (6.7gb) - This is the same as above, in CSV form without reviews or metadata.


- **5-core** (14.3gb) - This is the subset of the data in which all users and items have at least 5 reviews (75.26 million reviews).

- **Per-category data** - This is the review and product metadata for each category.


- **Small subsets for experimentation**: We can also use one of the smaller datasets. In the course, we used the “Toys and Games” dataset, which contains 1,828,971 reviews.


# Dataset released by

[Jianmo Ni, Jiacheng Li, Julian McAuley](https://nijianmo.github.io/amazon/index.html)

[_Empirical Methods in Natural Language Processing (EMNLP)_, 2019)](https://nijianmo.github.io/amazon/index.html)



Learn about the dataset we used in this course in detail and its source.


Amazon Review Data (2018)

Learn about the dataset we used in this course in detail and its source.

Introduction

Data Input/Output

Data Transformation

User Defined Function (UDF)

Wrapping Up

Appendix

Amazon Review Data (2018)

Description