PacktLib: Fast Data Processing with Spark

Fast Data Processing with Spark

Credits

About the Author

About the Reviewers

www.PacktPub.com

Preface

Installing Spark and Setting Up Your Cluster

Running Spark on a single machine

Running Spark on EC2

Deploying Spark on Elastic MapReduce

Deploying Spark with Chef (opscode)

Deploying Spark on Mesos

Deploying Spark on YARN

Deploying set of machines over SSH

Links and references

Summary

Using the Spark Shell

Loading a simple text file

Using the Spark shell to run logistic regression

Interactively loading data from S3

Summary

Building and Running a Spark Application

Building your Spark project with sbt

Building your Spark job with Maven

Building your Spark job with something else

Summary

Creating a SparkContext

Scala

Java

Shared Java and Scala APIs

Python

Links and references

Summary

Loading and Saving Data in Spark

RDDs

Loading data into an RDD

Saving your data

Links and references

Summary

Manipulating Your RDD

Manipulating your RDD in Scala and Java

Manipulating your RDD in Python

Links and references

Summary

Shark – Using Spark with Hive

Why Hive/Shark?

Installing Shark

Running Shark

Loading data

Using Hive queries in a Spark program

Links and references

Summary

Testing

Testing in Java and Scala

Testing in Python

Links and references

Summary

Tips and Tricks

Where to find logs?

Concurrency limitations

Memory usage and garbage collection

Serialization

IDE integration

Using Spark with other languages

A quick note on security

Mailing lists

Links and references

Summary

Index