Guides

• Why mrjob?
  • Overview
  • Why use mrjob instead of X?
  • Why use X instead of mrjob?
• Fundamentals
  • Installation
  • Writing your first job
  • Running your job different ways
  • Writing your second job
  • Configuration
• Concepts
  • MapReduce and Apache Hadoop
  • Hadoop Streaming and mrjob
• Writing jobs
  • Defining steps
  • Protocols
  • Defining command line options
  • Counters
  • Input and output formats
• Runners
  • Testing locally
  • Running on your own Hadoop cluster
  • Running on EMR
  • Running on Dataproc
  • Configuration
  • Running your job programmatically
• Spark
  • Why use mrjob with Spark?
  • Writing your first Spark job
  • Running on your own Hadoop cluster
  • Running on EMR
  • Passing in libraries
  • Command-line options
  • No setup scripts
  • Multi-step jobs
  • External Spark scripts
  • Custom input and output formats
• Config file format and location
  • Precedence and combining options
  • Option data types
  • Using multiple config files
  • Clearing configs
• Options available to all runners
  • Making files available to tasks
  • Temp files and cleanup
  • Job execution context
  • Other
  • Options ignored by the local and inline runners
  • Options ignored by the inline runner
• Hadoop-related options
  • Options specific to the local and inline runners
  • Options available to local, hadoop, and emr runners
  • Options available to hadoop and emr runners
  • Options available to hadoop runner only
• Configuration quick reference
  • Setting configuration options
  • Options that can’t be set from mrjob.conf (all runners)
  • Other options for all runners
  • Additional options for DataprocJobRunner
  • Additional options for EMRJobRunner
  • Additional options for HadoopJobRunner
• Job Environment Setup Cookbook
  • Uploading your source tree
  • Uploading your source tree as an archive
  • Running a makefile inside your source dir
  • Making data files available to your job
  • Using a virtualenv
  • Other ways to use pip to install Python packages
• Hadoop Cookbook
  • Increasing the task timeout
  • Writing compressed output
• Testing jobs
  • Inline runner
  • Local runnner
  • Anatomy of a test case
• Cloud Dataproc
  • Dataproc Quickstart
  • Dataproc runner options
• Elastic MapReduce
  • Elastic MapReduce Quickstart
  • Cluster Pooling
  • EMR runner options
  • EMR Bootstrapping Cookbook
  • Troubleshooting
  • Advanced EMR usage
  • The mrjob command
• Python 2 vs. Python 3
  • Raw protocols
  • Bytes vs. strings
  • python_bin
  • Your Hadoop cluster
• Contributing to mrjob
  • Contribution guidelines
  • A quick tour through the code
• Interactions between runner and job
  • Job Interface
  • Format of --steps