mrjob.parse - log parsing

Utilities for parsing errors, counters, and status messages.


Return True if uri can be parsed into an S3 URI, False otherwise.


Return True if uri is a URI and contains :// (we only care about URIs that can describe files)

mrjob.parse.parse_mr_job_stderr(stderr, counters=None)

Parse counters and status messages out of MRJob output.

  • stderr – a filehandle, a list of lines (bytes), or bytes
  • counters – Counters so far, to update; a map from group (string to counter name (string) to count.

Returns a dictionary with the keys counters, statuses, other:

  • counters: counters so far; same format as above
  • statuses: a list of status messages encountered
  • other: lines (strings) that aren’t either counters or status messages

Parse an S3 URI into (bucket, key)

>>> parse_s3_uri('s3://walrus/tmp/')
('walrus', 'tmp/')

If uri is not an S3 URI, raise a ValueError


If path_or_uri is not a URI already, convert it to a file:/// URI.