IOStreams

IOStreams is an incredibly powerful Ruby streaming library that makes changes to file formats, compression, encryption, or storage mechanism transparent to the application.

Features

Streaming avoids high memory utilization since the file (or other source such as AWS S3) is read or written a block at a time.

Develop applications using the local file system, then use for example AWS S3 for all file storage in production, without changing any of the source code.

File Extensions

File Storage

File formats

Examples

Write to a local file:

IOStreams.path("example.txt").write("Hello World")

Write to AWS S3:

IOStreams.path("s3://bucket-name/path/example.txt").write("Hello World")

Write the same data into a compressed file by adding the .gz extension to the file name:

IOStreams.path("example.txt.gz").write("Hello World")

Compress and encrypt the data into a PGP encrypted file:

path = IOStreams.path("example.txt.pgp")
path.option(:pgp, recipient: "receiver@example.org")
path.write("Hello World")

Write a file to a SFTP server:

path = IOStreams.path("sftp://example.org/path/example.txt", 
                      username: "example", 
                      password: "topsecret")
path.write("Hello World")

Write PGP encrypted file to AWS S3:

path = IOStreams.path("s3://bucket-name/path/example.txt.pgp")
path.option(:pgp, recipient: "receiver@example.org")
path.write("Hello World")

Read an entire file into memory:

IOStreams.path("example.txt").read
# => "Hello World"

Read an entire file into memory from S3:

IOStreams.path("s3://bucket-name/path/example.txt").read
# => "Hello World"

Decompress an entire gzip file into memory:

IOStreams.path("example.txt.gz").read
# => "Hello World"

Decrypt and decompress the entire PGP file into memory:

IOStreams.path("example.txt.pgp").read
# => "Hello World"

Streaming Examples

Read 128 characters at a time from the file:

IOStreams.path("example.csv").reader do |io|
  while (data = io.read(128))
    p data 
  end
end

Read one line at a time from the file:

IOStreams.path("example.csv").each do |line|
  puts line
end

Read the first line from a file:

IOStreams.path("example.csv").each.first

Read the first 5 lines from a file:

IOStreams.path("example.csv").each.first(5)

Display each row from the csv file as an array:

IOStreams.path("example.csv").each(:array) do |array|
  p array
end

Display each row from the csv file as a hash, where the first line in the CSV file is the header:

IOStreams.path("example.csv").each(:hash) do |hash|
  p hash
end

Read the first 5 records from a csv file:

IOStreams.path("example.csv").each(:hash).first(5)

Write data to the file.

IOStreams.path("abc.txt").writer do |io|
  io << "This"
  io << " is "
  io << " one line\n"
end

Write lines to the file. By adding :line to writer, each write appends a new line character.

IOStreams.path("example.csv").writer(:line) do |file|
  file << "these"
  file << "are"
  file << "all"
  file << "separate"
  file << "lines"
end

Write an array (row) at a time to the file. Each array is converted to csv before being written to the file.

IOStreams.path("example.csv").writer(:array) do |io|
  io << ["name", "address", "zip_code"]
  io << ["Jack", "There", "1234"]
  io << ["Joe", "Over There somewhere", 1234]
end

Write a hash (record) at a time to the file. Each hash is converted to csv before being written to the file. The header row is extracted from the first hash write that is performed.

IOStreams.path("example.csv").writer(:hash) do |stream|
  stream << {name: "Jack", address: "There", zip_code: 1234}
  stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"}
end

This time write the CSV data to a compressed zip file, by adding .zip to the file name.

IOStreams.path("example.csv.zip").writer(:hash) do |stream|
  stream << {name: "Jack", address: "There", zip_code: 1234}
  stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"}
end

Getting Started

Start with the IOStreams tutorial for a great introduction to IOStreams.