IOStreams is an incredibly powerful Ruby streaming library that makes changes to file formats, compression, encryption, or storage mechanism transparent to the application.


Streaming avoids high memory utilization since the file (or other source such as AWS S3) is read or written a block at a time.

Develop applications using the local file system, then use for example AWS S3 for all file storage in production, without changing any of the source code.

File Extensions

File Storage

File formats


Write to a local file:

IOStreams.path("example.txt").write("Hello World")

Write to AWS S3:

IOStreams.path("s3://bucket-name/path/example.txt").write("Hello World")

Write the same data into a compressed file by adding the .gz extension to the file name:

IOStreams.path("example.txt.gz").write("Hello World")

Compress and encrypt the data into a PGP encrypted file:

path = IOStreams.path("example.txt.pgp")
path.option(:pgp, recipient: "")
path.write("Hello World")

Write a file to a SFTP server:

path = IOStreams.path("s", 
                      username: "example", 
                      password: "topsecret")
path.write("Hello World")

Write PGP encrypted file to AWS S3:

path = IOStreams.path("s3://bucket-name/path/example.txt.pgp")
path.option(:pgp, recipient: "")
path.write("Hello World")

Read an entire file into memory:

# => "Hello World"

Read an entire file into memory from S3:

# => "Hello World"

Decompress an entire gzip file into memory:

# => "Hello World"

Decrypt and decompress the entire PGP file into memory:

# => "Hello World"

Streaming Examples

Read 128 characters at a time from the file:

IOStreams.path("example.csv").reader do |io|
  while (data =
    p data 

Read one line at a time from the file:

IOStreams.path("example.csv").each do |line|
  puts line

Read the first line from a file:


Read the first 5 lines from a file:


Display each row from the csv file as an array:

IOStreams.path("example.csv").each(:array) do |array|
  p array

Display each row from the csv file as a hash, where the first line in the CSV file is the header:

IOStreams.path("example.csv").each(:hash) do |hash|
  p hash

Read the first 5 records from a csv file:


Write data to the file.

IOStreams.path("abc.txt").writer do |io|
  io << "This"
  io << " is "
  io << " one line\n"

Write lines to the file. By adding :line to writer, each write appends a new line character.

IOStreams.path("example.csv").writer(:line) do |file|
  file << "these"
  file << "are"
  file << "all"
  file << "separate"
  file << "lines"

Write an array (row) at a time to the file. Each array is converted to csv before being written to the file.

IOStreams.path("example.csv").writer(:array) do |io|
  io << ["name", "address", "zip_code"]
  io << ["Jack", "There", "1234"]
  io << ["Joe", "Over There somewhere", 1234]

Write a hash (record) at a time to the file. Each hash is converted to csv before being written to the file. The header row is extracted from the first hash write that is performed.

IOStreams.path("example.csv").writer(:hash) do |stream|
  stream << {name: "Jack", address: "There", zip_code: 1234}
  stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"}

This time write the CSV data to a compressed zip file, by adding .zip to the file name.

IOStreams.path("").writer(:hash) do |stream|
  stream << {name: "Jack", address: "There", zip_code: 1234}
  stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"}

Getting Started

Start with the IOStreams tutorial for a great introduction to IOStreams.