Packmatic 1.1.1 has been released. Now it is possible to construct your archive with streams…

Added

• Added Packmatic.Source.Stream.
  • Added support for Streams that output IO Lists.
• Updated Packmatic.Source.
  • Added ability for any Source to use any data type as its Source State.
  • Added ability for any Source to return an updated Source State with new data.

Changed

• Revised Packmatic.Source.Dynamic.
  • Removed custom resolver; any entry notation now accepted.
• Updated development & test dependencies.

Hi all, Packmatic 1.2.0 is out, replacing ibrowse with hackney, and tested against more modern OTP versions.

Packmatic 1.1.0 has been released:

• Added support for custom Sources.

  • Any module which implements Packmatic.Source can be used as a Source.

• Added support for Encoder Events.

  • Added the on_event option to the Encoder which can be used to receive events.
  • See documentation for Packmatic.Event.

• Revised Packmatic.Source.

  • Added callback validate/1 for entry validation.

• Revised Packmatic.Manifest.Entry.

  • Moved validation of Initialisation Arguments to Sources.

• Revised Packmatic.Source.File.

  • Added explicit cleanup logic.

• Revised Packmatic.Source.URL.

  • Added explicit cleanup logic.

• Revised Packmatic.Encoder.

  • Fixed acceptance of IO Lists, in case of custom Sources returning these instead of binaries.

An upcoming version will add ability to consume IO Lists from Streams.

Packmatic 1.0.0 is now released. This release includes ahead-of-time verification enhancements for the Manifest, among other things. Changelog has been updated.

I had some similar requirements a few months ago:

1. UTF8 file names
2. Stream file input
3. Stream archive output

For our specific use case we needed to generate very large zips of video data without worrying about disk space. The goal was to create a multi part upload in S3 as the zip was created and to avoid disk writes altogether. The goal was to limit memory use to that of the largest video file. This was an unusual enough use case that I created a new library for it: zap.

Due to the specifics of the use case it doesn’t compress at all (though it could quite easily). Through the use of streams it accumulates inputs and periodically emits chunks of output suitable for S3. Since switching to zap we haven’t had any disk space issues.

Edit: I’m sharing in this thread in case others find themselves in a similar situation. Zap has a specific use case that is much narrower than packmatic, but we probably could have made packmatic work for our needs.