Compression (defalate, gzip)
It is a common case to compress big pieces of data for transmitting it between server and client. Since of v.0.4.4 RESTinio adds a concept of data transformators to its tooling. While itself the transform mechanics is independent it comes nicely integrated with RESTinio.
In this section, zlib transformation is introduced. It allows you to perform compress/decompress operation on the data using zlib library. Zlib supports the following compression algorithms: deflate and gzip.
Zlib transformator is not included with <restinio/core.hpp>
, to use it one needs
to include <restinio/transforms/zlib.hpp>
.
All functions and classes are located in restinio::transforms::zlib
namespace.
The basic usage of zlib transformator will start with something like the following:
#include <restinio/transforms/zlib.hpp>
// ...
void my_handler( auto req )
{
namespace rtz = restinio::transforms::zlib;
// ...
}
zlib_t
The main class for zlib transform is restinio::transform::zlib::zlib_t
.
It handles all the low level staff of using zlib.
All higher level functionality that minimizes boilerplate code and makes
compression and decompression logic less verbose is based on zlib_t
.
params_t
zlib_t
must be initialized with parameters (restinio::transform::zlib::params_t
)
that define the operation: compress or decompress and destination
format: deflate, gzip or identity. Note though identity
is an imaginary compression format it means that no compression will be used
and no header/trailer will be applied. identity is very helpful for cases when
compression (decompression) might or might not be used and
you want a single piece of code to handle these cases.
Class params_t
keeps the following transform parameters:
- operation
- Types of transformation. Enum
params_t::operation_t
with the following items: compress, decompress; - format
- Formats of compressed data. Enum
params_t::format_t
with the following items: deflate, gzip, identity. - level
- Сompression level (makes sense only for compression operation). Must be an integer value in the range of -1 to 9. -1 means default level and 0 means no compression but zlib header and trailer are still present.
- window_bits
- Must be an integer value in the range of 8 to MAX_WBITS (constant defined by zlib, usually 15). For decompress operation, it is better to keep the default value of windows beats (MAX_WBITS). Also for decompress operation window_bits can be zero to request the use of window size obtained from zlib header of the compressed stream (See inflateInit2() description for details).
- mem_level
- Defines memory usage of compression operation (makes sense only for compression operation). Must be an integer value in the range of 1 to 9. 1 stands for minimum memory usage and 9 stands for maximum memory usage. The amount of memory that can be used affects the quality of compression.
- strategy
- Compression strategy (makes sense only for compression operation). Must be an integer value defined by one of zlib constants: Z_FILTERED, Z_HUFFMAN_ONLY, Z_RLE, Z_DEFAULT_STRATEGY. (See deflateInit2() description for details).
- reserve_buffer_size
- The initial size of buffer used for output.
When using zlib transformator the outout buffer is used.
The initial size of such buffer must be defined.
zlib_t
instance will use this parameter as the initial size of out buffer and as an increment size if out buffer must be enlarged.
params_t
supports
fluent interface
for setting all the mentioned parameters except operation and format
that must be in constructor. params_t
has a default constructor
that defines identity transformation. Note that in case of identity transformation
all parameters are ignored as there is no place for them to be used.
RESTinio provides several functions for making parameters for a given operation:
params_t make_deflate_compress_params( int compression_level = -1 );
params_t make_deflate_decompress_params();
params_t make_gzip_compress_params( int compression_level = -1 );
params_t make_gzip_decompress_params();
params_t make_identity_params();
So instead of of writing something like the following:
namespace rtz = restinio::transforms::zlib;
params_t params{
rtz::params_t::operation_t::compress,
rtz::params_t::format_t::gzip,
-1 };
params.mem_level( 7 );
params.window_bits( 13 );
params.reserve_buffer_size( 64 * 1024 );
It is better to write the following:
params_t params =
restinio::transforms::zlib::make_gzip_compress_params()
.mem_level( 7 )
.window_bits( 13 )
.reserve_buffer_size( 64 * 1024 );
zlib_t API
Having parameters zlib_t
object can be instantiated.
zlib_t
is non-copiable and non-movable class with a single
constructor available:
zlib_t( const params_t & transformation_params )
- Initialize zlib transformator with given parameters.
As previously mentioned zlib_t
hides all low level details of working with
zlib. Based on parameters it initializes all appropriate structs
and calls necessary functions for doing compression or decompression
and deinitializes structs when the stream is completed.
The interface of zlib transform routine is the following:
- const params_t & params() const
- Get parameters of current transformation. Gives an access to parameters a given object is about.
- void write( string_view_t input )
- Append input data. Pushes given input data to zlib transform. Parameter input points to data to be compressed or decompressed.
- void flush()
- Flushes underlying zlib stream. All pending output is flushed to the output buffer.
- void complete()
- Complete the stream. All pending output is flushed and a proper trailer data is produced and inserted into stream.
- std::string giveaway_output()
Get current accumulated output data. On this request, a currently accumulated output data is returned. Move semantics is applied to the output buffer. Once current output is fetched
zlib_t
object resets its internal out buffer. In the following code:1 2 3 4 5 6 7 8 9
namespace rtz = restinio::transforms::zlib; rtz::zlib_t z{ rtz::make_gzip_compress_params() }; z.write( A ); consume_out( z.giveaway_output() ); z.write( B ); z.write( C ); consume_out( z.giveaway_output() );
function consume_out() on line(9) receives a string that is not an appended version of a string received on line(5).
- auto output_size() const
- Get accumulated output data size.
- bool is_completed() const
- Gives operation completion state.
Usage examples
Simple usage example:
namespace rtz = restinio::transforms::zlib;
rtz::zlib_t z{ rtz::make_gzip_compress_params( 9 ) };
z.write( input_data );
z.complete();
// Get compressed input_data.
auto gziped_data = z.giveaway_output();
Advanced usage example:
namespace rtz = restinio::transforms::zlib;
rtz::zlib_t z{ rtz::make_gzip_compress_params( 9 ) };
std::size_t processed_data = 0;
for( const auto d : data_pieces )
{
z.write( d );
// Track how much data is already processed:
processed_data += d.size();
if( processed_data > 1024 * 1024 )
{
// If more than 1Mb is processed do flush.
z.flush();
}
if( z.output_size() > 100 * 1024 )
{
// If more than 100Kb of data is ready, then append it to something.
append_output( z.giveaway_output() );
}
}
// Complete the stream and append remeining putput data.
z.complete();
append_output( z.giveaway_output() );
Easy interface
RESTinio provides a number of helper classes and functions for doing compression/decompression easier.
Single function for compression/decompression
RESTinio has a set of handy functions helping to perform zlib transform in one line:
//! Do a specific transformation.
std::string transform( string_view_t input, const params_t & params );
std::string deflate_compress( string_view_t input, int compression_level = -1 );
std::string deflate_decompress( string_view_t input );
std::string gzip_compress( string_view_t input, int compression_level = -1 );
std::string gzip_decompress( string_view_t input );
So instead of writing something like this:
namespace rtz = restinio::transforms::zlib;
rtz::zlib_t z{ rtz::gzip_compress() };
z.write( data );
z.complete();
body = z.giveaway_output();
It is possible to write the following:
body = restinio::transforms::zlib::gzip_compress( data );
Body appenders
RESTinio provides a special template class
body_appender_t<Response_Output_Strategy>
which is tied with response_builder_t<Response_Output_Strategy>
(see Response builder).
The purpose of body_appender_t
for a given response builder is to help to set
a compressed body data. It not only compresses the data but also sets an appropriate
value for Content-Encoding
field (deflate, gzip, identity).
For making body appender there is a set of factory functions:
// Create body appender with given zlib transformation parameters.
// Note: params must define a compression operation.
template < typename Response_Output_Strategy >
body_appender_t< Response_Output_Strategy > body_appender(
response_builder_t< Response_Output_Strategy > & resp,
const params_t & params );
// Create body appender with deflate transformation and a given compression level.
template < typename Response_Output_Strategy >
body_appender_t< Response_Output_Strategy > deflate_body_appender(
response_builder_t< Response_Output_Strategy > & resp,
int compression_level = -1 );
// Create body appender with gzip transformation and a given compression level.
template < typename Response_Output_Strategy >
inline body_appender_t< Response_Output_Strategy > gzip_body_appender(
response_builder_t< Response_Output_Strategy > & resp,
int compression_level = -1 );
// Create body appender with gzip transformation and a given compression level.
template < typename Response_Output_Strategy >
inline body_appender_t< Response_Output_Strategy > identity_body_appender(
response_builder_t< Response_Output_Strategy > & resp,
int = -1 );
There are 3 specializations of body_appender_t
for each type of
response builder.
body_appender_t< restinio_controlled_output_t >
Helper class for setting the body of
response_builder_t<restinio_controlled_output_t>
(see RESTinio controlled output response builder). It combines semantics of setting body data with response builder and zlib compression. the main functions of it are:body_appender_t & append( string_view_t input)
. Appends data to be compressed.void complete()
. Completes the compression and sets the response body.
Sample usage:
namespace rtz = restinio::transforms::zlib; auto resp = req->create_response(); // creates response_builder_t<restinio_controlled_output_t> by default. resp.append_header( restinio::http_field::server, "RESTinio" ) .append_header_date_field() .append_header( restinio::http_field::content_type, "text/plain; charset=utf-8" ); auto ba = rtz::gzip_body_appender( resp ); ba.append( some_data ); // ... ba.append( some_more_data ); ba.complete(); resp.done();
body_appender_t< user_controlled_output_t >
Helper class for setting the body of
response_builder_t<user_controlled_output_t>
(see User controlled output response builder). Sample usage:auto resp = req->create_response<user_controlled_output_t>(); resp.append_header( restinio::http_field::server, "RESTinio" ) .append_header_date_field() .append_header( restinio::http_field::content_type, "text/plain; charset=utf-8" ); auto ba = restinio::transforms::zlib::gzip_body_appender( resp ); ba.append( some_data ); ba.flush(); // ... ba.append( some_more_data ); ba.complete(); resp.done();
body_appender_t< chunked_output_t >
Helper class for setting the body of
response_builder_t<chunked_output_t>
(see Chunked transfer encoding output builder). It combines semantics of appending body data with chunked response builder and zlib compression. It has the following functions:body_appender_t & append( string_view_t input)
. Append data to be compressed. Function only adds data to anderlying zlib stream and it doesn’t affect target response right on here.body_appender_t & make_chunk( string_view_t input = string_view_t{} )
. Append data to be compressed and adds current zlib transformator output as a new chunk. After adding data flushes zlib transformator. Then ready compressed data is taken and used as a new chunk of target response.void flush()
. Flushes currently available compressed data with possibly creating new chunk and then flushes target response.void complete()
. Complete zlib transformation operation and appends the last chunk.
Sample usage:
resp.append_header( restinio::http_field::server, "RESTinio" ) .append_header_date_field() .append_header( restinio::http_field::content_type, "text/plain; charset=utf-8" ); auto ba = restinio::transforms::zlib::deflate_body_appender( resp ); ba.append( some_data ); ba.append( some_more_data ); ba.make_chunk(); // Flush copressed data and creates a chunk with it. ba.flush(); // Send currently prepared chunks to client // ... // Copress the data and creates a chunk with it. ba.make_chunk( even_more_data ); ba.flush(); // Send currently prepared chunks to client // ... ba.append( yet_even_more_data ); ba.append( last_data ); ba.complete(); // Creates last chunk, but doesn't send it to client. ba.flush(); // Send chunk created by complete() call // ... resp.done();
Handle decompressed body
To simplify handling of the requests with a body that might or might not be compressed RESTinio provides a helper function for handling those cases evenly:
// Call a handler over a request body.
template < typename Handler >
auto handle_body( const request_t & req, Handler handler );
Handler
is a function object capable to handle std::string
as an argument.
If the body is encoded with either ‘deflate’ or ‘gzip’ then it is decompressed and the handler is called. If the body is encoded with ‘identity’ (or not specified) the handler is called with original body. In other cases, an exception is thrown.
Sample usage:
auto decompressed_echo_handler( restinio::request_handle_t req )
{
return
restinio::transforms::zlib::handle_body(
*req,
[&]( auto body ){
return
req->create_response()
.append_header( restinio::http_field::server, "RESTinio" )
.append_header_date_field()
.append_header(
restinio::http_field::content_type,
"text/plain; charset=utf-8" );
.set_body( std::move( body ) ) // Echo decompressed body.
.done();
} );
}
Samples
See compression and decompression samples for full working examples using zlib transformator.