File support (sendfile)

Since of v.0.4.3 RESTinio supports a so-called sendfile operation. Briefly, it allows one to send a piece of data from a file (or the whole file) as a part of response content or the whole response. Just like this:

 restinio::request_handling_status_t handler(
     restinio::asio_ns::io_context & ioctx,
     restinio::request_handle_t req )
 {
     if( restinio::http_method_get() == req->header().method() &&
         req->header().request_target() == "/" )
     {
         return
             req->create_response()
                 .append_header( restinio::http_field::server, "RESTinio hello world server" )
                 .append_header_date_field()
                 .append_header( restinio::http_field::content_type, "text/plain; charset=utf-8" )
                 .set_body( restinio::sendfile( "hello_file.txt" ) )
                 .done();
     }

     return restinio::request_rejected();
 }

In above sample request handler the whole file hello_file.txt will be considered for write to a client. RESTinio will itself determine the size of the file and will account it when calculating Content-length field value.

restinio::sendfile() function returns an object of type restinio::sendfile_t which is responsible for holding a given file descriptor while it will be used for output. Also, it contains some parameters of a file write operation and allows to set these parameters. There are two main parameters.

There is a couple of overloads for sendfile():

inline sendfile_t sendfile(
    const char * file_path,
    file_size_t chunk_size = sendfile_default_chunk_size );

inline sendfile_t sendfile(
    const std::string & file_path,
    file_size_t chunk_size = sendfile_default_chunk_size );

inline sendfile_t sendfile(
    string_view_t file_path,
    file_size_t chunk_size = sendfile_default_chunk_size );

All the overloads mentioned to this point try to open a file with a given path and obtain its descriptor, once they succeed, they determine total file size and create sendfile_t object that becomes the owner of the file descriptor. In the case of error, an exception would be thrown.

And there is one more version of the function sendfile() used by all other versions:

inline sendfile_t sendfile(
    file_descriptor_t fd,
    file_size_t total_file_size,
    file_size_t chunk_size = sendfile_default_chunk_size );

It receives file descriptor and file size directly and assumes they are correct. Be careful when using this overload. It is preferable to use one of the safer versions of sendfile() mentioned above.

Once sendfile_t object instantiated it can be tuned to represent a certain piece of data in the file by giving an offset and size:

1
2
3
4
5
6
 auto sf = restinio::sendfile( params.data_file() );
 // Set the offset.
 if( params.data_size() )
     sf.offset_and_size( params.data_offset(), params.data_size() );
 else
     sf.offset_and_size( params.data_offset() );

Function offset_and_size() receives two parameters: offset and size. The second one is optional and by default it is assigned the maximum possible value. Size values greater than N (size of the file from the given offset) are shrunk to N. If the size of the data portion is known then offset_and_size() is used like in line 4 of the above sample. If it is needed to send all the data beginning from the given offset then offset_and_size() is used like in line 6.

Another parameter that can be set for sendfile operation is chunk_size. It defines the maximum size of data to be sent in one iteration, see Implementation notes.

File meta

Since RESTinio v.0.4.7 contains file_meta_t class that incapsulates meta data associated with file. It has the following interface:

class file_meta_t
{
  public:
    // ...

    file_size_t file_total_size() const noexcept;
    std::chrono::system_clock::time_point last_modified_at() const noexcept;

    // ...
};

Currently it makes file size and last modification time point available.

Sample usage:

auto sf = restinio::sendfile( file_path );
auto modified_at = restinio::make_date_field_value( sf.meta().last_modified_at() );
// ...
resp.append_header( restinio::http_field::last_modified, std::move( modified_at ) )

Implementation notes

RESTinio has separate implementations for different platforms using platform native API:

  • posix: Linux and FreeBSD ports;
  • win: Windows;
  • default: using regular <cstdio> API.

Internally the demand for performing sendfile operation is wrapped in a writable item instance and is treated as an ordinary buffer until it is selected as a next item for output. Sendfile operation is always selected alone for the further output (naturally, there is no ground for scatter/gather IO). Sendfile operation in general runs in the following manner:

  1. get the chunk of data from the file (at most the size of chunk_size) and initiate write-to-socket operation;
  2. wait asynchronously for the write operation to finish;
  3. once write operation is complete check if all the necessary data is written, if yes, then go to step 1, else go to step 4;
  4. back to the normal circle of doing output on a given connection.

Posix

For raw sockets (non TLS) RESTinio utilizes ::sendfile() function (Linux sendfile, FreeBSD sendfile). It gives an efficient way to send data from a file to socket avoiding moving data between kernel space and user space, more on this here.

Windows

For raw sockets (non TLS) RESTinio utilizes ::TransmitFile() function (see documentation here). It gives an efficient way to send data from a file to socket avoiding moving data between kernel space and user space.

Other platforms

On the platforms that are not recognized as Posix or windows there is an option to use a default sendfile implementation where file is processed with <cstdio> functions. But there is a limitation on file size that can be handled: 2Gb. By default it is not enabled, to enable it, define RESTINIO_ENABLE_SENDFILE_DEFAULT_IMPL macro before including RESTinio headers.

TLS sockets

When RESTinio runs with TLS, then there is no way to use ::sendfile() or ::TransmitFile() functions because the data must be ciphered (user space operation) before writing to a socket. For TLS sockets RESTinio applies read from file to buffer + write from buffer to socket schema. And for windows platform RESTinio uses Overlapped I/O.