Skip to main content
Version: Next

Delivery methods

AvailableCloud AvailableSelf-Managed CommunityAvailableSelf-Managed Enterprise Compare

Airbyte supports two methods for delivering source data to the destination.

  • Replicate records
  • Copy raw files

This article explains the difference between these methods, when you should use each one, and how to configure this option in Airbyte.

Replicate records​

When you replicate records, you extract and load structured records into your destination of choice. This method allows for blocking and hashing individual fields or files from a structured schema. Data can be flattened, typed, and deduped depending on the destination.

For most connectors, this is the only option you have and it's chosen by default. It's ideal for working with structured data like databases, spreadsheets, JSON, and APIs. In some connectors, this method also moves unstructured files included in the structured data.

Copy raw files​

When you copy raw files, you copy files without parsing their contents. Airbyte copies bits into the destination exactly as they appear in the source. In this case, Airbyte is focused on data movement over structure. To support AI use cases, this option also delivers structured metadata that describes those files.

This choice is ideal for unstructured text, non-text data like multimedia, and compressed files. However, it's only available on specific connectors designed to handle unstructured data, like those related to blob storage solutions.

To ensure your source and destination credentials remain isolated, Airbyte uses a temporary local staging location. See the illustration below.

Moving raw files from a source to a destination without regard for their contents or structure

Supported versions and limitations​

Supported Airbyte versions​

  • Cloud: All Workspaces

  • Self-Managed Community and Self-Managed Enterprise:

    • Version 1.2.0 or later to copy files.

    • Version 1.7.0 or later to copy files with metadata.

Limitations​

  • Maximum file size: 1-GB per file.

Configure copy raw files as your delivery method​

You can only configure the delivery method to copy raw files for specific, file-based connectors. In most cases, the option isn't present and you don't need to set it.

  1. Add your file-based source, if you haven't already. When you configure your source, under Delivery Method, choose Copy Raw Files.

  2. Add your file-based destination, if you haven't already. While setting up the destination connector, choose the file format of your metadata by setting the Output Format option. For help, see Change the metadata format below.

  3. Add your connection, if you haven't already. In the schema, enable the stream(s) containing the files you want to sync, and select which fields you want in your metadata. For help, see Change what's in the metadata, below.

Configure file metadata​

Two elements of the metadata are configurable: file format and what's in the file. You configure each of these in difference places in Airbyte.

Change the metadata format​

To change the format of the metadata, change the output format on the Destination connector.

  1. In the left navigation, click Destinations.

  2. Choose your destination connector from the list, or create a new one.

  3. Under Output Format, choose the file format you want. You can also choose whether you want to compress the file, or if you want to flatten it.

    Choosing JSON Lines format as the Output Format on the Destinaton configuration for the S3 destination

  4. Click Test and save. Next time you sync, your metadata has the chosen format.

Change what's in the metadata​

You choose what appears in your metadata when setting up your connection.

  1. In the left navigation, click Connections.

  2. Choose your connection from the list, or create a new one.

  3. Click the Schema tab.

  4. Open the stream that contains your files.

  5. Select and deselect fields as needed.

    article_attachments stream with various fields selected and deselected

  6. Click Save changes. Next time you sync data, your metadata in your destination updates.

Metadata sample​

Here is a sample of metadata in .jsonl format, describing a single Word document.

{
    "_airbyte_raw_id": "9e089a10-b347-4acf-abc8-f093b93b84bb",
    "_airbyte_extracted_at": 1748991118573,
    "_airbyte_meta": {
        "sync_id": 38401084,
        "changes": []
    },
    "_airbyte_generation_id": 2,
    "_airbyte_data": {
        "folder": "",
        "file_name": "Test Plan.docx",
        "bytes": 7838,
        "source_uri": "https://example.com",
        "id": "1i3cTG1UPQx-ZHft-78neonQ9tDNjkZ-E4bK46eGShFk",
        "created_at": "2025-03-04T14:30:28.000000Z",
        "updated_at": "2025-03-04T14:31:11.253000Z",
        "mime_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
        "_airbyte_file_path": "/Files/2025_06_03_1748991116728_/Test Plan.docx"
    }
}
Was this page helpful?