A client (such as a web service, application, or custom ETL pipeline) sends an HTTP POST request to Doris’s Stream Load API endpoint.
The request contains the target database and table, along with the data to be loaded. The data can be in formats like CSV, JSON, or Parquet.
2. Data Segmentation:
The data is transmitted to the Doris FE in chunks. For large datasets, this chunking ensures the data can be processed without overwhelming the system.
Doris supports streaming data in a continuous flow, allowing for near real-time data loading.
3. Data Parsing and Transformation:
The Doris FE parses the incoming data based on the specified format.
It validates the data against the schema of the target table. Any malformed data can be rejected based on predefined rules.
4. Data Sharding and Partitioning:
Doris uses a primary key-based sharding mechanism to determine which tablets (a unit of storage in Doris) the parsed data should be written to. This sharding ensures that the data is evenly distributed across the cluster.
Data is then further partitioned based on the table’s partitioning scheme, ensuring that it is stored in the correct partitions within the tablets.
5. Data Writing and Indexing:
The parsed and partitioned data is written to the corresponding tablets. Doris maintains a columnar storage format to optimize storage and query performance.
During the writing process, Doris creates and updates various internal indexes, such as Bloom filters and segment files, to support efficient data retrieval.
6. Commit and Acknowledgment:
After the data is written to all relevant tablets, Doris commits the transaction. This involves ensuring that all replicas have received the data and are consistent.
Once the transaction is successfully committed, Doris sends an acknowledgment back to the client, indicating that the Stream Load operation has been completed.
7. Commit or Rollback:
Doris uses a two-phase commit protocol to ensure data consistency.
Prepare Phase: The BE nodes prepare to commit the data by writing it to temporary storage.
Commit Phase: Once all nodes have prepared successfully, the FE instructs the BE nodes to commit the data, making it visible for queries.
If any part of the process fails, a rollback is triggered, and the data is not committed.
8. Compaction and Optimization (Post-Load):
Doris periodically performs compaction and optimization operations to merge small segments into larger ones, remove duplicate data, and optimize storage layout. This process improves query performance and reduces storage overhead.
The compaction process is managed by the BE nodes and is triggered based on system policies or thresholds.
9. Error Handling:
If any errors occur during the load process (e.g., data format issues, network failures), Doris has mechanisms to retry the operation or report the failure to the client. Error logs are maintained for debugging purposes.
10. Concurrency and Fault Tolerance:
Doris supports concurrent Stream Load operations, with mechanisms to handle data deduplication and recovery in case of partial failures.
The system ensures that all or none of the data in a Stream Load operation is ingested, maintaining data integrity.
Key Features:
High Throughput: Doris is designed to handle high-throughput data ingestion with low latency, making it suitable for real-time analytics.
Fault Tolerance: Doris ensures data consistency and availability through replication and robust error-handling mechanisms.
Scalability: The distributed architecture of Doris allows it to scale horizontally, handling larger volumes of data and higher load frequencies.