fllink预写日志和两阶段提交的差别

Question

flink保证sink 端exactly-once有幂等写入和事务写入两种方式。事务写入的实现方式又有预写日志和两阶段提交。请问预写日志和两阶段提交的差别在哪里？两个好像都是先写入缓存，等checkpoint确认了，再写入磁盘。请问两者的差别在哪里？

Answer 1

The WAL sink writes all result records intoapplication state and emits them to the sink system once it receives thenotification that a checkpoint was completed. Since the sink buffersrecords in the state backend, the WAL sink can be used with any kindof sink system. However, it cannot provide bulletproof exactly-onceguarantees, adds to the state size of an application, and the sinksystem has to deal with a spiky writing pattern.In contrast, the 2PC sink requires a sink system that offers transactionalsupport or exposes building blocks to emulate transactions. For eachcheckpoint, the sink starts a transaction and appends all receivedrecords to the transaction, writing them to the sink system withoutcommitting them. When it receives the notification that a checkpointcompleted, it commits the transaction and materializes the writtenresults. The mechanism relies on the ability of a sink to commit atransaction after recovering from a failure that was opened before acompleted checkpoint.The 2PC protocol piggybacks on Flink’s existing checkpointingmechanism. The checkpoint barriers are notifications to start a newtransaction, the notifications of all operators about the success of theirindividual checkpoint are their commit votes, and the messages of theJobManager that notify about the success of a checkpoint are theinstructions to commit the transactions. In contrast to WAL sinks, 2PCsinks can achieve exactly-once output depending on the sink systemand the sink’s implementation. Moreover, a 2PC sink continuouslywrites records to the sink system compared to the spiky writing pattern2of a WAL sink.