Interesting work on big data streaming & dynamic batching algorithms

While I was working on a solution for large scale Apache Kafka logging, I got interested in processing performance considerations.

Then I found an interesting work of Yuan Zhong on optimal buffer size and dynamic batching.


Maybe this is too adanced to get applied in my daily work, but I like the thing that engineers and doctors took this problem seriously and employed math to get optimal solutions.

The works like this get applied in your routers, switches, CPUs and software.


http://www.columbia.edu/~yz2561/dynamic_batching.pdf

https://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-133.html