Use java and shell realize tools about stream load
The method of stream load is limited by `curl` performance. As a result, it's too slow to import a big file.
I want to realize a Java client to transfer the big file in parallel, can increase the performance.
First, I need to sequential read the big file, due to the characteristics of the disk, the performance of sequential read is of less concern about thread number.
Second, the data that was read will compose a long string, and be put in the blocking queue. when the main thread will block, the size of the blocking queue is equal to a limited size.
Third, the worker thread will consume the blocking queue, and send data through the network. when the worker thread is stopped, they receive an empty string from the main thread.
Using curl with one thread, the transmission speed is limited by disk bandwidth and at tens of megabytes.
With the new tools, the transmission speed can run over the 10 Gigabit network and satulated the disk bandwidth.
However, the transmission speed may also be affected by the sequential read-write performance of the hard disk. If you want faster transmission speed, you need a hard disk with better performance
Moreno Garcia
Murphy
Rohit Satardekar
Alex Zhu
evelyn.zhaojie
xyz
evelyn.zhaojie
lichaoyong
yan.zhang
Napoleon
imay
林郎俊赏丶
Zhao Chun