How to Speed Up Uploading Multiple Files via HTTP

A few months ago, I worked on a research task to find out how to improve the speed of multiple HTTP connections from one server (playing a client role) to another server (playing a server role when uploaded files are eventually saved) at the same time. In this specific issue, both servers belong to the same network. That is, we can make any change we want though allocated resource does matter.

After measurement and looking at a few layers, we found out three points to improve this connection:

1. Implement persisent connections (keep-alive) on the server side, then the client side. If this is not implemented, after uploading a file, the related TCP connection is closed. When the next file is uploaded, a new TCP connection is constructed and this process takes quite a bit of time. It even takes more time than transferring the huge data via our network.
2. Implement con-currency connections. That is, instead of using a single TCP connection, the client side starts a few connections at the same time, and send data via these connections. We may test and choose the optimal number to optimize the concurrency.

How to gather these data to make the decision? During this task, I could find out the following ways:

• In WP, we can use this action requests-curl.after_request to log info for each HTTP request. The examle is below.
• For AWS SDK for PHP and S3, this info is saved in $['@metadata']['transferStats']. See more stats section of this guide.
• For PHP libarary Guzzle, it’s the on_stats option per this guide.

Good reference https://ops.tips/gists/measuring-http-response-times-curl/

Example code for WordPress:

<?php
/**
 * Log all HTTP request info to a CSV file.
 * 
 * Usage: save this PHP file as a plugin/mu-plugin and check results in the newly generated CSV file.
 */
add_action( 'requests-curl.after_request',
	function( string $headers, array $info = [] ) {

		if ( ! isset( $info['url'] ) ) {
			return;
		}
		htdat_write_csv( __DIR__ . '/http-stats.csv', $info );
	},
	9999, 2 );

function htdat_write_csv( $file = 'http-stats.csv', $info = []  ) {
	// See https://www.php.net/curl_getinfo for all available fields
	$fields = [
		'url',
		'namelookup_time',
		'connect_time',
		'pretransfer_time',
		'starttransfer_time',
		'total_time',
	];

	if ( ! file_exists( $file ) ) {
		file_put_contents( $file, implode(',', $fields) . PHP_EOL );
	}

	$values = array_map( function ( $field ) use ( $info ) {
		return $info[ $field ] ?? 'N/A';
	}, $fields);

	file_put_contents( $file, implode(',', $values) . PHP_EOL, FILE_APPEND );
}