0

I've got a program that takes 3 arrays (which are the same length, can contain 500 items or so) and writes them to a text file.

However I'm getting an issue with writing larger files. The arrays are coordinates and timestamps of a canvas drawing app so I can control the length. I've found that once files start getting larger than 2mb it doesn't save the file. The maximum file I've managed to save has been 2.18mb. From a related question PHP: Having trouble uploading large files I've determined that the cause is most likely due to having this hosted on a free hosting server. I've looked at phpinfo() and here are the 4 relevant numbers:

memory_limit 16M
max_execution_time 30
upload_max_filesize 5M
post_max_size 5M

Here is the relevant writing code: 

// retrieve data from the JS
$x_s = $_GET['x_coords'];
$y_s = $_GET['y_coords'];
$new_line = $_GET['new_lines'];
$times = $_GET['time_stamps'];

print_r($_GET);
$randInt = rand(1,1000);

// first want to open a file
$file_name = "test_logs/data_test_" . $randInt . ".txt";
$file_handler = fopen($file_name, 'w') or die("Couldn't connect");

// For loop to write the data
for ($i = 0; $i < count($x_s); $i++){
    // If new line want to write new line!
    if (!$new_line[$i]){
        if ($i!=0){
            // If not the first line
            fwrite($file_handler, "LINE_END\n"); }          
        fwrite($file_handler, "LINE_START\n");
    }

    // Write the x coord, y coord, timestamp
    fwrite($file_handler, $x_s[$i] . ", ". $y_s[$i] .", ". $times[$i]. "\n");

    // If last line then write last LINE_END
    if ($i == (count($x_s)-1)){
        fwrite($file_handler, "LINE_END\n"); }
}

fclose($file_handler);

I've setup a php server on my localhost and have access to the error log. This is what I am getting.

[Fri Mar 23 20:03:02 2012] [error] [client ::1] request failed: URI too long (longer than 8190)

PROBLEM RESOLVED: The issue was that I was using GET to send large amounts of data, which was appended to the URI. Once the URI reached 8190 characters it had an error. Using POST solves this.

Community
  • 1
  • 1
Stuart Lacy
  • 1,963
  • 2
  • 18
  • 30

4 Answers4

1

Maybe script exceeds max execution time.

Add this

set_time_limit(0)

at the beginning of your code.

Leri
  • 12,367
  • 7
  • 43
  • 60
1

upload_max_filesize and post_max_size determine the maximum size of data that can be posted. But this is probably not your problem since some of the data is written (if you reach the data limit, the script does not execute).

Your script has two restrictions: max_execution_time and memory_limit. Have a look at your apache error log file to see if you are getting an error message (saying which limit is reached).

You can also try logging inside the for loop to see the progression of time and memory usage:

if(($i % 100) == 0) { // log every 100 entries
  error_log(date("H:i:s ").memory_get_usage(true)."Bytes used\n", 3, 'test.log');
}

It may also be that the Suhosin patch is preventing you from sending too many data points: http://www.adityamooley.net/blogs/2012/01/09/php-suhosin-and-post-data/

chiborg
  • 26,978
  • 14
  • 97
  • 115
  • Is it possible to look at the apache error log if this file is hosted on a free web server? – Stuart Lacy Mar 23 '12 at 13:03
  • I think it should be `if($i % 100 == 0)` :) – piotrekkr Mar 23 '12 at 13:10
  • Some hosting providers give you access to your own log file, otherwise you can redirect the error output to a directory of you choice by calling `ini_set('error_log', '/path/of/your/choice/error.log');` – chiborg Mar 23 '12 at 13:11
  • I had a look for "suhosin" in phpinfo() but didn't find anything so assume that the patch hasn't been installed. I've put that code in my loop, but again how do I check the error log? EDIT: posted after you typed that, will try finding error log – Stuart Lacy Mar 23 '12 at 13:17
  • Ok tried looking in error log but no log files are getting created! I put the ini_set() line at the start of the PHP file – Stuart Lacy Mar 23 '12 at 13:24
1

1) check max_input_time

ini_set ( 'max_input_time', 50 );

2) check in phpinfo() - do you have Suhosin patch?

You should look at apache error_log - You should find which limit is reached.

Try

ini_set('error_reporting', E_ALL);
error_reporting(E_ALL);
ini_set('log_errors', true);
ini_set('html_errors', false);
ini_set('error_log', dirname(__FILE__).'script_error.log');
ini_set('display_errors', true);
Bartosz Kowalczyk
  • 1,479
  • 2
  • 18
  • 34
  • See my answer to chiborg, is it possible to view the apache error log if this is hosted on a free web server? – Stuart Lacy Mar 23 '12 at 13:04
  • You can ask root about configuration or least ask about error_log – Bartosz Kowalczyk Mar 23 '12 at 13:11
  • See my answer to chiborg: Don't have Suhosin patch installed And with that code still not getting any errors logged or any log files created. I just put that second code block at the start of the php file. – Stuart Lacy Mar 23 '12 at 13:23
  • I'm sorry. I don't know what is problem. You should ask your root about error_log from your script. I think - problem is configuration apache not php. Can you check time of processing ? Maybe you spot any timeout problem. Check - do time of end is const or not. – Bartosz Kowalczyk Mar 23 '12 at 17:00
  • I set up a PHP server on my computer and managed to get the error log. This is what I'm getting [Fri Mar 23 20:03:02 2012] [error] [client ::1] request failed: URI too long (longer than 8190) – Stuart Lacy Mar 23 '12 at 20:08
  • You see. You send data inside of URI (adress) but every browser have your limit of char in URI. Your error log say that you have URI longer than 8190 char. You could use $_POST to provide data. Or check long of URI. If too long than send rest in second request. – Bartosz Kowalczyk Mar 25 '12 at 16:30
0

PHP (an hence the web server) is protecting itself. Perhaps use a different mechanism to upload a large file - i would imagine they come from known (an trusted) sources. Use a different mechanism, for example SFTP.

Ed Heal
  • 59,252
  • 17
  • 87
  • 127