Here is my code to fetch data from websites which have there some data stores in txt files, I just give the urls and it starts fetching the data
foreach ($collector_urls as $key => $value) {
$k=0;
try{
$file=fopen($value,"r");
while (!feof($file)) {
$str=fgets($file);
$new_str=explode(".",$str);
if(is_numeric($new_str[0])){
// if (!in_array($str,$ip_array))
$array[$ip_count++]=$str;
}
}
$log[$name_in_key]=json_encode(array('status'=>'success','count'=>$k,'key'=>$name_in_key,'url'=>$value));
}catch(Exception $e){
print("Something wrong with file ".$value."<br>");
$log->$name_in_key=array('status'=>'failed','count'=>0);
}
}
while the code does so , some warnings arrive such has invalid resource content for fopen, fgets and isnumeric
data is very very huge so I might consider about 300000 rows of error each time it executes , what is actually my problem is it fills almost 25 gb of meemory when executed inside apache log with errors and warnings
is there a way I can tell apche not to log warnings and errors inside log file or some other method also should I need to change some part of code which I am doing wrong
Need suggestions , thanks in advance