使用PHP删除.txt文件中的重复行

Question

I have multiple txt files with a directory. The text files all contain the same header. I am reading in all the txt files and outputting it all to one file.

Since each individual file contains the same header, It is inserting all of them into the new merged file. How can I remove all the headers in the new merged file and leave one just at the top?

I have been looking at the sort command in unix.

sort filename | uniq

This command works, but removes all other data that is duplicate. Is there anyway to remove just the specific string "This is a header" but leave one at the top?

Current Code

$header = array( "XX-XXXXXXXXX-XXXXXXX-X        XXXXXXXXXXXX" );


$files = glob( "/path/to/folder/*.txt" );

$output_file = "newfile_".date( "YmdHis" ).".txt";

$out = fopen( $output_file, "w" );

foreach( $header as $inputHeader ) {

    fwrite( $out, $inputHeader );
}

    foreach( $files as $file ) {

        $in = fopen( $file, "r" );

            while ( $line = fgets( $in ) ) {

                if( $header !== $line ) {

                    fwrite( $out, $line );

                }

            }

        fclose( $in );

     }

fclose( $out );

The Line that is duplicated many times

Answer 1

Try inputting the header at the start of writing, then check for it later on when you are reading the lines

//cache our header lines
$header = "Header line";

$files = glob( "/path/to/files*.txt" );

//print_r($files);

$output_file = "newfile".date( "YmdHis" ).".txt";

$out = fopen( $output_file, "w" );

//input the header line at the top of our new file

fwrite( $out, $header);



foreach( $files as $file ) {

    $in = fopen( $file, "r" );

        while ( $line = fgets( $in ) ) {
            //header check, dont output header lines to new file
            if($header !== preg_replace('/\s+/', '', $line)){
                 fwrite( $out, $line );
            }
        }

    fclose( $in );
}

fclose( $out );