The file you are looking for is app/code/core/Mage/CatalogSearch/Model/Layer.php
. The name of the class is Mage_Catalog_Model_Layer
. Inside it you see the function prepareProductCollection($collection)
.
I suggest you rewrite this model (see this or this) and override the aforementioned function and add/implement you sorting logic. An example of overriding this function would be
class XXX_XXX_Model_CatalogSearch_Layer extends Mage_CatalogSearch_Model_Layer
{
public function prepareProductCollection($collection)
{
parent::prepareProductCollection($collection);
$collection->addAttributeToSelect('some_attribute');
$collection->setOrder('some_attribute', 'asc');
return $this;
}
}