PHP - 当键不唯一时，将数组值附加到另一个数组

Question

I have a query that returns me these results

array (size=6)
  0 => 
    object(stdClass)[14]
      public 'ReplyTo' => int 18
      public 'ReplyFrom' => int 21
  1 => 
    object(stdClass)[15]
      public 'ReplyTo' => int 19
      public 'ReplyFrom' => int 21
  2 => 
    object(stdClass)[16]
      public 'ReplyTo' => int 19
      public 'ReplyFrom' => int 22
  3 => 
    object(stdClass)[17]
      public 'ReplyTo' => int 20
      public 'ReplyFrom' => int 22
  4 => 
    object(stdClass)[18]
      public 'ReplyTo' => int 17
      public 'ReplyFrom' => int 22
  5 => 
    object(stdClass)[19]
      public 'ReplyTo' => int 18
      public 'ReplyFrom' => int 22

I need this kind of array to use the data:

array (size=2)
  ReplyFrom => 
    array (size=1)
      0 => ReplyTo
      1 => ReplyTo
  21 => 
    array (size=1)
      0 => 18
      1 => 19

I have tried this so far but it always overrides the value instead of adding it.

foreach($stm->fetchAll(PDO::FETCH_OBJ) as $quote) {
    $test[$quote->ReplyFrom] = array($quote->ReplyTo);
}

My question is: How can I append the values with the same key to the array instead of overwriting it?