使用laravel 5.2中的数据透视表对记录进行排序

Question

I need some help regarding sorting the record using pivot table in laravel 5.2.

models:

Product belongsToMany PremiumAdCategory
User hasMany Product

below is the relationship in Product model:

public function premium_ad(){
    return $this->belongsToMany('App\PremiumAdCategory', 'premiumadcategory_product', 'product_id', 'premiumadcategory_id')
                    ->withTimestamps()
                    ->withPivot(['coupon_id', 'total_amount', 'discount_amount', 'net_amount','payment_method'])
                    ->latest('pivot_premiumadcategory_id');
}

below is pivot table schema

Schema::create('premiumadcategory_product', function (Blueprint $table) {
    $table->increments('id');
    $table->integer('premiumadcategory_id')->unsigned();
    $table->integer('product_id')->unsigned();
    $table->integer('coupon_id')->nullable()->default(0);
    $table->string('total_amount');
    $table->string('discount_amount');
    $table->string('net_amount');
    $table->string('payment_method');
    $table->timestamps();
    $table->foreign('premiumadcategory_id')->references('id')->on('premium_ad_categories')->onDelete('cascade');
    $table->foreign('product_id')->references('id')->on('products')->onDelete('cascade');
});

I want to fetch all the records of a particular user from product table and sort them based on the pivot table. this is the code i use.

$product = Product::whereUserId(4)->with(['premium_ad' => function ($query) {
    $query->orderBy('pivot_premiumadcategory_id', 'DESC');
}])->get()->toArray();

But i can't able to sort the code, below is the dump query.the output of the query doesn't contain the sorted code.

select `premium_ad_categories`.*, `premiumadcategory_product`.`product_id` as `pivot_product_id`, `premiumadcategory_product`.`premiumadcategory_id` as `pivot_premiumadcategory_id`, `premiumadcategory_product`.`coupon_id` as `pivot_coupon_id`, `premiumadcategory_product`.`total_amount` as `pivot_total_amount`, `premiumadcategory_product`.`discount_amount` as `pivot_discount_amount`, `premiumadcategory_product`.`net_amount` as `pivot_net_amount`, `premiumadcategory_product`.`payment_method` as `pivot_payment_method`, `premiumadcategory_product`.`created_at` as `pivot_created_at`, `premiumadcategory_product`.`updated_at` as `pivot_updated_at` from `premium_ad_categories` inner join `premiumadcategory_product` on `premium_ad_categories`.`id` = `premiumadcategory_product`.`premiumadcategory_id` where `premiumadcategory_product`.`product_id` in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) order by `pivot_premiumadcategory_id` desc

Please take a look and let me know what i am missing. Thanks in advance.