I've recently inherited a project where the database was modeled interestingly (i.e. lack of some index and foreign key definitions). The project uses GORM and from what I can tell, those Models have tags which define everything correctly.
I can't think of a reason why using the ORM for database "modeling" wouldn't work. The closest I can come is performance, but at the scale this would need to run, the point seems moot. Are there any downsides to this running things this way?