I've never had an issue with this when establishing silvopasture - in fact, when compost is fungally-rich, the mycorhizzae can actually help tree roots get established faster. But if you're planting native trees, I'm sure they'll do just fine without any compost at all.