Yiannopoulos cites moral and religious objections, while insiders report broader issues within Kanye West's company.