The rise of social media-savvy startups like Ulta and continuing success of beauty specialists like Sephora have come at the expense of the traditional anchors of the beauty business, department stores.