Is it worth taking an Investment Banking job with lower pay vs. a high-paying finance job in a conglomerate?

The investment bank is the best bank in the country, but is smaller than the conglomerate. I am very passionate about my finance and capital market skills, which I use in my current corporate finance job. Investment Banking may open up more opportunities for me in the future, while working in the conglomerate is a steady less-dynamic job. I want more excitement, but do I have to sacrific my currrent pay by 50%?