Understanding the physics of galaxy formation is arguably among the most complicated problems in modern astrophysics. Recent cosmological simulations have demonstrated that feedback by star formation, supernovae and active galactic nuclei appears to be critical in obtaining realistic disk galaxies, to slow down star formation to the small observed rates, and to move gas and metals out of galaxies into the intergalactic medium. However the particular physical processes underlying these feedback processes still remain elusive. In particular, these simulations neglected cosmic rays and magnetic fields, which provide a comparable pressure support in comparison to turbulence in our Galaxy, and are known to couple dynamically and thermally to the gas. I will present our recent efforts to model cosmic ray physics in the cosmological simulation code AREPO and demonstrate that cosmic rays matter on all scales relevant for galaxy formation. Diffusive shock acceleration of cosmic rays at supernova remnants modifies the expansion velocity of the shock wave and the inclusion of cosmic-ray physics in models of the interstellar medium allows for self-regulated outflows from the star-forming disk. I will also present global simulations of galaxy formation that couple cosmic rays to the magneto-hydrodynamics and demonstrate how powerful galactic winds can be launched which reduces the available amount of gas for star formation. Finally, I will discuss the non-thermal radio and gamma-ray emission of Milky-Way like galaxies and how the next-generation instruments such as SKA and CTA can be used to infer properties relevant for galaxy formation.