Collecting the best books mentioned on hacker news, reddit and other places
My two biggest shelf pulls are Real-Time Rendering  by Akeine-Moller, Haines and Hoffman. 3D Math Primer for Graphics and Games  by Dunn & Parberry
I also read a ton of presentations and papers. Highly recommend the famous PBR SIGGRAPH course notes , especially the intro to light & physics by Naty Hoffman. GPU-Driven Rendering Pipelines  is another recent goodie.
Don't think there is any such book specifically for UE and its source. But there's a lot of good books on realtime rendering and graphics programming in general.
GPU Gems, Shader X and GPU Pro are good series for learning specific graphics programming techniques.
For a general game engine overview: Game Engine Architecture by Jason Gregory (Naughty Dog)
Game Programming Patterns: https://www.amazon.co.uk/Game-Programming-Patterns-Robert-Ny...
Realtime rendering overview: https://www.amazon.co.uk/Real-Time-Rendering-Third-Tomas-Ake...
It's fun to explore the source though, and NVIDIA has some cool experimental branches of the engine with their stuff integrated. https://github.com/NvPhysX/UnrealEngine
I'm not gavin, but I like the style of http://www.amazon.com/Math-Primer-Graphics-Development-Editi... and recommend it to anyone getting started. You don't need any more than high-school maths and some familiarity with programming in your language of choice (though the example code is in C++).
If you're just getting started and want a low-impact way to prototype things try WebGL! (I'm an unbiased WebGL contributor and user).
OpenGL seems to be the way to go on most platforms; DirectX is probably easier on Windows. There are libraries that take care of the platform-specific code for you if you don't want to write your own that are quite good: SDL2, glfw, etc.
I recommend http://www.amazon.com/Math-Primer-Graphics-Development-Editi... when you're ready