Collecting the best books mentioned on hacker news, reddit and other places
My two biggest shelf pulls are Real-Time Rendering  by Akeine-Moller, Haines and Hoffman. 3D Math Primer for Graphics and Games  by Dunn & Parberry
I also read a ton of presentations and papers. Highly recommend the famous PBR SIGGRAPH course notes , especially the intro to light & physics by Naty Hoffman. GPU-Driven Rendering Pipelines  is another recent goodie.
Part of why it's difficult is that there are a tons of concepts belonging to 'real-time rendering' which three.js is built on top of. These are things like how a 3D scene is projected from a particular viewpoint using a 'camera', how lighting works, the split in vector/raster techniques for representing 3D shapes vs their surfaces, the role of shaders in modern GPU architectures etc. I'd recommend the book 'Real-time Rendering': https://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akeni...
Edit: I'd also add: I wouldn't bother learning much OpenGL/WebGL to begin with (except shader programming in GLSL, since there's no good alternative abstraction for that). If you end up liking working with 3D graphics, go back and learn some about it since it'll help you understand performance concerns better—but meanwhile, knowing it is just an optimization you don't need yet. It's true three.js is built on top of it, but the significant principles you need to use three.js effectively fall under real-time rendering, not OpenGL.
Don't think there is any such book specifically for UE and its source. But there's a lot of good books on realtime rendering and graphics programming in general.
GPU Gems, Shader X and GPU Pro are good series for learning specific graphics programming techniques.
For a general game engine overview: Game Engine Architecture by Jason Gregory (Naughty Dog)
Game Programming Patterns: https://www.amazon.co.uk/Game-Programming-Patterns-Robert-Ny...
Realtime rendering overview: https://www.amazon.co.uk/Real-Time-Rendering-Third-Tomas-Ake...
It's fun to explore the source though, and NVIDIA has some cool experimental branches of the engine with their stuff integrated. https://github.com/NvPhysX/UnrealEngine
Maybe my expectations are too high, but that's just a collection of random comments loosely related to graphics. It's a terrible introduction, and it doesn't give any concrete information on how to get started.
My advice for people interested in graphics would be to jump in and get started with a tutorial on the web and then get a book or two to learn more. My recommendations would be:
Introduction To Ray Tracing - It focuses on ray tracing, but many of the topics (vectors, matrices, shading, view transforms, etc.) are applicable regardless of the actual rendering method, and it does a good job explaining all of it.
Real Time Rendering - This also covers all the math, but focuses on rendering at a higher level and covers more ground.
Finally, Computer Graphics: Principles and Practice - This is a great general reference for more specific topics. It's like Real Time Rendering expanded to cover more topics and going into more depth. Not a beginner reference, but great to have when you need more information on something.
While F&VD does a great job covering many different aspects of image generation, for this kind of work Real-Time Rending is probably more useful. It is firmly grounded in a world of programmable GPUs (shaders & the like), and wastes little time talking about rendering techniques that are not relevant to modern graphics hardware.
"Real-Time Rendering" also has a newer edition: http://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akenin...
Also, if people are looking for an even bigger list of graphics books the Real-Time rendering blog has a comprehensive list:
Sigh, the canonical reference on 3D graphics "in software" has been the Foley and Van Dam tome "Computer Graphics: Principles and Practice in C" , followed closely by "Principles of Interactive Computer Graphics"  by Newman and Sproul. Followed closely by the Graphics Gems series, and then Watt's eponymous "3D Computer Graphics"  and "Real Time Rendering"  by Moller and Haines.
All wonderful texts and can tell you everything you want to know about doing 3D graphics in software. They won't help at all (generally) for GPU based graphics sadly.
Hey, glad I was able to give you some useful information! If you ever find yourself in Toronto, say, for an indie game jam (http://www.tojam.ca/home/default.asp hint hint), you can buy me a beer ;)
If you're looking for something lower level, the book you probably want is the white book - Computer Graphics: Principles and Practice. I have the 2nd edition from 1992, which is the still the standard intro graphics textbook for many CS departments. Though Amazon says there will be a 3rd edition coming out at the end of this year!
Another highly recommended book (also recommended in another comment here) is Real-Time Rendering, but I've only used bits and pieces from this one, so I don't know how good it is for folks just starting out. Still probably one you'll want to add to your shelf if you continue on in the field.
Oh, and also also, head over to YouTube with some snacks and a drink, sit down, and watch the weekly Overgrowth game developer videos from Wolfire Games. It's both inspirational to see what other people are doing, and a great demo of concepts that you'll read about in GEA, such as animation blending: http://www.youtube.com/playlist?list=PLA17B3FAA1DA374F3&...
Any book on computer graphics would be a good start for the graphics side.
I like 3D Computer Graphics by Alan Watt as an introduction and overview http://www.amazon.co.uk/3D-Computer-Graphics-Alan-Watt/dp/02...
and Real-time Rendering for more advanced subjects http://www.amazon.co.uk/Real-time-Rendering-Tomas-Akenine-Mo...
The session from Assembly 'The Basics of Demo Programming' is online and may also be useful http://www.youtube.com/watch?v=TbcZyAO6K7c or have a look at The Demo Effects Collection http://demo-effects.sourceforge.net/
WebGL Browserscene: Creating demos on the Web http://www.youtube.com/watch?v=XZLqwXdXjqY
Deconstructing a browserscene demo http://www.youtube.com/watch?v=pOjWOA-iPnA
I have some more bookmarks on pinboard http://pinboard.in/u:z303/t:demoscene/