Before you dive in to hard problem, don't you need to create certain basic CS foundation? For example, can you start working on database kernel without studying even basic algorithms like red-black trees and B-trees? It would be interesting to know what are your go-to books for CS and other resources you use to dive in to particular topic? For example, do you start with Arxive or plain old web searches or taking cues from expert you know in person or look for books on Amazon or attend conferences?
Another thing is that creating algorithms are easy, especially the ones with bugs :). The difference between a joe-the-programmer creating an algorithm and a CS PhD creating an algorithm is that later produces a proof of correctness and performance while former relies on intuition for same thing. This is the hardest and most critical part of being a CS researcher. It would be interesting to know your approach for acquiring these skills.
Yes and no. You do need to have broad enough experience with programming that you are comfortable with the basic implementation algorithms even if not up to speed on the nuances of them. However, a modern database kernel typically has few or no B-trees or red-black trees in it, using more specialized and purpose-built algorithms instead. Probably one of the most important lessons I learned when I first started poking at database kernels is that the algorithms in the literature were demonstrably not being used in commercial databases because the behavior did not match what you would expect. The algorithms you learn in a CS curriculum are often not really correct for any real system built with a high degree of skill.
I've only ever owned a few CS-related books (e.g. Li and Vitanyi). Trawling search engines for CS papers and references to CS papers with potentially useful information is how I did most of my work. Honestly, Wikipedia is a good starting point to find search terms but you still have to follow threads in the literature that are poorly or non-existently documented on the Internet. There are important areas of literature that still have very limited presence on the searchable web. This is a huge bias I see with many people fresh out of school -- if it is not on the web it doesn't exist.
On your last point, that is the most difficult part. To be frank, most PhDs in computer science do not have the skills to successfully design new algorithms either. Oddly, chemical engineering provided a very good conceptual framework for thinking about this; there are no pure mathematical solutions and you optimize complex systems in part by understanding every knob in a holistic context. One of the things I've noticed about most people with a computer science education is that they make a large number of implicit assumptions about the nature of computational and algorithmic systems that are not required and sometimes not correct, often without realizing it. Most of the algorithm work I've done amounts to violating implicit but foolish assumptions in the orthodox literature that no one had seriously examined in decades.
It is not just computer science, in most fields of endeavor the vast majority of things are done the way they are because that is the way they've always been done as set forth by a smart person a long time ago. The underlying assumptions and implicit constraints are rarely reexamined. I found a lot of upside to studying theoretical knobs that had been essentially frozen in literature since at least the 1980s.
Another thing is that creating algorithms are easy, especially the ones with bugs :). The difference between a joe-the-programmer creating an algorithm and a CS PhD creating an algorithm is that later produces a proof of correctness and performance while former relies on intuition for same thing. This is the hardest and most critical part of being a CS researcher. It would be interesting to know your approach for acquiring these skills.