Nicholas Peterson

A workflow diagram is to a process, what a map of the airport is to the airport. It's meant to be accurate about the parts we care about and omit the details we don't.

Real-time data is cool, but sometimes overkill. Ask yourself, do I have data actually coming in all the time? Or does it get supplied at some fixed interval? A continuous computation full of neat math isn’t very useful if you get a big xml file once a day. Also, make sure you understand some reasonable use case and build with it in mind. A real-time data warehouse project is a really great way to set a few million dollars on fire.

most of the advice here seems to just be restatements of the prompt. Here is some actual advice for anyone with the bad luck to be pursuing this kind of a project. First, figure out what questions the business has about their data. If the answer is, "We're not sure", you're about to spend a lot of time and money on a complete boondoogle. Figure out some decent questions or don't proceed. Once you have some questions, try and answer them without building a new data warehouse. Most of the time you can. Sometimes you can't. When you can't, analyze why. Is it performance, missing data, or something else? Use those answers to determine what would be useful to the business. Also, don't waste money on some special ETL tool, just write code.

You're not going to design a correct schema upfront, since it's the time where you know the least about what actually needs to be built. Focus on your ability to evolve the schema as requirements shift or your understanding of them changes.

Change is the only constant, but keep in mind that nothing is new in this world, even if the words we use for them are. What distinguishes "data science" from statistics, or "programming" from logic? Focus on what's causing a new trend and why people think it's a good tradeoff.

Perpetual learning is good in general. That said, I'd avoid spending too much time learning about new features and different dbms's. Instead, focus on improving your knowledge of the fundamentals that make up the database management system (dbms) you currently use. How are transactions handled (if at all...)? How is the data stored and retrieved (b-trees, key/value, ...)? Solid fundamentals help build a mental model to quickly understand what trade-offs different dbms's are making. They'll be useful your whole career.