I thought of this 7 years ago and I still think it rings true. In my experience, school projects were tightly scoped and had things like a basic build system and boilerplate runnable code (e.g. main) already provided. Maybe even some tests. It trains a specific muscle in a certain way.
With personal projects, you do all of this yourself and incidentally also train the equivalent “stabilizer” muscles. Things like: making the repo, writing the main and boilerplate code, setting up unit tests, setting up CI, setting up dependencies and their management.
And in my experience, it’s all these “stabilizer” activities that define what it means to be a senior engineer. Junior engineers work within an existing project; senior engineers own the entire thing, including these “little” (critical) parts.
The point here isn’t to necessarily criticize CS education; there are good, practical reasons for setting up such project structure to ensure all students have a good experience, given the time constraints. The point is that students should be wary of not doing any personal projects at all throughout a CS education. Maybe it would be possible so students don’t have to do this in their free time, and build it into some kind of flexible course. I suppose this is the point of final capstone projects, but I think senior year is far too late to be doing this for the first time.
I see a strong parallel between creativity and gardening. Like a fruit, growing from seed to ripe, ideas do the same.
Idea seeds are those vague, half-formed thoughts that quietly appear in your mental space. They occupy a strange state where you are aware of them, but would struggle to write them down or explain to someone else coherently. That is the defining characteristic of an idea which is not mature yet.
Slightly more mature ideas are in the growth phase. They form of the idea has started to emerge, and you can half-express it, but there might be logical errors as you still haven’t fully thought it through yet.
Ripe ideas are fully developed and can be coherently expressed with the fewest words necessary.
Ideas can grow along this path explicitly or implicitly. You can explicitly grow ideas by sitting down and trying to write them. Or they can grow implicitly in the background as you live your life and expose yourself to experiences and other ideas.
Sitting down to write an idea is like trying to harvest it. It’s easiest and works best when the idea is ripe — the idea will almost pour out of your brain onto the page in this case. If you try to harvest an idea before it’s ripe, it will be harder.
The analogy might break down slightly here. With ideas, the act of trying to harvest them actually accelerates their journey towards ripeness. Trying to express an idea forces you to understand it better and clarify it. This is one way of growing an idea: the explicit way.
Ideas can also grow implicitly. They can quietly sit on your mental back burner as you live your life and expose yourself to experiences and other ideas. Maybe another idea acts as a bee that pollinates the first idea and together they form a complete, ripe idea. But in order for them to be able to safely sit on the back burner as long as it takes, they should be still captured in some kind of way, or else you’ll probably forget them.
Note that an idea’s state of being written down vs in head, and public vs private are each orthogonal qualities. Ripe ideas aren’t the only ones that can be written down and published. Idea seeds can still be attempted to be written down (although it will be hard), and also published.
I first wondered this question as a confused 17 year-old, considering options for university — this blog post is for that kid and people like him. It’s my opinion on this after working professionally in both fields and earning a degree in one them (CE).
I think tracing each back to its roots is a good way to gain an intuition for the difference. Computer Science is ultimately derived from Mathematics, while Computer Engineering derives from Electrical Engineering.
The boundary is blurry, but CS is classically more concerned with topics like algorithms & runtime (“Big O”), data structures, and the theory of computation (“What kinds of problems can we solve with computers, what kinds can’t we?”). You’ll do many more formal proofs in CS.
CE concerns itself more with how digital computing systems work in the real world, starting from foundations in electronics (i.e. transistors), to digital logic, building up towards embedded systems, the hardware-software interface, and computer architecture. The latter is where things get blurry and bridge to the “lower levels” of CS.
In practice, most practitioners of either have some awareness of the other, although I suspect (without evidence) that CE’s tend to know more about CS than vice-versa, on average. However I also expect that CE’s tend to write “worse” code than CS’s, on average — “writing good code” and “knowing CS” do not necessarily correlate. 1
So which should one choose?
I can only offer this scattered set of anecdotes and advice:
I knew many more CE majors that switched to CS, than the other way around. (Mostly people that just wanted to become “programmers” and realized that circuits & electronics are not for them. I’m not sure if they enjoyed the theory courses more, though.)
I have heard and personally agree that it’s easier to go “up” the stack than “down”.
Don’t give any experience building computers or doing IT support too much weight in the decision — those topics are neither CE nor CS and rather fall into the field of “IT” which is largely separate. You might be surprised how possible it is to study either CE or CS, work in these areas professionally, and not know how to physically build a computer. But in general, building a computer falls closer to CE than CS — but you will be learning how to design (some of, at a very high level) these lego pieces you are putting together.
Personally I’ve always been foremost curious “how computers actually work” and CE has served me well.
Lastly, what about “software engineering?”
Software engineering is an even newer field, derived from Computer Science, but describes the job descriptions of many (if not most) people that study either CS or CE. My view of it is that a degree in it focuses on the most practical elements of CS, focusing less (or not at all) on the theory. I don’t expect much of CE is covered it in, except for maybe a cursory overview of digital logic and computer architecture. But frankly, I don’t know any software engineering majors and am not qualified to really speak on this.
How did you choose between them?
Not rigorously. My decision was mostly based on the presence of other “engineers” in my life and a friend that studied CE. These were not good rationale for the decision, but I lucked out and had a good outcome anyway. I think I would have been fine either way, and naturally gravitated up or down the stack to the point I’m at now. Being CE did allow me to take some very cool classes in college like a very modern, practical compilers course using LLVM (unlike the more theory based compilers course taught in the CS school) (which had a direct impact on my ability to get a great job after) and a fun embedded systems class.
I was learning about the different types of RAM recently (e.g. SRAM and DRAM) and it occurred to me that “computer memory” is just an abstraction. This is obvious once you think about it but I think as a programmer, it’s very easy to not realize this. The idea of memory as a linear array of bits is an abstraction created and implemented by an electrical device.
Most programmers when they think of memory are thinking of virtual memory, which is a completely different abstraction. While it’s also a linear array of bits, the abstraction is created by the operating system and lives at a higher level.
One level below, the abstraction the operating system itself uses — “physical memory” — is the one I’m talking about, created by a a set of electrical devices connected to the CPU with wires (the memory bus).
I’m projecting without any basis, but I presume the reason so few programmers think of memory as an abstraction is because the abstraction is so strong. Nearly all of the time, it “just works” — you write bits and read them out later.2 The abstraction can leak slightly during programming disciplines that require awareness of low level details like the memory hierarchy and cache coherence (i.e. lockfree) but this is a leak of the abstraction of the memory hierarchy. The core abstraction of physical memory stays in tact — for example, programmers never need to be aware of the internal refresh mechanism of DRAM.3
Of course, you can go infinitely far with this — it’s turtles all the way down.
Sometimes, you just need to be willing. No special skills or particular competence necessary — simply just being willing to do something others can but won’t do.
I learned and have profited from this realization at work in the past year. A certain set of important tasks on my team are somewhat hands-on in nature and not compatible with remote work. Our colleague that usually does these left the team. I stepped up and filled this gap.
The work is not particularly difficult — my teammates are smarter than me and could surely do it also. But I’m the only one that was willing and able to sacrifice remote work and commute every day. So I’ve been doing it.
Although it’s not the hardest, the work is nonetheless very important — critical, even. So I’ve been getting a lot of credit, brownie points, and even a raise from doing it.
For the last three years, I’ve invested into learning to write. The main reason: strong writing skill is generally a shared quality among all the greatest thinkers, leaders and generally wise people of history.
How has it gone?
Really well. It’s paying off in many ways, and I only feel more inspired to keep doing it. I find that it gets more fun then more you do — the more ideas you have and the easier and faster it is to express them.
It’s paid off in my personal life with my offlinemark project. My writing (and general public creative efforts) has reached some unexpected places and put me in touch with some very interesting people. My most prized accomplishment is being cited in the Linux Kernel security mailing list.
It’s paid off in many ways at work. As a software engineer, I do a lot of writing at work, and I pride myself in having great written communication there. I write a lot of documentation, hold it up to a high standard, and people notice. I get points for this because most engineers don’t like writing docs, but I’ve trained myself to enjoy it.
Beyond these, modern life generally involves a good amount of written communication. I’ve found that putting in these reps has simply made this part of life a bit easier.
P.S. Something else I’ve observed is that I find myself having more ideas and being more aware of them. Whether they’re good or insightful is a different matter, but I’d like to believe that over time they’ll get there.
This has become crystal clear to me at my current job. As much as I’ve had success teaching myself things about computers, nothing, nothing, beats spending time with experts and learning from them.
I had never truly experienced this before and was shocked at how much I was able to learn, in such little time — about C++, git, Linux, and more.
It can be a significant price to pay, but nothing beats literally becoming an employee at a company with experts in a field you care about and joining their team. This is ideal from a learning perspective because as a team member, the experts become invested in your growth and are incentivized to transfer knowledge to you. You just need to be ready to receive it.
Once you’ve tasted this, this truth is also bittersweet. You can only work at so many places. If you have many interests, you now have to live with the fact that for most areas, your learning will be stunted relative to what it could be.
My deep gratitude to the experts in my life: STK, DIR, CSK, MKL, RYB, MZA — thank you.
FIFOS (“First-in-first-out”s aka Named Pipes) are are somewhat obscure part of Unix based operating systems — at least based on my personal experience. I’ve been using Linux for ten years and simply have never needed to interact with them much.
But today, I actually thought of a use for them! If you’ve ever had a program which had human readable output on stdout, diagnostics on stderr, and you wanted to observe the two as separate streams of text, without intermingled output — then FIFOs might be for you. More on that in a bit.
Youtube version of this blog post:
FIFOs 101
As background, FIFOs are one of the simpler IPC (Interprocess Communication) mechanisms. Like normal (un-named) pipes, they are a unidirectional channel for data transmission. Unlike normal pipes, they can be used between processes that don’t have a parent-child relationship. This happens via the filesystem — FIFOs exist as filesystem entities that can be operated on with the typical file API (open, close, read, write), also similar to normal pipes.
The simplest experiment you can try with them is to send data between two processes running cat and echo.
First, create the FIFO:
$ mkfifo my_fifo
Then, start “listening” on the FIFO. It will hang waiting to receive something.
$ cat my_fifo
In a separate terminal, “send a message” to the FIFO:
$ echo hello > my_fifo
This should return immediately, and you should see a message come in on the listening side:
$ cat my_fifo
hello1
FIFO-based logging, v1
Today I was working on a long-running program that produces a lot of human reading terminal output as it runs. There are some warning conditions in the code that I wanted to monitor, but not stop execution for.
I could print to stdout when the warning conditions occurred, but they’d get lost in the sea of terminal output that normally comes from the program. You can also imagine situations where a program is writing structured data to stdout, and you wouldn’t want a stray log text to be interspersed with the output.
The typical solution is to write the logs to stderr, which separates the two data streams, but as an exercise, we can try writing the logs to a FIFO, which can then be monitored by a separate process in another terminal pane. This lets us run our program with its normal output, while also monitoring the warning logs when they come.
In python, it would look like this (assuming the fifo is created manually):
import time
fifo = open('logging_fifo', 'w')
def log(msg):
fifo.write(msg + '\n')
fifo.flush()
i = 0
while True:
print(i)
i += 1
if i % 5 == 0:
log(f'reached a multiple of 5: {i}')
time.sleep(.1)
In a separate pane, we can monitor for logs with a simple:
$ while true; do cat logging_fifo ; done
reached a multiple of 5: 5
reached a multiple of 5: 10
reached a multiple of 5: 15
reached a multiple of 5: 20
reached a multiple of 5: 25
Note that the listener process must be running or else the writer will block!
FIFO-based logging, v2
An even more elegant and UNIX-y way to do this would be to still write to stderr in our writer application, but use shell redirection to redirect stderr to the fifo.
import time
import sys
i = 0
while True:
print(i)
i += 1
if i % 5 == 0:
print(f'reached a multiple of 5: {i}', file=sys.stderr)
time.sleep(.1)
Which you then invoke as:
$ python3 writer2.py 2> logging_fifo
This yields the same result, just is more semantically correct.
I’m 10 years into my tech career and yet I constantly feel so, so stupid.
It’s entirely self-originating; not because others are mean to me. (Ok maybe partly also because I’m lucky to be around smart people).
Why do I feel this way, despite mountains of evidence to the contrary — that I’m not stupid, and know a thing or two about what I’m talking about? Is my self esteem really that bad?
I think it’s also because I’ve been learning so much. The more I learn, I more I realize I don’t know. And I posit that the weight of a known unknown feels disproportionately bad to the relief of converting it to known known.
So, the equation looks like:
All Knowns = Known Knowns + Known Unknowns
Perception Of Intelligence = Known Knowns / All Knowns
(Smaller = Less Intelligence = More Stupid)
This makes sense to me. I think I accumulate known unknowns at 10x the rate of known knowns — it takes so much more energy to learn something, than to learn merely of the existence of something.
