I’ve experienced this mistake as both an IC and a manager. As an IC, it’s a mistake I made many, many times early in my own career. As a manager, I made the mistake of not recognizing and fixing it in the engineers I’ve managed.  

Correcting this habit is the most common feedback I now give, so I thought it might be good to write out how it happens and how it can be avoided.

The typical scenario goes something like this:

1. An engineer takes on a relatively big / important project, usually building a feature or piece of infrastructure.  The bigger the project, the likelier the situation to follow this pattern.
2. The engineer goes off on their own and starts by trying to figure out how to build the thing.  Often this is verbally framed as “exploratory work”, “designing”, or something else without a concrete deliverable or timetable.
3. Throughout this initial period there are standup updates of the form “I’m exploring X, or working through problem Y, but should have something for others to take a look at soon”
4. Finally, after several weeks, the engineer shares an update, and one (or many) of the following bad things transpire: 
   1. They have started building the thing and are about to send out a really big initial PR for it (oy!).  For reasons I talk about here, you always want to send the smallest PRs you can, and generally for big features you should start with a design or prototype, not a PR.
   2. They have actually been solving the wrong problem because the feature wasn’t well specified (which is normal, and speccing a feature out more completely is often part of building it).   E.g. they assumed that their feature would work way X, but the rest of the team was thinking it would be Y, so the design or code isn’t right.
   3. They have been trying to figure out the solution to some very specific part of the problem, which actually might not be important at all because we could easily trim back the product requirements to avoid it.  (E.g. they have been devising a locking solution for something that doesn’t even require concurrency).
   4. They have designed a very involved, complex, overall solution which if they had got feedback on earlier we might have redirected to a totally different spot.  E.g. the solution has multiple new microservices where there could have just been a library.

These outcomes are all bad for the same two reasons:

1. There was a huge cost in time without a lot of forward progress on the project.
2. We end up in a position of dealing with the Sunk Cost Fallacy where it hurts to abandon the work already done in order to get back on track – if we aren’t careful, we’ll end up shipping the wrong thing because we didn’t want to swallow the sunk cost.

Why does this happen?

For early career engineers, it often happens because they lack practice working on teams.  They train in school environments where they do classroom projects on their own, or work on long-term intern projects in a silo.  They are used to having to figure it all out up front, submit it for review, and then get some sort of result, like a grade or return offer.

For more senior engineers, it can happen because they like to work on their own and may be overconfident in finding solutions.  It can also happen if the team culture is toxic and engineers fear getting criticism early in the design process.

Building software on a team shouldn’t work this way though.  When you work on a team, you shouldn’t be in competition with other schoolmates or interns or teammates – you are working cooperatively to ship the best possible product, as quickly as possible.  And there’s a huge advantage in leveraging the team’s collective wisdom to build better and faster.  

Early career engineers don’t always know this – they need to be taught it.  Senior engineers can be overconfident and need to be reminded of it.  And as a manager, you need to be on the lookout for this flawed approach in order to keep your team productive.

To sum up, just as a product is likely to be best if it is developed iteratively with users, the work of an engineer will be best if developed iteratively with the product team.

Concretely, if you’re an engineering manager, this means the following:

1. Always encourage engineers to show their work as quickly as possible – an engineer on a project should never go more than a week without showing something, and usually it should be more like a day.  
2. The thing they show could be a design doc, a PRD, a prototype – anything that puts the team in a position to give meaningful feedback as early in the development process as possible. 
3. If the engineer is unclear on a first deliverable, help them decide on what (e.g. a design doc) and by when (e.g. in two days).  And so on for the next deliverables.
4. Encourage engineers to get something end to end launched internally as quickly as possible.  In other words, rather than developing the feature in a waterfall fashion, where the engineer does the complete data model, followed by the complete controller logic, followed by the complete view, encourage doing just a partial data model, controller, view that works end to end for some small part of the feature.  This will give everyone a better sense of the feature from a user’s perspective and will also be the easiest way to see how the pieces of code fit together.
5. Development should always be done behind feature flags so that features can be checked in incrementally.
6. Demo, demo, demo – continue to demo the feature as it gets fleshed out.
7. Separately, you should encourage everyone on the team to give constructive feedback, and be on the lookout for any feedback that is overly negative or toxic, since that discourages developers from working iteratively.

Hope this is helpful!!

One of the hardest and most important parts of starting and growing a company is hiring a great team of engineers.  It’s probably the most important thing you will need to get good at as a founder of a company, and, it’s one of those things that you’ll need to figure out on your own.

The reason it’s so important – in case it’s not obvious – is that the caliber of people you bring aboard is the biggest determinant of success at an early stage company.  Nothing will sink you faster than hiring (and retaining) B or C level talent. It will slow down the development process, lead to crappy product, cause management overhead, make joining you less appealing for A players, and is generally the worst thing you can do.

It’s easy to do hiring wrong (I’ve done it wrong).  Some of the dangers include

• Hiring the wrong people – which leads to having to fire them (or worse, retaining them), which is painful
• Not hiring enough people – which slows your velocity and can be deadly from a business perspective
• Wasting a ton of your time – you can easily spend all your time sourcing, interviewing, etc and not enough time building the actual business. But expect to spend a bunch of your own time on it at least to start
• Wasting a ton of your team’s time – this happens when you bring unqualified candidates too far down the funnel and other people get sucked into interviewing and grading bad fit candidates
  
• Spending a lot of money – recruiting is a huge business and once you start hiring you will be inundated with offers from recruiters and recruiting platforms to give you your business

Startups vs. Established Companies

As an aside, if you’re at Google or a place like it, this section won’t be as useful.  Google has unique advantages compared to a startup in terms of name-brand, amount of inbound applications, what they can pay and offer in perks.  

What Google or Facebook managers call “hiring” is actually a completely different thing from startup hiring.  It’s really “selection” from a highly curated applicant pool, without having to worry about compensation, applicant interest, sourcing, etc.   It has more in common with what the admissions committee at Harvard does. Most of the quality applicants come to you, and you sort, filter and select, and then, if someone meets the bar, you have to sell them on taking your (usually very competitive) offer.

Startup founders need to operate higher in the funnel, and typically have a much tougher pitch.  They need to go out and find quality engineers who already have other, more secure, higher paid jobs, cut through the noise to convince them to talk, sell them on a dream, convince them to do a real interview process (which might lead to their rejection), and ultimately hop ship to work on a much riskier, usually lower paid venture.  It’s hard.

I still have a lot to learn on how to do hiring well.  But I can share some of what I learned at my last company where I was really proud of the team we put together.

Hiring is a sales process

If you are at a startup, you should view hiring as a sales process.  You are selling the opportunity of working at your company. Engineers are the buyers, and it’s very much a buyers market.  The best candidates need to be sold – do not expect them to come knocking on your door.

You should think of bringing talent onto your team in terms of a sales funnel.  This particular funnel has three steps: 1) generating interest, 2) evaluation of fit, 3) offering and closing.

At the top of the funnel, there is a combination of marketing / branding, inbound traffic, outbound outreach and sourcing, referrals, recruitment, and other ways of getting candidates interested.

Once candidates are interested, they progress into an evaluation phase.  You should think of this in terms of finding whether the opportunity is a “fit.” Is the engineer right for your company and is your company right for the engineer.  It needs to be both.

Finally, for candidates you believe are good fit, you make an offer and go into closing mode to try to get them to sign.

What I describe below is what I think of as the “standard process,” which I used with success at my company.  I’m sure there are a lot of other ways of approaching this (and I’d love feedback on how to improve it).

Who do you want to hire & why should they work at your company?

You’ve got a startup that no one has ever heard of.  Your task is to get high-quality engineers excited to come work there.  You can’t compete in terms of salary or job security. How do you do it?

You need to start by deciding who you are looking for, the more specific the better.

Here’s what I care about:

General intelligence.  I don’t care about experience with particular technologies; I want the smartest people I can find. (I recently had someone I respect disagree here and favor hiring specialists at very early companies in order to cut out ramp up time – I don’t quite buy that strategy though)

Technical thinking.  I don’t require a CS degree (I don’t have one), but I do favor candidates who have demonstrated an ability to think rigorously.  For me this could mean a STEM degree, a philosophy or economics degree, past technical experience at a job, impressive side-projects, etc.

Team players.  No assholes.  I want people who put the good of the team and the product above their personal ambitions.

Product-first mindset.  See the section on Product-first vs. Code-first.  I have some questions below on how to suss this out.

For me this leads to a rubric like:

• EducationUndergrad degree from a good school, or self-taught ok if a strong track record
• Experience: worked at a place with a known rigorous hiring process (google, fb, or a smaller, well known startup like Oscar, Paperless Post, Betterment, Vine, etc)
• Any undergrad major fine, but STEM a bonus
• Enough CS knowledge to pass a Google-style interview

On the flip side, you need to consider what your target engineers are going to care about and how can you stand out.

As mentioned, you probably aren’t going to be able to compete based on:

• Name recognition.  You aren’t Google, Facebook or Slack.
• Salary and benefits.  Google pays its top engineers 7-figures and starting engineers good six figures.  You probably can’t.
• Perks.  Free catered food, on-site massage, volleyball courts, renting out the MOMA for holiday parties.  Maybe if you are realllllly well funded, but probably not.
  
• Number of users.  You aren’t scaling out to millions of users right at the start, so by that measure the impact of your product will be lower.

However, you have some advantages compared to bigger companies.

• Velocity.  Engineers at big companies tend to be frustrated by how long it takes to develop, how many meetings they have, how small of an area that are able to work on, how hard it is to launch, and so on.  At your startup you can (and must) move much faster. You should lean into this when pitching engineers.
• Product ownership and input.  Engineers at your small company can own a much bigger piece of the product.  I always sell this pretty hard because it also selects for the type of engineers I want – people who care about the customer and user experience.
• Culture.   Engineers at small startups have a chance to really shape the culture of the team.  I emphasize how smart everyone on the team is, how we have no assholes, how we self-direct and iterate on our process as a team, how we take ideas from everyone.
• Upside.  There’s not only the potential upside of startup equity, but upside in terms of growth and leadership if the company grows.  Generally the people who are there first are most likely to end up in leadership positions if the company succeeds.
• Technical challenges.  Emphasize the unique technical challenges that you need to execute on in order for your company to succeed.  Engineers want to stretch themselves and work on challenging tech problems. But be careful here not to oversell.  A lot of the challenge at an early stage startup is in figuring out what to build, not how to build for scale. You need engineers who are OK changing direction frequently and building something less than the perfect system.
• Vision and Impact. You have an opportunity to really sell the vision and mission of your company, probably in a way that a tech giant cannot at this point.

Branding & Marketing

You need to make your company as appealing as possible to an engineer who has never heard of you.  We underinvested in this at my last company and constantly ended up in a position where prospective engineers did not understand what the company did and why they should work there.

What this means in practice is that you need to invest in professional assets that:

• Clearly explain what your company does
• Explain the business opportunity
• Explain the mission – why does your company make the world better?
• Showcase the culture
• Talk about the tech you are building
• Spotlight the leadership
• Spotlight investors and strategic partners if you have any
• Make it clear in general why someone who has never heard of you should take a big risk to come join you

The assets you want in place are:

• A pro looking website.  It can be minimal, but should explain clearly the things above.  Here is a good example: https://www.cockroachlabs.com/careers/
• A “one-pager” that you can send someone as a pdf that explains the company, mission, opportunity, culture and tech.
• Bios of the co-founders and leadership.
• A great job description.

By professional, I mean that you should pay a designer to make them look legit.  You don’t need Red Antler, but you do need them to look good.

Writing a job description

At a minimum, the job description should succinctly describe what you are looking for and give a feel for the company culture and values.  Here’s an example from my last company.

We are looking for an outstanding software engineer to continue building and innovating on our customer-facing product.

This is an ideal opportunity for a full-stack engineer looking to work in a high-velocity, high-impact environment, owning the development process from design to implementation to launch.

YOU

• Write clear, self-documenting code
• Are a problem solver, a pragmatist, a fast-learner, not dogmatic
• Have experience building web and mobile apps (Javascript, Node, React a plus)
• Are confident with performance optimization, and identifying bottlenecks and scalability concerns
• Are familiar with a range of modern data stores (relational, document, key-value, search, etc.)
• Have a good background in algorithms, systems, and design
• Work well with a multidisciplinary team, and communicate actively
• Think independently and love sharing your technical knowledge with others
• Aren’t afraid of a little database administration or dev-ops work
• Probably have a technical degree and 1+ years of real-world development experience

WE

• Are a small, dedicated team of smart, experienced engineers, product folks, and creatives
• Love Github, Slack, Asana, AWS, Meteor, Node
• Hate useless meetings, value your time, and enjoy a good work/life balance
• Are data-driven, and take product ideas from the whole team
• Work quickly to test and launch new ideas
• Value design and beautiful user experiences

BENEFITS

• 100% covered Medical, Dental, and Vision Plans
• Health & wellness perks through HealthKick
• Flexible Working Hours
• 401K Plan
• Free Meals and Snacks with a Fully Stocked Fridge and Pantry
• Cold Brew Coffee in the Summer, Fresh Hot Coffee in the Winter, and Seltzer on Tap Year Round
• Subsidized Commuter Benefits for MTA & CitiBike
• Monthly Team Outings
• 20 Paid Vacation Days & Unlimited Sick Days
• Office Located in Little Italy

Expect a competitive application process that will include an initial coding assignment, phone screen, and in-person interview. Thank you for your interest. We look forward to hearing from you!

We are an equal opportunity employer and value diversity in our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

However, the job description and company branding surrounding it present an opportunity to go above and beyond and let you stand out.

Take a look at what Vanta does in their description: https://www.keyvalues.com/vanta

The thing I love about this is it leads with the culture and mission is likely to be appealing to exactly the kind of engineer they want to work there – someone who is very customer-centric and willing to do more than just engineering to make the product succeed.  The effort they put into the job description shows how much they care about hiring the right people.

The Recruiting Workflow

Once you have your assets in place, you need to start running a recruiting workflow.  You’ll want to track this workflow using some sort of Applicant Tracking System (ATS).  

There are a bunch of ATS solutions out there, and I don’t have strong opinions on what is best.  We used a product called Lever, which I don’t actually recommend. But there are others and the important thing is that you have one central database that effectively de-dupes candidates and stores where they are in your recruiting funnel.  This could be a spreadsheet to start if you don’t have budget for a real system, but there are some free options out there which are better than sheets for this purpose and I recommend you use something with more structure.

The above chart shows high level what the recruiting workflow looks like.  On the far left are the various places candidates come from. As you move right, there are various steps for progressing them through your funnel.  Let’s start by looking at the sources.

Referrals

Referrals are the best source.  They can come from your own network, from employees, or from investors.  They are best because (a) you have an easy in for contacting the candidate, (b) they are pre-vetted, (c) the pre-existing relationship can help you close the candidate.

For employees, you should offer a referral bonus to incentivize them.  There are different schools of thought here – I’ve heard people say that employees should love the company enough that they want to tell their friends to come work there and that therefore referral bonuses should not be necessary or create a bad incentive.  

I don’t really buy this.  Recruiting is so hard and referrals are so valuable that I would do everything in your power to motivate employees to take the time to reach out to their talented friends and co-workers.  There’s a reason Google offers referral bonuses even though everyone in the world wants to work there already.

You should set up times to sit down with engineers on your team and go over their linkedin networks and see if they have folks who might make sense for them to reach out to.

For investors, you should make sure the assets I mentioned above are strong and then ask them to distribute those assets to their network.  Good investors will have in-house people focused on talent since it’s such a huge determinant of success. Form good relationships with those talent folks and make it easy for them to sell your company to people in their network.

Sourcing candidates & Cold Outreach

After referrals, your best option is sourcing candidates yourself and doing cold outreach.  

Sourcing candidates is basically a manual process.  It entails compiling a big list of potentially qualified engineers, designers, etc along with your best way to contact them.  You can sometimes bootstrap this process by getting an existing list from an investor or fellow entrepreneur. These lists are very valuable and you should ask around and try to get one.

Even if you get a list of engineers and their emails, you’ll need a way of expanding it.  One of the “secret” ways I had of doing this at my last startup was paying people remotely via Upwork to source candidates off of LinkedIn.  The way this works is you create a Google sheet with fields like:

• Name
• LinkedIn
• Email
• Years of Experience
• Company
• Undergrad School
• Grad School
• Highest Degree

And then give instructions to a remote worker on the qualifications you are looking for.  I’d provide them with a list of schools, companies, job titles, and so forth, and set them to work reading LinkedIn looking for possible candidates.  Email is generally not listed on LinkedIn, so they will need to use a tool like EmailHunter to guess the email address.

As this list builds up, you start emailing potential candidates.  The ethics of this are a bit gray. Technically, it is not spam in the legal sense because the email was not automatically scraped and you are not selling a product, but it always felt iffy to me to reach out and try to get in front of someone using this method.  Up to you whether you go this route, but at a startup you need to be aggressive and I think it’s basically on the right side of the line. A lot of engineers were flattered to get the outreach. Some were annoyed.

On a technical note, I would create a separate email address to do the outreach from.

You can also try contacting people directly through LinkedIn, but I found that (a) it was very expensive and (b) not nearly as effective as directly getting into someone’s inbox.

Recruiting Platforms

Another option is recruiting platforms like Hired and Vettery.  These platforms aggregate engineers who are looking for jobs and make them searchable to employers.  It’s generally free for employers to browse the platform; they only pay if they actually make the hire.  

The fee is steep though, usually somewhere between $15k-25k, or 15-20% of the first year’s salary.  You can negotiate discounts as a startup, and can also get discounts for bulk usage. But overall it’s a pricey solution.

It’s also a relatively low quality solution in my experience.  The main issue is with the quality of talent on these platforms.  As mentioned above, the best engineers and designers typically have jobs.  If they are looking for new jobs, they often have connections to go through or will target their searches.

I found that the people on these platforms tended to be B-level talent.  We interviewed many of them, but literally zero of them made it though our interview process on the engineering side.

Still, there are probably some strong candidates on there and if I were hiring again, I would definitely look.

Inbound candidates

As a rule, inbound candidates (people who found your company and applied directly) tend to be the lowest quality.  They can still be good, so it’s worth posting your job description widely and seeing what comes in, but expect a high signal to noise ratio.

External recruiters

Outside recruiters are another option.  They generally work on contingency so that you only pay if you make a hire through them.  If you run a startup long enough they will start contacting you with potential placements.

The upside is that they do the sourcing work for you and sometimes have quality candidates.  

The downside is cost – like the recruiting platforms they take up to 25% of first year salary, although you should always try to negotiate a discount.  

Internal recruiters

If you have raised enough money and are hiring enough engineers, you should hire an internal recruiter.  Typically they cost around $60k / year. So if you are hiring more than 5-6 engineers per year, it will be beneficial to have someone on staff who can fully manage the hiring process.  It’s much more bang for the buck than paying an external recruiter 20k per hire – a good internal recruiter should be able to get you ~1 engineer per month depending on who you are looking for, and they will save you personally a ton of time and overhead.

The internal recruiter should be responsible for the whole hiring process, from writing and reviewing the JD, to filtering inbound applications, managing outbound, setting up phone screens and interviews, corralling feedback, and so on.  All of this overhead adds up quickly, and if you don’t have someone else handling it, you will find yourself as a founder, cto, or vp eng spending a ton of time setting up meetings and responding to emails. Far better to use your time on higher leverage activities like pitching high quality candidates and closing.

The Pitch

You need to practice a version of the pitch that is specific to engineers.  It’s a different pitch than for investors or random people you network with.

The specifics of the pitch are going to depend on your business, but some things that I like to hit on are:

• Mission – how is your company making the world better
• Culture and Values – what do you care about, what type of company is it
• Business Opportunity – how will the company grow and make money
• Traction – it’s working…
• Leadership – why are you and your cofounder and other leaders great
• Tech challenges – what are the most interesting things to work on
• Product process – here’s how we develop new products
• Team – the other engineers on the team are great because of X, Y, Z
• Customers – our customers are X, Y, Z and they need us because of Y
• Investors and Funding – We raised Xm from these great investors
• Product roadmap – here’s what we plan on building over the next year
• Personal growth and mentorship – if you work here, you will have the opportunity to grow in X, Y, Z ways
• Startups are great – emphasize the benefits of startups over big companies – velocity, culture, ownership

You should practice this pitch.  You’ll find that engineers ask the same questions over and over again, and every time they ask, you should make a note and try to adjust the pitch to explain again and again.

For my last company SelfMade, the pitch I gave went something like this:

• Mission: “We are trying to help small businesses grow”
• Details: “We do that by making them have awesome instagrams that drive traffic and conversions”
• Background: “The company is series A, we’ve raised 11.5m from great investors, we are based in NYC, have been around for 3 years”
• Opportunity: The opportunity is really big and growing – every month 20k new Shopify stores start.  Amazon does half its business through its marketplace. Instagram is THE channel that all of these businesses need to figure out, and we are helping them.
• Traction and growth: We have 750 paying customers and are doing about $5mm a year in revenue, up from $1mm at the start of the year
• Me: I was a principal engineer at Google. At SelfMade, I manage product and engineering.  My co-founder is a second time entrepreneur with an exit under his belt at a company you may have heard of.
• Product: The product is a platform for producing Instagram feeds and ads at scale.  Explain how it works….
• Tech challenges: The challenges are in optimizing the efficiency and quality of the content we produce.  We use ML for X, Y, Z
• Product roadmap: We are expanding to support paid media in addition to organic instagram content.  We are also expanding to support more platforms and higher end customers.
• Investors: We’ve raised 11.5mm in funding from top tier investors in NYC and Silicon Valley.
• Personal growth: I’ve mentored a lot of engineers, and would help mentor you.  You will have an opportunity to own major parts of the application
  Our startup is great: you’ll have ownership, be able to move quickly, meet new friends, and so on

Initial phone screens

If you have someone who is interested, the first step after email is to set up a brief phone call to pitch the company and hear about their background.

If you have an internal recruiter, they should set this call up and take it for most candidates.  The exception is if you have a particularly promising candidate who has come recommended, then you should consider taking the call and doing the pitch yourself.  And if you don’t have an internal recruiter, then, well, you’re taking the call.

The call should take 15-30 minutes max, and should cover:

• Intro
• Candidate’s background
• Candidate’s interests and desired role
• Company background and pitch
• Role description and pitch
  Next steps

I always have the candidates give me their background before giving the company pitch because it lets me tailor the pitch to what I think will resonate with them.  E.g. if they are interested in frontend, I pitch the opportunities to work on app frontend. If they are looking for mentorship, I talk more about my background managing engineers.  And so on.

The goals of this call are twofold:

1. Determine if you want to continue the process with the candidate.
2. If yes, then get the candidate to commit to the next step.

This call should be high-level and non-technical.  It’s purely a screen on

• Basic intelligence – is the person smart and articulate
• Basic background – do they have roughly the right background for the role
  Basic culture fit – did they do anything that would make you think they are obviously not going to fit at your company

If all of these things are a “yes,” then you should try to get them to commit to moving forward on the call.  

If they have questions or concerns and seem high quality, you can offer to connect them with another leader or engineer at your company for more color.

Coding tests (optional but recommended)

The next step I recommend is a short online coding test.  We used a platform called Codility, but there are a number of platforms out there that work for this.

The idea is to make sure the candidate can code well enough that they won’t totally flail in an onsite interview.  This is not the place to ask hard coding questions – it’s just a sanity check. It should take the candidate no more than 1-1.5 hours.

That said, even on what I judged to be relatively easy online coding tests, this would filter out at least 50% of applicants who couldn’t get the coding test half right.   

We probably ended up eliminating some qualified candidates with this filter, but I don’t think the false positive rate was very high.  Programming is hard, and there are a lot of people who can sort of do it, but not really do it, and if you want a great team you need to aggressively weed out people who are not strong.  That said, you should take the tests yourself and make sure you can do them. If you can’t do them, they are too hard.

The real downside to this filter is not that qualified candidates don’t do well on the test, but that qualified candidates might not want to take the test at all.  It takes time, can be stressful, and, if they don’t do great, it hurts the ego. You don’t want qualified candidates dropping out of the funnel here, so if you see someone who has a great resume (top school or top company) who was good on the initial screen, then I would definitely skip this step.

That said, try hard to avoid having unqualified candidates in for onsites.  When someone comes in and starts bombing questions it puts both you and the candidate in an awkward position.  You either have to ask them to leave early or else waste a lot of valuable engineering time interviewing them. Being asked to leave an interview early is a bad experience for candidates; keeping unqualified candidates around for full interviews is a waste of time for your engineers.  So weed people out as early as you can.

Technical phone screens (optional but recommended)

Another possible (and recommended) step is a technical phone screen with an engineer.  This can be done in addition to or in lieu of the coding test.

The format is a 45-60 minute phone call or video hangout where an engineer on your team poses one or two coding questions to the candidate and has them work through them using google docs or some other collaborative editor.

Some advantages compared to the coding test are (1) that you can learn more about how someone thinks by observing them work through a problem (2) it creates a chance for the candidate and engineer to have a personal interaction and (3) can give another person on the team a chance to evaluate intelligence and culture fit in a way that you don’t get from someone taking a coding test.  It also makes cheating impossible (and yes, people cheat on the coding tests fairly often).

The disadvantage is mostly that it takes time from an engineer, and, if you are in rapid hiring mode and have a relatively small group of engineers to conduct the calls, these interrupts can add up to a lot of lost productivity.

Overall though I recommend this step for getting the most qualified candidates into the onsite.

On-site interviews

The onsite interview is the most important part of the process.

For a full-time engineering or product hire, you should plan on having 5-6 separate sessions with the candidate, plus some more informal time where the candidate gets to have lunch, meet some non-technical folks, and get a feel for the office.

Most of the engineering interviews should cover a mix of:

1. Coding
2. Algorithms and runtime analysis
3. System design

And there can be a secondary focus on

1. Some particular technology (e.g. iOS or React)
2. Live coding or debugging
3. Product & user focus

The engineers giving the interviews should coordinate beforehand to make sure there is enough coverage (e.g. at least one systems focused question instead of all coding).

An interview session should last about 45 minutes:

• 5 minutes on introductions, getting the candidate comfortable, explaining the format
• 35 minutes on one main question
• 5 minute wrap up at the end where the candidate has an opportunity to ask questions

Everyone on the team should be able to conduct an interview, but new hires will need time to shadow experienced interviewers before jumping in on their own.  I also encourage people to develop their own questions.

A good interview question

• Should
  • be easy enough that it can be solved in 35 minutes
  • be non-trivial in the sense that candidates will need to spend at least a few minutes thinking about possible solutions before diving in
  • lend itself to different possible solutions that have different performance characteristics (often there is a “brute force” solution and then a more efficient solution)
  • be “extensible” so that if a candidate gets the basic solution quickly you can make it harder by asking variations of it
  • Should be “hintable” so that if a candidate gets stuck early on you can nudge in the right direction without fully giving it away
• Should not
  • require any specialized knowledge beyond basic coding and algorithms (unless you are interviewing for a specialized role)
  • be an “aha” type puzzle – no questions where you need a flash of insight to get it right (how does an insect get out of a blender, or why are manhole covers circular type questions)

The point of the interview, and I always stress this prior to starting with a candidate, is not just to see if the candidate can get to the correct answer.  It’s much more about seeing how the candidate approaches a problem. I want to see whether the candidate:

1. Takes the time to understand the question
2. Asks clarifying questions before starting to solve the problem
3. Writes down example inputs and their corresponding outputs to think through various cases
4. Breaks the problem down into simpler subparts
5. Starts with the simplest solution even if it’s slower (I always ask them to do this)
6. Takes instruction well when they are heading down the wrong path
7. Asks questions when they get stuck
8. Outlines the algorithm before writing the code
9. Writes code that is at least basically syntactically correct in the language of their choice
10. Describes in words what they are doing and how they are thinking
11. Proves to themselves and the interviewer that their proposed solution works
12. Can engage in a discussion about different possible solution strategies
13. Can describe the performance of their solution and think through possible optimizations
14. Can describe test cases they would write to validate their solution

Sample Interview Question - Pascal's Triangle

For instance, say the question is to write a function that determines the value at a particular position in Pascal’s Triangle.  

I would give the candidate a function signature like:

function pascalVal(row, col) {
  // your code here
}

The first thing I would want to see is if the candidate understands the question.  Good questions would be:

• What is the indexing of row and column (0-based)?
• What are some example solutions?
  • pascalVal(0, 0) = 1
  • pascalVal(2, 1) = 2
  • pascalVal(3, 2) = 3

And then I would want to see if they could get to either a recursive or dynamic programmimg solution.

• If they got the recursive solution,
  • What’s the runtime performance (2^n)
  • How could they improve that performance (memoizing, taking advantage of symmetry, etc)
  • Can they also write up the iterative solution
• If they got the dynamic solution
  • What’s the runtime performance (n^2)
  • How could they improve memory usage
  • Can they also write up the recursive solution

There are various points in this question where candidates get stuck, and I have moves at each of them to help them keep going.

The goal of the interview is that the engineer is able to fill out a simple rubric about the candidate:

1. What was the question?
2. What was the code the candidate produced?
3. How would you rate their coding (if applicable)?
4. How would you rate their problem solving?
5. How would you rate their design (if applicable)?
6. How would you rate their knowledge and experience?
7. Are they a culture fit? Why or why not?
8. Would you hire them (1. strong no, 2. weak no, 3. weak yes, 4. strong yes)

Interview etiquette

A few notes on interview etiquette:

• Whiteboard interviews can be very intense for candidates and it’s important that the interviewer be mindful and understanding.  There is nothing gained by increasing the pressure – it’s already a very contrived setup so you should try your hardest to be sympathetic and put the candidate at ease.
• Always emphasize that getting the solution is not everything and that you care just as much about how they think through the problem, the questions they ask, whether they would fit the culture and so on.
• Do not check your phone or email during an interview.
• Have someone on the team who is dedicated to making the experience good – this person would typically be the recruiter, but it also could be an engineer.  They should greet the candidate, make sure all the interviews are running on time, make sure the candidate has everything they need.
  Remember that if you make an offer you will likely need to sell the candidate on joining, and how they were treated during the interview will make a difference.

Post interview

Hiring meetings

After everyone has submitted their feedback, you should get together as a group and make a decision.  This can be pretty informal – everyone goes around the room reading their feedback and making an argument one way or the other.  

A note on bias here: everyone needs to submit their feedback independently before the hiring meeting.  If you don’t enforce this, then people end up biasing each other depending on who speaks first.  It’s ok to be swayed if someone addresses your concern in the hiring meeting, but if more than a couple of the interviewers are changing their minds I tend to want to pass.

Typically I require a unanimous “yes” in order to move forward, which effectively gives every interviewer a veto.  This undoubtedly leads to false negatives – people we should have made offers to that we ended up passing on. However, the cost of a false positive (a bad hire) is much higher than a false negative, so I’m ok with this bias.  A bad hire ends up wasting a lot of management time, money, opportunity cost in terms of hiring someone better, and so on. Keep your hiring bar high.

Reference checks

The final step before making an offer is checking references.  Ninety percent of the time the references will reinforce your existing opinion and make you more confident in the offer.  But sometimes they will give you pause.

Even though it takes a bit of extra time, it’s worth it – there have been many times where I was going to make an offer, but the reference call made me change my mind (e.g. I’ve uncovered candidates lying about their previous positions; I’ve had people tell me in not so many words that the candidate was just fired, and so on).

I try to get three references from each candidate, where two of them had a supervisor relationship (not peers, friends or coworkers).

The reference call itself should take about 10 minutes.  I prefer a call to an email because it’s more likely to get honest answers and let you read between the lines.  But if you can’t get a call, then use an email – it’s way better than nothing.

Here’s the call template I use:

Intro – “Hi I’m Zach, thanks for taking the call, it shouldn’t take more than 10 minutes…”

• Company overview – “We do X, Y, Z.  The company is X people”
• Role overview – “We are making X an offer for software engineer”
• Questions
  1. What is your relationship to X?
  2. Can you describe X’s role & responsibilities?
  3. Why did X leave his job? (optional)
  4. How would you rate X’s overall job (or school) performance?
  5. What percent is X in for programmers? Top 5%? Top 10%? Top 25%?
  6. For a specialist (e.g. an AI person)   How would you rate X’s expertise in the field?
  7. How would X be at translating research into engineering?
  8. How is X as a programmer vs. a researcher?
  9. Do you think X would do well at an early stage startup?
  10. How were X’s communication skills?
  11. How was X’s work ethic?
  12. How quickly did X work? Is X more of a pragmatist or perfectionist?
  13. If I were to be his manager, do you have any advice on how i could help X succeed?
  14. What could X improve on?
  15. How does X deal with conflicts?
  16. What was it like to work with X?
  17. Would you want to work with X again? If so, why? If not, why not?
      Is there anything else I should know about X?

Question 13 is bold because it’s often most telling.  It’s a way of asking about the candidate’s weaknesses without explicitly asking about them.  You need to keep in mind that the person you are getting the reference from most likely has been chosen by the candidate to say nice things about them – that’s the idea.

Still, people want to be honest and if you give them a way to give honest feedback which might be less positive without feeling like they are betraying the candidate, they will do it.  That’s why this is a good question – it says, tell me how I can help the candidate, but what it’s really asking is what does the candidate need help with.

Everyone has weaknesses, so hearing about areas for improvement about a candidate is not at all a disqualifier – it can help you manage them.  But, if enough of the references give the same negative feedback, you might consider whether you missed something at the interview and whether the candidate really is a good fit.

That said, the things that are most likely to make me rescind an offer at the reference stage have to do with a candidate not being honest.  For instance, if a candidate told me they had a certain role at a company and I find out the role was different, that’s a huge flag; I had that happen with someone pretending they were full time at google, and I only found out at the reference that he was a contractor (very big difference).

Offers & Closing

The offer should come via a phone call from you – it’s more personal and exciting than getting an email.

It should include salary, start date, title, equity, benefits and a ton of enthusiasm about the candidate.

Some notes…

Explain your equity grant.  

• Don’t count on candidates understanding startup equity – in the interest of being transparent you should be clear how the vesting works, what the strike price is, what the exercise period is, and what percentage of the company the equity amounts to.  
• Re: disclosing percentages vs number shares – just tell the candidate the percentage; trying to give a large sounding number of shares without explaining what it’s actually worth makes the offer harder to think about and is disingenuous.
• One thing that can be helpful is showing what the equity would be worth in dollar terms in different exit scenarios.

Have other folks reach out.

• It’s a good idea to have the folks who interviewed the candidate reach out with short notes saying how excited they are and offering to answer questions.
• Off for other people involved in the company (e.g. investors and advisors) to hop on a call and answer questions – having an outside perspective can be helpful for candidates

Give the candidate some time, but don’t keep it open ended.

• There should be a date by which you and the candidate agree they will give you an answer.  I think 2-4 weeks from the day you give the offer is fair.
• You want to give some time so the candidate gets to make an informed choice, doesn’t feel pressured, and, if they say “yes” it’s clear that the job is something they really want.
• On the flip side, if it starts to really drag out what is probably happening is that they don’t want to work there that badly and they are probably using your offer as leverage in the rest of their interview process to ratchet up what they can get.
  Having a lot of people that you’ve put the time into interviewing who are shopping your offer for something better is bad for morale at your company, and, if the candidate does eventually join, it can feel off for everyone (like they are attending their safety school).

On the other hand, I don’t think it pays to be too aggressive in the deadline.  Don’t give an exploding offer – this is a good way to end up with a bad fit candidate and mess up the reputation of your company.  You might be able to strong-arm some people this way, but it’s really a shitty thing to do from a cultural perspective.  Fundamentally the candidate has the leverage in these situations and using a pressure sales tactic to try and reverse that is bad.

The product-first programmer cares about that stuff too, kind of, but only as a means to an end. For product-first programmers, the code is the scaffolding, the support, the steel beams in the building, but not the end product. The end product is, well, the product, not the code, and what matters to them is how well that product actually solves the underlying problem. Does the building stay upright? Do the elevators work? Is the A/C functioning? Do people like being there? Product-first programmers love building and launching and seeing users use what they’ve built. The product is the thing.

Anyone who has worked at a place like Google has met plenty of code-first programmers. They are the teammates who are always refactoring code and nit-picking spelling in your function comments. They are in the micro-kitchen complaining about “spaghetti code,” “technical debt,” and the lack of rigor in other teams’ code review processes. They are probably not fixing bugs or launching features. You can probably tell I’m not a huge fan of the code-first approach.

When I interview programmers I’m always amazed at how many of them seem to think the code-first approach is what I’m looking for. Trying to impress me, they ask: “What’s your unit test coverage like?” Pretty close to zero; this is a startup. “Do you guys use hot new technology X.” Not yet, no, how would that help us build the right thing faster? “Is there a lot of technical debt?” We will have to rewrite everything at some point, but it doesn’t matter right now because we haven’t even figured out the right thing to build.

They have an understandable but fundamental misconception of what programming is all about. Programming is about building products that solve problems for users not about writing beautiful code for its own sake.

And to be clear, this is true at all levels of the stack, whether your users are external customers, third-party developers, or internal consumers of an API. What matters is how the code works, not how the code looks.

Does that mean that I encourage writing bad code? That I don’t care about what technology we use or how the software is architected? Absolutely not – I care a lot! But I care because I believe if you make the right engineering choices, you will ultimately get to a better product.

I recently had an engineer I’m managing ask me how his code was. I responded by asking two questions back: “Did the feature work well?” and “Did you build it quickly?” “That’s what matters to me,” I said.

And typically, if the answers to those two questions are yes and yes, it’s likely (although not certain) that the code was actually pretty good, because good product usually implies good code. By definition the opposite doesn’t exist by the way – there is no such thing as good code that produces bad product.

I call this my rule of good code:

If the product doesn’t work well, the code is not good.

In other words, there is no such thing as good code in the abstract; good code can only exist if it’s producing a working product (again, define product loosely here to mean product at any level of the stack).

Over and over I see some engineers producing great product quickly, and when I see the code it’s usually well factored and smartly built. It’s not surprising since it’s hard to quickly build good product in a complicated system with code that is poorly architected or factored or not tested.

Conversely, when I see programmers not launching features quickly, the issue is often overengineering. Or when engineers do launch quickly but the quality is bad, then the issue is usually under-engineering or sloppy code.

There’s no zero sum game here: you should strive for good code, produced quickly, leading to great product. This is what great programmers produce.

The best programmers I know are product-first, but actually have more coding knowledge than the coding-first programmers. They know when to use the chainsaw vs the hand saw vs the chisel. When you need to really get something right (usually the deeper in the stack you are) vs. when you can fake it and just move quickly. When it’s better to just write a normal for-loop rather than a custom iterator. These choices and tradeoffs are what becoming a great programmer is all about, and the ultimate feedback mechanism is how well the product works.

If you are an engineering manager and not hiring interns year round, you are making a mistake.  This is even more important at a startup where top talent is extremely hard to find.  If you are at Google, maybe it’s less important because of the abundance of full time talent.  But even at Google I found that interns often made outsize contributions.

Why do I like interns so much?

There are a bunch of reasons.

Interns are easy to hire.

The way to do it is establish a relationship directly with the career placement office or computer science department at the schools you want to hire from.  When intern season comes, they will reach out to you directly about the hiring process. If you are proactive you should be able to get a bunch of inbound traffic from students looking for summer jobs.

The interview process for interns should be much lighter weight than for full time.  I recommend an initial phone screen with a hiring manager to get a feel for culture fit and baseline experience, followed by a short coding test and technical phone screen.  

The main things you are looking for are (a) is this person going to fit culturally, (b) is their attitude good, (c) is their general intelligence high and (d) do they have enough baseline CS knowledge that they are not going to be in over their head in a professional setting.

Interns are available year round

Look into schools that have CoOp programs like Waterloo and Northeastern.  I cannot recommend highly enough that you hire from Waterloo – we have had 10 excellent interns from there at my current company.  They are the same quality that you would find at a top eng school in the US, but are available all year.

Interns will work on anything

Most of what they are learning is how to be productive in a professional software development setting, so the particular tasks matter less.  They have not yet developed the sense of entitlement and need to work on particular tasks that ~some~ full time engineers get 🙂

Interns are fun

Most of them are, anyhow.  Having a rotating group of students who are learning how to code in the office is just plain fun.  They enjoy coding in a way that I remember enjoying it when I took my first CS class and stayed up all night coding my first project.  The positivity they bring improves the whole office.

Mentoring interns is a growth opportunity for engineers

Every intern should be managed by an engineer, usually one who is not already a manager.  It’s a great opportunity to train engineers to have their first reports. I always have engineers successfully mentor interns before asking them to manage.

Interns are relatively inexpensive

Good interns definitely are not free, and at first you might balk at what you have to pay.  At a minimum you will need to pay 50% of the rate you pay a junior engineer. If you are competing against Facebook and Google, you will need to pay more (as of writing this, you might need to pay up to $7k/month).  

However, if you are getting high-quality interns it’s worth it.  In my experience the best ones quickly become as productive as your junior engineers, for a much lower price.

Internships are a pipeline for full time hiring

I haven’t had quite as much success with this at my startup as we had at Google, but if you bring on interns and they have a great experience it’s definitely a leg up for hiring them post graduation.

How to host an intern

It’s pretty easy, actually, but there are a few things to get right.

• Pair each intern with a dedicated mentor.  Every intern should have a host engineer to answer questions and meet with them regularly to check in on progress.
• Start interns on very small tasks.  Start with trivial bug fixes, followed by small features.
• Give every intern a “big” project.  After a few weeks, interns should get to focus on bigger, more impactful projects.
  • DO: Make this project something that your product actually needs.  Interns should make a real impact.
  • DO: Make it meaty enough that it’s anywhere from a few weeks to a couple months work in total.
  • DO NOT: have the intern work on it alone.  This is a good way to end up with a project that you have to completely throw away because the code is written wrong.  Always have a real engineer working with an intern.
  • DO NOT: make this an experimental side project.  The best way to get the most out of interns is to have them work like real engineers on real projects that you actually need.
• Give frequent feedback.  Like you would for a full-time engineer.

Gotchas

There are a few things to look out for to make sure your interns are net positive contributors.

The biggest risk of hiring interns is that you will throw away their code.  I do not recommend that you put them on something that is outside of the main part of your code base, and I can’t emphasize enough that they should not work on their projects alone.  The unsuccessful internships I’ve hosted have always followed this pattern.

A second risk is that an intern will unwittingly do something really costly, like bringing down your app.  This is on you, as an eng manager or host, to prevent. It’s a real issue; at SelfMade, one of our interns accidentally deleted one of our production databases; another deleted our Facebook app.  

Do not let interns work on super-sensitive parts of the code; lock down what actions they are able to take; carefully review their work, make them write tests, etc.  But the truth is I have seen senior engineers bring down services far more often than interns.

The final risk is that interns won’t be productive and will be a net negative in terms of the resources needed to manage and train them.  This is possible if the interns you hire are too junior or not high caliber. You need to tackle this high up in the funnel by only getting interns from CS programs at good schools.  I’ve had some random high-school students intern and it was more work to clean up their code than if they had never written it.

Secret Weapons

We had so much success with interns at my last company that we thought of them as a sort of secret weapon, an advantage over our competitors in the war to get great engineering talent.

I led a project at Google that was over a year late. Granted, it was a big project – a rewrite of Google Sheets. But I was off by a whole year, which seems crazy in retrospect. How did this happen?

I made a few mistakes right at the outset.

I, like most engineers, have a bias towards optimism in my coding ability. I thought my and my team’s code wasn’t going to have as many bugs as it ended up having. I misjudged the number of tasks we had to complete. I underestimated the number of external dependencies we had. And so on… it’s just a lot easier to think of what you need to do if the project goes perfectly than if things go wrong.

I also over-promised to get buy-in. I’m typically pretty reserved in making promises, but for big projects there is a built-in incentive to say they are going to go as well as possible and under-represent the risks. If you give too conservative of a timeline you run the risk of not getting the go-ahead from decision makers – no one wants to fund a 4-year software project (for good reason).

I also just flat out didn’t think through how much work was really going to be involved.

During the first internship I ever had – at a dot-com-bubble company that’s long since gone – the CTO spoke to this question. He said “Here’s how I estimate: I take whatever time the engineer says and multiply by 𝝅. Why 𝝅? Because it’s more fun than 3, so why not?” I thought it was really weird at the time.

Almost 20 years later, the multiply by 𝝅 rule seems pretty accurate. I’ve seen very, very few projects or large features launch on time, but scores that have taken over 𝝅x as long as they were supposed to.

Projects tend to start slow. When we first started on the Sheets rewrite we wanted to do everything The Right WayTM. All our classes were perfect. We agonized over code reviews. We were determined not to make the mistakes of the prior version. Of course, all of this “perfection” came at a cost of time. But at the beginning the time pressure wasn’t intense, so it seemed fine.

As we got further along, we started to lose the luxury of time. Stuff came up that we couldn’t or didn’t foresee. A year into the project we realized the performance of our new architecture was actually way worse than expected, worse than the version we were replacing. Oops.

We had to stop all forward progress and spend the next four months working on speeding up the app (thankfully we were able to). It would have been hard to plan for this particular issue, but I should have seen the risk of something like this happening.

The specifics vary, but the general pattern is the same – shit happens and your project gets delayed. And once it’s delayed, there’s usually no way to get it back on schedule. The instinct around adding more engineers usually ends up in Mythical Man Month-land slowing things down further. The best thing is often to just start cutting features. No one really likes this though.

How can you minimize the risk in the first place? Ever since the sheets rewrite, I always start by taking the project’s big tasks and breaking them into smaller ones. However, the idea is not that you get to a more accurate estimate of the big task by dividing it into smaller ones and adding them up.

The idea is more that by breaking up the big tasks you can get a better sense of the true surface area of your feature or project – it’s a reality check.

For our Sheets rewrite I would have ended up with a huge list of features, functions, and so on – everything that goes into a spreadsheet app from the UI of formula editing, to pivot tables, to sorting, and so on. I couldn’t have really said that writing pivot tables was going to take X weeks and charts Y months with any accuracy. But by looking at those features I could have realized at some subconscious level how big the effort really was. I should have been more scared.

As an aside, “divide and add up” is not a good estimation method. The problem is that most of the time in software development isn’t spent writing code for features. It’s spent in integration, debugging, testing, handling changing requirements and so on.

There’s an 80/20 rule at play here, and goes against the instincts of the estimator – the 80% is spent after the initial code is written, but it’s very hard to list out how this time will be spent beforehand. When you divide and add up you leave out those hard-to-name tasks and end up with an underestimate.

Deadlines

So what’s the actual way to deliver on time? Unfortunately, the best thing I’ve seen is to set a deadline.

I say “unfortunately” because from the developer’s perspective, deadlines suck. Developers tend to think things “take as long as they take” and adding on an arbitrary date doesn’t change the fact that the software takes a certain amount of time to write and debug. It just adds pressure. It creates a death march mentality.

I totally get this perspective.

However, setting a launch deadline and orienting the team around it has all sorts of benefits that actually do move projects along faster.

The proper way to set a deadline is to establish the “when” and leave a little bit of flexibility for the engineering team around the “what;” meaning you say that product/feature X needs to launch by day Y, but there is some flexibility around what exactly will launch depending on what is feasible. But the goals should be aggressive.

Once you have a deadline, time becomes a limited resource; you become more conscious of wasting it.

Deadlines let you backtime; if the deadline is date X, then you know you need to have rolled out to internal testers by X – 4 weeks, and that means the code needs to be complete by X – 8 weeks, and so on, which means your first feature needs to be done in 2 weeks. Better start coding!

Deadlines create competitive motivation. You don’t want to be the sole engineer on the team who makes the project ship late.

Immovable deadlines really motivate. If your deadline is tied to a public facing event (say Google IO or the launch of GDPR), then you most likely will have something ready to ship.

The issue, of course, is that deadlines can seem arbitrary and if you use them too often it ends up feeling to engineers like a never ending emergency.

The best way to overcome this is to:

• Not always be in deadline mode – sometimes there needs to be space for exploration, design, cleanup etc.
• Save deadlines for the most important features
• Try to pick deadlines that have a plausible business rationale (e.g. an external conference or a competitor releasing a product)
• Get the leaders on the team totally bought in on the deadline and make it a fun cultural thing (print tee shirts, and so on)
• Make the deadline as realistic as you can without sandbagging

In retrospect, I wish our Sheets rewrite had a deadline. I bet we could have gotten it done at least six months sooner. There would have still been unforeseen issues, feature cutting, and so on, but we would have approached the whole project with a greater sense of urgency and clearer prioritization from the get-go.

Also, be careful of rewrites in general! That’s a subject for another post though…

When multiple developers are working on the same codebase, it’s important that everyone follow the same procedure for making changes.  This is software engineering 101, but the details matter and new engineers don’t always come out of school knowing how to do this.

Here’s how I have done it with my teams in the past.  

Let’s assume for simplicity’s sake that we have the following set up (if you don’t have something like this, then start by getting it in place):

• Your code is stored in some shared repo, say Github.
• Everyone is using git or something similar for version control.
• There is a continuous build, test and deployment system.
• There are at least three environments code can run in: local (not committed), staging (not real users, but hosted in a prod-like environment), and production.
• The goal is to write and test code locally, test it in staging with real-ish data and services and other pending changes, and push it to production as quickly as possible and with as few bugs as possible.

How to make a change?  Bear with me if this seems basic – i’ll add some color below.

1. Create a local branch off whatever your team’s “head” branch is.  Usually this is develop if you are using gitflow, but it could also be master or some other branch.  It’s where the up to date code lives and features and bug fixes start from.
2. Make a small, incremental change, test it locally, push it to staging and test there.
3. Review your own code before sending it to someone else to review.
4. Assign a code reviewer and ask them for a code review.
5. Respond to or acknowledge every code review comment from the reviewer and ask them to look again until they approve.
6. Deploy your code to production and merge the branch into master.
7. Repeat.

That’s the basic procedure.  Here are the rules on how to do this workflow well…

Writing Code

Incremental changes

Changes should be as small as possible while still doing something meaningful; small, incremental changes are

• Easier to review: it’s easier for a reviewer to grok a small change and also less of an interrupt so you are likely to get a review more quickly
• Easier to test: smaller changes mean less surface area for testing and more confidence that they work
• Less likely to have merge conflicts when you sync locally
• Less likely to cause merge conflicts for others when you submit
• Easier to roll back
• Pretty much better in every respect

But don’t make the change so small that the overhead of the review and commit would make it make sense for the change to be bigger. Be pragmatic.

Feature-flags

Use feature-flags or some similar mechanism to check-in incomplete code

• The most common reason I see big changes is because engineers don’t want to check in half-working code.  Feature-flags are the solution here.
• It’s fine for code that is not fully functional to be committed so long as it is not executed.
• Feature flags also have other important uses for staged rollouts and canarying, so it’s good to get them in place right from the start.

Thoroughly test your own code

• If you’re fortunate enough to have support in the form of a QA team or a group of manual testers, that’s great, but ultimately you are responsible for the quality of what you launch.

Get code reviewed early on

• The beginning of a feature is the most important time to get a code review.  It is preferable to make an initial commit on a new feature as early as possible because most of the important design decisions are made at the beginning of implementing it.  
• A common mistake I see in feature development is not getting the code reviewed until way too much of it has been written, at which point it becomes much more painful to adjust in case a reviewer requests a big design change.
• I’ve seen a lot of sub-par code checked in because the initial change was too big and the author didn’t want to redo it and the code reviewer didn’t want to push the author because of the amount of time it would take.  That’s a bad outcome.

Don’t group unrelated changes in a single review

• It makes the change harder for a reviewer to understand.
• It makes it harder to selectively roll back if one part of it causes an issue but the other part is fine.
• But again, be pragmatic.  You don’t need to send five distinct code reviews if that would end up taking everyone a lot more time.

Code should make it all the way to head on every review

• Don’t have code incrementally reviewed on a single branch as you build out a feature and then submit that one branch once the feature is fully built
• This leads to committing one big change at the end, which, for reasons stated above, is bad
• Use feature-flags to submit incomplete code – that way you can separate committing and pushing the code from enabling the code.

Keep working while your code is being reviewed

• The technique I recommend is “branching off of your current branch” to keep going.
• E.g. if you are having “feature_branch” reviewed, then you should locally do “git checkout -b feature_branch_2” from feature_branch and keep working.
• As you make changes for the initial review in “feature_branch” you merge them into feature_branch_2 so it picks them up.
• Finally, when you submit feature_branch, you merge master into feature_branch_2 and it becomes the next thing you have reviewed.

Do not share feature branches

• Under no circumstances should engineers share feature branches.  By “sharing”, I mean if two or more engineers are working on a single feature off of a non-head branch.  
• There are many reasons this is a bad idea, but the primary one is that this encourages the submission of very large changes, and for the reasons listed above, large changes are bad.
• It also makes it hard for other engineers on the team to review code early enough to catch errors.

Use a linter for enforcing style

• Using a linter like eslint or prettier that enforces a particular code style saves a lot of time commenting on style issues in code reviews and also keeps the code consistent.
• I’ve seen teams put in presubmit checks that verify the code style is followed.  I don’t recommend this because it’s likely to annoy anyone external to your team who is trying to make a small fix in your codebase and doesn’t have your linter configured.  Either make it so the linting is automatic or don’t rigorously enforce at check in time. Someone will fix it on the next PR.

Code reviews

Code reviews are the most important quality check there is in writing new code.  If there is one quality control practice I would prioritize above all others it’s a culture of good code reviewing.

For the author

• Always merge master into your feature branch before starting the review process – otherwise your diff might show spurious changes and you could end up with a change that needs merge conflict resolution after the review
• Review your own code before you ask someone else to review it!  Make sure what you present to the code reviewer doesn’t have changes you know you should make.
• Do not expect code reviewers to find bugs!  They sometimes will, but that is not their job.
• Assign the review to the person who knows the area of code you are working on best, not the person who does the easiest reviews.  But try to spread your reviews around if you can.
• New team members should act as “secondary” reviewers on a bunch of code reviews before reviewing code on their own.  
• If you are really trying to scale a codebase, you can introduce a concept of “readability;” any engineer who has “readability” in a programming language is allowed to approve a change on his or her own.  Otherwise, they need another reviewer to approve. Google has this process in place, but they also have a huge codebase they are trying to scale. I would not recommend this for a startup.
• Continue working by branching off of your feature branch while you are waiting.  This is a powerful way not to get blocked. See the technique of branching off a branch above.
• If you are blocked on a review, you should be aggressive in pinging the reviewer, and, if that doesn’t work, a manager.  Fast code reviewing is a key part of a healthy team and everyone should feel responsible for reviewing code quickly.  Speed of review is inversely correlated with the size of the change, so if you want your changes reviewed quickly, keep them small.
• When a review comes back, address every comment, even if just acknowledging that you made the suggested change.
• Test your code again after you have made the suggested changes!  Blindly commiting the reviewer’s suggested change is a common source of bugs and build breaks.
• Code reviews from more senior engineers are how you become a better coder – remember that.  Don’t get defensive when an engineer requests changes.

For the reviewer

• Treat code reviews as high priority.  If you can’t immediately direct your attention to the review, you should let the author know when you expect to get to it.  I don’t think any review should sit for more than a couple hours unaddressed.
• Focus on design issues first and foremost.  The most impactful reviews ask a few key questions about design issues that are likely to cause problems down the line if not addressed early on.
• Keep the tone focused on learning and improvement and not critical.  Engineers are sensitive about criticism and the more you focus on making the code better and not pointing out what someone has done wrong, the better – i usually frame my feedback as suggestions.

Example good feedback:

• “It looks like this might run in n^2 time – is there a way to make it linear?” “I think this is making a network call within a tight loop.  Maybe add a bulk api on the server side so you can do the work in one call?” “I’d think about putting all of these methods into their own class and memoizing the calls if you can”

And not so good feedback:

• “Style guide for JS Doc calls for two newlines between top level file comments.”
• “NPE”
• “I didn’t understand how this worked but if you’re sure it produces the right results, then ok.”

Presubmit checks

• A “presubmit” check is a trigger that automatically runs before your code gets merged and deployed.
• Examples of presubmit checks are scripts that check the formatting and linting of code, ones that check the code has been reviewed, etc.
• The most valuable presubmit routine you can have is one that takes the code you are about to merge, integrates it into the production “head” in another sandbox environment, and builds and runs all of your tests against it.  The important thing is that this happens before the code is merged, so you avoid breaking the build and tests.
• See this awesome post by Josh Haas on how they implement this at Bubble.

After submitting and pushing the code

• Code is almost never “done.”  There are always bugs that arise, which brings me to my next point…
• FIX YOUR BUGS. If you push a bug, fix it. If you create a bug somewhere else in the product because of your change, fix it or roll back your change.  Great engineers are always fixing not only their own bugs but other bugs they find in the product.

IF YOU BREAK A BUILD, ROLL BACK. This depends somewhat on the nature of the break, but a broken build slows everyone down and your default should be to roll back your change rather than trying to roll forward with a fix.

What’s the best way to actually do this?  There are a lot of methodologies out there – agile, scrum, kanban, waterfall, and so on.  I don’t really subscribe to any of them, but instead I can tell you the general outlines of what I’ve done in the past, some principals that work, some pitfalls to look out for.

OKRs

Let’s start with the question of goal setting.  

A good practice that I took away from Google is starting with quarterly OKRs.  OKRs stand for “objectives and key results.”  You make a list of the things you want the team to get done (e.g. improve onboarding) and for each one of those things specify the metric that lets you measure it (e.g. % of users who complete onboarding goes from 60% to 80%).  

These Objectives should generally be outcome driven goals (e.g. improve performance, improve conversion, improve customer satisfaction) rather than output goals (e.g. minimize JS, put button in a different spot on page; add chat widget to homepage).  

The Key Results can be numeric metrics (if possible) or output oriented if it’s not yet possible to attach a reasonable number to the goal.  If you are too early in your development process then the result you are shooting for might just be building and launching the thing, rather than moving a metric.

Here are some OKRs from my last company:

• Improve our ability to edit photos to match a customer’s brand
  • Increase positive outcome rate to 70%
  • Reduce revisions citing “Doesn’t match style” to 3%
• Improve the performance of our image specialists
  • Increase positive outcome rate to 70%
  • Reduce revisions citing “Looks fake” or “Didn’t follow my instructions” to 10%
• Improve the growth rate of our customers and their engagement in our app
  • Increase % of members using the planner weekly to 50%
  • Increase median time in app to 4:00
  • Increase members’ followers by 10% MoM (median)

Notice that we were going for very specific changes in metrics.  I’ll speak more to the question of whether you should be prioritizing completing features or driving customer outcomes in a bit.

You make this list at the beginning of the quarter and spend the next three months working on it and tracking your progress.  You’ll want to check on how you are tracking on these goals every month at a minimum. Importantly, at the end of the quarter, before making your next list you go back and grade yourself on how you did on the last one.  

The grades should be pretty high-level; I like to use a green/yellow/red framework: green = you nailed it; yellow = some progress but incomplete; red = you missed it.  The grading keeps you accountable to yourself and the rest of the company.

Quarterly OKRs should not appear out of thin air as a diktat of a single product leader.   Instead there should be a collaborative process for arriving at them. The exact process doesn’t matter that much, but it needs to synthesize data from a lot of different points.

Those data points ideally include input from users, engineers, PMs, designers, sales, marketing, and other stakeholders.  Common ways of getting this data are surveys, retros, feedback-emails, bug reports, NPS surveys, etc.

Good ideas come from everywhere and gathering input with a wide net is a good idea.  But, at the end of the day, someone needs to decide what the real priorities are and be able to justify them to the larger team.  

Deciding what not to do is just as important as deciding what to do.

At a small startup the ultimate decider is the CEO or CTO.  At a bigger one it might be a VP of Product or Engineering. At a place like Google it’s typically the tech lead, eng manager or PM.  OKRs roll up through an organization, so there will be OKRs of increasing scope the higher up in the org you get.

The way the decider should pick what to work on is by looking at a lot of different factors and arriving at a stack rank.  Then you choose as many of the items from the top of the list as you have resources to get done.

And these decisions should happen pretty quickly.  If you are spending more than a week or at most two deciding on what to work on, then you are spending too long.  You need to move fast.

Which brings me to the topic of prioritization…

Prioritization

I’ve had to prioritize a lot of projects and the only thing I’m certain of on the topic is that no matter how you prioritize someone is going to think you are doing it wrong.  

To start, let’s admit that prioritization is a hard problem, and you are not going to come up with a perfect solution. You will never know if the priority order you came up with was the right one.  

The best you can do is use a reasonable framework and come up with your best guess and get on with building.  

That said, it’s good to start by judging candidate projects along a few big dimensions:

• Impact: how big of a deal will it be if you get the project done – how much does it help you hit the OKR?
• Effort: How much work is it? How long will it take?
• Risk: Can you actually build it?  And, if you do, are you sure it will achieve the desired outcome?
• Definition: How scoped is the work?  Do you actually know what you want to build?
• Resources: Do you have the right resources available to work on the project?

E.g. say you are building an asset library for a photo editing tool.  You might weigh one feature that lets users find content on the web and upload it themselves against another that uses machine learning to find and identify content.  The first would probably be lower risk and lower effort, but also lower impact. The second (the ML solution) could be risky, take longer and be harder to scope but might have a bigger payoff.  It also might require special resources in terms of engineering talent and data.

The goal, per above, is to make sure you hit the OKRs.  Typically you do this by maximizing impact and reducing effort and risk.  In our hypothetical asset library example you would weigh the difference in impact, effort, etc and make a judgement call on whether the extra risk and effort was worth the payoff.

The Real World is Messy

In the real world though things are rarely so simple and if you do prioritization just based on the above, you’re likely to land in a world of pain.  There are other things to consider…

You need to make sure the engineers on the team are happy and working on things they find interesting.  If you don’t you may have attrition. You also will have less productivity as engineers stop working as hard, start coming in late, feel demotivated and so on.  You want missionaries not mercenaries on your team.

You need to make sure that engineers are on a growth path and being prepared for promotion at some point.  Every engineer wants better skills, a higher salary, and more responsibility.  Some projects lend themselves more to growth than others.

You don’t want anyone idling.  Everyone should be engaged in productive work at all times.  This can be tough to manage because the features you most want people working on might not be specced for a while.  If specs aren’t ready, you can also have engineers working on the product discovery process by prototyping and feasibility testing.

You want to make sure that pressing bugs and other one-off issues are addressed.  Often these will not fit neatly under one of your OKRs, but you still need to fix them.

You want to be flexible enough that if some new but important issue arises you aren’t slavishly following the OKRs.  This happens all the time at startups.

You want redundancy in who knows different areas of the codebase in case an engineer decides to move on.

You have to accommodate vacation, parental leaves, sick days.

You have to recognize that engineers are not fungible.  They know different parts of the stack, different technologies, different parts of the codebase, and will work on projects at different rates with different quality outputs.

You sometimes need to build things to increase company value in some strategic way that might not be directly tied to a user story (say for an investor demo).

The things you build can’t just be the product of a stack rank.  They need to blend into a cohesive product.

You may want to bundle certain features for a specific launch.

And so on…

In short, figuring out who should work on what, and in what order, is a big constraint maximization problem that has no clear best answer.  Putting together a simple spreadsheet that weighs the options can be helpful, and I recommend it.

At the end of the day, you want a set of goals that are high impact, cohesive, and the team can rally behind.  You want something that can be packaged into a neat effort that you can attach a launch deadline to. The results should be measurable and something everyone on the team can work towards.

But ultimately it’s a judgement call to be made (and defended) by the product decider.  Like I said, someone is going to be unhappy. But it’s your job to get agreement to move the project forward if not agreement on the actual priorities.

Week by Week – Sprints

Once you have your overall priorities and roadmap set, it’s time to start working on a week by week cadence.

What has worked best for my teams in the past is dividing up tasks into weekly sprints.  Two-week sprints are ok as well. Every sprint should be kicked off by a short weekly meeting where each engineer and designer publicly commits to what they are going to get done that week.

A few points here:

• The things that you want to get done that week should be visible in some public place.  It could be a shared spreadsheet, a tool like Asana or Trello or Jira, or just a plain document.  Don’t obsess on the format. What matters is that it’s visible to all.
• It’s important to start the week with everyone publicly saying to the whole team that they are going to get X done.  For more senior engineers on the team, they should be coming up themselves with what X is. For less senior folks, the engineering manager should assign tasks and communicate them   Either way it’s important that the engineers themselves commit. This creates accountability within the team.
• X should be sized appropriately for a week.  This can be tricky, but check with the engineer or designer to make your best guess.  I’ve tried to use point systems or other sizing methodologies, but I have never found them to work particularly well.  But I’m not against them. Be pragmatic and don’t get caught up looking for the perfect process.
• X should be scoped and defined so that everyone knows what it actually is.

Here’s an example weekly task list from my last company:

As mentioned, the exact format doesn’t much matter – this one was a google doc that linked out to Asana tasks that had more granular detail on the tasks.  We could have done this same type of list in a spreadsheet or in Asana itself, but we liked the simplicity and readability of the Google Doc.

After the engineers and designers have made their commitments for the week, I suggest sending them to some larger group outside of the product team.  This could be a higher up manager, the sales or marketing team, the CEO, etc. Sending the weekly plan outside of the product team creates whole team accountability and gives visibility as to what is currently being worked on.

The thing you send should also highlight what you got done last week.

Day by Day – The Daily Standup

I recommend starting every day except the day of the product team meeting with a short team standup.  I’ve seen standups work a number of ways (and not work at all).

The point of the standup is to discover issues that could prevent folks from reaching their weekly goal.  It’s the main tool an engineering manager and PM has for understanding what is going on, what is blocking the team, etc.

The mechanics are simple: get every person on product team in the same room in the morning at a fixed time and have each person say:

1. What they got done yesterday
2. What they are going to get done today
3. What they are blocked on or need help with
4. And that’s it.

Each person should talk for less than 30 seconds.

Good issues to surface:

• Engineers blocked on code reviews
• Engineers blocked on design input / not sure how to build something
• Engineers stuck on difficult bugs and needing help
• Engineers being behind on tasks and needing help
• Engineers and dev/ops stuck on deployment cycles and bad merges
• Designers needing input and/or feedback
• Two people working on the same thing by accident
• Two people working in a way likely to cause a conflict
• Bad bugs that have come in during the week and how they are being fixed

Things to look out for:

• Do not get into deep technical discussions in standups.  Anytime this is happening you should move the discussion to after the standup.
• Do not try to define features or debate priorities during standup.  Keep things short.

The standup is the main way that engineering managers and PMs are able to see if the team is on track for the week.  If not, it’s an appropriate time to adjust and rebalance tasks, unblock folks, and so on.

Standups tend to deteriorate in attendance and quality over time.  I’ve seen this again and again.  Engineers start coming in late, the standup gets cancelled a few days when lots of folks are out, the standup goes badly a few times, standup updates start to come in via slack rather than in person.

When this happens it is the job of the team leaders to get it back on track.  The standup deteriorates because engineers would rather not do it.  It requires them coming in at a fixed time, it pulls them away from their computers and forces them to talk in front of a group.  But it’s important to do because without it, you, as a leader, will not know what’s going on with the team.

At my last company I literally resorted to taking attendance.  Each engineer had a “standup streak:” the number of consecutive standups they had attended in a row.  There were rewards for hitting certain milestones – ten straight standups got you a free coffee, and so on.  It felt like elementary school, but, when we pushed it, it worked.

A few folks have asked what I have against Slack standups.  I’ll probably write a section on when I think Slack is appropriate (not that often), but I definitely don’t like it for standups because:

1. Slack updates aren’t interactive in the same way as in person – it’s much harder to ask questions quickly
2. New slack messages push the standup updates out of the window (i find this to be a general issue with any kind of group slack)
3. They tend not to all come at the same time, which causes the standup to actually take longer and cause more distraction

The Perfect Process

As an aside, I’ve come across many folks in the product world whom I consider overly-dogmatic about product process.  People tend to hold something like religious beliefs on how things need to be done.

This is not me.  I’ve tried a lot of different systems, tools, and so on, and what I think works best is a pragmatic approach.  Do what makes the most sense in the context of your product, team, stage, company, culture and so on.

For instance, if you are a very early stage startup with no product, it doesn’t make a lot of sense to base your OKRs on moving customer metrics.  You should base your OKRs on getting your MVP built as quickly as possible.  The Key Results should be MVP features.

The specific product planning tools you use rarely matter in my experience.  At Google, it was a set of in-house tools. Externally I have used Asana, spreadsheets, documents, github tasks.  The more advanced tools have various bells and whistles, which can enable some advanced workflows.

What actually matters is commitment to a process and constant vigilance by the management. And more than that, what matters is the culture you build. If the culture favors pragmatism and a commitment to putting the user first, then you will likely be successful no matter what the particulars of the process are.