Entry-level white-collar work exists on a spectrum. At one end are the roles that are almost entirely task execution: data entry, basic content production, routine research compilation, document formatting, and first-pass screening. These are being absorbed into workflows that one person now manages with AI assistance. At the other end are roles that use task execution as training for something harder: learning to read a brief properly, understanding what good output looks like, and developing the judgment that makes someone a valuable senior contributor. These are not the same thing, and conflating them misses the real problem.

What AI is doing is hollowing out the first category faster than employers are redesigning the second. Companies are not replacing 10 junior analysts with 10 AI analysts and one senior. They are replacing 10 junior analysts with AI and not thinking carefully about where the next generation of senior analysts comes from. The developmental pipeline depends on the early repetitive work that AI is now eating.

The early career AI exposure data is precise about this. Employment in the most AI-exposed entry-level occupations fell 13% in 2025. That is not theoretical displacement. It is people who did not get jobs that would have existed without the technology. The roles above that level are largely stable. The path to those roles is narrowing.

There is no clean resolution here. You cannot preserve inefficient work just to train the next generation of workers. But companies that automate entry-level functions without redesigning how people develop into senior roles will find themselves facing a talent gap in four to six years that is expensive and slow to fix.

The most practical thing individuals can do is understand which part of their current role involves execution and which involves judgment. The execution is being compressed. The judgment is not. Investing in developing judgment while the market still rewards execution is not comfortable advice, but it is accurate.

A workflow is a sequence of tasks, often across multiple people, that produces a business outcome. Customer support escalation: message comes in, gets triaged, routed, answered, logged, and closed. Internal reporting: data gets pulled, transformed, written up, reviewed, and distributed. Recruitment screening: applications arrive, get filtered, ranked, summarized, and forwarded. These workflows did not require one smart person. They required several people with defined roles coordinating across a handoff chain.

AI agents can now handle that entire chain. Not all of it, not always reliably, and not without oversight. But the workflow architecture changes. Instead of five people touching a process sequentially, you might have one person supervising an agent that runs the whole thing. Jack Dorsey at Block was pointed about this when he cited it as a factor in laying off nearly half his workforce: smaller, flatter teams enabled by AI are doing the same work.

The Gartner finding that only 20% of companies actually reduced headcount due to AI is accurate yet somewhat misleading. The bigger effect is that companies are not backfilling when people leave. The workflow still runs. Fewer people run it. That does not show up as a layoff. It shows up as a hiring freeze and a productivity number that looks good on a slide.

Middle-layer roles are feeling this first. Positions built primarily on coordination, relay, and report generation are shrinking because those are the functions agents execute cleanly. Senior judgment roles and roles requiring physical presence or emotional trust are holding. Entry-level roles, which were the training ground for senior roles, are where the disruption is most structurally significant.

If you are building an AI product and you are thinking about which tasks it automates, zoom out. The question worth asking is which workflows it replaces and what the headcount implications of that replacement are. That is a harder question and a more honest one.

The models caught up. Claude Opus 4.6, GPT-5, and Gemini 3.1 Pro handle multi-step reasoning, tool selection, retrieval, and context management natively in ways the 2023 models did not. The scaffold was compensating for model limitations. Most of those limitations are gone.

What you get now, if you reach for LangChain by default, is an abstraction layer over a model that does not need abstracting. The debugging experience when something breaks is worse than with direct API calls, because you are now tracing through the framework’s logic to find the problem. The surface area is larger. The control is lower. For the teams I know that have removed it and replaced it with direct API calls, the debugging time dropped sharply.

This is not a criticism of LangChain as a project. It shipped a stable 1.0 in October 2025. LangGraph handles genuinely complex stateful multi-agent workflows well. LangSmith is a good observability tool. If you are building something that needs those specific capabilities, the ecosystem earns its weight.

The issue is that teams use LangChain as a default starting point when they do not have a specific reason to. They add a dependency, they will spend months understanding before they discover the underlying model could have handled their use case with 200 lines of direct API calls. The abstraction was not solving their problem. It was deferring the work of understanding what they actually needed.

Before reaching for a framework, be specific about what it buys you. If the answer is ‘quicker to prototype,’ that is a legitimate reason. If the answer is ‘I have always used it,’ run the experiment of not using it. You will find out quickly whether the complexity is earned.

That window has closed. GPT-4-level performance, which was the benchmark two years ago, now costs under a dollar per million tokens. DeepSeek R1 achieved reasoning performance competitive with OpenAI’s best model at a reported training cost of $5.6 million, compared to hundreds of millions for comparable US lab models. Open-source models from Alibaba, Mistral, and others have narrowed the gap to the frontier to the point where, for a growing set of tasks, they are competitive or better. The model itself is becoming a commodity.

This matters because a lot of companies built their AI strategy around which model they call. That is not a strategy. That is a vendor choice. When the vendor choices are increasingly interchangeable and the switching cost is near zero, the model you call says nothing about whether your product is defensible.

What actually differentiates outcomes now is further up and further down the stack. Proprietary data that you have accumulated and that a competitor cannot easily replicate is a real moat. Workflow depth, meaning how deeply your AI is embedded in a user’s actual work rather than sitting as a chat window beside it, is a real moat. The speed at which you can ship and iterate is a real moat, because a technical advantage measured in months is now measured in weeks. The model is infrastructure.

The companies that understood this early are designing model-agnostic architectures from day one. They are not betting the product on Claude staying ahead of GPT or vice versa. They are building the data layer, the integration layer, and the user experience in ways that compound over time regardless of which model is under the hood.

If your current AI strategy could be fully described as ‘we use GPT-5,’ you have a vendor choice, not a strategy. The question worth spending time on is what your product would have that a competitor could not replicate in six months by calling the same API.

The problem it was solving is mostly gone. Claude Sonnet 4.6 and GPT-5.4 both ship with 1 million token context windows. Gemini 3.1 Pro had that capability months earlier. For a knowledge base of 10,000 documents or a codebase you want an agent to reason over, you can load the relevant material directly into context. No chunking strategy. No embedding model choice. No similarity threshold to tune. You give the model what it needs and let it work.

I am not saying vector databases are dead. For retrieval over millions of documents with strict latency requirements, they still make sense. But I have watched teams maintain Pinecone pipelines for internal knowledge bases with a few hundred pages of content. The infrastructure was serving a scale problem that did not exist.

What is replacing naive RAG is something more deliberate. The better teams are building what people now call context engines, systems that decide dynamically what the model needs before each step. Sometimes, that is retrieved documents. Sometimes it is a live API call. Sometimes it is cached history. The decision is made per query, not by a fixed pipeline that always retrieves regardless of whether retrieval helps.

The models are also better at knowing what they do not know. Ask a frontier model a question about something in its training data, and it answers. Ask it about something that requires live or private data, and it says so. The retrieval decision is no longer purely the engineer’s job. The model is a participant in it.

If your team built a RAG pipeline in 2024, it was probably the right call at the time. The question now is whether you are maintaining infrastructure for a problem the model can largely handle on its own. Run the experiment. Load your knowledge base into context and compare the output quality. For most mid-sized knowledge bases, you will find the retrieval layer is patching a limitation that no longer exists.

It’s also usually a waste of time and money.

Pre-trained models - GPT variants, BERT, Vision Transformers etc represent billions of dollars and millions of hours of research already sunk. These models have learned general patterns across massive datasets. They’re proven. They work.

What they don’t do is understand your specific domain. That’s where fine-tuning comes in. You take a pre-trained model and retrain it on your proprietary data. Your domain. Your patterns. Your edge cases. This requires nowhere near the compute, time, or expertise that training from scratch demands.

The efficiency difference is staggering. A team building a text classification system from scratch might spend six months on architecture, training infrastructure, hyperparameter tuning, and validation. The same team starting with BERT fine-tunes in two weeks. That’s not a marginal difference. That’s the difference between shipping next quarter and shipping next month.

The cost follows the same curve. Training from scratch requires specialized infrastructure - GPUs, distributed training frameworks, constant monitoring. Fine-tuning runs on commodity hardware. Development costs drop by an order of magnitude.

But here’s what matters more: domain specificity. Your proprietary data is where the real value lives. Fine-tuning captures that value without the overhead of training from scratch. The model learns your terminology, your patterns, your edge cases. It becomes genuinely useful instead of generically mediocre.

This approach scales across domains. Legal firms fine-tune language models on contract databases. Healthcare providers fine-tune on clinical notes. E-commerce platforms fine-tune on customer behavior. In each case, the domain-specific layer is what creates competitive advantage. The base model is just infrastructure.

The efficiency frontier in 2025 isn’t about finding better models. It’s about better systems engineering around those models. Teams that treat AI as a long-term architectural investment—modular components, strong governance, incremental execution—outperform those chasing quick wins.

This means pragmatic tool selection. Don’t build custom when pre-trained does the job. Don’t optimize prematurely. Don’t treat the model as the system—the system is everything around it.

The winners aren’t the ones with the fanciest models. They’re the ones who move fastest, maintain control, and compound learning over time. That comes from building on what works instead of rebuilding what already exists.

Start with pre-trained. Build your advantage on top.

Here’s what works instead: pick one non-critical component. A batch job that runs overnight. A reporting function. An internal tool. Something isolated enough that failure doesn’t spread. Run the AI system there for a month while keeping the old approach running in parallel. See what breaks. Fix it. Prove it works. Then expand.

This isn’t risk aversion. It’s how you actually validate assumptions before they become expensive.

The best pilots aren’t glamorous work. Batch jobs nobody notices if they’re slow. Reporting that takes two hours to generate—if the AI version cuts it to 30 minutes but makes occasional errors, you’ve learned something valuable. Internal tools used by a small team. Completely isolated services.

This sandbox is where you surface real problems. Integration gotchas that only appear under load. Performance assumptions that don’t hold. Edge cases nobody anticipated. Bias patterns that show up when you run at scale.

Running AI in parallel with existing systems reveals patterns quickly. Let the AI make recommendations while humans still decide. Compare outputs. Watch for misses, false positives, systemic biases. Catching these before full deployment is the difference between shipping confidently and shipping recklessly.

The organizational benefit matters too. When pilots succeed, stakeholders believe the next initiative might work. Non-technical people see results. Budget approval gets faster. You’ve built credibility—you’re not chasing technology, you’re shipping value.

Scaling before proving is how projects become disasters. Starting small, running long enough to see patterns, comparing against the old way—this takes longer upfront. It pays for itself a hundred times over when you avoid the catastrophe that happens when you skip it.

Boring implementation beats catastrophic deployment.

It was also predicting wrong on half their data because the data was corrupted. I was intruiged and decided to get familiar with some of the mistakes they made. I’ll use this knowlede sometime in the future.

They’d been feeding it timestamps from three different systems, some in UTC, some in local time, some just wrong. They had duplicates they didn’t know about. They had null values they’d filled with averages without understanding what the averages meant. The model wasn’t the problem. The data was.

This is the efficiency trap nobody talks about. Teams chase sophisticated models when they should be building pipelines.

Models only work as well as their input. You can have the most advanced architecture in the world. If it’s learning patterns from bad data, it’s learning noise. Fix the data and suddenly your simple model outperforms the complex one.

The infrastructure that matters is boring. Apache Kafka to stream data reliably. Snowflake or Databricks to manage it at scale. Clear data contracts that specify what fields mean, what format they’re in, when they arrived. Version control on transformations so you know what changed when. Validation gates that catch bad data before it reaches your model.

This stuff isn’t exciting. It doesn’t get talks at conferences as much as state of the art models. But it pays dividends immediately because most AI systems are starving for quality data. They’re eating whatever gets thrown at them.

There’s another payoff: the same infrastructure serves both analytics and AI. Traditional batch ETL architectures - data warehouses that update nightly—can’t support real-time decision-making. But a well-built data pipeline with Kafka and proper streaming can feed both your analytics dashboards and your AI systems, real-time, at the same time.

Via word of mouth, I heard about a fraud detection team that had terrible latency until they rebuilt their data pipeline. The model was fine. But it was working with data that was hours old. They moved to a streaming architecture. Latency dropped from 45 minutes to 30 seconds. That’s not tuning the model. That’s plain old infrastructure.

Here’s what matters: spend on data infrastructure first. Build pipelines that are robust, observable, and scalable. Then add your model on top. The opposite—fancy model, sketchy data - is how you end up debugging predictions you don’t understand.

Data infrastructure is where the real efficiency lives.

Then the lawyers get involved.

Think of two hypothetical ways it can play out. Once with a lending system that rejected applicants based on a pattern the model learned that nobody intended. Second, with a content moderation system that over-corrected and took down legitimate business accounts. Both times, the technology was sound. The governance was nonexistent.

Here’s the thing: AI amplifies what you already do. If you have strong code review practices, AI-generated code gets better because reviewers catch more issues. If you have weak ones, everything gets worse because now you’re generating code faster than you can possibly validate it.

Most teams discover this too late - when they’re already at scale.

The fix is unglamorous. You need code review processes that work for AI-generated code. Different focus areas than traditional reviews, but mandatory. You need version control for models and prompts, not just code. You need validation frameworks that specify how decisions get approved before they affect customers. You need clear escalation paths so a questionable prediction goes to a human before it becomes a problem.

But here’s what actually matters: human-in-the-loop validation isn’t optional for production systems. It’s architectural. You design it in from the start or you’re bolting it on desperately later.

A fintech team I worked with learned this the hard way. They launched a system without proper escalation paths. Six months in, they realized they had no idea which decisions had human review and which didn’t. Audit trail was a mess. They had to pause the system for a month to retrofit governance. Could have been avoided with two weeks of thinking at the beginning.

The regulatory angle pushes this further. Financial services, healthcare, anything regulated - auditors want to see decision trails. They want to understand why the system did what it did. If you built governance in, you have logs and traces. If you didn’t, you’re explaining why you can’t explain anything.

Version everything. Trace everything. Validate automatically where you can. Escalate to humans where you can’t. This isn’t bureaucracy. This is the infrastructure that lets you ship confidently instead of nervously.

Start with this foundation and scale becomes manageable. Skip it and scale becomes a liability.

Then they get the cost estimate. It’s in the hundreds of millions. The timeline is three years, minimum. And somewhere around month six, they realize they’ve broken seventeen critical processes and there’s no going back.

This is why wrapping beats replacement every single time.

Your legacy system works. It’s been handling your business logic for fifteen years. It’s slow, it’s undocumented, it’s written in COBOL nobody understands anymore—but it works. Ripping it out to start over isn’t modernization. It’s a bet-the-company gamble disguised as engineering.

The smarter move is to leave it alone and build around it.

Create an API layer between your legacy system and the outside world. Add event streams that capture what’s happening. Then layer your AI on top—it sees the events, makes decisions, writes back through the APIs. The legacy system stays intact. The business logic never gets touched. You modernize without the risk.

This sounds simple. It is, relatively speaking. But most teams skip it because they’re seduced by the idea of starting fresh.

Generative AI changed the equation. Now you can actually understand what those legacy systems are doing. I worked with a financial services team stuck maintaining a 30-year-old system written in COBOL. They had three people who knew how it worked, all approaching retirement. Instead of rewriting, they fed the codebase to a language model. It extracted business rules, mapped dependencies, translated chunks to Python. Not magic- they still had to validate everything. But they went from “this is unmaintainable” to “we can actually work with this” in weeks.

McKinsey ran the numbers on this approach. Teams using gen AI for legacy refactoring hit 70% better productivity than traditional methods. That’s not marginal improvement. That’s transformative.

The payoff is triple. You reduce risk- no massive rewrite means no massive failure risk. You keep the business logic intact - nothing gets lost in translation. And you modernize incrementally - you wrap one piece, prove the pattern works, wrap another.

A year in, you’ve added AI capabilities without touching the legacy system. Two years in, you’ve reduced its load because you’re handling new work in the modern layer. Eventually it becomes less critical, not because you murdered it, but because you built better alternatives around it.

That’s how you actually retire legacy systems. Not with a bang. With patience and APIs.

Then your model needs updating. You redeploy. Everything goes down. Or you want to scale the inference layer but the data pipeline can’t handle it. You’re stuck because everything’s coupled.

I’ve seen teams spend six months building sophisticated ML systems only to realize they can’t upgrade a dependency without breaking production. They can’t swap out a model that’s underperforming. They can’t test changes without risking the whole thing. That’s the cost of pretending modularity is optional.

Efficient AI systems are built differently. Data ingestion, model serving, and orchestration operate as independent pieces. You change one without touching the others. That’s not just cleaner engineering - it’s the difference between iterating in weeks and iterating in months.

The practical payoff shows up fast. A team I worked with had a fraud detection system that could only update their model quarterly because redeployment was risky. We split the architecture: the model ran in its own service behind an API. The data pipeline fed it independently. Suddenly they were swapping models weekly, testing approaches in parallel, scaling the prediction layer without touching anything else.

Cloud-native principles - containers, Kubernetes, managed services—aren’t buzzwords if you actually need to run at scale. They let you define infrastructure as code, spin up environments consistently, and avoid the “it works on my machine” nightmare. More importantly, they force you to think in modules because distributed systems require it.

For legacy environments, API-driven patterns are lifelines. You don’t rip out old systems. You wrap them. Build an API layer between your legacy database and your new AI components. They talk through well-defined contracts. The legacy system doesn’t care that you’ve upgraded your model. Your model doesn’t care about the creaky old business logic underneath.

This approach costs more upfront. You’re thinking about boundaries, contracts, failure modes. You’re not just stitching things together. But that cost compounds into savings - you iterate faster, fail safer, scale without panicking.

The teams that move fastest aren’t the ones who build the fanciest models. They’re the ones who can change components without everything collapsing. Start with modularity. You’ll thank yourself in six months.

The pattern’s always the same. Someone gets excited about a new model and decides this is the one. The engineering team builds something sophisticated. Six months and $500K later, they’re explaining to executives why there are no measurable returns.

The turning point isn’t better algorithms. It’s asking a simple question first: what problem are we actually trying to solve?

Teams usually get this backwards. They pick the flashiest technology. They build systems designed for scale before proving the concept works. Then they wonder why they’re burning budget on infrastructure nobody uses.

The successful ones start different. A financial services company didn’t ask “how do we implement cutting-edge ML?” They asked: “where are developers wasting time?” Answer: debugging stack traces in legacy systems. One engineer spent 4 hours weekly on it. They built an AI tool for that one thing. $50K cost. 200 hours saved annually. Problem solved.

Another team wanted to “leverage AI” broadly. Vague and expensive. We dug into actual pain points and found their testing was completely manual. Test case generation became the focus. Unsexy work, but they shipped 30% faster with measurable returns in weeks.

Most AI problems aren’t AI problems—they’re execution problems. You’ve got bottlenecks: expensive manual work, slow turnaround, incomplete data. AI addresses some of that. But only if you identify them clearly before building.

Stack trace analysis. Code refactoring. Test generation. Domain automation. These aren’t glamorous. They don’t get conference talks. But they move the needle because they solve specific, measurable problems.

When you know exactly what you’re building and why, you stop overengineering. You don’t build for scale when you need speed. You don’t implement multi-model orchestration when an API wrapper works. You don’t burn three months debating architecture when you could ship in two weeks.

This also builds organizational credibility. Ship something that works, show the returns, and stakeholders trust the next project. Build something cool but pointless, and you’ve spent political capital you won’t recover.

The critical efficiency gain happens at the beginning. Before tool selection, before architecture, before hiring. Define the business problem. Make it specific. Quantify success.

Then pick the technology that solves it.

Start there and you’re already ahead of most teams.

Person 1: Hi

Person 2: Yes, tell me

Person 1: I have a question

Person 2: sure, go ahead

Person 1: I was thinking about contributing to this project……

By the time Person 1 got to ask their question, multiple hours had elapsed, and Person 2 had to break their attention numerous times each just to try to move the conversation ahead.

What do you think you should do? I'd like you to please get to the point. Don’t ask to ask. It’s not a movie where you must set up the suspense and focus on character development. Just ask the question.

Person 1: hey, I wanted some help from you. I wanted to contribute to this project. Can you introduce me to some critical people and send me some reading material to understand the background. I would really appreciate your help.

Depending upon how the message is drafted, it can come off as crass since you didn’t develop the foundation for asking your question. The question is whether they consider it rude.

An individual regularly approached by people for help will value their time significantly. To ensure our actions are aligned with theirs, it’s the best course of action to “Just ask the question.”

Different interviewers, have different attention spans, but you can't rely on the interviewer whom you have never met before to be the one with a longer attention span. It's safer to assume that everyone you meet have a short attention span and I can completely understand if that is truly the case.

Zip straight to the Situation

It’s a well known secret that all behavior questions should be answered in the Situation-Task-Action-Result (STAR) format. The immediate goal is to craft a strong hook so that the interviewer is invested in your answer. That hook is the mystery, or in our case, the situation. Remove all fluff when you are explaining the situation, and leave it for the time when the interviewer asks you a follow up question.

In fact, this strategy can be very effective as the background and specifics that you didn't provide might end up being the next follow up question. Take care that you provide just enough introduction and background, else the interviewer might lose interest.

Imagine a question “Tell me the most challenging project that you worked on”.

Even thought it might not look like a question for STAR format,

For this question, you can start by

“In my current company Apple, I was working on Apple Health iOS app, where we were expected to integrate the app with Hospitals, Networks, Labs online records so that users can have all their health data from all providers in one place”

We started with (a) company name (b) product name (c) the project or situation and then followed by (d) why this is important.

Explaining the impact of the project is important since it showcases why you are working on it. It showcases you understand the objective rather than just doing what you are told to do.

Move quickly to your actions

Do not spend too much time on the Situation. It just needs to be a quick opener, setting the stage for your answer.

The Task section is a quick overview of what had to be achieved. e.g.

“I had to work with public relations team to coordinate relate to the providers that would be supported and I had to coordinate with the Data Lake team to find ways to actually fetch the data”

The Task section can also contain why it was a challenging project in the first place.

The Action would be something along the lines of

“I reached out to the TPM on public relations team with a quick outline of the problem. I setup a meeting with her and 2 days before the meeting sent her the agenda of the meeting including objective of my project and the possible challenges I anticipate. In parallel, I reached out to the SDM on Data Lake team about the capabilities I am looking for. I reached out to the Security team about how to secure health data locally on the iPhone”

Wrap it up quick

Make the results sound appealing by getting to the actual numbers quickly. Some phrases that convey such numbers are

• “We completed the project in 4 months, which was a month ahead of time”

• “We reduced the latency from X → Y (Z% decrease) in the 4months timeperiod”

• “We released the product and we saw 7% month-over-month growth of daily active users”

• “We saw 7% decrease of storage usage by switching to xz compression, that translates to $900K of monthly cost savings which is 12% of the storage costs”

Attaching the $ value to the answers is an impactful method. Make sure the numbers make sense and it does not sound like you are making it up, even if the cost savings was real. It’s very hard to change the perception if the interviewer concluded that you cooked up a number to impress them.

First mistake that we make it tailoring the resume to make it look good and fit our aesthetic sensibilities. You are not your audience of your resume. It doesn’t matter what opinion you have of your resume. What does a recruiter think about your resume? Are they impressed? Did they lose interest?

The actual question should be “How does a recruiter scan your resume?”. Given that they have hundreds of resumes to scan through, why does your resume stand out? Are they looking for a list of technical skills or your work experience? Which one would you present first? Is Education section more important than Experience?

While this is an old study, in 2012 BusinessInsider reported that recruiters look at your resume for just 6 seconds. That’s not a lot of time, which means they are skimming it. If the resume is not eye-catching, the details are irrelevant. The layout matters more than contents if you want to cross the first hurdle - getting shortlisted.

The one on the right has clearly marked section, providing guiding lines for the recruiter to jump where they wish to.

While there are a millions of solutions to fix your resume, there is no “Correct” format. Just guidelines to improve your odds

1. Overview, Experience, Skills, Projects, Education etc should have their own section, with clear seperation of sections

2. The most recent experience matters.

3. Draw attention to the Company Name and Title by bolding them. Don’t bold duration, location and description. When everything is bold, nothing is bold.

4. If you are in tech, writing details of each job experience, broadly follow these steps

   1. Start the sentence with an action. e.g. “Improved”, “Reduced”, “Saved Cost” etc

   2. Use numbers to justify your achievements. They should be believable.

5. For Education, just bold the name of the University. If the recruiter is interested, then they can read more.

6. Skillset - Just have two lines. First line is a comma seperated list of technical skill and second line for non-technical skills. These are probably for ATS, not for humans.

7. Recruiters are likely to scan your resume from top to bottom looking at just the leftmost word. If leftmost word is catchy, they might read the rest. That’s why we start the sentence with action. That’s why the leftmost side should have name of the comany.

Let’s have a look at how an experience section could look like

CompanyName Location, State

Title/Position From - To (Duration)

1. Improved user satisfaction tracked using NPS from X to Y (17% improvement), by taking specific actions

2. Reduced request latency from X to Y (8% reduction), by investigating bottlenecks and optimizing critical code paths

3. Saved cost of $400K by analyzing log access patterns and updating the archival policy.

As Bruce Schneier, a renowned security technologist, states, "Security is not a product but a process."Every individual within an organization, from developers to managers, is involved in embracing security as an integral part of daily operations.

The impact is significant

I cannot stress enough the importance of this collective approach to security. The facts speak for themselves: according to IBM, the average cost of a data breach in 2021 was a staggering $4.24 million, with the healthcare industry facing an even more alarming average cost of $9.23 million per incident. These aren't just numbers – they represent a significant financial impact and a potential erosion of customer trust, not to mention the harm inflicted on victims.

I've seen the consequences of major cybersecurity incidents, and they're sobering. Take the 2017 Equifax breach, which exposed the sensitive information of 147 million people. Equifax and FTC reached an agreement where Equifax would pay between $575M - $700M. This still fails to undo the damage to the victim, who might have to worry about their stolen identity for the rest of their lives.

The 2020 SolarWinds supply chain attack compromised numerous government agencies and corporations. The SolarWinds breach was so impactful that the Government Accountability Office (GAO) published “SolarWinds Cyberattack Demands Significant Federal and Private-Sector Response (Infographic).”

Let me be perfectly clear: adopting a security mindset goes far beyond implementing best practices in code. It encompasses operational security measures and demands a heightened awareness of potential threats. Consider the 2011 RSA breach, which compromised SecurID two-factor authentication tokens. This wasn't a result of sophisticated hacking—it was initiated through a simple phishing email. This incident highlights the importance of staying alert against social engineering attacks, which manipulate human psychology instead of exploiting technical vulnerabilities.

We need to change our thought process

I'm convinced that to think like a security engineer, we must cultivate a mindset of constant vigilance and healthy skepticism. This means questioning assumptions, anticipating potential vulnerabilities, and considering the security implications of every decision we make.

Ross Anderson, in his seminal work "Security Engineering" rightly points out that security is often compromised not by breaking specific mechanisms but by exploiting oversight and complacency. I firmly believe that by fostering a culture where every team member actively contributes to security efforts, we can create a more robust defense against cyber threats.

To develop this mindset, it's important to stay informed about current threats, participate in security training programs, and integrate security considerations into every stage of the software development lifecycle. I'm particularly excited about "shift-left security," which promotes incorporating security practices earlier in the development process. The results speak for themselves: a study by Puppet Labs found that organizations implementing DevSecOps practices spend 50% less time remediating security issues. That's not just efficient – it's smart security.

Wrapping up

In conclusion, I want to emphasize this point: while not everyone needs to become a security engineer, adopting their mindset is absolutely crucial in today's threat landscape.

By internalizing the fact that security is everyone's responsibility, remaining vigilant against various forms of attacks, and integrating security considerations into all aspects of our work, we can significantly enhance our resilience against cyber threats.

As the cybersecurity landscape continues to evolve, I'm convinced that this collective approach to security will become increasingly vital in protecting sensitive data and maintaining trust in our digital ecosystems. It's time for all of us to step up and think like security engineers – our digital future depends on it.

Code coverage measures the percentage of code that is executed during testing. However, it's important to note that simply running each line of code once doesn't ensure that all potential issues will be detected. We also need to test different combinations of values. It's possible that someone might have 15 tests for one method and zero tests for the next 5 methods, which might seem surprising, but not all code paths are equally important.

Subscribe now

Martin Fowler, a renowned software developer, aptly puts it:

"Test coverage is a useful tool for finding untested parts of a codebase. Test coverage is of little use as a numeric statement of how good your tests are."

I strongly believe in focusing on the quality and effectiveness of tests, rather than just the quantity. Instead of aiming for 100% coverage, I am confident that investing time in designing comprehensive test cases that cover critical paths and edge cases is more beneficial.

I believe that striving for complete code coverage can often result in diminishing returns. The effort required to increase coverage from 95% to 100% can be disproportionately high, leading to a significant rise in the number of test cases. This expansion can make the test suite more fragile and challenging to maintain over time.

As noted by Steve McConnell in his book "Code Complete":

"Trying to achieve 100% coverage often results in a significant amount of additional testing effort for minimal gain."

Moreover, I must always consider the long-term maintenance cost of a fully covered codebase. As our software evolves, keeping all tests up-to-date can become a time-consuming task that may slow down our development and hinder our ability to ship new features quickly. This brings me to a crucial question: What do our customers truly care about? In most cases, they prioritize a stable, feature-rich product delivered in a timely manner over a theoretical measure of code quality.

In my opinion, while code coverage is certainly helpful, it must not overshadow other vital aspects of software development. I advocate for a well-rounded approach that incorporates strategic test coverage alongside other quality assurance methods.

As suggested by Glenford Myers in "The Art of Software Testing":

"The objective of testing is to find errors. Therefore, a good test case is one that has a high probability of detecting an as-yet undiscovered error."

I believe that by focusing on creating effective tests that address potential problem areas and essential functionalities, my team and I can optimize our time and resources, ultimately providing more value to our customers.

Many engineers tend to adopt an engineering-oriented mindset, concentrating on delivering technically sound solutions. They take pride in writing elegant code, optimizing performance, and maintaining clean codebases. While this approach often leads to robust and efficient software, it can sometimes create a gap between the product and its users. As Robert C. Martin pointed out in his book "Clean Code: A Handbook of Agile Software Craftsmanship" (2008),

"The only valid measurement of code quality is WTFs/minute"

This quote emphasizes that even technically perfect code is of little value if it doesn't effectively meet user needs.

Subscribe now

I have discovered that adopting a customer-oriented mindset shifts the focus to user satisfaction and business value. Engineers with this approach actively seek user feedback and are willing to make technical compromises for a more user-friendly product. This aligns closely with Tim Brown's Design Thinking principles. In his Harvard Business Review article "Design Thinking" (2008) (p86), Brown states,

"Innovation is powered by a thorough understanding, through direct observation, of what people want and need in their lives and what they like or dislike about the way particular products are made, packaged, marketed, sold, and supported".

The focus on deeply understanding user needs aligns perfectly with a customer-oriented approach, which should be the preferred mindset. In today's competitive software market, user experience and customer satisfaction are crucial differentiators. As Peter Drucker famously said in his book The Practice of Management (1954),

"The purpose of a business is to create and keep a customer"

I believe this principle applies equally to software development: a technically perfect product that fails to meet user needs is ultimately a failure. Throughout our careers, we have all experienced using external software for payroll or travel booking. We may have considered the user interface terrible and hoped for startups to disrupt the market. Unfortunately, that's not always the case because individual needs often differ from the needs of the customers of the product, in this case, your company. Those external software vendors are focused on their customers - your employer - not you.

I see these mindsets as complementary rather than mutually exclusive. In my view, the optimal approach is to strike a balance, with a preference for the customer-oriented end. This balanced approach is effectively demonstrated in the Agile methodology, which places equal emphasis on technical excellence and customer collaboration. The first principle of the Agile Manifesto (2001) states,

"Our highest priority is to satisfy the customer through early and continuous delivery of valuable software."

By wholeheartedly embracing a customer-centric mindset and upholding high engineering standards, software teams can confidently create products that are both technically sound and highly valued by users.

Over the years, I have become much more seasoned in the industry. This includes changing my mindset and being more flexible in my thought process. I have entertained ideas that would be considered offputting right out of college. Software Engineers should focus on writing code. Right?

Software Engineers should be good writers. They should be great writers. If a tree falls in the forest and no one hears it fall, did it even fall? If a software engineer delivered a project and no one knew about it, did they even deliver it? The comparison is intentionally outrageous to draw attention to the point. You need to express yourself and talk about your achievements.

None of us should expect others to “just know” about our achievements. Do you know everyone’s achievements? Even if they somehow know about your achievements, do they have a proper picture, or is it a misrepresentation? When you expect others to know about your work, you are handing over the power of narrative. Seize it! You should control the narrative of your accomplishments.

If you know how to write well, you will be much more effective in communicating your ideas. Writing is more than just a bunch of words strung together. It’s a story. It should be compelling and hook the reader. It should be precise and express what you mean.

I strongly suggest reading William Zinsser’s book “On Writing Well” about the importance of writing well. It is a game-changer. When I compare my writing style today to two years ago, I can see a stark difference. I get to the point quickly. I strive to start with the punchline. I aim to embrace the letter and spirit of the book in the coming future.

When asked the question “How can we reduce tech debt?” it more than often turns into a discussion of priorities. In a lot of those moments, the discussion is valid, but just like a significant chunk of the discussions, we are just bikeshedding. Instead, focus on

1. What is your definition of tech debt? Do people in this room (or the zoom call) mostly agree with your definition?

2. Why do you want to reduce tech debt?

3. Is this thing actually tech debt?

Is debt bad? Can tech debt be compared to traditional debt or mortgages? Do you necessarily have to pay it back? From whom have you borrowed the debt?

Subscribe now

If you have been an intermediate experienced software engineer and approached the management or leadership to present your “One True Plan to slash Tech Debt by Half in three financial quarters™” it’s very likely you walked back with a sour taste in your mouth, the taste rivaling spoilt milk. Reading Listening between the lines, you can hear them screaming “Ummm, no, I am trying to tell you NO without sounding rude”. The management is either hostile because they don’t understand it or much more likely to have heard this proposition a millionth time. My money is on the latter.

Just ask Netscape when they tried re-writing... you hear that right. Rewriting Netscape from scratch while Microsoft ate its lunch with Internet Explorer (which resulted in another tragedy for the internet). Big-gigantic-huge changes without any short-term gains are extremely risky. We consistently make similar mistakes and need people to review our proposals and whip us back into shape. I personally prefer presenting my proposals of this nature to people who possess no idea of the product so that they can ask fundamental questions which I might have ignored as it’s quite easy to put your blinders on.

Why do you want to reduce technical debt?

Bear with me cowboy! I know you want to bring home the sheep and celebrate with your buddies. Tell me the reason why you want to reduce technical debt? Is it because it offends your sensibilities that something is tied together by duct tape? Maybe the tape is indeed strong enough to hold that thing together till the sun devours our planet. Maybe it’s an aviation-grade speed tape that can hold jet engines together?

At this moment, I am not raining on your parade. We, humans want a reason for doing something? We humans ask ‘What’s in it for me?’. If you are pitching your tech debt reduction bill to your manager or shark tank, you need to explain ‘why’. You are a salesperson and the management is your client. Tell them the value proposition.

Share

Does the customer care? How will it benefit the customer?

You can go and ask the customer - Do you want a button which does this? They might understand what that button does. The customer might provide an enthusiastic response which is a ‘yes’ or at worse make you run around because no one wants to spend time understandin if this customer feature will be useful to them.

Now go and ask the customer if they want to reduce the technical debt? The people in the conference room will be looking at each other trying to understand what it means. The finance people might wonder if this is a new kind of bond to raise funds.

At this point, its upto you to understand if this even benefits the customer. Try to keep asking ‘why’ till you reach the actual question. Think of this conversation

Question: Why do we need to reduce tech debt?

Answer: Our CorpseKeeper module has tight coupling with GothamBeautifier which makes it difficult to maintain DogeCrypt for multiple version and deployment takes over 12 hours?

Question: How does the tight coupling make it difficult to maintain multiple versions?

Answer: It makes the developer life difficult as we have to test every version extensively before moving ahead to release.

Question: How will quicker deployment time help us and the customers?

Answer: If we have to deliver a quick fix then 12 hours is too long and customer is kept waiting.

Question: How long should the deployment take? How often are these quick fixes? How much time would it take to bring down the deployment time? What can break?

Answer: …

The conversation can keep going on until the benefits to the customer is determined.

Will it ever bite you in the future?

Its entirely possible that the hack using bubblegum and duct tape is not really solving any major problem or providing any valuable service? How many customer use that code path? Even the most successful companies make aweful products that fail spectacularly. Maybe the customer uptake will never be high enough to make any difference.

Even worse, the software or the component can suffer a slow agnoizing death. At that time, is reducing tech debt a priority?

Time is limited

Take off your software engineer hat and put on your account manager hat. Would it be better to use the time to add another half broken feature? Maybe! If it increases product sales, then why not? There is no shortage of adequately working ‘Temporary Hack, Fix Later’ codeblocks.

10,000 lines of revenue generating rotting code is better than 10,000,000 lines of elegantly written code which passes all linting requirements, but is not really seeing any action.

Maybe good code doesn’t exist at all. Maybe it does and we should focus first on practial requirements like minimum viable product, requirements clarification, timely releases, adequate bug fixes.

This is not a ‘good code’ slander, but a pushback against ‘good code’ getting more attention than it deserves at the cost of multiple other equally important factors.

Leave a comment