Bridging Bytes and Bolts for an Integrated Software and Hardware Development Lifecycle

New technologies and market challenges call for streamlining physical and virtual development — we’ll explore 5 means to enhance efficiency, collaboration and integration of those interlinked disciplines.

tl;dr: In a fast-paced market, it’s crucial to integrate hardware and software development lifecycles closely, paving the way for a more efficient process, improved collaboration and seamless integration. We can achieve this through collaborative tooling and with an adapted and iterative version of the V-Model to build viable, resilient, and innovative products that our customers love.

It’s October 29, 2018 when a Boeing 737-Max operated by the Indonesian airline Lion Air crashes into the Java Sea 13 minutes after takeoff, killing all 189 people on board.

Boeing’s 737 Max airplanes that have been involved in 2 fatal crashes and several more incidents in context of their flight stabilisation system MCAS set a tragic example of how underestimating the complexity and necessity to integrate hardware and software development can lead to deadly consequences.

The MCAS (Maneuvering Characteristics Augmentation System) was designed to avoid stalling and it should have served as a support system for the pilot to control the aircraft. Instead, relying solely on data from one sensor, discrepancies and failures in the sensor’s readings led to erroneous software commands, resulting in the nose of the aircraft being pushed downwards, the pilot unable to manually counteract with the system’s commands — with fatal consequences.

The reliance on a single sensor for the MCAS underscored the dangers of insufficient integration between hardware and software as redundancy and robustness in the system’s design should have been key aspects in the collaboration between hardware and software components to ensure the safety and reliability of complex systems like an aircraft.

For everyone who has been working in an environment where hardware and software both contribute to the success of a product, the struggles to streamline both lifecycles to a shared “one-product mentality” may have become apparent very quickly, not only for developers and engineers but also for the end customer who might sense that the product is a compilation of different sub-parts instead of one entity.

The adaptation of agile principles in software development, in contrast to hardware environments, has started already two decades ago, and founding a standard operating model for modern software companies today to build competitive and adaptable products and attract talent.

Agile frameworks such as Scrum, Kanban or Lean Startup offer an iterative approach to product development:

Scrum is an agile framework emphasising iterative development, collaboration, and adaptability through defined roles, artefacts, and events, notably the iterative planning and execution of backlog items and the delivery of incremental value through a sprint cadence.

Kanban is a method for managing work, focusing on visualising workflow, limiting work in progress, optimising flow, and promoting continuous improvement.

Lean Startup is a methodology for rapidly developing businesses and products by building, measuring, and learning from a Minimum Viable Product (MVP) to validate assumptions and make informed decisions. It is geared towards validating business ideas and products in uncertain and dynamic market conditions.

And while LEAN principles, originating from the manufacturing sector, have indeed also transformed production processes across various hardware-centric industries and brought about improvements in efficiency and waste reduction, the shift to agile methodologies and iterative development in this domain is not as straightforward due to the characteristics of the manufacturing process and its inherent sequential nature.

However, there are ongoing efforts to tailor agile principles to hardware development and bridge the gap between the bytes and bolts of hardware and software lifecycles — because ultimately, the final product will be only as performant/ resilient/ attractive/ innovative as its weakest link.

Staying within the metaphor of bytes and bolts — the self-driving car industry is another good example for the intricate collaboration between hardware and software components. While the “bolts” represent physical components of a self-driving car, such as sensors, cameras, lidar systems, GPS modules, and actuators, “bytes” are the digital essence of the autonomous driving system’s intelligence, comprising algorithms, machine learning models, control systems, and software frameworks. Each bolt is a crucial piece of the physical puzzle, contributing to the car’s ability to perceive its environment, make decisions, and execute actions safely. But without matching bytes, the processing of data from the hardware sensors, interpretation of road conditions or obstacles, and calculations for optimal driving paths could not happen, potentially resulting in similar fatal consequences as Boeing faced with its aircrafts.

Working as an Organisational Development Consultant with clients across manufacturing industries, e.g. Mobility and Aviation, I have seen them struggling to find the right tools to bridge the bytes to the bolts. So imagine that we were to build a self-driving car where hardware and software truly need to be seamlessly integrated for building a great product. To get this car safely on the highway, it is important to first understand what is causing the scuffle between those interlinked disciplines.

Depicting the Differences

• As mentioned, agile methodologies in software engineering foster an iterative development, allowing for frequent updates, which is much harder to achieve with hardware.
• The same applies for testing and integration which in software development primarily occurs in virtual environments and can be heavily automated. Contrastingly, hardware development involves physical components such as a GPS module or a sensor that need to be tested and validated in laboratories.

Having those two significant differences in mind, let’s dive into some theory around the so called V-Model, which will explain and exemplify the challenges that come with traditional hardware development and it’s (perceived) incompatibility with an iterative development approach. The V-Model is a conventional system engineering process model, commonly affiliated with waterfall development. In this model, each stage needs to be completed before progressing to the next, reflecting a sequential and structured methodology. It is prevalent in hardware-centric contexts and the development of mission-critical systems, particularly in domains such as automotive and aviation, but also energy, and military applications. When working with hardware-heavy domains, I stumbled across it frequently as the main way of managing the product development lifecycle.

The V-Model is graphically represented in the shape of a “V.” The left side of the “V” signifies the initial stages of the development process, involving the decomposition of requirements and the creation of system specifications. In this phase, the project team dissects and defines the requirements to form a clear understanding of the system’s architecture and functionalities.

Conversely, the right side of the “V” symbolises the latter stages of the development process, focusing on component integration and testing to ensure that they function as intended. The model aims to emphasise the critical relationship between the early design decisions and the later stages of implementation, integration, and testing which happen sequentially.

Comparing the V-Model with agile frameworks (Scrum, Kanban, Lean), its structure marks a significant difference to the iterative approaches within the software domain. Why this model therefore needs an adaptation will be explored later in this article.

Back to more differences.

• A very obvious point to mention as a difference between hardware and software development is for sure the aspect of tangibility. In software, changes are easily implemented, and modifications to algorithms or a machine learning model occur swiftly without physical constraints. This is a different story for a hardware part such as a sensor, where changes require a physical change of the unit.
• Modifications in hardware which necessitate manufacturing and physical alterations introduce complexities in terms of time and cost. Software development generally incurs lower and flat costs over time, accompanied by a faster time-to-market. Conversely, hardware development entails higher costs due to materials, manufacturing, and testing, with a longer time-to-market (AND consequently “time-to-money”) and rising costs towards the end of the development cycle.
• While iterative development for hardware components is partially feasible, it often requires the creation of several physical prototype stages, leading to more time-consuming and expensive testing cycles. Software relies on virtual prototypes and mock-ups for testing and early user feedback.
• Lifecycle durations differ significantly as well, with software featuring a shorter lifecycle characterised by rapid updates. Technologies may become obsolete quickly. Hardware, on the other hand, boasts a longer lifecycle with a focus on long-term reliability and less frequent upgrades, especially once the product has been released. Imagine a client would need to frequently bring their car to a workshop to exchange a sensor in order for the self-driving system to keep working. In contrast, a software update could be distributed to all cars with a network connection within a few minutes.
• Lastly, one does not come around talking about regulatory requirements. Software compliance predominantly revolves around adherence to data protection and cybersecurity standards, such as the European GDPR legislation. In contrast, hardware compliance spans a broader spectrum from safety standards to electromagnetic compatibility, environmental standards, and more. Meeting regulatory requirements involves physical certification processes. Unlike software, hardware must undergo meticulous examinations to ensure it not only meets but exceeds safety and compatibility standards set by regulatory authorities. A tedious and often time-consuming verification phase and necessary reworks are not seldomly delaying the go-to-market of a product. But rightfully so — as we saw with the Boeing example in the beginning.

Exploring the toolbox — 5 Means of Synchronising Lifecycles through (Agile) Processes

After having examined the differences of hardware and software lifecycles, the question remains — how to bring this together to work more seamlessly? First of all, I may need to disappoint any expectation of a clear-cut solution to this complex question — a single, definitive answer remains elusive. Instead, what we will be exploring in the following are five means, not solutions, devised to grapple with this inherent complexity. Consider these approaches as sources of inspiration taken from a great variety of tools in a toolbox, offering strategic insights on how to build a more cohesive bridge between the worlds of hardware and software development.

1. Integrated Planning: Companies may establish a shared planning process that involves both hardware and software teams in a shared end-to-end value stream. This ensures alignment in terms of project goals, timelines, and resource allocation. By fostering a unified planning approach, teams can work towards common objectives, enhancing overall project cohesion. For our self-driving car this would for example mean to have a value stream concerned with the routing functionality of the car. Software engineers provide insights into the computational requirements for real-time route planning, while hardware engineers consider the sensor and processing capabilities needed for data collection and analysis and create a shared roadmap with regular integration points to ensure compatibility and functionality.
2. Cross-Functional Teams: Form cross-functional teams with members from both hardware and software disciplines. In this collaborative structure, hardware and software engineers can work closely to co-design sensor systems and routing algorithms which promotes a shared understanding of the entire development process, breaking down traditional silos. Continuous communication within these teams ensures that hardware and software considerations are integrated from the outset, reducing potential conflicts during development.
3. Agile Principles: Embracing agile principles, using methodologies and frameworks, such as Scrum or Kanban, and adapting them to the specific needs of hardware development is probably the most crucial and complex endeavour as there is no one-fits-all solution. Teams need to incorporate practices like stand-up meetings, sprint planning, and frequent reviews. Retrospectives allow for iterative refinement of processes and the adaptation of best practices over time. This dynamic approach enhances flexibility and responsiveness in the face of evolving project requirements, but it is often hard to find “the right fit” at the beginning, requiring a steep learning curve on how agile methodologies and tooling can work in the specific context.
4. Iterative Development, Continuous Integration and Testing: Regardless of the (agile) framework, one crucial aspect to a successful and adaptable product development process is an iterative approach where both hardware and software components are developed and validated incrementally to reduce integration challenges and identify issues early. Looking at the self-driving car, in specific the routing value stream, this would include hardware-in-the-loop simulations which allow software engineers to test routing algorithms with realistic sensor inputs, while hardware engineers assess sensor performance under various routing scenarios and by continuously deploying new and adapted characteristics to the system to test varying use cases, conditions and scenarios with low effort. The same approach could have been applied to test Boeing’s MCAS system working with the respective sensor. Though hardware prototypes might be costly, it is essential to not cut expenses here — successful companies such as Apple and Tesla rely on rapid and iterative prototyping to validate design concepts, identify flaws, optimise manufacturing processes, test user experience, and maintain a competitive advantage in their respective industries. Through iterative prototyping, they can ensure that their products meet the highest standards of quality, functionality, safety and user satisfaction. The costs for prototypes will still be a fraction of the costs of a failing end product.
5. Common Tooling: It is not uncommon for hardware and software teams to utilise different tools an platforms for project management and communication. But establishing common tooling as a single source of truth can facilitate collaboration between hardware and software teams. Shared documentation outlining interfaces and dependencies ensures a clear understanding of how changes in one area may affect the other. This common tooling approach enhances transparency and minimises the risk of miscommunication, promoting a more cohesive development process.

Again, it is needless to say that implementing processes and tooling are still no guarantee for success. One important constant throughout change management efforts and improvements should be strong leadership support and sponsorship. Leadership empowerment extends beyond providing direction for the product to creating an environment where individuals and teams can thrive, fostering a culture of collaboration, innovation, responsibility and growth. Empowered teams are encouraged to take ownership of their work and are held accountable for outcomes rather than outputs, both for how they work and what they produce.

On a transformational journey, pitfalls will still be encountered despite best efforts. In the following section, we will have a closer look into some common pitfalls and how to mitigate them.

Common Pitfalls

• Clear communication is vital for successful collaboration. Inadequate or inconsistent communication can lead to misunderstandings, delays, and integration issues.
• Equally important is aligning priorities between hardware and software teams. Differences in focus may cause conflicts and delays, emphasising the need for a shared understanding of project and product goals. A strong direction set by (product) leadership can help to set a north star and empower teams to subsequently set and align their priorities.
• Effective dependency management is as well pivotal to avoid bottlenecks. Poorly managed dependencies can lead to integration challenges, e.g. when interfaces and specs are not well-defined.
• Inadequate testing poses another potential risk. Rigorous testing throughout the development process is indispensable to ensure the resilience and reliability of components and their integration into the system.
• One of the more intricate and hard-to-solve challenges are skillset differences and variances in understanding agile processes between teams and individuals. Bridging this gap requires a shared comprehension of each other’s domains but also the willingness of employees (especially those that are not used to agile / incremental development) to learn and change their means of working. This should be accompanied by seasoned agilists and engineers who are familiar with agile concepts and who put an effort in understanding the realities and work patterns of those colleagues who have not been working agile before.
• Resistance to change is therefore another common challenge, necessitating adept change management strategies and clear communication about the benefits of synchronisation. One strategy would be to identify multipliers (personnel who foster collaboration practices within teams and disciplines) and sponsors (in leadership positions) to structurally work on a mindset change within the organisation. Building lighthouse teams and identifying role models would be another.
• Lastly, the scale and complexity of projects can pose unexpected challenges. Breaking down large projects into manageable increments and ensuring effective synchronisation and integration for each increment is vital for success.

In conclusion, navigating the numerous pitfalls in the development of complex critical systems requires a combination of strategic planning, effective communication, and a commitment to adaptability in the face of evolving challenges. And perhaps a new approach to a dusty model…?

Why the classic V-Model fades into the Agile Dawn

The identified pitfalls in the collaborative efforts between hardware and software resonate with the characteristics of the classical V-Model that was presented earlier. Imagine that our self-driving car’s hardware engineers would build their sensor suite of cameras, GPS and lidar sensors or design processors and GPUs without ever depicting and cross-testing the stages of their prototypes with the actual routing algorithms until they had produced the final units after months (or years) of development. With high probability, the software would not run on the hardware, errors would be encountered that needed to be investigated in a very late stage of the project, pushing the release timeline further and costing the company money.

The inherent nature of the V-Model accentuates the risk of compartmentalisation, hindering the collaboration and communication flow required between hardware and software teams. These pitfalls underscore the importance of exploring alternative methodologies.

The Agile V-Model

Transforming the traditional V-Model requires a comprehensive strategy that nurtures collaboration, flexibility, and adaptability. Encouraging simultaneous development of hardware and software components, including design, coding, and testing concurrently, propels a swifter and more iterative development process.

Incorporating incremental prototyping into the V-Model is pivotal, facilitating early testing and validation, embracing iterative development involves breaking down the project into smaller increments, aligning with agile principles of delivering functional increments and gathering (user) feedback early on. Adapting towards continuous integration and testing, functional or integration issues of code and hardware components can be encountered early in the development process.

Adding again the idea of an end-to-end value creation, the incremental development for our self-driving car routing could start with developing and testing a baseline functionality, focusing on essential components such as basic perception, localisation, and simple route planning, implementing rudimentary algorithms for obstacle detection using a single type of sensor (e.g. cameras) and a basic decision-making logic for following predefined routes. After successfully integrating and verifying the first increment, the next iteration with new or multiple prototypes could start where the system’s perception capabilities would be enhanced by integrating data from multiple sensors (e.g. lidar, GPS) using sensor fusion techniques, incrementally improving object detection and tracking algorithms that can handle various scenarios, including different lighting conditions or complex traffic environments.

Implementing agile ceremonies — such as sprint planning, daily stand-ups, and retrospectives — within the V-Model framework is imperative. These ceremonies offer platforms for teams to make their status transparent, align goals, discuss challenges, and perpetually refine their processes, allowing for development and integration within a single iteration (“V-Sprint”) or alternating between development and integration sprints.

Additionally, a shift towards a more flexible approach to requirements management is essential. Acknowledging that requirements can evolve throughout the project permits adjustments based on continuous feedback and dynamic shifts in priorities and market conditions. Substituting rigid planning with adaptive planning is equally crucial, necessitating regular reassessment of project goals, timelines, and priorities in response to the evolving needs of the project and the organisation. Just imagine that technological advancements like new sensor technologies become available during the course of the project but the company would not be flexible enough to adapt those and profit from improved performance or potentially lower resource usage or costs.

By integrating these agile principles and practices, the evolved Agile V-Model fosters greater collaboration, facilitates faster delivery of value, and enhances responsiveness to changing requirements and technological advancements, ultimately promoting a more adaptive and efficient development process that also secures aspects such as safety and resilience and hopefully helps to avoid deadly incidents such as with the Boeing 737-Max.

Wrapping it up

Though these ideas should generally not be anything “new” (considering that LEAN and Agile practices are out there for decades), many hardware-centred product companies are only recently starting to adapt their classic V-Model development and integration of hardware and software disciplines, e.g. in IoT.

On the highway of product development, where innovation is the accelerator and adaptability is the key to survival, the classic V-Model stands as a relic of a dusty era like a combustion engine.

It’s time to shift gears and embrace a more dynamic approach in the development of physical products, especially in industries such as manufacturing, plant engineering, automotive and aviation. Their products should shine not only through solid, safe and cleverly built hardware but foremost innovative software features, built iteratively through a continuous flow of deliverables and improvements.

About the author

🦄 Lisa is a consultant at Netlight, working as an Organisational Coach and Product Manager. She has gathered expertise in Coaching, Product Leadership and Project Management, but also worked actively in developing backend applications. Bridging the gap between technical and business perspectives, she is committed to providing end-to-end support for projects, from their strategic conception to development and launch.

Having worked in various project setups and industries such as aviation, mobility, e-commerce, logistics, and SCM, she learned to quickly adapt to new environments and contexts. Her dedication to her work is fuelled by her passion for people, new technologies, and the desire to enable organisations to build exceptional products.

About Netlight

Netlight is a leading digital consultancy company providing a full range of consultancy services from tech and data to organisational coaching and management. Our 2000+ consultants across 11 offices in Europe help to make aspiring digital companies and their leaders successful.

Disclaimer: Agile frameworks and V-Model figures created by author. All other images created using AI image generation.