2023年10月27日 星期五

2023年10月8日 星期日

Automotive Grade Linux (AGL)

Open concerns observed in AGL:
* No common kernel version/branch
* Lack of functional security certificates
* Lack of cyber security certificates
* Not hard real-time
* Full of IP-polution (license-vs-legality)

Source 4 Critical Questions You Should Ask About “Automotive Grade Linux” (linkedin.com)

4 Critical Questions You Should Ask About “Automotive Grade Linux”

By Kaivan Karimi

I get nervous each time I hear that someone is developing a Linux-based product for the automotive market, even if it is Automotive Grade Linux (AGL).

To be fair to the AGL community, they have the best intentions as reflected by the organization’s objectives: 1) to build a Linux-based open-source software platform for automotive applications that can serve as the “de facto industry standard” through a collaborative effort of automakers, suppliers and technology companies and 2) to effect “rapid innovation and faster time-to-market for new products” by reducing industry fragmentation and allowing members to reuse the same code base. While these are solid objectives, I disagree with the choice of Operating System (OS) they have made for their mission.

The idea behind using open-source software in automotive started in the mid-2000s, with a focus on infotainment and cabin functions. At that time, the concept of connected car was in its infancy and the word ‘connected and autonomous vehicles’ was a thing of the distant future. Today we know that most cars are becoming connected and that self-driving cars with various levels of autonomy are right around the corner.

In the growing age of connected and autonomous vehicles, what the automotive industry needs in its software more than anything is safety and security. When it comes to choosing AGL for critical systems on which human lives depend, such as advanced driver assistance systems (ADAS), functional safety and autonomous driving, Linux-based OS’s pose four crucial questions that should be considered.

1. Is AGL free?

Often, the idea of open-source software is equated to being low or no cost – otherwise thought of as ‘free’. This is far from the truth for the AGL OS.

The idea of free software may be appealing, until you have to hire an army of engineers to maintain it; and once you do that, you realize you have traded the price of licensing an off-the-shelf proven piece of software with two to three times more cost of added operating expense. In other words, you trade a smart capital investment in a quality OS with continued Opex in another system, which increases your organization’s liabilities.

I vividly recall one of our customer’s experiencing this pain. A couple of years ago, I had visited this customer’s beautiful campus, which had an abundance of open space for their engineers to brainstorm, as well as room for foosball and air-hockey tables. Last summer, I went to see this customer again, who after two years of the “Free Linux Experiment” was ready to come back to us for their future products. This time, I only saw crammed cubicles, with no free floor space. The business manager I met with explained to me that the old business case behind using Linux went out the door shortly after they adopted it, as they had to hire four to five times more engineers than originally planned, buying new tools and implementing new processes to monitor IP-pollution related issues. To this day, every time they ship a product they are nervous about whether they will soon see their company’s name in the news due to a security hack.

Costs grow exponentially when you layer in the requirements of testing/code coverage, verification of software of unknown provenance (see ‘Is AGL Safety Certified?’), proven development processes, meeting specifications and tight revision control. With BlackBerry QNX there are no hidden fees to hit your OPEX, and you can focus your engineers to work on value-added differentiating features of your products, versus non-differentiating foundational “core-plumbing” software. Since QNX is a POSIX-based Real Time Operating System (RTOS) designed specifically for embedded applications, it allows Linux developers to keep their programming model and port their application and middleware code, usually with zero-to-minimal changes, to QNX.

2. Is AGL Easy to Manage?

Patch management and version alignment is a big issue for embedded Linux. The sheer number of possible combinations of configurations and the fact that some updates may not work with other updates becomes challenging. This means that when a critical update arrives, it may not just be a matter of simply adding it to the release. Many patches or changes have ripple effects and dependencies that can be far reaching. This only gets worse when you talk about GENIVI, where “multiple suppliers can be compliant to GENIVI Specification, each with different starting points, with much less interoperability and reuse”.

An OEM especially needs to consider pros and cons of AGL very carefully. Last month, I was meeting with one of the OEMs who is switching from AGL to QNX for their next generation products. A head of software development for one of their products was telling me about the configuration management headaches he had to deal with. With at least 4 Tier 1’s providing a different version of the AGL kernel for their particular products, they were now managing 4 different AGL kernels for different patches. Each change in the code base really meant more than one change, and with hundreds of changes during the development cycle, they had a dedicated team focused on managing these Linux distribution differences.

3. Is AGL Safety Certified?

Linux is considered to be a software ounknown provenance, aka SOUP. SOUP is a term often used in the context of safety-critical systems. The issue with SOUP is that since the software is developed from a plethora of sources, the origin and quality of the software code is unknown and, therefore, cannot be relied upon to perform safety-related functions. Due to its very nature, Linux doesn’t provide records of the development processes, and there are no requirements, traceability matrices or development plans to back up the design of the software. This means that the onus is on you to provide proof that the software does what it’s supposed to do. Given the size of the Linux kernel (tens of millions of lines and growing), and the constant change going on in the source base, this is a daunting task, and adds a significant cost.

As a SOUP, the Linux-based OS begs several questions such as, when you use open source and “leverage the work of 100s of other companies,” what happens if a bug arises? Who is liable in the consortium with hundreds of contributors? Who is responsible for performing safety impact analysis on all these issues and providing a fix? And with a monolithic architecture, every change to the code base would invalidate the existing safety certification, leading to the need of re-certification, which can be extremely costly and time-consuming.

With BlackBerry QNX, the software comes pre-certified to the highest levels ISO 26262. Also, you know the software you are getting and can track that code back to its original source. Furthermore, the freedom from interference provided by QNX’s microkernel architecture makes certification significantly cheaper and less time consuming. This is because most file systems, protocol stacks and device drivers exist outside of the kernel as user-space applications. QNX is ASIL-D safety certified, meeting all safety standard requirements of the automotive industry. Automotive Safety Integrity Level (ASIL) is a risk classification scheme defined by the ISO 26262, Functional Safety for Road Vehicles standard. Claims that are still in R&D labs and yet to produce anything commercial, after over 10 years and millions of dollars, do not count.

Also, Linux is a general-purpose OS (GPOS), which means it does not provide mechanisms to ensure timely responses to events and interrupts like a real-time OS, such as QNX’s, does. This is achieved through careful system design for low latency, and a priority-based scheduling algorithm, in which the highest priority task ready to run always runs immediately, ahead of a lower priority task. Linux has a real-time scheduling option, but it’s what is called "soft" real-time which means real-time “most of the time” but not all the time. This is because typical latencies in Linux are orders of magnitude higher than a real-time RTOS like QNX.  Real-time performance is especially important in mission-critical applications, in which tasks must run in a deterministic manner. If the designer doesn’t have complete control of scheduling, unpredictable and unwanted system behavior can and will occur. 

4. Is AGL Secure?

An AGL OS is an invitation to hackers, and this is where Linux fails the most. In this day and age, where major hacks are an everyday occurrence covering the front page of every newspaper, and chief security officers are focused on fortifying their products and endpoints against modern cybersecurity threats, why be so foolhardy to advertise your vulnerabilities to all hackers around the world? 

Secure-Linux is an oxymoron, like jumbo shrimp. There is nothing “jumbo” about something that is less than the length of one’s hand, just as there is very little that is secure about a Linux-based OS. As a result, I am clearly confused (yes, another oxymoron) about why automotive manufacturers would consider Linux in this regard.

Linux’s security problems are systemic and long-standing, and there is no way to fix it to the level of security required for mission-critical applications. The table below from Statista shows the security vulnerabilities of the various flavors of Linux:

No alt text provided for this image


The vulnerability in this table is defined as a mistake that can be directly used by a hacker to gain access to a system/network. Among the CVEs (Common Vulnerabilities and Exposures) reported here, many had CVSS (Common Vulnerability Scoring System) score of “9” or higher, meaning the vulnerabilities were deemed severely critical in nature.

Now let us get specific about AGL and their special flavor of Linux. Below is a table from BlackBerry’s last security summit, where using specific tools the AGL camp’s infotainment implementation was analyzed from a security stand point:

No alt text provided for this image


Of the 1,295 vulnerabilities, 176 of them were deemed “critical” and 857 were deemed “major” in nature.

With over 150 million lines of codes planned for most mid-size sedans in 2019, and the fact that typically for every 100 lines of codes, there are 8-15 vulnerabilities introduced to the code base by software engineers, why would one want to make the situation worse by using Linux?

It’s time to get serious about safety and security requirements of automotive software. At BlackBerry, we have an unmatched portfolio of products for the Automotive industry, which includes:

·        BlackBerry QNX’s foundational certified software for mission critical applications such as ADAS platform;

·        BlackBerry QNX’s market proven solutions supporting Virtual Cockpit Controllers (Instrument Cluster, Infotainment systems, Telematics, wireless gateway, and Cabin functions)

·        Best-in-class innovative security solutions needed for connected and autonomous cars; and,

·        Software and security life cycle management technologies that are optimized for the future of the automotive market.

Software is becoming more pervasive across all domains of the car, and only one OS can cover this broad a reach – BlackBerry QNX. The forward-thinking OEMs know they need a solid base that enables every aspect of the foundational software for the car and allows them to use Android where needed for things like apps within a secure and controlled container that our hypervisor provides.

In the automotive market, our portfolio is backed by 20 years of on-time delivery for more than 280 Start Of Production (SOP) programs over more than 40 auto platforms and 240 auto models.

If we were discussing the general embedded market, everything here would also easily apply to medical or industrial equipment needing safety certification (e.g. General Embedded safety - IEC 61508 SIL 3 - and Medical devices - IEC 62304) and true security. 

****************************************************************

Adding an updated chart with information from NIST (National Institute of Standards and Technology):

No alt text provided for this image


#ADAS #Autonomous #Software #Hardware #Software Complexity #Hardware Complexity #cybersecurity #automotive #autonomousvehicles #connectedcar #virtualization #infotainment #operatingsystems #qnx #android