Assessing the quality of management in any organization is as much an art as it is a science. Moreover, the approach to assessing varies widely from organization to organization. However, this does not have to be the case, as these authors point out. Managers can develop a set of attributes, or measurements, that boards of directors and senior executives can repeatedly use to determine whether or not an organization is on track.

Assessing the quality of management in any organization is as much an art as it is a science. Quite often, the approach varies widely from organization to organization, as does the definition of quality. Our own experience and research have led us to conclude that this, in fact, does not have to be the case. Our work shows that it is both useful and possible to develop a comprehensive set of performance ideals or attributes for an entity that is notoriously difficult to manage—the research organization. We describe and discuss those attributes in this article. We also believe that attributes can be developed for other types of organizations, such as software firms or insurance companies, as well as government agencies and universities. In the end, the development of attributes provides boards of directors and senior executives with a critical frame of reference they can use to determine whether an organization is on track or not, and, if not, why not.

ATTRIBUTES: WHAT ARE THEY AND WHY THEY ARE NEEDED

The idea of attributes comes from the question, “How could you tell if an organization is well managed?” In other words, is there a set of performance ideals that could be used to assess the quality of management in any organization? To be most useful, these attributes would have to be end-results oriented questions that go beyond the development and implementation of best practices. From a CEO’s perspective, best practices are means to the end in mind. So, our initial question leads to several others. What are the intended results of best practices? What should boards and CEOs be looking for to be sure that best practices are in fact moving the organization towards the intended results? Can these results be stated in terms that are useful, observable and, preferably, measurable?

The development of attributes by high-performance research organizations serves many purposes:

1. Over the past six years, the Office of the Auditor General of Canada has presented a series of reports to the House of Commons on various aspects of federal science and technology activities. Our work underlines the need for a universal description of a well-managed research organization.

2. Companies are increasingly dependent on the results of research for new and improved products to maintain competitive advantage. Governments view industry-driven science and technology as economic engines, and they are increasingly dependent on their own science and technology programs for dealing with public policy issues such as climate change and the impacts of toxic substances. Furthermore, governments are placing more emphasis on achieving results, e.g., Results for Canadians, an initiative of the federal government, and the Government Performance and Results Act in the United States.

3. Assessing the performance of, and return on investment (ROI) from, research is a challenge faced by private and public sector executives as well as politicians. Research is a risky activity, and not all research activity leads to expected results. Furthermore, the benefits from research sometimes take years to materialize.

4. The information available to boards and CEOs for assessing the performance of research organizations is inadequate. The information tends to focus on past performance (e.g., published papers and patents) and on process and activities. However, high performance in the past does not guarantee the same for the future. In today’s fast-paced world, executives need better and more current information. Furthermore, performance assessments that focus on process beg the question: What has happened as a result of having implemented good practices?

Approaches to assessing the performance and ROI of research organizations generally fall into three categories:

a) Retrospective evaluation—examining the relevance and impact of research conducted in the past.

b) Current evaluation—examining the organization’s vision, strategies, target clients, practices and people.

c) Future evaluation—examining planned research, its relevance, potential benefits and likelihood of success.

The methodologies are most advanced for retrospective evaluations, and tend to involve costly studies conducted by third parties. The results of retrospective evaluations influence future funding and investment, and identify the lessons learned and insights for the way ahead.

The more important management challenge lies in future performance. Learning from the past is important, but far more critical are the questions of “Where do we go from here?” and “How do we get there?” Another key question is “How can one gauge whether the research organization is well managed and focused on relevant end results?” The main approaches to answering this question are:

1. The use of performance measures: The most promising approach is based on the balanced-scorecard concept developed by Kaplan and Norton.

2. Drongelen and Bilderbeek have examined R&D performance measurement practices in the Netherlands, indicating that the balanced scorecard provides an appropriate framework. (Inge C. Kerssens-van Drongelen and Jan Bilderbeek, “R&D performance measures: more than choosing a set of metrics,” R&D Management 29, 1, 1999.)

3. The identification, development and implementation of best practices: Best practices implemented in research organizations are based on quality management, Baldridge, ISO 9000 and other such constructs. These efforts are both good and important. However, the implementation of best practices does not help executives address questions such as: Having implemented the best practices, so what? How would one know if they have had the desired effects? What are the desired effects or ideal outcomes?

In order to move beyond the current approaches, we must first identify the attributes of high-performance research organizations. We define “attributes as the characteristics of research organizations that produce, over time, and have a reputation for producing, results that are used and that have a significant impact. That impact can be on generating wealth, developing public policy, protecting the environment or improving people’s well being.

Attributes are not a recipe for action. They are performance ideals that organizations strive to achieve, but will never reach fully. As such, they are offered as statements of the direction that management should take, rather than a plan for getting there. In Identifying and developing attributes, we:

• Focused on what senior managers of research organizations consider to be important—“the view from the top”
• Endeavoured to articulate attributes so that they could be observed, evaluated and measured
• Looked for strategies, policies, practices and systems that research organizations use to improve performance in relation to the attributes.

DEVELOPING ATTRIBUTES

We used an iterative approach to develop the attributes, one that involved a combination of fact-finding and analysis, followed by review and challenge. A panel of experienced experts and well-known research organizations carried out the review and challenge. The organizations that participated in our study were drawn from important sectors of the economy and represent the broad range of research activities undertaken by government: knowledge creation, wealth generation, policy, regulatory and security functions, and the provision of a major research infrastructure.

The Alberta Research Council, a provincial corporation owned by the government of Alberta. Its purpose is to advance the economy and well-being of Alberta by providing technology and innovation to meet current and emerging needs of industry and government.

Argonne National Laboratory, a multiprogram research and development centre owned by the U.S. Department of Energy and operated by the University of Chicago. The laboratory’s mission is basic research and technology development to meet national goals in scientific leadership, energy technology, environmental quality and national security.

Army Research Laboratory, of the Army Materiel Command, is the U.S. Army’s corporate, or central, laboratory for materiel technology. ARL’s mission is to execute fundamental and applied research to provide the army the key technologies and analytical support necessary to assure supremacy in future land warfare.

Office of Research and Development of the U.S. Environmental Protection Agency, the scientific and technological arm of the U.S. Environmental Protection Agency.

Goddard Space Flight Center, NASA’s Center of Excellence for Scientific Research. Goddard’s mission is to expand knowledge of the Earth and its environment, the solar system and the universe through observations from space.

Merck Frosst Canada and Co., a large fully integrated pharmaceutical company. It has R&D programs aimed at discovering novel therapeutic agents for the treatment of allergic, respiratory and inflammatory diseases, and for diabetes, osteoporosis and neuronal injury.

The National Institute of Standards and Technology, a non-regulatory federal agency in the U.S. Department of Commerce. NIST’s mission is to strengthen the U.S. economy and improve the quality of life by working with industry to develop and apply technology, measurements and standards.

Nortel Networks, a Canadian-based global corporation with 70,000 employees in 150 countries worldwide. The corporation is committed to working with its customers and global partners to create a new era of high-performance networks that are changing the way the world communicates and shares ideas.

We identified the preliminary attributes referred to in Exhibit 1 by applying the four perspectives on performance in Kaplan and Norton’s balanced scorecard. We rethought these substantially, using concepts from numerous frameworks for leadership, criteria for quality management and input and feedback from the above-mentioned organizations. As indicated in Exhibit 2, this led us to conclude that the key perspectives on performance for research organizations are:

• -People
• -Leadership
• -Research management
• -Organizational performance

These perspectives bind the attributes around themes we identified in the course of our work. Because the first three perspectives (people, leadership and research management) produce the fourth, organizational performance, they are critical in their own right and demand equal attention. Conversely, organizational performance directly impacts the attributes of the other three perspectives.

THE ATTRIBUTES OF HIGH PERFORMANCE RESEARCH ORGANIZATIONS

The proposed attributes are interdependent and work together to provide an overall picture of good research management. While there is overlap among the attributes, each one sets out an ideal outcome for a different dimension of the management of research.

We have used the word “right” several times. The idea is that when management is faced with choices, some are better than others, and one, the right or ideal one, is the best of all. Management is responsible for selecting a recipe for action, based on key available information and judgment—within the context of its vision, goals and values.

10 ATTRIBUTES OF PERFORMANCE RESEARCH ORGANIZATIONS

PEOPLE

1. Management knows what research and other talent it needs to accomplish the mission, and recruits, develops and retains the right mix of people.

2. Employees are passionate about their own work, have confidence in management, and are proud of their organization.

LEADERSHIP

3. The current and anticipated needs of dependent constituencies drive the organization and its research programs.

4. Employees and dependent constituencies share management’s vision, values and goals.

5. The portfolio of programs represents the right research at the right time and at the right investment.

RESEARCH MANAGEMENT

6. Research projects embody excellent science, involve the right people, are on track and within budget.

7. Research projects leverage external resources.

8. Organizational knowledge is systematically captured and turned into needed work tools.

ORGANIZATIONAL PERFORMANCE

9. The organization is widely known and respected.

10. The organization meets the needs of dependent constituencies.

The remainder of this section describes each of these attributes briefly.

1.Management knows what research and other talent it needs to accomplish the mission, and recruits, develops and retains the right mix of people.

Today’s research organizations require highly competent, multi-skilled professionals. Researchers need to be first-rate scientists as well as effective communicators and team workers. Their technical competencies need to be aligned with the organization’s current and anticipated needs. Research managers must have demonstrated technical competence as well as strong leadership and management skills. Management invests significant time and effort to identify the core competencies required to synthesize and apply knowledge effectively, and recruit and retain staff.

Research organizations use alternative approaches to recruitment (for example, secondments and use of contractors), build bridges to future talent pools (for example, outreach activities with schools, colleges and universities) and have mechanisms to respond to shifts in core competency requirements (for example, transition assignments, retraining and outplacement).

2. Employees are passionate about their work, have confidence in management, and are proud of their organization.

Employee morale lies at the heart of productivity and organizational success. Satisfied employees are creative, innovative, efficient and effective. Furthermore, they:

• trust management
• are treated with respect and feel valued and relevant
• have opportunities to learn, upgrade their knowledge and skills and reach their full potential
• are encouraged to contribute ideas and feel free to speak out on issues of concern without fear of retribution
• are shown empathy in the challenges that they face on the job and away from it
• believe that their overall compensation and job-related benefits are fair and reasonable

Employees tend to be passionate about their work and proud of their organization if they feel that they are contributing to its success, are recognized for their contributions and are empowered to achieve agreed-on results. This requires a work environment where the roles are reasonably clear, and teamwork and participative management are results-oriented. Ensuring that such conditions exist requires that management be proactive. The use of organizational climate surveys can assist management in gauging the health of the work environment. Management’s prompt response to survey results is important to maintaining employee confidence.

3. The current and anticipated needs of dependent constituencies drive the organization and its research programs. Dependent constituencies are those individuals, groups or organizations that absolutely depend on the results of the organization’s research to carry out their own responsibilities. Their effectiveness depends on the research support they receive. Determining current needs and anticipating future needs of dependent constituencies requires focusing on outcomes or end results. It also requires the involvement of management and scientific staff from both the research organization and its constituencies as well as other knowledgeable persons. Being driven by dependent constituency needs implies that the organizational effort be relentlessly aligned with those constituency’s needs.

Because the very existence of an organization depends on its ability to respond effectively to constituency needs, constituency groups and their needs must be clearly defined in order to channel efforts and resources into achieving what is relevant and important to those groups.

The research needs generally far exceed the resources available, and hence, priorities must be set. These are best established through the involvement of dependent constituencies, and with an understanding of the limits on the organization’s capacity to deliver.

4. Employees and dependent constituencies share management’s vision, values and goals. The attribute assumes that the organization’s senior managers provide the leadership to develop a vision supported by values, goals and strategies, and communicate them to employees and dependent constituencies. Without buy-in from employees and dependent constituencies, management’s vision, values and goals are little more than a dream.

It is easier to point to people who were or are leaders than to describe leadership. Nevertheless, by describing leadership from a number of angles, a clearer picture emerges.

• Leadership functions include creating and promoting values and expectation, setting directions (the vision), projecting a strong constituency focus, aligning the systems, policies and resources with the vision and mission, and empowering employees to be productive.
• Leadership in a research environment recognizes that researchers are highly trained professionals who are guided by the standards, expectations and structures provided by the research disciplines in which they work. Leadership is less about directing and controlling, and more about establishing and promoting a shared vision and shared values. It is also about establishing a pathway, harnessing the organization’s talent and resources to achieve the vision, supporting research professionals by creating and maintaining a positive research environment, and building relationships with dependent constituencies.
• Leaders serve as examples and sources of inspiration. Integrity and trust are established when a leader’s actions are consistent with the personal and organizational values. Their actions influence the behaviour of managers, teamwork among scientists and staff, and the sharing of knowledge and learning conditions that are essential for creativity and innovation in a research organization.

5. The portfolio of programs represents the right research, at the right time and at the right investment. This attribute is about getting the best value for money, taking into account that most research organizations serve a hierarchy of dependent constituencies. “Portfolio” refers to a suite of research programs, or groups of relatively homogeneous or highly interrelated research projects. Right research at the right level of investment means selecting research based on a set of criteria.

Criteria that can be used to manage research portfolios include:

• -importance to the dependent constituencies
• -fit with the organization’s mission, goals and overall priorities
• -need for involvement
• -benefit-cost considerations
• -likelihood of success, and
• -overall balance in the portfolio.

Achieving and maintaining the optimum portfolio requires multiple lines of input, review and challenge, including:

• -independent evaluations of programs
• -maintaining a lookout for attractive new opportunities
• -management-driven reviews that assess the performance of current programs, compare existing with potential new programs, and result in identifying and making the necessary changes.

6. Research projects embody excellent science, involve the right people, are kept on track and within budget. The focus is on doing the right research projects and doing research projects right. This is important since projects are the core business of research organizations. Once the projects have been determined, excellent science should be the basis for all the work.

This attribute includes:

• -ensuring that the project produces, and is based on, excellent science and technology and that it stands up to the scrutiny of world-class experts
• -involving the appropriate and best persons available in the planning, conduct, technology transfer and review of projects
• -periodically reviewing projects using increasingly demanding assessment criteria over time (referred to as stage-gating). These reviews normally include assessments from both a technical perspective and the dependent constituent’s perspective.

They also include:

• ensuring that research equipment and facilities are appropriate to the nature and requirements of the projects;
• ensuring that the research results are communicated and/or transferred to all relevant constituencies. This usually requires the involvement of team members beyond the completion of the research: and
• managing the project, including the timetable and the budget, on a continuing and active basis to reflect the dynamics of the research environment.

7. Leveraging external resources involves collaborating with research performers and dependent constituencies, as well as relying on and/or building on the findings of other research groups. “External” simply means outside the specific unit/division carrying out the research. It includes interactions with staff in other parts of an organization. Leveraging is important since no research organization has all the required expertise and resources to identify and meet the needs of its dependent constituencies. Furthermore, the involvement of dependent constituencies is a way of accelerating adopting and increasing the likelihood of success.

Leveraging increases the likelihood of achieving objectives by optimizing the impact of the available human and financial resources and bringing to bear the best expertise from a variety of sources. It also helps focus the research organization on the end result and expedites the technology-transfer process.

8. Organizational knowledge is systematically captured and turned into needed work tools. This attribute entails creating organizational knowledge proactively and then exploiting it by making it accessible to all staff for continuous learning and delivering value to the current and prospective dependent constituencies. Work tools such as new methods, management practices or technology resulting from organizational learning are needed to exploit knowledge.

Knowledge is the key asset of a research organization. Organizational knowledge encompasses personal and collective knowledge (know-how, expertise, experience and wisdom). The management of organizational knowledge, sometimes also referred to as intellectual capital, pulls together the knowledge that is dispersed throughout the organization, and makes it accessible and usable by others inside the organization and its dependent constituencies. It involves the continuous recycling and creative use of shared knowledge and experience. The management of organizational knowledge ensures that a particular competence remains with the organization, even when individuals leave.

9. The organization is widely known and respected. An organization commands respect when knowledgeable observers judge that its output of science is of high quality and at the leading edge. (Observers are individuals in the same or complementary areas who are not necessarily among the organization’s constituencies, such as world-class researchers in other countries.) Respect results from an organization’s ability to maintain its reputation for excellence over a long period of time and to have its research staff sought after as participants in collaborative activities and partnerships and as members of prestigious committees. The participation of a significant fraction of the organization’s research staff on international committees charged with the development of international standards and regulatory regimes is an indication of the organization’s credibility and reputation.

An organization is also respected when dependent constituencies believe that it is performing an essential service, is accessible and responsive to their needs, is reliable, demonstrates flexibility, and is service-oriented.

10. The organization meets the needs of its dependent constituencies. This attribute is achieved by providing dependent constituencies with expertise and research findings that are relevant, timely, significant and of high quality, and can be communicated clearly to the organization’s constituencies. This attribute represents the research organization’s raison d’être, and is the ultimate test of the quality of its management. The pursuit of all the previous attributes contributes to the realization of this final attribute. While it is important to be well-respected, satisfying the needs of those who depend on the organization defines its success.

To ensure and demonstrate that this attribute is realized, research organizations take measures to make sure that their constituencies understand their research findings. They also assess the relevance, timeliness and significance of their research to identifying opportunities for improvement.

Networking and personnel exchanges with constituencies are among the best ways to promote effective communication and understanding of results. Also, expert review is an effective means of evaluating the relevance of timeliness and the quality of research.

OTHER TYPES OF ORGANIZATIONS

As we said at the outset of this paper, the idea of attributes comes from the question: How can one tell if an organization is well managed? This question can be replaced with a more bottom-line oriented inquiry such as: How can one tell if an organization is on course to exceed shareholders’ expectations?

The skeptic will ask: Why do boards and CEOs need attributes? They already have a plethora of tools such as performance indicators and success factors. Yet, some of the best organizations and companies stumble because they:

• Do not anticipate or respond to new technologies
• Turn a blind eye on disconnects between demand, production and inventories
• Lose their focus on the customer’s priorities in product improvement and development.

What’s missing, in our view, are in-depth, insightful probes to ascertain whether an organization is on track. In addition to certain standard things that every organization must do in order to stay on track, there are at least some things that it must do and that are unique to its line of business. Several of the attributes presented earlier in this paper are unique to research organizations (e.g., attribute #5 dealing with the management of the portfolio of research programs). Similarly, organizations in other lines of business have attributes that are unique to, and critical in, their line of business. Without keeping an eye on the performance or behaviour pertaining to these attributes, boards and CEOs face the risk that their respective organizations will veer off course in a rough, white-water world.