P/0025/97, P/0026/97, P/0030/97

IN THE MATTER OF AN ARBITRATION

Under

THE PUBLIC SERVICE ACT

Before

THE PUBLIC SERVICE GRIEVANCE BOARD

BETWEEN

Majkot/Ferguston/Imonti

Grievor

• and -

The Crown in Right of Ontario (Ministry of the Attorney General)

Employer

BEFORE

D.J.D. Leighton

Vice-Chair

FOR THE GRIEVOR

James K. McDonald Counsel Sack Goldblatt Mitchell Barristers and Solicitors

FOR THE EMPLOYER

Jane Hooey Counsel, Legal Services Branch Management Board Secretariat

HEARING

November 7, 1997; December 4, 1997; February 4-5, 1998; April 20-21, 27-28, 1998; May 5-6, 1998; June 4 & 18, 1998; September 2, 1998

Decision

The grievances before the Board were consolidated since they all relate to a job competition for a new position of Control and Risk Analyst in the Ministry of the Attorney General. There were two major parts to the competition: the screening test and the simulation exercise. The grievors, Mr. Arthur Mohipp, Mr. Rick Bertozzi, Mr. Andrew Robertson; Ms. Barbara Mitchell, Mr. Ted Majkot, Ms. Lynn Imonti and Mr. Jim Ferguson complained that the Employer:

Breached the “Selection Directive” and “Selection Guidelines,” and acted in a manner that was unfair, unreasonable and in bad faith in being denied the promotions.

Four of the grievors who were eliminated from the competition by the screening test, Mr. Mohipp, Mr. Bertozzi, Mr. Robertson and Ms. Mitchell, settled their grievances with the Employer half-way through the hearing.

The remaining three grievors, Mr. Majkot, Ms. Imonti and Mr. Ferguson, were not successful in the simulation exercise and were denied the promotion. They alleged that the Employer did not consider all pertinent factors and qualifications of the candidates and the selection process was conducted in a manner that was unfair, unreasonable and in bad faith. The grievors withdrew the allegation of bad faith at the hearing. They alleged that the competition was so seriously flawed that they should either be awarded the position with retroactive pay and benefits, or a fresh competition ordered. The Employer took the position that the process of the competition was fair, though not perfect, and that the grievances should be dismissed.

The position of Control and Risk Analyst was a new one, which was initially posted for employees within the Ministry. There were ten available positions, and only eight people applied originally. Three were successful in this competition. The Employer then sought candidates in a second and third round of hiring, initially within the government, and then with a final posting that included applications from the public. Counsel for the Employer informed me that third parties were notified of this hearing and their right to participate in it.

EVIDENCE

As part of an amalgamation and reorganization of the audit branches of the Ministry of the Solicitor General and Correctional Services and the Ministry of the Attorney General, the Employer decided to change the focus of the audit branch away from compliance auditing to a management assurance role. This shift was part of a government-wide change to a “control and risk” approach of auditing. As part of this reorganization, the Control and Risk Analyst positions were created at the AM-19 classification. As part of this process, the Audit Management position, classified at AM-17, was abolished. Thus the grievors, Mr. Majkot, Mr. Ferguson and Ms. Imonti, were declared surplus. The new positions were posted on March 24, 1997:

A challenging opportunity exists within the Integrated Corporate Services Division to provide assurance to the Deputy Ministers of the Ministry of the Solicitor General and Correctional Services and the Ministry of the Attorney General that the ministry has effective, efficient and economical control systems in place that ensure ministry objectives are achieved.

In a client based service organization, you will be performing high level, corporate focussed control and risk assessment based reviews encompassing all administrative, operational, financial and program delivery activities.

You will provide audit and consulting services which include the facilitation of control and risk assessment exercises, development and implementation of self assessment tools, development and the provision of training in the area of control design.

Qualifications:

Thorough knowledge of auditing principles and practices, control and risk self assessment techniques and control frameworks, management and operational control systems, controllership/ governance and principles/ practices of accounting and financial impact analysis and organizational change processes with an emphasis on current management theories and practices normally acquired through a combination of related education and/or demonstrated experience.

Demonstrated ability to identify, research and analyse complex business problems and make sound decisions independently. Effective organizational and project management skills. Excellent oral and written communication, consulting, presentation, facilitation and interpersonal skills to deal effectively with all levels of senior management. Ability to act with diplomacy and tact in difficult situations. Strong client service orientation.

Knowledge of computer applications, controls and computer assisted audit techniques to extract and analyse information contained in computer programs, PC skills for word processing, spreadsheets, graphics, email and database applications.

The grievors applied for the positions and were successful in the pre-screening exercise. However, they were not successful in the simulation exercise which was developed by a consultant.

The evidence is clear that the Employer attempted to establish a fair competition process for this position. There were four steps to the competition process beginning with 1) a pre-screening exercise which was an exam to assess analytic and report writing skills set by an independent professor from Nippissing University; 2) a simulation exercise which was developed with the assistance of the consulting firm, Hay Management Consultants (the consultants); 3) a self- assessment of each candidate’s computer skills; and 4) reference checks per Management Board directives. Candidates for the position had to be successful at each step in order to move onto the next.

The Employer decided to use the “best fit” model for hiring the new control and risk analysts. That is, in selecting managers for the new position, the Employer wanted to ensure that they had the technical and management skills and the core competencies required for the position. The Employer decided on a simulation exercise to test the candidates as to the required core competencies. The simulated exercise, which was developed by the consultants, was a case study or problem which each candidate had to analyze and present to the selection panel of three people. Each candidate had four hours to prepare the presentation which included a three page report and whatever else they saw fit to attach to that. It could include overheads or anything that they could create in four hours that would facilitate in the presentation.

The candidate’s task is to play the role of a Control and Risk Analyst who has been asked to review documentation and prepare recommendations regarding a new pilot program for the “Ministry of Early Education.” The candidate will be required to review several pieces of information presented in a variety of formats in order to deal with these issues. The candidate is required to write in point form a report, up to three pages in length, under the headings given in the written scenario instructions including: a brief description of the program, issues noted, and recommendations.

In addition to this point form report, the candidate will give a presentation to a panel of assessors regarding their findings. The presentation will be followed by a question and answer period, in which the candidate will be asked a series of structured questions by panel members. After the simulation, each assessor will rate the candidate on each key competency. Candidates will be assessed on the content of their point form report and their presentation. The assessors will then meet and through discussion arrive at a agreed upon rating on each of the competencies for the candidate.

All the materials were given to the selection panel after the presentation was over. A control and risk analyst simulation evaluation form was developed by the consultants and is attached to this decision as Appendix 1.

While the panel was acting as a selection committee, it also role played in the presentation in that one of the individuals played the part of an Assistant Deputy Minister with particular concerns and a particular agenda with regard to the case. The selection panel was made up of Angela Lockridge, Acting Director of the Audit Branch, Don Forsey, Audit Manager, and Loretta Eley, who was not an auditor but a client of the audit service. At the time of hearing Ms. Eley’s home position was Superintendent of the Toronto Jail and she was acting as Project Manager for a special project on female offenders for Correctional Services.

The consultants worked with senior managers of the government to identify “core competencies” for the position. Mr. Ken Kristopherson, Director of the Audit Services Branch, at the time the competition process was developed decided that candidates had to demonstrate every competency. The competencies identified were the following: Achievement Orientation; Analytical Thinking; Business Appreciation/ Mindedness; Conceptual Thinking; Impact and Influence; Information Seeking; Listening, Understanding and Responding; Project Accountability; Report Writing Skills; and Self-confidence. Different levels were selected for each competency. For example, Achievement Orientation required a Level 3 competency, while Analytical Thinking required a Level 4. Level 3 for Achievement Orientation was defined as:

Improves performance; makes specific changes in the system or in work methods to improve business outcomes (e.g., does something better, faster, at lower cost, more efficiently, improves quality, customer satisfaction, morale, revenue) without setting any specific goal.

“Behavioural” examples of Level 3 were provided for those who would assess the candidates. Again, using Achievement Orientation as an example, behavioural examples of Level 3 would be:

Specifically mentions HLP shortfalls and how they could be improved through recommendations (e.g., discusses how renting out empty office space could generate new sources of revenue to offset overall program costs).

Each competency was then rated on a scale of O to 2. Zero meant that the competency level had not been demonstrated. A zero (O) would mean that the candidate had made no reference to how recommendations would improve the current situation. To score a one (1) the candidate

makes general reference to improvement when making recommendations; mentions need for improvement in the program (e.g., utilization rates), but did not provide specific measures (e.g., ’utilization could be improved by X percent by doing y’).

For a candidate to score a two (2) he or she had to present

several alternative options with specific examples and supporting data, demonstrating how each option would improve the current situation in both the short and long term.

Each core competency defined the level, gave behavioural examples of the level, and defined what counted as a zero, one or two on the scoring scale. (See Appendix 1)

The consultants trained the selection committee on how to assess the candidates. They were instructed not to mark during the presentation, but they were permitted to take notes. After the presentation was finished, each committee member did an initial rating on each competency. A consultant from Hay Management acted as facilitator and asked each Committee member to explain their rating. Dr. Smidterly, who acted as a facilitator, gave evidence that the first person to comment as to why a score was given tends to affect the others in their assessment. Thus they had a practice of starting with a different person on the selection committee each time they discussed the ratings for a candidate’s competency.

After a discussion between the three panel members, a consensus score had to be agreed upon for each competency. In addition to the training given by the consultants, Mr. Forsey took on the role of a candidate in a trial run of the case study. He prepared for four hours, and then presented the case study. As a result of that test run, there were some small changes made to the case.

The consultants were of the view that consensus marking was the best method of scoring, in part because it avoided the “halo” effect a theory that people select those that look like themselves or present well, even though there is no content in the presentation. The consultants were also of the view that taking an overall average of a panel is prone to bias. Using consensus scoring, therefore, avoids rating errors in their view.

The evaluation results of the candidates for the Control and Risk Analyst position can be found in Appendix 2. Ms. Imonti received a zero consensus score for the competency Impact and Influence and, therefore, lost the promotion. She had received a zero from Ms. Lockridge, and one from Ms. Eley, and a zero from Mr. Forsey in the individual marking. Mr. Ferguson received a zero consensus mark in Achievement Orientation, after having been marked individually in this competency, with a-zero from Ms. Lockridge, a one from Ms. Eley, and a zero from Mr. Forsey. Mr. Ferguson also received a zero consensus mark in Listening, Understanding, and Responding, after having been assessed individually with a zero from Ms. Lockridge, a two from Ms. Eley, and a one from Mr. Forsey. He was likewise unsuccessful, therefore, in getting the promotion. Mr. Majkot received a zero consensus score in Listening, Understanding, and Responding, after having been assessed individually with a score of zero from Ms. Lockridge, zero from Ms. Eley, and a one from Mr. Forsey.

Ms. Eley was asked why she gave Mr. Ferguson a one in Achievement Orientation, but then agreed to a consensus mark of zero in the end. It was her evidence that she would have been persuaded by the other panel members that the level had not been reached by the candidate. When questioned as to why she went from a two in her assessment of Mr. Ferguson’s competency on Listening, Understanding, and Responding, to a zero in the consensus mark, she said that Mr. Lockridge’s view and her assessment of zero had been persuasive. When questioned as to why she gave Ms. Imonti a one in Impact and Influence, but then agreed to a consensus score of zero, she could not explain the change. When Mr. Forsey was questioned about why he changed his evaluation of Mr. Ferguson on Achievement Orientation from a one to the consensus mark of zero, he said he must have been convinced in the consensus discussion to give him a zero. He didn’t recall the arguments made that changed his mind. He gave the same evidence to explain why he went from a score of one to agreeing to a zero for Mr. Ferguson in Listening, Understanding and Responding. He also noted that Mr. Ferguson was the third candidate on April 17, 1997. He testified that the three panellists spent a lot of time reaching consensus on this particular candidate’s scoring. Mr. Forsey was asked the same question regarding his scoring of Mr. Majkot for Listening, Understanding and Responding, in that he went from a one to the consensus score of zero. He could not remember why he changed his score or why Mr. Majkot got a zero in the consensus.

The three grievors gave extensive evidence on their professional qualifications and experience as auditors, which I will not summarize here. They also gave extensive evidence on the competition process and how they had addressed the key behavioural examples as outlined in the various competencies for the job. While I have reviewed this evidence carefully in coming to my decision, I will only refer to it as needed in my reasons for the decision.

SUBMISSION OF THE GRIEVORS

Counsel for the grievors, Mr. McDonald, argued that the issue before me was a narrow one. He argued that the question before me was whether or not the grievors had established a flaw in the competition so serious and so significant so as to affect the results of the competition process. He argued that the grievors had shown on a balance of probabilities that the competition was flawed for the following reasons. He argued that the use of consensus marking to determine success or failure in the job competition was a serious flaw. He also argued that the consensus marking scheme combined with what he terms the “single knockout” which was the Employer’s decision not to allow a candidate to proceed if there was a single zero consensus score was unfair. He argued further that the use of the simulation exercise to the exclusion of any other review of the candidates credentials or experience was flawed.

Counsel submitted that this was not a case of relative ability. It was a competition to determine whether the candidates were qualified for a new position. There were, in fact, more positions than applicants in the first round. Counsel argued that the grievors were clearly qualified to perform the job and that the evidence of each grievor supported that finding. Each of the candidates persuaded at least one of the competition committee members that they ought to get a one. Counsel noted that the Grievance Settlement Board (GSB) had found that consensus marking is a defect in competition cases and also that the government had acknowledged this in its directives and guidelines on recruiting and staffing. In the Employer’s Human Resources Directives and Guidelines (January 199 1) the guideline notes that there are two possible scoring methods which selection panels can use. Panels can use consensus rating or score averaging. With regard to consensus rating, the guideline states:

The consensus approach has been criticized by the Grievance Settlement Board as subjective and open to undue influence by one or two board members. It is important to have a clear and complete board report that demonstrates objective and accurate decision making. On such evidence that has been presented to the board no fault was found.

Counsel argued that consensus marking was flawed for the same reasons as found by the GSB, essentially because one or two of the competition panel may dominate the decision making. There is the possibility of collusion.

Counsel emphasized that when members of the selection committee were asked why they changed their individual mark, in most cases from a one to a zero, they generally could not remember why they had changed, but both Mr. Forsey and Ms. Eley said they must have been convinced by the other committee members. Counsel reviewed the evidence of the candidates on the competencies where they had been assessed as zero and their own evidence that they had delivered the behavioural examples for the specific level required of the competency in their presentations. He noted again that they had all convinced at least one individual that they had met that level, at least in the .initial assessment before the consensus discussion.

Mr. McDonald cited OPSEU (Bent) and Ministry of Transportation and Communications 1733/ 86 (Fisher); OPSEU (Vaillancourt) and Ministry of Health 1620/87 (Wilson), OPSEU (Riddock) and Ministry of Correctional Services 1206/89 (Emrich), OPSEU (Walker) and Ministry of Transportation and Communications 514/84 (Verity) and OPSEU (Peters) and Ministry of Revenue 1423/90 (Kaplan) in support of his argument that consensus marking was a defect in this case.

Counsel also reviewed other flaws with the assessment scheme for the simulation. He noted that there was no model answer or model presentation that, in effect, there was no marking scheme for the simulation. He argued that the assessment sheets for each competency, which I have attached as Appendix 1, were confusing and complicated and were, therefore, rife for wrong results. Counsel also argued that the competition process was flawed in that it relied on the test results to the exclusion of any other evidence that the individual grievors had the competencies to do the job. He argued that their personnel records ought to have been reviewed. Counsel relied on OPSEU (Sauve) and Ministry of Transportation 1695/91 (Gray) to support his argument that past related experience should not be excluded in assessing candidates for promotion.

In conclusion, Counsel argued that the consensus rating scheme resulted in a significant procedural flaw so that the grievances should be granted. This flaw, particularly with the combination of the decision that a single zero would result in a candidate’s failure, aggravated the defective rating scheme. Counsel was of the view that the flaw-was so significant that if a different rating scheme had been used there would have been a different result. Counsel suggested that if the board used an averaging score system it would show the candidates as scoring high. Ms. Imonti had three zeros in total, or a 27 out of 30. Mr. Majkot had six zeros in 30, or 23 out of 30. Mr. Ferguson had three zeros total, or 27 out of 30. Thus Counsel asked that this board award positions to the grievors with full compensation, retroactive to the date of the grievance. He argued in the alternative I should order a fresh competition with directions that the procedural flaws not be repeated.

EMPLOYER’S SUBMISSION

Counsel for the Employer, Ms. Hooey, argued that the process of competitions had to be fair, but not perfect. She argued that in hindsight that some things could have been done differently in this competition. However, the process was designed to look at and assess merit, not seniority or history with the government. She stated that the Employer wanted people who were able to do the job immediately without any training. She argued that the process was not flawed and that the grievances should fail. Ms. Hooey submitted that if I was to find that the competition procedure was flawed, I could not also find that the grievors ought to be awarded the job. The only appropriate remedy in that case would be to order a fresh competition. Counsel argued that the Employer had decided that a single zero would eliminate a candidate from the job competition. But she noted that other things also screened out candidates. For example, people who did not achieve a high enough score in the pre- screening exercise were eliminated at that point. She noted that there were four steps to the competition process. The pre-screening, the simulation, the computer skills test, and references.

With regard to consensus marking she argued that the purpose of this method of marking was to make sure everyone felt part of the process. She argued that the consultants were of the view that consensus marking was the better way to score a competition. Counsel noted that the consensus discussion was free between the committee members. She also argued that the committee members did not change their marks, that the individual assessment marks were just a starting point in the process. She stated that there were good reasons to have three people on the panel with different perceptions. The consultants were there as quality assurance. She noted that the grievors were necessarily biassed in their view in that they had covered the material and, therefore, should have gotten a mark. She argued that a difference of opinion on this was not enough to show a flaw in the competition process.

In Counsel’s written submission to the board, she argued that three issues were before the board:

1. whether or not there was evidence that the competition process was flawed;

2. whether or not the board should rely on GSB decisions which have found consensus scoring to be defective, and

3. if the board should find flaws in the process, does the grievance prevail.

Counsel argued that the jurisdiction of this board is limited to reviewing whether or not a competition was flawed or otherwise not in good faith. Counsel cited Brander and Ministry of Correctional Services P/006/92 (Willes) and McAdam and Ministry of Finance P/0015/95 (Leighton).

Counsel argued that GSB decisions finding that consensus scoring resulted in a defect were distinguishable: the circumstances of the consensus scoring were different in both nature and substance. Referring to Bent Counsel submitted

…that the decision in Bent with respect to consensus marking was an error and should not be followed by this board. The decision is in direct conflict with evidence of The Hay Group which indicated that consensus scoring is more reliable and fair than averaging or totalling the marks.

As to the third issue of whether the grievances should prevail if the board finds flaws, Counsel argued that the Employer had conducted the competition in good faith and gone to great lengths to ensure that the process was as objective as possible. She argued that irregularities in the competition process are not enough to justify a finding of bad faith by the board, or to require that a competition be re-run. Counsel argued further that even if flaws were found, the flaws in the process had to be so serious that the outcome would have been different. Counsel cited the Bent case as support for this submission.

Counsel noted that if the board found a significant flaw an order that the grievors be given an opportunity to compete in future competitions for the position, is a possible remedy. Ms. Hooey cited Simpson and Ministry of the Attorney General P/0001/89 (Willes) as authority for this submission. In the alternative, Counsel for the Employer-argued that the remedy for the grievors, if the board found that there was a serious flaw in the competition, would be to order a re-run of the competition. Counsel cited Lacombe and Ministry of Colleges and Universities P/0008/ 89 (Willes). In conclusion, the Employer submitted that it was not up to the board to second guess the competition panel and that, barring unfairness, the grievances should not succeed and, therefore, the grievances should be dismissed.

DECISION

The issue before me is whether the competition for the Control and Risk Analyst position was so flawed that it effected the outcome of the process, wrongly depriving the three grievors of a promotion. The key issue in this case is whether the consensus marking scheme was fair or not. While this is the first time this issue has come before the Public Service Grievance Board, it has been considered by its sister board, the GSB. In Bent, supra, the board found that consensus scoring, used by a selection committee, was a defect. The board held

This board finds that the method of consensus scoring is inappropriate and should not be continued by this or any other Ministry in the selection process. The purpose of having three individuals on a selection committee is presumably so that each person can bring their own input into the decision making process. A consensus method of scoring means that the parties must agree on the scoring for each question, and that raises a spectre of either a majority or two overruling the third person each time, or a dominant member of the panel exercising his superior position or knowledge over the other members. Furthermore, it leads to the possibility that there would be ‘horse-trading’ between selection committee members. For example, a committee member could agree to a certain scoring on one question in exchange for another committee member changing his position on another question. There is, in this board’s opinion, no advantage at all in consensus scoring over individual scoring.

The board in Bent went on to find that it would be more appropriate to use an averaging of scores.

In Vaillancourt, supra, the board addressed the issue again. Referring to consensus marking by the selection committee as “coordinating” the marks of applicants. The board here referred to MacIntyre and Ministry of Community and Social Services (1989), 141/85 (Knoph) where the chair in that case found it was better for interviewers not to consult about their assessments because of the risk of “collusion.” The board in Vaillancourt went on to say that collusion did not have to amount to “secret agreements for a wrongful purpose, especially between person wishing to defraud another ...” That board held that it was not necessary to show this kind of corruption to invalidate a scoring process. The board held

I would be willing to set aside a set of scores where the evidence showed that the three members of the panel actually surrendered their individual judgments.

In Riddock, supra, the board reviewed the Bent case and agreed that consensus marking constitutes a defect in the interview process.

Finally, in Peters, supra, the union’s counsel argued that there was evidence that one of the members of the selection committee had been influenced to change her score and had, thus, surrendered her individual judgment in assessing the grievor. While the board did not find any evidence of collusion or conspiracy, the board was concerned not only with a fair process, but the appearance of fairness in the process of hiring.

However, we are troubled, as the grievor must surely be, by a marking process that at one point indicated she was ‘relatively equal’ to Mr. Johnston, applying the standards established by some panels of this board, but which later indicated, as a result of marks being changed for all of the applicants, that she was no longer ‘relatively equal.’ The fact that the questions themselves were questionable further undermines the credibility of the process. For this, the employer is entirely at fault for it had carriage of the job competition process, and its responsibility was not just to run a fair competition, but to run a competition that was perceived by applicants and employees as fair. The questionnaire and the changing of marks reflects poorly on that perception and for that, the employer is entirely to blame.

While the PSGB is certainly not bound to follow previous GSB decisions and there may be good reason in certain cases not to, in this case the analysis and reasoning for finding consensus marking defective is helpful and I am persuaded that it is right.

A review of the Control and Risk Analyst simulation evaluation results, found at Appendix 2, raises real doubts as to the fairness of the process that was used in evaluating the candidates for this competition. The method of using consensus evaluation led to completely inconsistent marking of all the candidates. Mr. Ferguson received a zero from Ms. Lockridge, a one from Ms. Eley, and a one from Mr. Forsey on Achievement Orientation, and yet received a zero in consensus mark. In Analytical Thinking, he received a zero from Ms. Lockridge, a one from Ms. Eley, and a one from Mr. Forsey, and was given a one in consensus mark. For the competency Listening, Understanding, and Responding, he received a zero from Ms. Lockridge, a two from Ms. Eley, and a one from Mr. Forsey, and was given a zero in the consensus mark. The evaluation forms on their face raise a doubt about the fairness of the method of scoring. The evidence does not dispel this doubt. Although the evidence is clear that the employer went to great lengths to try to create a hiring process that would be fair, and as one. witness put it “legally defensible,”’ it was not fair.

Consensus scoring required the selection committee to reach mutual agreement on scores for each competency. There was evidence that when this was difficult, as at times it was, the consultants were influential in guiding the process. In Employer Counsel’s submission, the consultants agreed with the consensus marks that the panellists reached. I find this, in itself, somewhat problematic since the consultants had no knowledge of auditing in general or the audit position. More importantly, they were not on the hiring panel. So while they were there as facilitators of the process, it is not at all clear how they influenced the outcome. More problematic, however, is that when individual panellists were asked why they changed marks, in one case a mark from two to a zero, they couldn’t remember why they had done so in most cases. Both Ms. Eley and Mr. Forsey said that they must have been persuaded by the others. In the case of Mr. Ferguson the only one that would have been persuading them to change to a zero is Ms. Lockridge since she was the only one to give Mr. Ferguson zeros. I am convinced by the evidence that the consensus marking in this case resulted in some panel members surrendering their own judgment.

Forcing panellists to reach a consensus score is inherently unfair. I make this finding despite the evidence of the consultants that consensus marking is the best method of evaluating candidates. As Counsel for the grievors pointed out, the consultants in this case have a vested interest in defending the method of evaluation since they designed it. There was an argument advanced that consensus scoring allowed selection committee members to examine why or what rationale was behind a score and in doing that they might discover that, in fact, the competency had not been demonstrated when at first they thought it had been. As the board in Bent said

Even in an individual scoring system, the selection committee members could quite properly discuss their individual viewpoints and ask scoring questions of each other, however, when it comes down to the actual, each member of the selection committee must put down his own opinion without the necessity of reaching a consensus. Therefore, this board envisages a free and lively discussion among the selection committee members before individual scoring was exercised but ultimately, the member must score the candidate as he deems fit and not so as to please the other members of the selection committee.

There was nothing to stop the panellists discussing each candidate’s performance, but then they should have been allowed to score individually.

There were other problems with the design of the competition which in and of themselves would not result in a finding of unfairness to the grievors, but which aggravated the unfairness of consensus marking. I find it problematic that there was no model answer or model presentation that could have been used as a benchmark. While the selection committee was given evaluation forms for each of the ten competencies, the definition of the level of each competency was, at times, confusing. For example, in the competency Listening, Understanding and Responding, which required a Level 3 of ability, the definition was as follows:

Understands meanings; understands current, unexpressed or poorly expressed meanings; understands current unspoken thoughts, concerns or feelings. OR gets others to willingly take actions desired by the speaker; that is, uses understanding to get others to act the way you want them to.

As a behavioural example of Level 3 of this competency the first example includes:

Acknowledges partiality of ADM to the lifestyle training program and bias against the home learning program.

In order to be graded a one for this competency, a candidate had to demonstrate as follows:

Speaks to some unexpressed issues, but overlooks other concerns when making recommendations. E.g., speaks to ADMs opening remarks.

Even with training, it would be difficult to use this guideline to assess an individual as to this competency.

Mr. Majkot received a zero in this competency after the consensus marking, although he had received a one from Mr. Forsey in his initial evaluation. One of the behavioural examples of Level 4 for the competency of Impact and Influence is almost identical to a behavioural example in Listening, Understanding and Responding. A behavioural example of Level 4 for Impact and Influence includes “Acknowledges ADM’s bias in favour of the lifestyle training program in presentation.’’ Mr. Forsey, in his evaluation form, checked this behavioural example off. It was Mr. Majkot’s own evidence that he included this acknowledgement in his presentation and it can be found in writing in his notes. His notes were not used by the evaluators to come to their consensus mark.

Ms. Imonti got one zero in the consensus marking after Ms. Eley changed her mark from a one to a zero. A review of the marking sheet that Ms. Eley used in assessing Ms. Imonti for Impact and Influence shows that she specifically ticked off one of the behavioural examples of Level 4, and she also ticked off Level 1 that Ms. Imonti “used the language of the Ministry when presenting key points and recommendations (e.g., uses ‘buzz words’ and praises found in the ADMs letter).” Despite this, she was persuaded to go from a one to a zero. Further, I agree with Counsel for the grievors that the Employer’s decision that one zero consensus score would eliminate a candidate further aggravated the unfairness of consensus marking. It allowed the possibility of one dominant panel member vetoing a candidate if he or she could persuade the others.

A final concern was that Mr. Ferguson’s simulation was the third in one day and there was some evidence to suggest that the panel was working until at least 10:00pm. There was also evidence that the panel members did not rush and took the time to assess Mr. Ferguson. However, an already difficult complex process was made more difficult given he was the third of the day. There is no doubt that when people are tired it is harder to remember the details of a presentation in order to assess it.

These additional difficulties would not by themselves be enough to lead me to conclude that the competition was so flawed that the results had to be set aside. But these weaknesses in the competition process made the use of consensus marking more unfair.

Thus I must conclude that consensus marking in this job competition was a significant and serious flaw in the job competition process and I am convinced that if a different method of marking had been used it may well have given a different result. The next issue then is to consider what remedy is appropriate. Employer counsel argued that should I find that the competition was not fair to the grievors, I should not order that they be given the promotions with full back pay and benefits, but rather order a fresh competition. This board has consistently held that the remedy for a flawed competition is normally an order that the competition be re-run properly. As noted in Lacombe, supra, this is because:

Boards of Arbitration are rarely (if ever) prepared to substitute their judgement for that of the employer’s selection committees with respect to the relative qualifications and abilities of the candidates for particular positions.

In the case before me, I see no reason to make an exception to this remedy. While the evidence seems to support a finding that the candidates are qualified to fill the position, I cannot be sure. If I total the individual scores for Mr. Ferguson, they come to 33, which is the total that one of the successful candidates for the job received. The scores for Ms. Imonti and Mr. Majkot were 29 and 25, respectively. This may be enough to satisfy the selection requirements, but it is a decision which the Employer should make.

Even if the flawed part of the competition was excised, and the candidates assessed on the individual scoring, they would still have to satisfy- the requirements of the last two steps of the competition - the computer skills self-assessment and reference checks. I am in no position to assess this. Thus I find it would be inappropriate for me to order that the grievors be promoted to the position.

That said, however, it would be practical since the competition, though not perfect, was reasonably fair without the consensus marking, to assess the candidates on their individual scores.

If the parties agree the individual scores, without the consensus scores could be assessed by a fresh selection panel to see if the grievors, meet the selection criteria for the job. Then, if the candidates are evaluated as meeting the selection criteria, they may proceed to the last two steps of the competition. If the parties cannot agree on an assessment process of the “individual” scores in the first competition within 30 days of this decision, then a fresh competition must be run. If they can, and the grievors are successful they should be placed in the positions which were filled last.

In conclusion, for the reasons recorded above, the grievances are granted. It is hereby ordered that the Employer re-run the competition for Control and Risk Analyst for the last three positions filled if they cannot reach agreement as noted above.

I shall remain seized in order to assist the parties if there are any difficulties in implementing this award.

Dated at Toronto this 12th day of May, 1999.

Appendix 1 CONTROL & RISK ANALYST SIMULATlON EVALU ATION FORMS – see original decision

Appendix 2

Control & Risk Analyst Simulation Evaluation Results – see original decision