Rank Order Lists

Frequently Asked Questions:
Internship Training Directors - Preparation and Submission of Rank Order Lists

PREPARATION AND SUBMISSION OF RANK ORDER LISTS

1. What are my "Rank Order List(s)"?
2. What special ranking options are available to internship programs during each Phase of the APPIC Match?
3. Do I need to use more than one Rank Order List?
4. What considerations are important in developing my Rank Order List(s)?
5. As a Training Director, I plan to submit several Rank Order Lists during Phase I of the Match. Is it permissible to include an applicant on more than one of these lists?
6. What if an applicant ranks our internship program even though we are no longer interested in that applicant?
7. Are there restrictions on communication between applicants and programs during the Match?
8. How confidential are my Rank Order Lists?
9. I have a question with regard to a possible game that could be played by a site by creating and submitting multiple lists. Could a site with three positions, for example, improve their chances of getting their three top candidates by submitting three lists? Would this provide an advantage by allowing them to have three #1 choices, rather than ranking them as #1, #2, and #3 on a single list?
10. Under the Matching Program, can't people make "under the table" deals?
11. Where can I get more information about ranking applicants and the submission of Rank Order Lists?
    

PREPARATION AND SUBMISSION OF RANK ORDER LISTS

1. What are my "Rank Order List(s)"?

To participate in Phase I or Phase II of the Match, you are required to submit one or more Rank Order List(s) of internship applicants no later than the Rank Order List submission deadline for that Phase of the Match (the use of multiple Rank Order Lists for a single program is discussed below). These list(s) should consist of the internship applicants to whom you would like to be matched, ranked in order of preference (e.g., first choice, second choice, etc.). The order of preference is extremely important, as the matching process uses the relative rankings of applicants that you submit to determine the best possible matches of applicants for your program. There is no limit on the number of applicants that you may include on your Rank Order List(s).

You can only be matched to those applicants who: (a) you have included on your Rank Order List(s), and (b) have included your program on their Rank Order List. Of course, you should only include on your Rank Order List(s) those internship applicants who have applied to your program and to whom you are willing to be matched.

Many programs will choose to rank only the applicants who remain under consideration after the interview process, rather than all of the applicants who applied to their program. However, since there is no limit on the number of applicants who may be included on your list(s), you may rank as many or as few applicants as you wish. If you choose to rank applicants who you have not interviewed, you should notify applicants of this practice so that they will know to rank your site (otherwise, non-interviewed applicants will likely assume that you aren't ranking them).

If you decide that a particular applicant is unacceptable to you under any circumstances, simply omit that applicant from your Rank Order List(s). You cannot be matched to any applicant who does not appear on your Rank Order List(s). Be aware, however, that reducing the number of applicants on your Rank Order List(s) potentially increases your chances of having one or more of your positions remain unfilled.

2. What special ranking options are available to internship programs during each Phase of the APPIC Match?

During Phase I of the Match, internship programs have a number of special ranking options available to them:

1. Use of Multiple Rank Order Lists: Submitting more than one Rank Order List for a program can allow a program to meet certain requirements that might not be possible by using only one Rank Order List. For example, it is possible within the matching process to attempt to recruit a particular distribution of applicants for a program based on specific applicant characteristics.
   
   
2. Reverting Unfilled Positions from One Rank Order List to Another: This option allows a program to move a position from one Rank Order List to another List if that position cannot be filled from the original List.
   
   
3. Limiting the Number of Matches from any One University or School: This option allows a program to impose an upper limit on the number of applicants from any one university/school who are matched on a single program Rank Order List.
   

Due to the accelerated timeline for Phase II of the Match, the use of multiple Rank Order Lists and reversion of positions are NOT permitted in Phase II. However, in Phase II, programs may still use the option to limit the number of matches from any one university or school.

More information on each of these options can be found on the National Matching Services web site.

3. Do I need to use more than one Rank Order List?

While each internship applicant submits one Rank Order List, an internship program has the option to submit more than one such list. Experience indicates most programs prefer to submit only one list, allowing the matching process to produce the best match possible from a single list. However, some programs have special needs, and the use of multiple lists can provide a great deal of flexibility.

A program may have as many Rank Order Lists as they have intern positions to fill. For example, an internship site with four positions may:

• choose to fill all four positions from one list;
• choose to fill two positions from one list and two positions from another list;
• choose to fill three positions from one list and one position from another list;
• choose to fill one position from each of four lists

For example, assume that a hospital has five internship positions: three "general" positions, one "neuropsychology" position, and one "gerontology" position. In this example, the hospital may choose to have three Rank Order Lists, and instruct the Matching Program to select three interns from the "general" list, one from the "neuropsychology" list, and one from the "gerontology" list. Furthermore, if the "gerontology" position remains unfilled in the Match, the hospital may instruct the Matching Program to select an additional intern from the "general" list instead.

This example is only one in a range of possibilities. Programs may devise their own strategies and criteria in developing their lists. NMS provides individualized consultation to programs with special or complex needs.

4. What considerations are important in developing my Rank Order List(s)?

IMPORTANT: There is only one correct "strategy" for developing your Rank Order List(s): simply rank applicants based on your true preferences, without consideration for where you believe you might be ranked by each applicant. List the applicant that you want most as your #1 choice, followed by your next most-preferred applicant, and so on.

The Uniform Notification Day-based selection process used prior to the implementation of the Matching Program actually encouraged and rewarded strategizing and deal-making, resulting in tremendous stress for internship applicants. The Matching Program eliminates many of the incentives for this kind of behavior, and actually rewards participants for listing their true preferences by guaranteeing them the best possible match.

If you rank an applicant highly even if you believe that you have little chance of being ranked highly by the applicant, you do NOT reduce your chances of being matched with other applicants that you ranked as less preferred on your Rank Order List. Similarly, if you believe that one of your lower-ranked applicants is very interested in you and plans to rank you highly, you may safely rank other more preferred applicants higher without jeopardizing your chances of obtaining this lower-ranked applicant if you do match with the more preferred applicants.

Do NOT take into consideration where you believe you are ranked by applicants. Do NOT waste your time trying to develop "strategies" to supposedly increase your chances of getting your highly-ranked applicants. Do NOT engage in "making deals" with applicants, or other behaviors that violate Match policies. Due to the design of the matching process, any of these behaviors will ultimately hurt you because they will reduce your chances of getting your best possible match.

EXAMPLE 1: Let's assume that you have five internship applicants for one position that you are offering, and your true rankings are as follows:

1. Juan
2. Sandra
3. Paul
4. Nancy
5. Mary

In this example, Juan is your top choice. Assume that Juan doesn't appear to be very interested in you. Should you move Juan down your list, given your assumption that Juan may not rank you highly?

The answer is NO. If Juan is truly your top choice, the best (and only) strategy is to leave them at the top of your list. The matching process will attempt to match you with Juan without penalizing you and without reducing your chances of being matched with other applicants if you do not match with Juan. Changing your rankings based solely on how you perceive an applicant will rank you is a poor strategy that could negatively affect your outcome.

EXAMPLE 2: Let's again assume that you have five applicants for one position, and that the sample Rank Order List above reflects your true rankings. Furthermore, let's assume that your fifth choice, Mary, has expressed a very strong interest in you; in fact, Mary has told you that your program is a top choice. Is there any reason for you to move Mary up your Rank Order List, given the strong interest in you?

The answer again is NO. If Mary is truly your fifth choice, then leave Mary ranked fifth. The matching process will first attempt to match you to your higher-ranked applicants, but Mary will not be matched with any other site until all of your positions are filled. This allows you to try for your more-preferred candidates without risking the loss of Mary. Thus, you have been able to rank Mary according to your true preferences, without being penalized if your higher-ranked applicants don't work out for you.

With the Uniform Notification Day system used prior to the implementation of the Matching Program, receiving ranking information from an applicant sometimes provided you with a strategic advantage in making offers (particularly if you wanted to avoid having an offer "held" for an extended period). However, with the Matching Program, while it may feel good for you to receive "first choice" information from an applicant, this information is no longer important in developing your Rank Order List(s).

5. As a Training Director, I plan to submit several Rank Order Lists during Phase I of the Match. Is it permissible to include an applicant on more than one of these lists?

Yes. An applicant may be listed on more than one of a Training Director's Rank Order Lists. The applicant can be assigned a different rank number on each list.

6. What if an applicant ranks our internship program even though we are no longer interested in that applicant?

A match between a program and an applicant is only possible if: (a) the program includes that applicant on (one of) its Rank Order List(s), and (b) the applicant includes that program on their Rank Order List. Thus, if a program does not include an applicant on at least one of its Rank Order List(s), that applicant cannot be matched with that program.

7. Are there any restrictions on communication between applicants and programs during the Match?

Yes. APPIC Match Policy #7 states that applicants may never request ranking information from an internship site, while sites may never request ranking information from an applicant, even after the Match results have been released. Applicants and sites may voluntarily choose to share ranking information after the completion of Phase II, but are under no obligation to do so.

During Phases I and II, sites may choose to inform students as to whether or not they remain under consideration (e.g., whether or not they will be ranked). In practice, some will do so while others will not.

Sites and applicants are free to communicate about matters that don't involve the sharing of ranking information.

8. How confidential are my Rank Order Lists?

National Matching Services is committed to maintaining the confidentiality of both applicants' and sites' Rank Order Lists. This policy allows all participants to provide their true rankings without concern that this information may be disclosed to other parties.

9. I have a question with regard to a possible game that could be played by a site by creating and submitting multiple lists. Could a site with three positions, for example, improve their chances of getting their three top candidates by submitting three lists? Would this provide an advantage by allowing them to have three #1 choices, rather than ranking them as #1, #2, and #3 on a single list?

In the matching process, the ranking number itself is not important in determining who matches where, but rather it is the relative position on the list that matters. An applicant ranked #1 gets preference over an applicant ranked #2, etc., but the actual rank number of 1, 2 or 3 is not relevant. If a site has three positions, the applicants ranked #1, #2 and #3 are all treated like first choices, because positions are available for all of them.

A site submitting three lists has no greater advantage or disadvantage over a site submitting a single list. Even though a site that submits three lists (for three positions) can be described as having three "#1" choices (one on each list), it really provides no advantage over having one list with those same candidates ranked #1, #2, and #3.

The multiple list feature simply provides an opportunity to have the computer fill each slot from a different list of candidates; it doesn't give the site a strategic advantage or priority. Unless there is a specific reason for a site to use multiple lists (such as having different training tracks), it is certainly easier to "keep it simple" and use just one list.

10. Under the Matching Program, can't people make "under the table" deals?

It is possible for a site and an applicant to violate APPIC Match Policies by negotiating a "deal." However, this behavior will often penalize the person(s) attempting such a deal instead of providing them with an advantage. Thus, those who engage in such "under the table" deals will only end up hurting themselves.

For example, let's suppose a Training Director, Susan, has one internship position available. Susan has ranked an applicant, Frank, as number five on the program's Rank Order List. Frank has ranked Susan's site as third on Frank's own Rank Order List. However, several days before the deadline for list submission, Susan calls Frank and offers a deal: "I'll rank you number one if you rank me number one" (this is, of course, a clear violation of APPIC policies). Frank agrees to this arrangement. In this example, two results are then possible:

1. Susan and Frank both keep to their agreement listing each other as their number one choice, and are thus matched by the Matching Program. However, while both may feel that they've been successful, they've both actually lost out on better opportunities. Susan has lost the opportunity to obtain the first four candidates on the program's list, and Frank has given up the opportunity to obtain one of the two higher-ranked internship sites.
   
   If Susan hadn't attempted this "deal", listing Frank as a fifth choice would not have reduced Susan's chances of being matched with Frank. The matching process would simply have tried to obtain Susan's top four applicants before attempting to match with Frank. Similarly, Frank would have been better served by listing Susan as third, as it wouldn't reduce the chances of being matched with Susan in the event that Frank's top two choices did not come through.
   
   
2. One party keeps to the agreement by ranking the other party as their top choice, but the other party decides to back out of the "deal" and not change their rankings (under the Matching Program, verbal commitments or "deals" are non-binding). In this scenario, the party who changed their rankings has hurt themselves, while the party who does not change their rankings has gained a potential advantage.

As can be seen from this example, changing your rankings to anything but your "true" preferences is never advantageous. Thus, the only way that you can profit from an "under-the-table deal" is to be doubly-dishonest by (1) agreeing to make the deal in the first place (a violation of APPIC policies), and (2) reneging on your part of the deal, thus inducing the other party to change their rankings without changing your own.

This example points out one of the basic tenets of the matching program: the best "strategy" in submitting Rank Order Lists is to provide your true preferences, without regard to the likelihood of obtaining a particular site or applicant. As can be seen from this example, using any other strategy only hurts yourself.

11. Where can I get more information about ranking applicants and the submission of Rank Order Lists?

By December 31 of each year, the National Matching Services web site provides important information and detailed instructions concerning the preparation and submission of Rank Order Lists. All programs should review that information before preparing and submitting their Rank Order Lists.