-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[fix] fix hf_summary #2416
base: main
Are you sure you want to change the base?
[fix] fix hf_summary #2416
Conversation
looks like the duplication problem was gone
|
Breaking cycles in rankings is exactly the interesting part of ranked-pairs: def head_to_head_votes(pairs):
# 1. get all possible choices and fix an order to label the axis later
options = list(set([p[0] for p in pairs] + [p[1] for p in pairs]))
# 2. make square preference matrix
tallies = np.zeros((len(options), len(options)))
# count up all preferences (might be (j,i) order, not sure of how I ordered winner and loser in my original implementation)
for (i,j) in pairs:
tallies[i,j] += 1
return tallies, options making the matrix skew-symmetric, doing the tie-breaking, etc is already handled in the rest of the code. |
I would appreciate a test for this |
@MattAlexMiracle thanks, it did fix some of the bad results when comparison between different two different aggregation method. |
propose fix for #2415
in the previous code, the aggregation fails to handle such case:
Some issues not address in this PR
tie rank, where A > B = C > D; how do we format B, C?
this dataset has circular rank problem, ie B > C, C > B;