Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] fix hf_summary #2416

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[fix] fix hf_summary #2416

wants to merge 4 commits into from

Conversation

theblackcat102
Copy link
Collaborator

@theblackcat102 theblackcat102 commented Apr 9, 2023

propose fix for #2415

in the previous code, the aggregation fails to handle such case:

[('B', 'C'), ('A', 'B')]
old code
> ['B', 'A', 'C']
new code:
> [ 'A', 'B', 'C']

Some issues not address in this PR

  • tie rank, where A > B = C > D; how do we format B, C?

  • this dataset has circular rank problem, ie B > C, C > B;

@theblackcat102
Copy link
Collaborator Author

looks like the duplication problem was gone

[6] Prefix: ["Please summarize the following content:\nMy den is a total mess, I have plans for a business that I really need to work on and I have so much that I could and should be doing.. and I really want to do it, but I just can't get myself to do it.\n\nI've discovered something about my motivation though... Wednesday and Thursday I hung out with people and both days AND Friday I was totally active, I cleaned out my den closet, reorganized all my old papers and files and crap, did the dishes and the laundry.. it was awesome.  Today?  All I can do is refresh Askreddit and watch for orangereds...\n\nI suddenly realized that I'm a SIM... I'm that asshole SIM that has to have constant contact with people every damn day or I refuse to do anything because my mood bar is red.  \n\nIt all makes sense now.  I've been depressed as hell and totally useless ever since I moved - I don't have any friends here."]
[6] Continuations (3):
[6.0]: My motivation is directly related to how much social interaction I've had.  Is anyone else like this?  How can I overcome this, or make more friends so that I can be more motivated?
[6.1]: I'm a fucking idiot and I can't control my emotions.  I'm not sure how I can loose the belief that I'm going to get something done if I don't really have anything to do because I can't get myself to do anything.
[6.2]: I'm an asshole.
[7] Prefix: ["Write a summary for the following article:\nI'm not sure if this is the right place, but my situation is pretty straightforward:\n\nI graduated from college about two years ago and started working for a company as essentially a data analyst, with regards to pricing. My actual job role is more geared towards looking at pricing data/past sales and developing strategies in ways to enhance profitability and margin opportunities, but I have other interests.\n\nI really thrive off problem solving and business process optimization, and have recently developed a LAN website with many languages (PHP/AJAX/PLSQL/HTML5/CSS/JavaScript/jQuery) to make it user responsive...develop reports, save them, create timed tasks, automatically format/e-mail out/etc.  It's like an easier and more responsive view of IBM Cognos.\n\nThat's the type of thing I like doing -- finding business problems (processes that are redundant) and solving them. I've literally automated things and saved the company SEVERAL hours per week for SEVERAL employees and plan on saving them more in the future.\n\nShould I be talking to my boss and how I should transfer/create my own job title...or what? What type of job title should I be looking for other jobs with? Business Process Optimization Specialist? I have no idea. \n\nI also believe I'm pretty underpaid for the skill-set/amount of money I'm saving the company by automating some timed reports. I wouldn't mind staying around here, but I also would LOVE to work somewhat remotely and/or travel. Almost like a consultant."]
[7] Continuations (3):
[7.0]: I'm an IT guy that's been doing business process optimization for several years; I enjoy automating business processes, but don't know how to transfer to another job.
[7.1]: I've developed a really useful tool to help companies with pricing data analysis and a passion for business process optimization. To me, this sounds like a job title that I could maybe get somewhere better (or not).**
[7.2]: My job demands are different than what I enjoy doing. Can you tell me what type of job I should be looking for?
[8] Prefix: ["So I had a discussion with my Dad about some political (I guess that's the word) topics. We talked about church/state, racism, immigration prejudice, and sexism. General 'equality' stuff. We didn't see eye to eye at all. That would be fine, you can't expect everyone to agree with you, but the arguments that he was making were often 'you're 20 and acting like a 20 year old, which is fine' and 'you don't have enough life experience. You don't know. You'll change your mind when you're older'.\n\nWell I'm not ok with that attitude. Yeah I'm in my 20s but that shouldn't invalidate my opinion and it certainly doesn't mean that I'm inherently wrong or uninformed. I identify very strongly with my political views. They're a very important part of me. And yes my views may very well change in the next ten years, but that's because people are fluid and opinions can change. That doesn't belittle my opinions NOW. Or it shouldn't I don't think.\n\nI'm just really bothered by this and I'm wondering if it would be an ok idea to approach him and mention how I feel. I would like for him to respect my opinion in the same way that I do his. Should I just let it go? It's really bothering me.\nTLDR;"]
[8] Continuations (2):
[8.0]: Talked politics with my Dad, he repeatedly boiled my thoughts down to 'well you're just young, you'll grow out of those thoughts' and I'm wondering if I should tell him how much that bothers me.
[8.1]: I have political views that my Dad doesn't agree with. I'm confused on how to approach this...
[9] Prefix: ['Hey guys, just looking for a bit of advice for my situation, sorry if it\'s a bit long. Little bit of background, I\'m a 20 year old guy, interested in a 20 year old female friend, we are both juniors and both attend the same university.\n\nOkay, so we\'ve been friends since freshman year, and things have always been going pretty smoothly. We both suffer from social anxiety/depression, and have a lot of interests (politics, shows, movies, music, etc.) in common.\n\nOver the years she always joked that we "were pretty much a chick flick" and would end up getting together despite anything that came up. Even when we got drunk and made out this semester (3 separate times), I still kinda laughed the \'chick flick\' thing off, but somewhere I knew she was right and we were getting a lot closer together. Anyway, she started getting a bit more hormonal, and when I was away for a weekend she slept with a mutual friend of ours. This is where the problem started.\n\nIt became more than a one-time thing, and while they aren\'t exactly dating, they\'re at least friends with benefits. She\'s even said she wasn\'t really into him at first, but now she likes the FWB. She also said she only got with him to start with because she wanted to lose her virginity, and I didn\'t put out enough.\n\nIt\'s gotten to the point where I know I\'m interested in her, and I\'m getting upset seeing her be FWBs with my friend. I want to tell her how I feel because I feel very strongly that we would work well together (not to mention that my friend is very judgmental towards people who drink, people who need anti-depressants, etc., and has made her very upset/cry multiple times with his statements).\n\nCan anyone help me figure out what I should say to her? I\'d like to prevent things from getting super awkward if possible, but I don\'t feel like I can stand around doing nothing anymore...\nPlease summarize the content above']
[9] Continuations (3):
[9.0]: Female friend was willing to move further, I was unsure, she starts being FWB with a friend, I know I'm interested now, what do I do.
[9.1]: Friend and I are close friends, and she's been dating my friend, and she's been FWBs with him a lot. She's really into him, and I want to tell her how I feel because I feel very strongly she's not into people who drink.
[9.2]: Not over a girl, but at least through a mutual friend. Want to enjoy the friendship, but don't want it to end like this. What should I say/do?
[10] Prefix: ['The GAP has showed up on every Egyptian persons facebook profile asking us to vote for the GAP female/male/child model of the year! The sleaziest people have submitted their pictures and Egypt has done a good job bringing them down with very awkward or horrible pictures that cannot be in any way used for modeling!  Only one contestant got through with a picture like that to the final round and he is in second place behind some little kid by 100 votes.  \n\nIntroducing the "models" \nMajid is a German Egyptian Engineer and his friend Wael who is an electronics music producer.\n\nWe need this photo to get insane amounts of votes. Let\'s make this happen.  Lets troll the gap with these models in a white robe (arab dress) and pants as their alter ego - the popular folk singer ZIZO KAMBUTAR.\n\nZizo\'s Music can be found here : \n\nPlease like the photo and share it somehow! We don\'t even know what the prize is to be honest but the gap must be trolled.\nPlease summarize the content above']
[10] Continuations (3):
[10.0]: Unintentional joke that could ruin the GAP's chances for getting the models in.

EDIT:  If you like this, please subscribe to the subreddit.
[10.1]: Enter contestants with photos of Egyptian girls in Arab dress and pants as their alter egos and get crazy amounts of votes.
[10.2]: Troll the gap with models!
[11] Prefix: ["Hey everyone\n26m Looking for some insight on a common argument I have with my girlfriend 25f. I have an ongoing medical issue I am trying to work out with my doctors and specialists, and I let my GF know if I've made an appointment regarding something that has happened recently because I think she at least deserves to know im going to the doctor because (insert reason) happened. Most of the times I tell her this, she tells me what she thinks is wrong with me and sometimes will suggest that I do 'X' whether that is take some over the counter drug, insist that my doctor ruin some test or even to the point one time where she will try to get me to take a prescription drug prescribed to her mom for the issue that she thinks I also have. I am not OK with doing things outside what my doctor recommended. Because of that I will usually get defensive as i am not asking for help, just letting her know I am going to the doctor and we argue about it. She does have some medical background in regards to sports related injuries, but is not a doctor trained in what my doctors think is wrong (gi issues). How can I communicate about this without being confrontational, or am I in the wrong for getting defensive ( totally fine with that if it's the case ). I also want to note that I do go to the doctors when something is wrong regardless of what it is, not the type to hold out for long.\nTLDR;"]
[11] Continuations (3):
[11.0]: I am sick with something unknown long term,  girlfriend gets upset with me if I don't follow her medical advice in addition to what doctor recommended. Girlfriend is not a doctor but is a medical professional in athletic injury discipline and my issues are GI related. How do I communicate my discomfort or am I in the wrong.
[11.1]: I'm an issue solver and have medical issues, sometimes my GF will suggest things i should do like take some over the counter drugs etc because she thinks im going to the doctor. How do I communicate about it?
[11.2]: Girlfriend won't listen to my advice on what to do in regards to my medical issues.
[12] Prefix: ["Please summarize the following content:\nIt's been awhile now, and I believe that I am feeling a little better and getting a whole new perspective of things and realizing that it wasn't my fault. It's been more than 4 weeks now, but every single night I get nightmares of my ex who has Borderline Personality Disorder, Anorexia, depression, and suicidal tendencies. \n\nThey aren't violent nightmares, nor scary ones. But they are the issues in which she used to guilt trip me with. And happy dreams in which I wake up from and I wake up to the sad reality that we aren't even together anymore.\n\nI dreamt a few times this week that we reconciled, and I woke up crying because I knew that it wasn't the case.\nLast night I dreamt that she was dying from anorexia (she is also anorexic and blamed me before) and that it was all my fault, but getting back together in my dream Results to me saving her.\n\nI am so tired. I can't fully move on with all this in my head. When I am conscious and awake, I am feeling better. It's whenever I sleep that memories and illusions of being with her and all of the guilt of her haunt me.\n\nI just want to be happy again. I feel like I'm drowning...\n\nShe cut all ties with me and doesn't respond anymore even when I tried to reach out to her a couple of days her, so I can't exactly ask her or talk to her anymore."]
[12] Continuations (3):
[12.0]: I'm so tired. I'm feeling better and have a clear mind when I'm awake, but it's whenever I sleep I dream of my ex, which makes it harder to move on. She is suffering from a personality disorder, eating disorder, and suicidal tendencies.
[12.1]: I miss my ex :(
[12.2]: I am feeling better and realized that I was not the cause of her problems. I wish I could sleep just to sleep and be happy again.

@theblackcat102 theblackcat102 marked this pull request as ready for review April 9, 2023 04:06
@MattAlexMiracle
Copy link
Collaborator

Breaking cycles in rankings is exactly the interesting part of ranked-pairs:
The idea of the ranked pair method is to build the graph in such a way that cycles are never created by removing edges that are dominated by a different edge (i.e. if A>B is more prevalent than B>A, you pick that one).
In that you have to be a little careful to do this during graph construction, since doing so later on can get you to misidentify the cycles, which gives you non-minimal dominance between edges (i.e. you don't maintain as much of the preference as is possible).
If you look at the code I implemented over at backend/oast_backend/utils/ranking.py, what you need is very similar, just that your code doesn't need to destruct the rankings into pairs first, since you already get pairs in the first place:
i.e. overriding head_to_head_votes should be enough. Something like:

def head_to_head_votes(pairs):
    # 1. get all possible choices and fix an order to label the axis later
    options = list(set([p[0] for p in pairs] + [p[1] for p in pairs]))
    # 2. make square preference matrix
    tallies = np.zeros((len(options), len(options)))
    # count up all preferences (might be (j,i) order, not sure of how I ordered winner and loser in my original implementation)
    for (i,j) in pairs:
        tallies[i,j] += 1
    return tallies, options

making the matrix skew-symmetric, doing the tie-breaking, etc is already handled in the rest of the code.
The only other thing you'd have to patch is line 102 and 103 of the ranked_pairs method, since I'm using the length of the ranking for iteration and since you don't have rankings but pairs, you'd have to change that to len(names).

@CloseChoice
Copy link
Collaborator

I would appreciate a test for this

@theblackcat102
Copy link
Collaborator Author

@MattAlexMiracle thanks, it did fix some of the bad results when comparison between different two different aggregation method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 participants