How do I stop GitX asking me for my SSH key password?

How do I stop GitX asking me for my SSH key password?

In order to let myself log into two different Heroku accounts I'm using
.git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
[branch "master"]
[remote "heroku"]
url = git@myrepo:myrepo.git
fetch = +refs/heads/*:refs/remotes/heroku/*
[remote "origin"]
url = git@github.com:peternixey/myrepo.git
fetch = +refs/heads/*:refs/remotes/origin/*
and ~/.ssh/config:
Host myrepo
HostName heroku.com
IdentityFile ~/.ssh/myrepo
IdentitiesOnly yes
GitX doesn't seem to be able to handle this

GitX doesn't seem to be picking up the correct key for the account. It
keeps trying to use id_rsa.pub when the correct key is myrepo.pub. There
is also a passphrase on the RSA key itself.
This is GitX(l) (as in L for Lima) and it's basically unusable in the
current configuration, how can I stop the password demands?