Open your options.cfg (make a backup first) in something, hopefully not, like notepad (not Word or Wordpad, a plain text editor which can handle "null" characters would be best) and find your way to (in the next section, ignore the "", they are just to denote key markers): "[Graphics Options]".
Then, under that section, find "Fullscreen Mode|" make sure it's set to 0
Then you can change "Display Height|" and "Display Width|" to your liking.
But be sure to run in -TestMode (I would suggest a more permanent -TestMode, since it means you can't mess up your BC options if you accidently forget to run in -TestMode).
TBH, I would actually suggest to run in Windowed Mode ( Fullscreen Mode set to 0) and use one of the standard, highest, possible resolutions, since the GUI won't look so good. In fact, BC will think that you are running 640x480 resolution since you aren't using any standard resolution (to BC anyway).
I have been looking into this issue, but sofar I haven't really found a satisfying sollution to it.
Maybe in my BC:Foundation.