The answer is 8
“I sit between you and a star”: The number 8 sits between the letter U (you) and the asterisk (a star) on a standard keyboard.
“but far below me is an empty space”: far below the 8 key is the space bar
“I can be infinite or finite depending on how you look at me.”: If you look at an 8 sideways, it’s the symbol for infinity. If you look at it normally, it’s an 8, which is finite.