That question mark is what’s left of a folder or disk image or mounted volume in the finder that went away or was ejected. And what’s left is an empty space (see image below) for the missing item, so OS X displays a question mark. But the question mark is easy to get rid of.
Right click or control click on it and a pop-up menu will appear. Select “Remove Item” and the question mark will disappear. Poof, it’s gone.

The strange question mark
But also realize that in that same menu, you can select “Customize Toolbar…” and you can drag and drop and rearrange the items in the tool bar. That might come in handy. You may find some functions you would like to have in the toolbar, and some you don’t need , so customize away.