I've used your code in the contact's module to test it. To solve your issue you have to display_module, because during display it processes callback hrefs.
You've got two options. Use pack module, or init/display
$this->record_picker = $this->init_module('Utils/RecordBrowser/RecordPickerFS', array($tab));
Base_ActionBarCommon::add('add', __('Select'), $this->record_picker->create_open_href(), __('Select records'));
$this->display_module($this->record_picker);
or
$this->record_picker = $this->pack_module('Utils/RecordBrowser/RecordPickerFS', null, null, array($mode));
Base_ActionBarCommon::add('add', __('Select'), $this->record_picker->create_open_href(), __('Select records'));
Regards,
Adam