It is possible to create a notification for shoutbox messages, but it's not really that easy or straightforward.
You should start off by having the ShoutboxCommon execute javascript (outside class definition) with eval_js_once(); to create DOM element to hold the notification and initiate a recurring JS that will communicate (via ajax request) with the server to check for new messages - please see line 204 in file modules/Apps/Shoutbox/Shoutbox_0.php.
If there are new messages, the DOM element should be displayed to notify the user.
If you need assistance with any particular part of the process feel free to ask your questions in this thread.
Kind regards,
Arek
