When enabling xp_cmdshell under the SQL Server Surface Area Configuration for Features, the following error occurs :
"Failed to connect to server localhost\MICROSOFT##SSEE. (Microsoft.SqlServer.ConnectionInfo)"
This is due to Small Business Server 2011 including a SQL instance for Microsoft SharePoint that cannot be accessed by the Surface Area Configuration tool.
One possible work around is to stop the SSEE instance of SQL Server, and then run the Surface Area Configuration and enable xp_cmdshell.
For more information, see this MS SQL Server forum article.
If this solution does not work, xp_cmdshell can be enabled through the sp_configure stored procedure. To do this :
1. Run SQL Server Management Studio.
2. Open a new query on the server level and enter the following :
sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'xp_cmdshell', 1;
GO
RECONFIGURE;
GO
3. Click the Execute button.
For more info, see this MSDN Library article.