To play custom songs or enhance the game, you can use modern toolsets:

This tool allows you to convert songs from other rhythm games like Clone Hero or Guitar Hero into the RB3 format ( .CON files) for use on your console.

Typically used for exported songs from previous Rock Band titles. 2. Custom Songs & Modern Enhancements

Use the RBtoUSB feature within Nautilus to drag-and-drop custom songs onto a USB drive formatted for Xbox 360.