In NextGen Options, go to Effects.
Next to JavaScript Thumbnail effect, select Custom
Next to Link Code line, type rel=”shadowbox[%GALLERY_NAME%]”
Click on Save Changes

for example:

from: class=”shutterset_%GALLERY_NAME%”

to: rel=”shadowbox[%GALLERY_NAME%]”