What do you mean by inexpensive? It's easier if you just give us a price range.

I checked a couple of high-end compact cameras - the Canon PowerShot G10 and Nikon P6000 - and neither one appears to have a bulb mode. Actually, you don't even get bulb mode on Canon DSLRs unless you buy an accessory shutter release. So finding a compact camera with a bulb mode could be really difficult. Have you thought about moving up to a digital SLR? Maybe it's time to make the move.