The Shark really works best with multi-cell batteries. That said, your configuration should be OK.
You need to be careful to not let the battery voltage drop below 3V as seen by the converter board. So, even if your battery may be have a partial charge it's possible that the battery sag, wire voltage drops, voltage drop across the switch could make the voltage less by the time it gets to the converter board. When it drops below 3V the converter will not work correctly and possibly flash, blink or do other things. I don't know to what extent or behavior under these conditions.
I believe you have the pot hooked up correctly. I would test with only a fresh cell or with multi-cell to determine where the issue is. If with two cells you have full range control then one cell is not a working configuration and you might want to consider using larger wires and or switch to lower the losses in the wires/switches to see if that will help the one cell configuration.
You can also measure the voltage at the converter board while it's running to see if it's high enough.
Bottom line. If there is any way to up the input voltage would help the shark out a lot.
Make sure the Shark isn't getting hot. If it's getting hot it gets very inefficient.
Wayne