To whom. The underlying construction is “the victims owed their lives to the fishermen”. When fishermen becomes the antecedent of the relative clause, the required preposition remains: “the fishermen, to whom the victims owed their lives”.
Whom is appropriate because it is the object of to and refers to people. Neither omitting to nor using which preserves that construction.