Hold Thea,

Yes, the fact is that I actually said something, but this is a relative clause introduced by "lo que..." and what comes next can be something known or unknown. The speaker here is saying that he didn't mind whatever he may have said - this would be expressed with the subjunctive. There is somehow an unknown element here or it could also be taken as known but the speaker is detaching himself from what was said. 

In my opinion, the use of subjunctive here could also be justified by the type of verb used: "importar a alguien"; this verb expresses a feeling, in a similar way that "me alegraba que..., "me molestaba que..." and these trigger the subjunctive. 

I hope this clarified it.

Saludos

Inma