Liverpool dominated large chunks of a topsy turvy match at the Emirates and took the lead, in a game match littered with defensive errors, through Philippe Coutinho but wasted several chances, Sadio Mane the main culprit.
Mohamed Salah added a second, but then Arsenal hit back with three goals in five chaotic minutes.
First Alexis Sanchez pounced on defensive hesitation, then Liverpool goalkeeper Simon Mignolet blundered to allow a leveller from Granit Xhaka, and finally Mesut Ozil put the Gunners ahead.
But incredibly then, an error from Arsenal goalkeeper Petr Cech allowed Roberto Firmino to haul Liverpool level late in the game.
A frustrated Klopp, who exchanged angry words with nearby fans as he threw down a water bottle after Firmino’s goal and appeared to have water thrown at him, said: “Of course we should have won but we had five minutes where we had obvious problems.
“It was not team defending. The first goal and the second were misjudgements. We opened the door for them in these five minutes. I’m disappointed.
“I was aggressively happy at our equaliser, I have made my apologies, but if the poeple were to apoligize to me it would take a bit longer! I did nothing, I threw a bottle onto the ground, I was not happy at that moment.
“It was not about controlling the game. In 99.9% of these situations Mignolet saves these balls – but not tonight. We should be three, four even five-nil up before we even defend a cross.”
Arsenal manager Arsene Wenger said: “I’m frustrated we did not win – but to be honest it could have been over at half time. Give credit to the players because even at 2-0 we did not show weakness and came back.
“I will keep private what we said at half time – but the positive thing was that we were only 1-0 down.
“In the first half we were paralysed, frozen, second best everywhere. I said to the players the good thing is we’re only 1-0 down. What we did was not good enough. We had nothing to lose.
“But at 3-2 we were naive defensively. If I tried to be fair overall maybe it’s a fair result.”
Jack Wilshere said: “We played like we were scared in the first half.” ends