The pair lost the first set 6-2 but responded well to win the second 6-3 to take the match to a deciding first-to-10-points tie-break.
With Djokovic and Kyrgios two points from victory, a double fault from the Serbian proved costly at 8-6.
It shifted the momentum of the tie-break back towards Mektic and Venus, who won the next three points to seal the win 6-2 3-6 10-8.
“Incredible,” Mektic said afterwards. “I was very happy when I saw them [Djokovic and Kyrgios], I could play them on the first of January.
“It’s an amazing feeling to start the year like this and we knew there would be points like that against players like this.”
Djokovic, pursuing his 100th career ATP title, is through to the quarter-finals of the singles, where he will play France’s Gael Monfils.
