This is making a large assumption, of course, that time travelers had any interest in attending his party, or that they even would have heard about it.*
* I do not believe in time travel.
Then a bunch of other people arrived and they had a wild party.
You can use it, but in doing so you remove the conditions for creating time travel. You can use it once and then you and your entire timeline cease to exist. In such conditions I might reserve the use of time travel for things like preventing an unlivable hellscape and not waste it on attending a party with balloons.