The simple answer is that the Babylonian Jewish poor - who were by definition not members of the better off households - were invited to enter homes only after the hosts had finished preparing the meal, drunk some diluted wine and consumed some slight food:
מה שנהגו לומר כל דכפין ייתי וייכול, כך היה מנהג אבות, שהיו מגביהין שלחנותיהס ולא היו סוגרים דלתותיהם והיו אומרים ככה כדי שיבאו ישראל העניים שכניהם לאכול , ולקבל שכר היו עושים זה.
ועכשיו... מפרנסין אותם בתחילה כדי שלא יחזרו על הפתחים, ואחר כך מגביהין את השלחן ואומרים כמנהג הראשונים. (רב מתתיהו גאון – אוצר הגאונים ג, פסחים סי' ד"ש [עמ' 112])After all, even Rav Huna, who famously invited the poor to eat in his house every single day, did so only when he ate a piece of bread (b. Taanit 20b) - while Rava stated (ibid) that he could not live up to that standard.
Thus on the seder eve the common Babylonian Jews invited the poor guests only after they themselves had drunk and eaten a bit - and were ready to remind themselves of the obligation to the other that is the message of Pesah (??). They thus began with "this is the bread of oppression" according to some versions - and according to other versions simply began with the invitation. Although the poor waited, the hosts were also more calm and receptive* - and could then offer the poor the diluted wine and appetizer in which the poor were also obligated (m. Pesahim 10:1).
In fact, even as regards the Paschal lamb, the poor may have been invited only before the slaughtering of the lamb - in the crowds of the Temple Mount and Jerusalem (cf. m. Pesahim 5:5-7 and cf. the views of R. Eliezer, R. Yossi b. R. Yehuda and R. Yossi ha-Glilee in m. Pesahim 9:2 & t. Pesahim 8:1 on the exemption from the Paschal sacrifice of a person who cannot reach the Temple Mount) - but would not necessarily have shown up to wait in the host's Jerusalem house where the lamb was eaten (m. Pesahim 9:3; 7:13; 7:9,12) and where the main group was waiting for the animal to be slaughtered and brought over (m. Pesahim 9:9).
*cf. Although a person is so obligated to act humanely toward her animals that he must feed them before he himself eats (Rav in b. Gittin 62a), she may drink before providing the animals with water.