[hlds_linux] another request about update releases

Hi,

Another thing I've been noticing about updating the servers anytime
recently is that most of the time, if not all of the time, a "104
connection reset by peer" occurs even when an update has (apparently)
fully completed.

If this did not occur, the steam client would exit with a normal return
code and the server could come back up. Instead, probably many of the
various scripts we all run our servers under see this non-zero return code
and assume it needs to make another pass. Which delays our servers coming
back up.

Here is example from this evening's update, demonstrating this:

(this is a nemrun-initiated update but nemrun or not doesn't matter):

-------
[Wed Jan 11 20:45:46 EST 2012] :: Using command: ./steam -dir
"/home/tf2/tf2/" -game "tf" -command update -retry
Checking bootstrapper version ...
Updating Installation
Updating 'Team Fortress 2 Content' from version 307 to version 308

8.20% downloading /home/tf2/tf2//orangebox/tf/bin/server.dll
19.32% downloading /home/tf2/tf2//orangebox/tf/bin/server.dylib
38.06% downloading /home/tf2/tf2//orangebox/tf/bin/server.so
49.03% downloading /home/tf2/tf2//orangebox/tf/maps/graphs/cp_dustbowl.ain
49.03% downloading /home/tf2/tf2//orangebox/tf/maps/graphs/cp_granary.ain
49.03% downloading /home/tf2/tf2//orangebox/tf/maps/graphs/cp_gravelpit.ain
49.03% downloading /home/tf2/tf2//orangebox/tf/maps/graphs/cp_well.ain
49.03% downloading /home/tf2/tf2//orangebox/tf/maps/graphs/ctf_2fort.ain
49.46% downloading /home/tf2/tf2//orangebox/tf/maps/cp_foundry_l_0.lmp
49.46% downloading /home/tf2/tf2//orangebox/tf/resource/mapinfo.txt
51.21% downloading /home/tf2/tf2//orangebox/tf/resource/tf_brazilian.txt
52.88% downloading /home/tf2/tf2//orangebox/tf/resource/tf_danish.txt
54.75% downloading /home/tf2/tf2//orangebox/tf/resource/tf_dutch.txt
55.58% downloading /home/tf2/tf2//orangebox/tf/resource/tf_english.txt
57.38% downloading /home/tf2/tf2//orangebox/tf/resource/tf_finnish.txt
59.29% downloading /home/tf2/tf2//orangebox/tf/resource/tf_french.txt
61.19% downloading /home/tf2/tf2//orangebox/tf/resource/tf_german.txt
63.03% downloading /home/tf2/tf2//orangebox/tf/resource/tf_hungarian.txt
64.85% downloading /home/tf2/tf2//orangebox/tf/resource/tf_italian.txt
65.98% downloading /home/tf2/tf2//orangebox/tf/resource/tf_japanese.txt
67.21% downloading /home/tf2/tf2//orangebox/tf/resource/tf_korean.txt
68.44% downloading /home/tf2/tf2//orangebox/tf/resource/tf_koreana.txt
69.90% downloading /home/tf2/tf2//orangebox/tf/resource/tf_norwegian.txt
71.77% downloading /home/tf2/tf2//orangebox/tf/resource/tf_polish.txt
73.58% downloading /home/tf2/tf2//orangebox/tf/resource/tf_portuguese.txt
75.25% downloading /home/tf2/tf2//orangebox/tf/resource/tf_romanian.txt
77.08% downloading /home/tf2/tf2//orangebox/tf/resource/tf_russian.txt
78.59% downloading /home/tf2/tf2//orangebox/tf/resource/tf_schinese.txt
80.47% downloading /home/tf2/tf2//orangebox/tf/resource/tf_spanish.txt
82.29% downloading /home/tf2/tf2//orangebox/tf/resource/tf_swedish.txt
83.71% downloading /home/tf2/tf2//orangebox/tf/resource/tf_tchinese.txt
85.52% downloading /home/tf2/tf2//orangebox/tf/resource/tf_turkish.txt
86.67% downloading /home/tf2/tf2//orangebox/tf/scripts/items/items_game.txt
86.72% downloading /home/tf2/tf2//orangebox/tf/sound/vo/spy_thanks01.wav
86.72% downloading /home/tf2/tf2//orangebox/tf/steam.inf

Updating 'Team Fortress 2 Materials' from version 159 to version 160
86.72% downloading
/home/tf2/tf2//orangebox/tf/materials/backpack/player/items/soldier/luckyshot.vmt
86.72% downloading
/home/tf2/tf2//orangebox/tf/materials/backpack/player/items/soldier/luckyshot_large.vmt
Read more on: : i3D.net Game Forums /hlds-valve-linux-newsletter/183074-hlds_linux-another-request-about-update-releases.html
86.72% downloading
/home/tf2/tf2//orangebox/tf/materials/models/player/items/soldier/luckyshot_blue.vmt
86.72% downloading
/home/tf2/tf2//orangebox/tf/materials/models/player/items/soldier/luckyshot_red.vmt
86.76% downloading
/home/tf2/tf2//orangebox/tf/models/player/items/soldier/luckyshot.dx90.vtx
86.76% downloading
/home/tf2/tf2//orangebox/tf/models/player/items/soldier/luckyshot.mdl
86.76% downloading
/home/tf2/tf2//orangebox/tf/models/player/items/soldier/luckyshot.phy
86.83% downloading
/home/tf2/tf2//orangebox/tf/models/player/items/soldier/luckyshot.vvd

Checking/Installing 'Base Source Shared Materials' version 8


Checking/Installing 'Base Source Shared Models' version 4

Checking/Installing 'Base Source Shared Sounds' version 4


Updating 'OB Linux Dedicated Server' from version 144 to version 145
87.45% downloading /home/tf2/tf2//orangebox/bin/datacache.so
88.36% downloading /home/tf2/tf2//orangebox/bin/dedicated.so
92.08% downloading /home/tf2/tf2//orangebox/bin/engine.so
92.41% downloading /home/tf2/tf2//orangebox/bin/libtier0.so
92.70% downloading /home/tf2/tf2//orangebox/bin/libvstdlib.so
94.61% downloading /home/tf2/tf2//orangebox/bin/materialsystem.so
95.74% downloading /home/tf2/tf2//orangebox/bin/replay.so
95.86% downloading /home/tf2/tf2//orangebox/bin/scenefilecache.so
96.03% downloading /home/tf2/tf2//orangebox/bin/shaderapiempty.so
96.29% downloading /home/tf2/tf2//orangebox/bin/soundemittersystem.so
97.43% downloading /home/tf2/tf2//orangebox/bin/studiorender.so
100.00% downloading /home/tf2/tf2//orangebox/bin/vphysics.so

Connection Reset, errno 104 "Connection reset by peer"
CAsyncIOManager: 0 threads terminating. 0 reads, 0 writes, 0 deferrals.
CAsyncIOManager: 153 single object sleeps, 0 multi object sleeps
CAsyncIOManager: 0 single object alertable sleeps, 0 multi object
alertable sleeps
[Wed Jan 11 21:12:23 EST 2012] :: Update failed or incomplete, retrying
Read more on: : i3D.net Game Forums http://forum.i3d.net/showthread.php?t=183074
[Wed Jan 11 21:12:23 EST 2012] :: Attempting update
[Wed Jan 11 21:12:23 EST 2012] :: Using command: ./steam -dir
"/home/tf2/tf2/" -game "tf" -command update -retry
Checking bootstrapper version ...


--------

As you can see, it made it to 100%, but rather than the depot server (or
master, or whatever we are getting update content from) returning a
success return code to the steam binary actually updating our server
files, it just "hangs up the phone" abruptly, forcing another update run
attempt to happen. Which will take a lot longer to finish if (as is
usually the case) a LOT more servers are trying to update by this time.

If there is anything possible to do to try to improve the code so this
doesn't happen, it would help a lot of servers get back up faster (not
just mine) so your customers can start playing again.

Not criticizing BTW.

Thanks

_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please visit:
https://list.valvesoftware.com/cgi-b...nfo/hlds_linux

Meetoo.



PharaohsPaw wrote:
> As you can see, it made it to 100%, but rather than the depot server (or
Read more on: : i3D.net Game Forums http://forum.i3d.net/showthread.php?t=183074
> master, or whatever we are getting update content from) returning a
> success return code to the steam binary actually updating our server
> files, it just "hangs up the phone" abruptly, forcing another update run
> attempt to happen. Which will take a lot longer to finish if (as is
> usually the case) a LOT more servers are trying to update by this time.
>
> If there is anything possible to do to try to improve the code so this
> doesn't happen, it would help a lot of servers get back up faster (not
> just mine) so your customers can start playing again.
>
> Not criticizing BTW.

--
# Jesse Molina
# Mail = jesse (AT) net
# Page = page-jesse (AT) net
# Cell = 1.602.323.7608
# Web = http://www.opendreams.net/jesse/



_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please visit:
https://list.valvesoftware.com/cgi-b...nfo/hlds_linux

Dont some scripts just check the steam.inf (or what that file was) for the
version number?

But indeed, that should be better.

On Thu, 12 Jan 2012 01:43:34 -0700, Jesse Molina <jesse (AT) net>
wrote:
> Meetoo.
>
>
>
> PharaohsPaw wrote:
>> As you can see, it made it to 100%, but rather than the depot server (or
>> master, or whatever we are getting update content from) returning a
>> success return code to the steam binary actually updating our server
>> files, it just "hangs up the phone" abruptly, forcing another update run
>> attempt to happen. Which will take a lot longer to finish if (as is
>> usually the case) a LOT more servers are trying to update by this time.
>>
>> If there is anything possible to do to try to improve the code so this
>> doesn't happen, it would help a lot of servers get back up faster (not
>> just mine) so your customers can start playing again.
>>
>> Not criticizing BTW.

_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please visit:
https://list.valvesoftware.com/cgi-b...nfo/hlds_linux

>
> Dont some scripts just check the steam.inf (or what that file was) for the
> version number?
>
> But indeed, that should be better.

Well, nemrun does check the steam.inf file to build a query to the master
to see if a newer version is available, and it does try to "verify the
update" after one is finished -- but it doesn't try to "verify" the update
(make sure a steam.inf exists, etc.) until the steam binary that actually
updates the server files exits with a 0 return code.

The problem I'm mentioning here is that for whatever reason after the
update has hit 100% (which I am pretty sure means, we have successfully
pulled in all of the files comprising this update), the master end
abruptly terminates the connection/session a lot of the time, (connection
reset by peer usually means exactly that, they RST our TCP connection on
us) rather than issuing some form of an "OK that's all the files, we're
done here" app-layer response to the dedicated server (steam binary
updating it) - which causes the steam binary to think it didn't finish
successfully, and repeat.

Of course, since we have all the updated files now, it won't pull any more
files in. But we have to go through the whole process of another attempt
anyway to get a "success" (or whatever it is) app-layer response from the
steam content server so that we can get a successful exit code out of the
steam binary.

Again I really don't have any visibility of their side of things but I do
suspect that there might be something code-wise on the master end that
handles returning success responses differently (ie, not politely) when it
actually HAS sent us files to get us up to date, vs. when it didn't need
to send us any files. If that makes sense.

Because what it has ended up meaning, at least most of the updates I
watched get served out, is that even when we get 100% of the updated
files, it always fails the first time and has to go through the whole
shebang again.

Lately this has meant the servers are down for another 30 minutes or so -
for no good reason. Just because the master hung up on us abruptly
Read more on: : i3D.net Game Forums http://forum.i3d.net/showthread.php?t=183074
rather than return whatever message the steam client needs to get from the
master to tell it "all done, HLDS Installation is Up to Date", the first
time around.





_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please visit:
https://list.valvesoftware.com/cgi-b...nfo/hlds_linux

Precisely. Heck, even when I updated manually, if I saw Steam end with a
Connection Reset By Peer message, I'd rerun the update. Printing an error
to the console and exiting with a non-zero error code when an update
succeeds is bad, to say the least.

This is one of the reasons the update servers are always bogged down; the
updater is getting the wrong message back from the content server. Unlike
the game itself, the server doesn't do an update check on start unless you
specifically tell it to auto-update. If you do tell it to auto-update,
then when it finishes "updating" once (be it pass or fail) the server
launches. If the update failed, your server either crashes on startup or
sits there with no players because they get bad version errors while trying
to connect.

nemrun addresses some of these problems, specifically the failed update
thing mentioned above, but to do so it has to check the content server
again. And I've found in the past that I sometimes have to manually run a
-verify_all update to fix missing files when a Steam update fails or my
server gets stuck in a crashing loop.

On Fri, Jan 13, 2012 at 3:19 PM, PharaohsPaw <listacct2 (AT) com> wrote:

> >
> > Dont some scripts just check the steam.inf (or what that file was) for
> the
> > version number?
> >
> > But indeed, that should be better.
>
> Well, nemrun does check the steam.inf file to build a query to the master
> to see if a newer version is available, and it does try to "verify the
> update" after one is finished -- but it doesn't try to "verify" the update
> (make sure a steam.inf exists, etc.) until the steam binary that actually
> updates the server files exits with a 0 return code.
>
> The problem I'm mentioning here is that for whatever reason after the
> update has hit 100% (which I am pretty sure means, we have successfully
> pulled in all of the files comprising this update), the master end
> abruptly terminates the connection/session a lot of the time, (connection
> reset by peer usually means exactly that, they RST our TCP connection on
> us) rather than issuing some form of an "OK that's all the files, we're
> done here" app-layer response to the dedicated server (steam binary
> updating it) - which causes the steam binary to think it didn't finish
> successfully, and repeat.
>
> Of course, since we have all the updated files now, it won't pull any more
> files in. But we have to go through the whole process of another attempt
> anyway to get a "success" (or whatever it is) app-layer response from the
> steam content server so that we can get a successful exit code out of the
> steam binary.
>
> Again I really don't have any visibility of their side of things but I do
> suspect that there might be something code-wise on the master end that
> handles returning success responses differently (ie, not politely) when it
> actually HAS sent us files to get us up to date, vs. when it didn't need
> to send us any files. If that makes sense.
>
> Because what it has ended up meaning, at least most of the updates I
> watched get served out, is that even when we get 100% of the updated
Read more on: : i3D.net Game Forums http://forum.i3d.net/showthread.php?t=183074
> files, it always fails the first time and has to go through the whole
> shebang again.
>
> Lately this has meant the servers are down for another 30 minutes or so -
> for no good reason. Just because the master hung up on us abruptly
> rather than return whatever message the steam client needs to get from the
> master to tell it "all done, HLDS Installation is Up to Date", the first
> time around.
>
>
>
>
>
> _______________________________________________
> To unsubscribe, edit your list preferences, or view the list archives,
> please visit:
> https://list.valvesoftware.com/cgi-b...nfo/hlds_linux
>
_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please visit:
https://list.valvesoftware.com/cgi-b...nfo/hlds_linux

TL;DR: valve update servers are hammered a second time by servers updating cos of this without a need. Oops, to get less traffic during updates they have to send that "acknowledge" that the update was fully done. If they would, the updates for others would go better etc etc.
Read more on: : i3D.net Game Forums http://forum.i3d.net/showthread.php?t=183074

/me shrugs, if they dont want to optimize their own traffic.... or maybe they like the numbers for servers that hit updates to be artificially be blown up. Dunno.


>________________________________
> From: Ross Bemrose <rbemrose (AT) com>
>To: Half-Life dedicated Linux server mailing list <hlds_linux (AT) valvesoftware.com>
>Sent: Friday, 13 January 2012, 23:06
>Subject: Re: [hlds_linux] another request about update releases
>
>Precisely.* Heck, even when I updated manually, if I saw Steam end with a
>Connection Reset By Peer message, I'd rerun the update.* Printing an error
>to the console and exiting with a non-zero error code when an update
>succeeds is bad, to say the least.
>
>This is one of the reasons the update servers are always bogged down; the
>updater is getting the wrong message back from the content server.* Unlike
>the game itself, the server doesn't do an update check on start unless you
>specifically tell it to auto-update.* If you do tell it to auto-update,
Read more on: : i3D.net Game Forums http://forum.i3d.net/showthread.php?t=183074
>then when it finishes "updating" once (be it pass or fail) the server
>launches.* If the update failed, your server either crashes on startup or
>sits there with no players because they get bad version errors while trying
>to connect.
>
>nemrun addresses some of these problems, specifically the failed update
>thing mentioned above, but to do so it has to check the content server
>again.* And I've found in the past that I sometimes have to manually run a
>-verify_all update to fix missing files when a Steam update fails or my
>server gets stuck in a crashing loop.
>
>On Fri, Jan 13, 2012 at 3:19 PM, PharaohsPaw <listacct2 (AT) com> wrote:
>
>> >
>> > Dont some scripts just check the steam.inf (or what that file was) for
>> the
>> > version number?
>> >
>> > But indeed, that should be better.
>>
>> Well, nemrun does check the steam.inf file to build a query to the master
>> to see if a newer version is available, and it does try to "verify the
>> update" after one is finished -- but it doesn't try to "verify" the update
>> (make sure a steam.inf exists, etc.) until the steam binary that actually
>> updates the server files exits with a 0 return code.
>>
>> The problem I'm mentioning here is that for whatever reason after the
>> update has hit 100% (which I am pretty sure means, we have successfully
>> pulled in all of the files comprising this update), the master end
>> abruptly terminates the connection/session a lot of the time, (connection
>> reset by peer usually means exactly that, they RST our TCP connection on
>> us) rather than issuing some form of an "OK that's all the files, we're
>> done here" app-layer response to the dedicated server (steam binary
>> updating it) - which causes the steam binary to think it didn't finish
>> successfully, and repeat.
>>
>> Of course, since we have all the updated files now, it won't pull any more
>> files in.* But we have to go through the whole process of another attempt
>> anyway to get a "success" (or whatever it is) app-layer response from the
>> steam content server so that we can get a successful exit code out of the
>> steam binary.
>>
>> Again I really don't have any visibility of their side of things but I do
>> suspect that there might be something code-wise on the master end that
>> handles returning success responses differently (ie, not politely) when it
>> actually HAS sent us files to get us up to date, vs. when it didn't need
>> to send us any files.* If that makes sense.
>>
>> Because what it has ended up meaning, at least most of the updates I
>> watched get served out, is that even when we get 100% of the updated
>> files, it always fails the first time and has to go through the whole
>> shebang again.
>>
>> Lately this has meant the servers are down for another 30 minutes or so -
>> for no good reason. * Just because the master hung up on us abruptly
>> rather than return whatever message the steam client needs to get from the
>> master to tell it "all done, HLDS Installation is Up to Date", the first
>> time around.
>>
>>
>>
>>
>>
>> _______________________________________________
>> To unsubscribe, edit your list preferences, or view the list archives,
>> please visit:
>> https://list.valvesoftware.com/cgi-b...nfo/hlds_linux
>>
>_______________________________________________
>To unsubscribe, edit your list preferences, or view the list archives, please visit:
>https://list.valvesoftware.com/cgi-b...nfo/hlds_linux
>
>
>
_______________________________________________
To unsubscribe, edit your list preferences, or view the list archives, please visit:
https://list.valvesoftware.com/cgi-b...nfo/hlds_linux