Finding where the problem is (lag issue)

General Discussion Started Last reply 30 posts
Ok, so I've been all day testing and investigating the lag issue that makes it impossible for some (most) to play, or to even move; with freezes of up to several minutes.


From my computer (in Argentina) to the server, there are around 15-16 hops, the last 5 being:

/previous route/
4.69.155.62 // vlan60.csw1.NewYork1.Level3.net
4.69.148.33 // ae-62-62.ebr2.NewYork1.Level3.net
4.69.137.121 // ae-3-3.ebr2.Dallas1.Level3.net
4.69.151.153 // ae-82-82.csw3.Dallas1.Level3.net
4.69.145.139 // ae-3-80.edge2.Dallas1.Level3.net
/server/ (IP varies depending on which one you play)

Now the thing is that up until (and including) that 4.69.155.62, the connection is working beautiful. 0% packet loss during a 1 hour period. The problems start at the next server, 4.69.148.33. From that server (including) and the next ones, you see a disastrous connection. Around 15-20% packet loss all the time, disconnections, etc.

So the conclusion is that this is definitely a problem not with NCsoft, but with Level3 (it's a network carrier for the ones that don't know). Anyways, NCsoft could (and SHOULD) be already complaining to the company and trying to change the routing so that jump from .62 to .33 goes through a different path.


I have also tested some routes from Europe and they all go through those servers as well, the ones that vary (as expected) are the 1st ones before getting to level3. Please people, if you can test and post your results it would help to determine if we are all affected or just some particular ips/isps/countries.


Tests over 1000 sent packets (this was NOT done at the worst time, so the results could be worse, or better depending on when you do them; I've seen around 25% packet loss at peak hour, and around 7-10% at night after 2 or 3AM GMT -3):

4.69.155.62 = 1000 sent, 995 ok, 5 lost (0.5%)
/problem is here, between these 2 nodes/
4.69.148.33 = 1000 sent, 834 ok, 166 lost (16.6%)
4.69.137.121 = 1000 sent, 850 ok, 150 lost (15%)
4.69.151.153 = 1000 sent, 870 ok, 130 lost (13%)
4.69.145.139 = 1000 sent, 853 ok, 147 lost (14.7%)


I have also found the following threads on different forums talking about the same issue, so it's not an isolated Lineage 2 issue, it's a more general thing.

http://community.o2.co.uk/t5/Home-Broadband-Home-Phone/Packet-Loss-and-Bonkers-Routing/m-p/494792
https://forums.station.sony.com/ps2/index.php?threads/loosing-packets-to-east-coast-servers.119787/page-2


As you see, there's always some problem within the Level 3 network.
Ok, so I've been all day testing and investigating the lag issue that makes it impossible for some (most) to play, or to even move; with freezes of up to several minutes.


From my computer (in Argentina) to the server, there are around 15-16 hops, the last 5 being:

/previous route/
4.69.155.62 // vlan60.csw1.NewYork1.Level3.net
4.69.148.33 // ae-62-62.ebr2.NewYork1.Level3.net
4.69.137.121 // ae-3-3.ebr2.Dallas1.Level3.net
4.69.151.153 // ae-82-82.csw3.Dallas1.Level3.net
4.69.145.139 // ae-3-80.edge2.Dallas1.Level3.net
/server/ (IP varies depending on which one you play)

Now the thing is that up until (and including) that 4.69.155.62, the connection is working beautiful. 0% packet loss during a 1 hour period. The problems start at the next server, 4.69.148.33. From that server (including) and the next ones, you see a disastrous connection. Around 15-20% packet loss all the time, disconnections, etc.

So the conclusion is that this is definitely a problem not with NCsoft, but with Level3 (it's a network carrier for the ones that don't know). Anyways, NCsoft could (and SHOULD) be already complaining to the company and trying to change the routing so that jump from .62 to .33 goes through a different path.


I have also tested some routes from Europe and they all go through those servers as well, the ones that vary (as expected) are the 1st ones before getting to level3. Please people, if you can test and post your results it would help to determine if we are all affected or just some particular ips/isps/countries.


Tests over 1000 sent packets (this was NOT done at the worst time, so the results could be worse, or better depending on when you do them; I've seen around 25% packet loss at peak hour, and around 7-10% at night after 2 or 3AM GMT -3):

4.69.155.62 = 1000 sent, 995 ok, 5 lost (0.5%)
/problem is here, between these 2 nodes/
4.69.148.33 = 1000 sent, 834 ok, 166 lost (16.6%)
4.69.137.121 = 1000 sent, 850 ok, 150 lost (15%)
4.69.151.153 = 1000 sent, 870 ok, 130 lost (13%)
4.69.145.139 = 1000 sent, 853 ok, 147 lost (14.7%)


I have also found the following threads on different forums talking about the same issue, so it's not an isolated Lineage 2 issue, it's a more general thing.

http://community.o2.co.uk/t5/Home-Broadband-Home-Phone/Packet-Loss-and-Bonkers-Routing/m-p/494792
https://forums.station.sony.com/ps2/index.php?threads/loosing-packets-to-east-coast-servers.119787/page-2


As you see, there's always some problem within the Level 3 network.

How I can do that test?

And why the problem is just with VIVO-Telefonica and OI-VELOX?
Different ISPs may have different contracts with different carriers, hence they may take a different route to the server. The important thing to notice is that it is not a Lineage 2 problem only, it's a general issue affecting EVERYTHING.

To test yourself download the following: http://www.nirsoft.net/utils/countrytraceroute.zip

Then write the server's ip address on the box (shilen's IP for example is 64.25.37.134), set it to 30 hops max, and click go. You will see what servers your connection go through to reach the destination (l2 server).

Then you can ping those IPs to see at which one the problems begin. To do that go to start menu > run and type CMD, and click ok or press enter.

Then type: ping -n XXX IP

replace XXX with the number of packets you want to send, and IP with the IP you want to ping.


PS: you can't ping the server's IP directly, because ICMP requests are filtered. If you want to ping the server's IP directly you need to download tcping and ping it to port 7777.
Different ISPs may have different contracts with different carriers, hence they may take a different route to the server. The important thing to notice is that it is not a Lineage 2 problem only, it's a general issue affecting EVERYTHING.

To test yourself download the following: http://www.nirsoft.net/utils/countrytraceroute.zip

Then write the server's ip address on the box (shilen's IP for example is 64.25.37.134), set it to 30 hops max, and click go. You will see what servers your connection go through to reach the destination (l2 server).

Then you can ping those IPs to see at which one the problems begin. To do that go to start menu > run and type CMD, and click ok or press enter.

Then type: ping -n XXX IP

replace XXX with the number of packets you want to send, and IP with the IP you want to ping.


PS: you can't ping the server's IP directly, because ICMP requests are filtered. If you want to ping the server's IP directly you need to download tcping and ping it to port 7777.

My last question, of who is the problem and how solve that problem? to who I shall call?
The problem is in the Level3 networks.

http://www.level3.com/


I doubt you can call anyone that can help. I checked their facebook page and they were already told about the issue several times, and I guess them being one of the most important global carriers in the world they should already know about it even before we noticed it ingame.


I have done more tests, and it looks like the route is good now until .33, and the bad part start at .121. Weird that it moved 1 hop up, maybe they are working on it?
Here you can see it better. The data travles like the arrow in the picture (from the bottom to the top).

It's very easy to note that everything is ok until you get to .33, and after you pass the .33 server, the connection quality drops like a *****'s panty :D


http://img10.imageshack.us/img10/2127/11m7.png
Anyway, so far i have not found any of NCSOFT official pronouncement on this subject. Especially if she suffers from this
Yup, at least they could issue an official statement saying they are aware of the problem but they are unable to do anything, or they could lie and say "we are working on it"... but nothing.
The problem is in the Level3 networks.

http://www.level3.com/


I doubt you can call anyone that can help. I checked their facebook page and they were already told about the issue several times, and I guess them being one of the most important global carriers in the world they should already know about it even before we noticed it ingame.


I have done more tests, and it looks like the route is good now until .33, and the bad part start at .121. Weird that it moved 1 hop up, maybe they are working on it?

I dont saw anything saying about "solving the problem" or anything about "some problem"

Can u put on link of FB or anything their talking about?
Whole weekend and Monday have gone by, and the issues are exactly the same. Nothing changed. Just did a quick test sending 5000 ping requests to the same IPs as I did on Friday, and the results are not surprisingly equal:

Packet loss to 4.69.148.33: 17/5000 (0%)
Packet loss to 4.69.137.121: 837/5000 (17%)


I don't see a coming solution to this crap tbh, at least not soon :(
I can confirm that these lag issues are not only affecting south america players but europe too - me and few friends got same issues (Czech republic), 5-30s freezes, disconnects...
It is not a problem with South America, I live in Spain, got a super fiber optics line (100 mb) I can play Eve Online (heavy fleet battles) with minimum choke or loss, other games are flying, 10-20 ping in League of Legends.

When it comes to L2 yesterday I wasn't even able to pass the PIN screen... imagine that! Got frustrated, force closed the game, opened League of Legends and gave L2 another black ball.

I really like L2. I came back after 4 years to play L2 again.

Very dissapointing.

Bots ingame advertising adena sales. The most annoying part is that they also spam this on the party matchmaking system and on shouts, not to mention the private shops. Implement a **** bot to ban the bots. Add a wordlist.

Hire 5 people to patrol the game wearing a banhammer for each server. It will raise the morale of the people. GM presence is very healthy.

LAGS - the killer of all games.

If you do it right the game will survive and probably will see more updates or better a full revamp. If not it will slowly die. Or maybe this is what they want ???
I've got 50% packet loss when not using a proxy like wtfast. The loss is happening in the exact same place where OP mentiones
in short version: it's a problem what ncsoft can't fix because its not their problem. the problem is in their servers hosting ISP network or something like this.
and if many peole talk about that problem, probably it's not only for l2 :) so keep calm, go outside, it's summer )))) (so many times people say this for me, and i wanted to say this for someone too, now it's good time)
If I were having 20%-50% lost I would sing in paper with blood.

Using program called PingPlotter suggested by tech support (I have an open ticket) ( Shilen Server)


http://i40.tinypic.com/ziv9tz.jpg


So, wonder what can be the way to get it fixed. Things changed when July month started, before things were perfectly fine and smooth.
bem la no meio do olho do cu do colorado.
If I were having 20%-50% lost I would sing in paper with blood.

Using program called PingPlotter suggested by tech support (I have an open ticket) ( Shilen Server)


http://i40.tinypic.com/ziv9tz.jpg


So, wonder what can be the way to get it fixed. Things changed when July month started, before things were perfectly fine and smooth.

You were never able to reach the server with PingPlotter, it's normal to have this result (100% losses) on the last jump
If I were having 20%-50% lost I would sing in paper with blood.

Using program called PingPlotter suggested by tech support (I have an open ticket) ( Shilen Server)


http://i40.tinypic.com/ziv9tz.jpg


So, wonder what can be the way to get it fixed. Things changed when July month started, before things were perfectly fine and smooth.

That is absolutely normal. You can't ping the server's IP, as ICMP traffic is blocked hence you won't get any answer to ping.

If you really need to ping the server, you need to use something that sends TCP packages instead of ICMP ones, like for example tcping (http://www.elifulkerson.com/projects/tcping.php).


With that tool, you can ping to port 777 like this

tcping 64.25.37.134 7777


or if you want to send for example, 1000 packages

tcping -n 1000 64.25.37.134 7777


Anyways, your trace doesn't go through the level3 network, do you have lags ingame? Long freezes of 30+ seconds? If you do, then there's another problem and it's not only a level3 network anymore.
It's not NCs problem directly, but this situation hurts portion of their customers and indirectly cuts into profits if they do not play.

They cannot fix it, but what they could do is acknowledge the problem and possibly let at least their service provider know about the problem so they can talk to their peers etc. Do not guarantee anything, just try and see if there is possibility to get at least heard some word about trouble as some major issues like fiber cuts or government issued internet blocks are known pretty fast and ISPs have access to reporting tools and sites.
It's not NCs problem directly, but this situation hurts portion of their customers and indirectly cuts into profits if they do not play.

They cannot fix it, but what they could do is acknowledge the problem and possibly let at least their service provider know about the problem so they can talk to their peers etc. Do not guarantee anything, just try and see if there is possibility to get at least heard some word about trouble as some major issues like fiber cuts or government issued internet blocks are known pretty fast and ISPs have access to reporting tools and sites.

Exactly, it's not the same if "Mr. Nobody" (any of us) calls level 3 and complains, than NCsoft or even the datacenter where they host their servers complaining directly to level 3. There a lot of money involved and I really doubt they haven't already.
A lot of spanish players have same problem, my computer works perfect, only have problems with l2.
I have 0 lag from 9:00am to 15/16:00pm here in Spain (daily instance reset start at 10:30 here)...after start the nightmare (no play at this hours make me angry delays of 1 hour).
I have exactly this problem for nine days, all started the fateful July 1 :mad:
using proxy(battleping,etc) fix all problem :D

my ping is even better than before.

gg :D
Anyway, so far i have not found any of NCSOFT official pronouncement on this subject. Especially if she suffers from this

Yup, at least they could issue an official statement saying they are aware of the problem but they are unable to do anything, or they could lie and say "we are working on it"... but nothing.

That's because NcSoft has no stake in the issue, and can't do anything about it. Some people will experience issues depending who their ISP's backbone providers are, and some won't. Nothing you or NcSoft can really do about it, until the Level3/Global Crossing, and or Cogent's of the world get their heads out of their ***.

You people need to realize that NcSoft has NOTHING to do with this issue, so stop asking them to fix something they have 0 control over. There is a reason why this is only affecting some of the player base. It's a transit issue with the backbone providers (The transit beween your ISP, and NcSoft's). Depending on who you are routed through, you may or may not experience the issue. From what I can see, it seems to be localized to Level3/Global Crossing's (Same Company) transit circuits. Regardless, stop nagging NcSoft... They have as much power in this issue as you.
It is not a problem with South America, I live in Spain, got a super fiber optics line (100 mb) I can play Eve Online (heavy fleet battles) with minimum choke or loss, other games are flying, 10-20 ping in League of Legends.

When it comes to L2 yesterday I wasn't even able to pass the PIN screen... imagine that! Got frustrated, force closed the game, opened League of Legends and gave L2 another black ball.

I really like L2. I came back after 4 years to play L2 again.

Very dissapointing.

Bots ingame advertising adena sales. The most annoying part is that they also spam this on the party matchmaking system and on shouts, not to mention the private shops. Implement a **** bot to ban the bots. Add a wordlist.

Hire 5 people to patrol the game wearing a banhammer for each server. It will raise the morale of the people. GM presence is very healthy.

LAGS - the killer of all games.

If you do it right the game will survive and probably will see more updates or better a full revamp. If not it will slowly die. Or maybe this is what they want ???

I feel bad for you if your only getting 100M over Fiber.
I feel bad for you if your only getting 100M over Fiber.


Can you please resume eating your American **** propaganda, and if you do not have something useful to contribute just go and try a school shooting, cinema rampage or tower bombing or whatever you wish to choose from. You got plenty of options in the US and A. Enjoy your 1gps optic fibers, and beware of the neighbor when you go to the grocery store, he might just put a sawed off shotgun up your stupid *** because he did not get a bike when he was a kid.
About lag, I disagree that the problem was a fiber optic cable in the Pacific or Atlantic, affect not only the Lineage 2, but any connection would be slow, and not only has that L2 is hosted in the United States, it is a problem but in NCsoft. Learn one thing, the problem happens and is happening with virtually matades of players both in South America and also in Europe, ie, it is not normal, funny to connect with the United States, uses fiber-optic passing through such oceans, but Lag only carries the L2. The fact is, this silent NCsoft, and in my country there is a saying: "WHO IS SILENT, CONSENT. Summarizing the saying goes, if she did not comment so far, why the problem is at the bottom there, she hired a service of an outsourcing company on connection, it must demand this company solution.

For even her site sometimes gets slow, when I access sites allocated in the United States to see movies, which makes actual download, or while you're watching the movie, it hangs down and there, and this type film for example, need a great connection, while the Lineage 2 suffers with a lag, where it is not a type of connection that requires a constant speed, but there is a tremendous lag. If it were only a few players and was an occasional lag that lasts only hours, but days and days and days, prejudicandos players level up, damaging players who bought and bought NCoins Runes Runes XP and now these days have lost and are losing because of the Lag who will replenish the loss of these players?

Who will replenish the lost time of the players before the intense lag that lasts for days? Funny that this lag only affects players who use leased servers in the United States, as players of other games that is also allocated in the United States do not suffer lag, and the data also undergo such fiber optics in such oceans.
You people need to realize that NcSoft has NOTHING to do with this issue, so stop asking them to fix something they have 0 control over.

I understand, but some people don't. Plus you are wrong that NC has nothing to do with that, in the past they informed about connections trouble for Comcast customers and some other events. They had nothing to do with Comcast, I don't give a **** about some American cable company, so should most of world just say **** and don't bother, my interwebs working fine?

If they at least open ticket with their ISP so they open ticket with upstream and see what's going on, it would help. Or their/upstream ISP changes trouting to go different path and route around the problem. If there is some issue and 25% of all traffic going through certain node gets blackholed, something bad is happening and at least one service provider should know about it. Angry individuals calling L3/Global Xing/AT&T/Savvis/Sprint/Whoever are not much of a leverage to have them at least acknowledge and investigate, angry business partners are.
using proxy(battleping,etc) fix all problem :D

my ping is even better than before.

gg :D


Any of them are free or all payable services ?
It will only take way when losing players and losing players who buy this NCoins because those without lag alone will not help keep the server buying Ncoins and without mentioning that it will generate a downside, because who wants to play or return to play, may fall in this route with lag. To sum it all, but she has to investigate, because the problem is not just a country and a continent, is several. I for one think about stop playing for now, until we solve the problem of lag, since there are usually as many servers in Lineage 2 official, they use more than one connection port and services, so one of these are in trouble.
Can you please resume eating your American **** propaganda, and if you do not have something useful to contribute just go and try a school shooting, cinema rampage or tower bombing or whatever you wish to choose from. You got plenty of options in the US and A. Enjoy your 1gps optic fibers, and beware of the neighbor when you go to the grocery store, he might just put a sawed off shotgun up your stupid *** because he did not get a bike when he was a kid.

The hell is your issue?

100m over a Fiber connection = Bad Speed. Not sure what the hell your rage rant is about. You do realize that the United States has some of the worst Networking infrastructure/speeds in the world? Apparently you missed the point of my post.